[
https://issues.apache.org/jira/browse/SPARK-61749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61749.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 736
[https://github.com/apache/spark-connect-rust/pull/736]
> Take the paths of the saves and loads of the native ML classes as PySpark
> does, and read the JVM class of a subclass's name
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61749
> URL: https://issues.apache.org/jira/browse/SPARK-61749
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The native ML classes of spark-connect-rust (pyspark.ml.connect.*) took the
> path of save, load, write().save and read().load as a str only, and raised
> PyO3's TypeError of any other value ("'int' object is not an instance of
> 'str'"), where PySpark takes it as the protobuf field of its Write and Read
> takes it (bytes of their UTF-8, None as the empty path that the server
> refuses, upb's "bad argument type for built-in operation" of anything else, a
> pathlib.Path too), and of the Python format of a Pipeline or a validator as
> os.path.join takes it (an os.PathLike, os.path's TypeError of anything else).
> Also, the load of a Python subclass read the JVM class of the native class,
> where PySpark's RemoteMLReader names the JVM class of the Python class, which
> the server refuses (CONNECT_ML.UNSUPPORTED_EXCEPTION); and getParam of a name
> that is not a str raised PyO3's TypeError where PySpark's getattr raises
> "attribute name must be string, not 'int'".
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]