Hyukjin Kwon created SPARK-61749:
------------------------------------

             Summary: Take the paths of the saves and loads of the native ML 
classes as PySpark does, and read the JVM class of a subclass's name
                 Key: SPARK-61749
                 URL: https://issues.apache.org/jira/browse/SPARK-61749
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


The native ML classes of spark-connect-rust (pyspark.ml.connect.*) took the 
path of save, load, write().save and read().load as a str only, and raised 
PyO3's TypeError of any other value ("'int' object is not an instance of 
'str'"), where PySpark takes it as the protobuf field of its Write and Read 
takes it (bytes of their UTF-8, None as the empty path that the server refuses, 
upb's "bad argument type for built-in operation" of anything else, a 
pathlib.Path too), and of the Python format of a Pipeline or a validator as 
os.path.join takes it (an os.PathLike, os.path's TypeError of anything else).

Also, the load of a Python subclass read the JVM class of the native class, 
where PySpark's RemoteMLReader names the JVM class of the Python class, which 
the server refuses (CONNECT_ML.UNSUPPORTED_EXCEPTION); and getParam of a name 
that is not a str raised PyO3's TypeError where PySpark's getattr raises 
"attribute name must be string, not 'int'".




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to