Hyukjin Kwon created SPARK-61749:
------------------------------------
Summary: Take the paths of the saves and loads of the native ML
classes as PySpark does, and read the JVM class of a subclass's name
Key: SPARK-61749
URL: https://issues.apache.org/jira/browse/SPARK-61749
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The native ML classes of spark-connect-rust (pyspark.ml.connect.*) took the
path of save, load, write().save and read().load as a str only, and raised
PyO3's TypeError of any other value ("'int' object is not an instance of
'str'"), where PySpark takes it as the protobuf field of its Write and Read
takes it (bytes of their UTF-8, None as the empty path that the server refuses,
upb's "bad argument type for built-in operation" of anything else, a
pathlib.Path too), and of the Python format of a Pipeline or a validator as
os.path.join takes it (an os.PathLike, os.path's TypeError of anything else).
Also, the load of a Python subclass read the JVM class of the native class,
where PySpark's RemoteMLReader names the JVM class of the Python class, which
the server refuses (CONNECT_ML.UNSUPPORTED_EXCEPTION); and getParam of a name
that is not a str raised PyO3's TypeError where PySpark's getattr raises
"attribute name must be string, not 'int'".
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]