Hyukjin Kwon created SPARK-61737:
------------------------------------
Summary: Make the native ML classes subclasses of PySpark's
classes, so that their objects are instances of every class of PySpark's MRO
Key: SPARK-61737
URL: https://issues.apache.org/jira/browse/SPARK-61737
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The objects of PySpark's ML classes are instances of every class of their MRO:
MLWritable, MLReadable, Identifiable, JavaMLWritable and JavaMLReadable,
HasTrainingSummary and the summary bases, the param mixins (HasMaxIter,
HasFeaturesCol, ...), PredictionModel and ClassificationModel, the Java
wrappers, and the class itself.
The native ML classes of Spark Connect Rust are PyO3 classes, registered as
virtual subclasses of PySpark's ABC bases (Estimator, Model, Params, Evaluator)
only: isinstance(model, MLWritable) is False for every native class (MLWritable
is a plain class), and so PySpark's own pyspark.ml.Pipeline and validators
refuse to save native stages ("... is not MLWritable"). Every one of the 140
classes of the full API has a class of PySpark's MRO that it is not a subclass
of.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]