Hyukjin Kwon created SPARK-61955:
------------------------------------
Summary: Make the writers and readers of the native ML classes
PySpark's plain objects, order the defaults of a fitted model as PySpark's
try_remote_fit, and iterate the stages of a Pipeline as its fit does
Key: SPARK-61955
URL: https://issues.apache.org/jira/browse/SPARK-61955
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
A round-3 review of the native ML classes (pyspark.ml.connect) against
pyspark-client 4.3.0:
1. The writers and readers of every native class (RemoteMLWriter,
RemoteMLReader) kept PyO3's methods and state.
- session(sparkSession=spark) raised "unexpected keyword argument
'sparkSession'".
- overwrite(1) and save(p, 1) raised PyO3's messages, not those of PySpark's
functions (MLWriter.overwrite, RemoteMLWriter.save).
- copy.copy, copy.deepcopy and pickle raised AttributeError ("no attribute
'instance'"/"'clazz'").
- shouldOverwrite was a bool field, so `w.shouldOverwrite = 0` raised TypeError.
- _sparkSession, _instance and _clazz were missing.
PySpark's are plain objects (ml/util.py:534, 570-603;
ml/connect/readwrite.py:31-39, 202-206).
2. SummaryBuilder.summary was a PyO3 method, of another binding error, and the
static methods of pyspark.ml.stat had a __module__ of None.
3. A fitted ALSModel's _defaultParamMap was in name order. PySpark's
try_remote_fit runs _resetUid, then _copyValues, which appends the estimator's
defaults that the model's constructor lacks: ['blockSize', 'predictionCol',
'coldStartStrategy', 'itemCol', 'userCol'].
4. Pipeline(stages=<one-shot iterator>) fitted its stages, where PySpark's
Pipeline._fit iterates the stages three times (pipeline.py:120-142), so a
generator gives an empty PipelineModel.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]