Hyukjin Kwon created SPARK-61955:
------------------------------------

             Summary: Make the writers and readers of the native ML classes 
PySpark's plain objects, order the defaults of a fitted model as PySpark's 
try_remote_fit, and iterate the stages of a Pipeline as its fit does
                 Key: SPARK-61955
                 URL: https://issues.apache.org/jira/browse/SPARK-61955
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


A round-3 review of the native ML classes (pyspark.ml.connect) against 
pyspark-client 4.3.0:

1. The writers and readers of every native class (RemoteMLWriter, 
RemoteMLReader) kept PyO3's methods and state.
- session(sparkSession=spark) raised "unexpected keyword argument 
'sparkSession'".
- overwrite(1) and save(p, 1) raised PyO3's messages, not those of PySpark's 
functions (MLWriter.overwrite, RemoteMLWriter.save).
- copy.copy, copy.deepcopy and pickle raised AttributeError ("no attribute 
'instance'"/"'clazz'").
- shouldOverwrite was a bool field, so `w.shouldOverwrite = 0` raised TypeError.
- _sparkSession, _instance and _clazz were missing.

PySpark's are plain objects (ml/util.py:534, 570-603; 
ml/connect/readwrite.py:31-39, 202-206).

2. SummaryBuilder.summary was a PyO3 method, of another binding error, and the 
static methods of pyspark.ml.stat had a __module__ of None.

3. A fitted ALSModel's _defaultParamMap was in name order. PySpark's 
try_remote_fit runs _resetUid, then _copyValues, which appends the estimator's 
defaults that the model's constructor lacks: ['blockSize', 'predictionCol', 
'coldStartStrategy', 'itemCol', 'userCol'].

4. Pipeline(stages=<one-shot iterator>) fitted its stages, where PySpark's 
Pipeline._fit iterates the stages three times (pipeline.py:120-142), so a 
generator gives an empty PipelineModel.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to