[ 
https://issues.apache.org/jira/browse/SPARK-61955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61955:
---------------------------------
    Fix Version/s: connect-rust-4.4.0

> Make the writers and readers of the native ML classes PySpark's plain 
> objects, order the defaults of a fitted model as PySpark's try_remote_fit, 
> and iterate the stages of a Pipeline as its fit does
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61955
>                 URL: https://issues.apache.org/jira/browse/SPARK-61955
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.4.0
>
>
> A round-3 review of the native ML classes (pyspark.ml.connect) against 
> pyspark-client 4.3.0:
> 1. The writers and readers of every native class (RemoteMLWriter, 
> RemoteMLReader) kept PyO3's methods and state.
> - session(sparkSession=spark) raised "unexpected keyword argument 
> 'sparkSession'".
> - overwrite(1) and save(p, 1) raised PyO3's messages, not those of PySpark's 
> functions (MLWriter.overwrite, RemoteMLWriter.save).
> - copy.copy, copy.deepcopy and pickle raised AttributeError ("no attribute 
> 'instance'"/"'clazz'").
> - shouldOverwrite was a bool field, so `w.shouldOverwrite = 0` raised 
> TypeError.
> - _sparkSession, _instance and _clazz were missing.
> PySpark's are plain objects (ml/util.py:534, 570-603; 
> ml/connect/readwrite.py:31-39, 202-206).
> 2. SummaryBuilder.summary was a PyO3 method, of another binding error, and 
> the static methods of pyspark.ml.stat had a __module__ of None.
> 3. A fitted ALSModel's _defaultParamMap was in name order. PySpark's 
> try_remote_fit runs _resetUid, then _copyValues, which appends the 
> estimator's defaults that the model's constructor lacks: ['blockSize', 
> 'predictionCol', 'coldStartStrategy', 'itemCol', 'userCol'].
> 4. Pipeline(stages=<one-shot iterator>) fitted its stages, where PySpark's 
> Pipeline._fit iterates the stages three times (pipeline.py:120-142), so a 
> generator gives an empty PipelineModel.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to