[ 
https://issues.apache.org/jira/browse/SPARK-61932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61932:
---------------------------------
    Fix Version/s: connect-rust-4.4.0

> Pickle the native ML objects at every protocol and refuse the pickles of the 
> DataFrames that keep a model as PySpark does, and give the models and 
> summaries PySpark's _java_obj
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61932
>                 URL: https://issues.apache.org/jira/browse/SPARK-61932
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.4.0
>
>
> A round-2 review of the native ML classes (pyspark.ml.connect) against 
> pyspark-client 4.3.0, each item probed on both clients:
> 1. At pickle protocols 0 and 1, the estimators, transformers, evaluators, 
> Pipelines, validators, OneVsRest and PrefixSpan raised "TypeError: Method 
> __init__ forces keyword arguments.", and unbound models raised an 
> AssertionError or a PicklingError ("it's not the same object as 
> pyspark.ml.connect.pipeline.PipelineModel"). PySpark's, Python objects, 
> pickle at every protocol, and a model of a server-side object raises the 
> error of the pickle of the lock of its RemoteModelRef at the protocol (a 
> PicklingError below 2, the TypeError "cannot pickle '_thread.lock' object" 
> from 2).
> 2. A DataFrame that keeps a model in its plan pickled as any DataFrame: a 
> plain pickle raised the AttributeError of the session's local function, and 
> cloudpickle and copy.deepcopy copied it. Such DataFrames are a transform of a 
> fitted model, a DataFrame that reads one, and a DataFrame of a method of a 
> model or of the summary of an evaluate. PySpark pickles the plan before the 
> session, so its pickle, cloudpickle and deepcopy of them raise the error of 
> the lock of the model's RemoteModelRef. For a DataFrame of a method of a 
> training summary (summary.predictions), PySpark's pickle raises the 
> AttributeError of the session, and cloudpickle the TypeError of the lock of 
> the summary's model (_remote_model_obj). PySpark's copy.deepcopy requests the 
> schemas of the DataFrames it copies on the way before it raises.
> 3. The models of server-side objects had a _java_obj of None, and their 
> summaries had no object id, so PySpark's own _call_java(name, *args) of them 
> failed: "AssertionError" for a model, and "'NoneType' object has no attribute 
> 'split'" for a summary. Over Spark Connect, PySpark's _call_java sends a 
> Fetch (try_remote_call). In PySpark:
> - a model's _java_obj is the RemoteModelRef of its object, after a fit, a 
> load, a copy, a fetch, or a class method such as 
> StringIndexerModel.from_labels;
> - a training summary has "<id>.summary", and the _summary_dataset and 
> _remote_model_obj that try_remote_fit sets, in that order;
> - the summary of an evaluate has the server's id of the summary and 
> __source_transformer__, the model;
> - a model with a training summary has _summary.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to