Hyukjin Kwon created SPARK-61932:
------------------------------------

             Summary: Pickle the native ML objects at every protocol and refuse 
the pickles of the DataFrames that keep a model as PySpark does, and give the 
models and summaries PySpark's _java_obj
                 Key: SPARK-61932
                 URL: https://issues.apache.org/jira/browse/SPARK-61932
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


A round-2 review of the native ML classes (pyspark.ml.connect) against 
pyspark-client 4.3.0, each item probed on both clients:

1. At pickle protocols 0 and 1, the estimators, transformers, evaluators, 
Pipelines, validators, OneVsRest and PrefixSpan raised "TypeError: Method 
__init__ forces keyword arguments.", and unbound models raised an 
AssertionError or a PicklingError ("it's not the same object as 
pyspark.ml.connect.pipeline.PipelineModel"). PySpark's, Python objects, pickle 
at every protocol, and a model of a server-side object raises the error of the 
pickle of the lock of its RemoteModelRef at the protocol (a PicklingError below 
2, the TypeError "cannot pickle '_thread.lock' object" from 2).

2. A DataFrame that keeps a model in its plan pickled as any DataFrame: a plain 
pickle raised the AttributeError of the session's local function, and 
cloudpickle and copy.deepcopy copied it. Such DataFrames are a transform of a 
fitted model, a DataFrame that reads one, and a DataFrame of a method of a 
model or of the summary of an evaluate. PySpark pickles the plan before the 
session, so its pickle, cloudpickle and deepcopy of them raise the error of the 
lock of the model's RemoteModelRef. For a DataFrame of a method of a training 
summary (summary.predictions), PySpark's pickle raises the AttributeError of 
the session, and cloudpickle the TypeError of the lock of the summary's model 
(_remote_model_obj). PySpark's copy.deepcopy requests the schemas of the 
DataFrames it copies on the way before it raises.

3. The models of server-side objects had a _java_obj of None, and their 
summaries had no object id, so PySpark's own _call_java(name, *args) of them 
failed: "AssertionError" for a model, and "'NoneType' object has no attribute 
'split'" for a summary. Over Spark Connect, PySpark's _call_java sends a Fetch 
(try_remote_call). In PySpark:
- a model's _java_obj is the RemoteModelRef of its object, after a fit, a load, 
a copy, a fetch, or a class method such as StringIndexerModel.from_labels;
- a training summary has "<id>.summary", and the _summary_dataset and 
_remote_model_obj that try_remote_fit sets, in that order;
- the summary of an evaluate has the server's id of the summary and 
__source_transformer__, the model;
- a model with a training summary has _summary.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to