Hyukjin Kwon created SPARK-61932:
------------------------------------
Summary: Pickle the native ML objects at every protocol and refuse
the pickles of the DataFrames that keep a model as PySpark does, and give the
models and summaries PySpark's _java_obj
Key: SPARK-61932
URL: https://issues.apache.org/jira/browse/SPARK-61932
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
A round-2 review of the native ML classes (pyspark.ml.connect) against
pyspark-client 4.3.0, each item probed on both clients:
1. At pickle protocols 0 and 1, the estimators, transformers, evaluators,
Pipelines, validators, OneVsRest and PrefixSpan raised "TypeError: Method
__init__ forces keyword arguments.", and unbound models raised an
AssertionError or a PicklingError ("it's not the same object as
pyspark.ml.connect.pipeline.PipelineModel"). PySpark's, Python objects, pickle
at every protocol, and a model of a server-side object raises the error of the
pickle of the lock of its RemoteModelRef at the protocol (a PicklingError below
2, the TypeError "cannot pickle '_thread.lock' object" from 2).
2. A DataFrame that keeps a model in its plan pickled as any DataFrame: a plain
pickle raised the AttributeError of the session's local function, and
cloudpickle and copy.deepcopy copied it. Such DataFrames are a transform of a
fitted model, a DataFrame that reads one, and a DataFrame of a method of a
model or of the summary of an evaluate. PySpark pickles the plan before the
session, so its pickle, cloudpickle and deepcopy of them raise the error of the
lock of the model's RemoteModelRef. For a DataFrame of a method of a training
summary (summary.predictions), PySpark's pickle raises the AttributeError of
the session, and cloudpickle the TypeError of the lock of the summary's model
(_remote_model_obj). PySpark's copy.deepcopy requests the schemas of the
DataFrames it copies on the way before it raises.
3. The models of server-side objects had a _java_obj of None, and their
summaries had no object id, so PySpark's own _call_java(name, *args) of them
failed: "AssertionError" for a model, and "'NoneType' object has no attribute
'split'" for a summary. Over Spark Connect, PySpark's _call_java sends a Fetch
(try_remote_call). In PySpark:
- a model's _java_obj is the RemoteModelRef of its object, after a fit, a load,
a copy, a fetch, or a class method such as StringIndexerModel.from_labels;
- a training summary has "<id>.summary", and the _summary_dataset and
_remote_model_obj that try_remote_fit sets, in that order;
- the summary of an evaluate has the server's id of the summary and
__source_transformer__, the model;
- a model with a training summary has _summary.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]