[
https://issues.apache.org/jira/browse/SPARK-61932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61932:
------------------------------------
Assignee: Hyukjin Kwon
> Pickle the native ML objects at every protocol and refuse the pickles of the
> DataFrames that keep a model as PySpark does, and give the models and
> summaries PySpark's _java_obj
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61932
> URL: https://issues.apache.org/jira/browse/SPARK-61932
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> A round-2 review of the native ML classes (pyspark.ml.connect) against
> pyspark-client 4.3.0, each item probed on both clients:
> 1. At pickle protocols 0 and 1, the estimators, transformers, evaluators,
> Pipelines, validators, OneVsRest and PrefixSpan raised "TypeError: Method
> __init__ forces keyword arguments.", and unbound models raised an
> AssertionError or a PicklingError ("it's not the same object as
> pyspark.ml.connect.pipeline.PipelineModel"). PySpark's, Python objects,
> pickle at every protocol, and a model of a server-side object raises the
> error of the pickle of the lock of its RemoteModelRef at the protocol (a
> PicklingError below 2, the TypeError "cannot pickle '_thread.lock' object"
> from 2).
> 2. A DataFrame that keeps a model in its plan pickled as any DataFrame: a
> plain pickle raised the AttributeError of the session's local function, and
> cloudpickle and copy.deepcopy copied it. Such DataFrames are a transform of a
> fitted model, a DataFrame that reads one, and a DataFrame of a method of a
> model or of the summary of an evaluate. PySpark pickles the plan before the
> session, so its pickle, cloudpickle and deepcopy of them raise the error of
> the lock of the model's RemoteModelRef. For a DataFrame of a method of a
> training summary (summary.predictions), PySpark's pickle raises the
> AttributeError of the session, and cloudpickle the TypeError of the lock of
> the summary's model (_remote_model_obj). PySpark's copy.deepcopy requests the
> schemas of the DataFrames it copies on the way before it raises.
> 3. The models of server-side objects had a _java_obj of None, and their
> summaries had no object id, so PySpark's own _call_java(name, *args) of them
> failed: "AssertionError" for a model, and "'NoneType' object has no attribute
> 'split'" for a summary. Over Spark Connect, PySpark's _call_java sends a
> Fetch (try_remote_call). In PySpark:
> - a model's _java_obj is the RemoteModelRef of its object, after a fit, a
> load, a copy, a fetch, or a class method such as
> StringIndexerModel.from_labels;
> - a training summary has "<id>.summary", and the _summary_dataset and
> _remote_model_obj that try_remote_fit sets, in that order;
> - the summary of an evaluate has the server's id of the summary and
> __source_transformer__, the model;
> - a model with a training summary has _summary.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]