Hyukjin Kwon created SPARK-62014:
------------------------------------
Summary: Keep the state of Python subclasses of the native ML
classes in their pickles and copies, give pyspark.ml.stat PySpark's docs, and
make the ML objects' pickles, uids, metadata and attributes PySpark's
Key: SPARK-62014
URL: https://issues.apache.org/jira/browse/SPARK-62014
Project: Spark
Issue Type: Bug
Components: ML
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
>From a review of the documented differences of ML (docs/compatibility.md,
>dev/parity/known-differences.md) against PySpark 4.3.0, and the other
>session's review of the last 100 commits:
- A Python subclass's __getstate__, __setstate__ and __reduce__ of a native ML
class are ignored: a pickle or a deep copy of a subclass of Tokenizer loses its
uid and Params (getInputCol raises "Param ... does not belong to"), and with
__getstate__ only, a pickled maxIter=3 becomes 100; copy.copy and Params.copy
ignore them too. uid is a native property, not in __dict__.
- The static methods of ChiSquareTest, Correlation, KolmogorovSmirnovTest and
Summarizer lack PySpark's signatures, annotations and docstrings, and the
classes PySpark's docstrings.
- DataFrames of the vendored pyspark.ml models pickle and deep copy, where
PySpark raises "cannot pickle '_thread.lock'"; the plan walk of the pickle
check does not enter subqueries.
- A validator subclass's own Params are not saved in its metadata; a subclass
of PipelineModel, OneVsRestModel or a validator model has the uid of its base
class; Pipeline, OneVsRest, the validators and their models have a _java_obj
attribute, which PySpark's do not; an object with sparkSession but no _plan
raises a TypeError where PySpark raises the AttributeError of _plan; the
_remote_model_obj of a summary is not the model's reference; members of the
native classes do not cloudpickle; a Pipeline of a one-shot iterator of stages
saves no stages.
- Several ML entries of the docs are stale or lack their reasons (uid
assignment, CrossValidator's setParams, OneVsRest's write, the cause of the
RandomForest fold failure, the non-active session, shallow copies).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]