Hyukjin Kwon created SPARK-61877:
------------------------------------
Summary: Make the native ML summaries and stat classes construct,
copy, pickle and document as PySpark's, and fetch StopWordsRemover's defaults
once
Key: SPARK-61877
URL: https://issues.apache.org/jira/browse/SPARK-61877
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
The native ML summaries and stat classes of the drop-in differ from PySpark
4.3.0's:
- The DataFrame members of a summary built as cls("<id>.summary") return a lazy
DataFrame that fails at execution, where PySpark raises AttributeError
'_summary_dataset'. A summary built by the user (cls() or cls("abc")) cannot be
copied or pickled, and its _java_obj is None, where PySpark's copies and
pickles as a plain object and keeps the given value. Pickling a fitted training
summary or an evaluate() summary raises the session's AttributeError, where
PySpark raises the TypeError of pickling a lock. Summary members have empty
docstrings.
- ChiSquareTest, Correlation, KolmogorovSmirnovTest, Summarizer and
SummaryBuilder cannot be constructed, where PySpark's are plain classes;
SummaryBuilder's signature and errors are JavaWrapper's.
- pyspark.ml.stat defines an __all__ that PySpark's lacks.
- StopWordsRemover fetches its server defaults twice: 4 requests at
construction where PySpark sends 2, and 5 at a load where PySpark sends 3.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]