Hyukjin Kwon created SPARK-61877:
------------------------------------

             Summary: Make the native ML summaries and stat classes construct, 
copy, pickle and document as PySpark's, and fetch StopWordsRemover's defaults 
once
                 Key: SPARK-61877
                 URL: https://issues.apache.org/jira/browse/SPARK-61877
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


The native ML summaries and stat classes of the drop-in differ from PySpark 
4.3.0's:
- The DataFrame members of a summary built as cls("<id>.summary") return a lazy 
DataFrame that fails at execution, where PySpark raises AttributeError 
'_summary_dataset'. A summary built by the user (cls() or cls("abc")) cannot be 
copied or pickled, and its _java_obj is None, where PySpark's copies and 
pickles as a plain object and keeps the given value. Pickling a fitted training 
summary or an evaluate() summary raises the session's AttributeError, where 
PySpark raises the TypeError of pickling a lock. Summary members have empty 
docstrings.
- ChiSquareTest, Correlation, KolmogorovSmirnovTest, Summarizer and 
SummaryBuilder cannot be constructed, where PySpark's are plain classes; 
SummaryBuilder's signature and errors are JavaWrapper's.
- pyspark.ml.stat defines an __all__ that PySpark's lacks.
- StopWordsRemover fetches its server defaults twice: 4 requests at 
construction where PySpark sends 2, and 5 at a load where PySpark sends 3.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to