[ 
https://issues.apache.org/jira/browse/SPARK-61877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61877.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 789
[https://github.com/apache/spark-connect-rust/pull/789]

> Make the native ML summaries and stat classes construct, copy, pickle and 
> document as PySpark's, and fetch StopWordsRemover's defaults once
> -------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61877
>                 URL: https://issues.apache.org/jira/browse/SPARK-61877
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> The native ML summaries and stat classes of the drop-in differ from PySpark 
> 4.3.0's:
> - The DataFrame members of a summary built as cls("<id>.summary") return a 
> lazy DataFrame that fails at execution, where PySpark raises AttributeError 
> '_summary_dataset'. A summary built by the user (cls() or cls("abc")) cannot 
> be copied or pickled, and its _java_obj is None, where PySpark's copies and 
> pickles as a plain object and keeps the given value. Pickling a fitted 
> training summary or an evaluate() summary raises the session's 
> AttributeError, where PySpark raises the TypeError of pickling a lock. 
> Summary members have empty docstrings.
> - ChiSquareTest, Correlation, KolmogorovSmirnovTest, Summarizer and 
> SummaryBuilder cannot be constructed, where PySpark's are plain classes; 
> SummaryBuilder's signature and errors are JavaWrapper's.
> - pyspark.ml.stat defines an __all__ that PySpark's lacks.
> - StopWordsRemover fetches its server defaults twice: 4 requests at 
> construction where PySpark sends 2, and 5 at a load where PySpark sends 3.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to