[ 
https://issues.apache.org/jira/browse/SPARK-61333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61333.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 461
[https://github.com/apache/spark-connect-rust/pull/461]

> Release pyspark.ml models whose summary keeps a DataFrame of them, as 
> PySpark's garbage collector does
> ------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61333
>                 URL: https://issues.apache.org/jira/browse/SPARK-61333
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> In the Python drop-in, a pyspark.ml KMeans, BisectingKMeans or 
> GaussianMixture model is never released on the server: every fit leaks one 
> server ML cache entry (offloaded to the driver's disk, and fits fail with 
> MLCacheSizeOverflowException past the session's maxStorageSize) and its 
> client objects. An elbow loop over k=2..5 leaves 4 models; official PySpark 
> 4.3.0 leaves none.
> These models keep the default summary dataset, model.transform(dataset), 
> whose plan keeps the model (__source_transformer__), a reference cycle that 
> Python's garbage collector frees in PySpark, where the DataFrame is a Python 
> object. In the drop-in, the DataFrame is a native object that keeps the 
> Python plan in a Rust owner, which the collector cannot see, so the cycle is 
> never freed. Other models (LogisticRegression, LinearSVC, ...) return the 
> training dataset as their summary dataset and have no cycle.
> The fix gives the native DataFrame garbage-collector support: it reports the 
> Python plans that only it keeps alive (every reference on the way held once), 
> and drops its plan when the collector breaks a cycle. A DataFrame that still 
> uses the model keeps it, as in PySpark.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to