[
https://issues.apache.org/jira/browse/SPARK-61333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61333:
------------------------------------
Assignee: Hyukjin Kwon
> Release pyspark.ml models whose summary keeps a DataFrame of them, as
> PySpark's garbage collector does
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61333
> URL: https://issues.apache.org/jira/browse/SPARK-61333
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In the Python drop-in, a pyspark.ml KMeans, BisectingKMeans or
> GaussianMixture model is never released on the server: every fit leaks one
> server ML cache entry (offloaded to the driver's disk, and fits fail with
> MLCacheSizeOverflowException past the session's maxStorageSize) and its
> client objects. An elbow loop over k=2..5 leaves 4 models; official PySpark
> 4.3.0 leaves none.
> These models keep the default summary dataset, model.transform(dataset),
> whose plan keeps the model (__source_transformer__), a reference cycle that
> Python's garbage collector frees in PySpark, where the DataFrame is a Python
> object. In the drop-in, the DataFrame is a native object that keeps the
> Python plan in a Rust owner, which the collector cannot see, so the cycle is
> never freed. Other models (LogisticRegression, LinearSVC, ...) return the
> training dataset as their summary dataset and have no cycle.
> The fix gives the native DataFrame garbage-collector support: it reports the
> Python plans that only it keeps alive (every reference on the way held once),
> and drops its plan when the collector breaks a cycle. A DataFrame that still
> uses the model keeps it, as in PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]