Hyukjin Kwon created SPARK-61333:
------------------------------------
Summary: Release pyspark.ml models whose summary keeps a DataFrame
of them, as PySpark's garbage collector does
Key: SPARK-61333
URL: https://issues.apache.org/jira/browse/SPARK-61333
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In the Python drop-in, a pyspark.ml KMeans, BisectingKMeans or GaussianMixture
model is never released on the server: every fit leaks one server ML cache
entry (offloaded to the driver's disk, and fits fail with
MLCacheSizeOverflowException past the session's maxStorageSize) and its client
objects. An elbow loop over k=2..5 leaves 4 models; official PySpark 4.3.0
leaves none.
These models keep the default summary dataset, model.transform(dataset), whose
plan keeps the model (__source_transformer__), a reference cycle that Python's
garbage collector frees in PySpark, where the DataFrame is a Python object. In
the drop-in, the DataFrame is a native object that keeps the Python plan in a
Rust owner, which the collector cannot see, so the cycle is never freed. Other
models (LogisticRegression, LinearSVC, ...) return the training dataset as
their summary dataset and have no cycle.
The fix gives the native DataFrame garbage-collector support: it reports the
Python plans that only it keeps alive (every reference on the way held once),
and drops its plan when the collector breaks a cycle. A DataFrame that still
uses the model keeps it, as in PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]