Hyukjin Kwon created SPARK-61971:
------------------------------------
Summary: Report the UDF functions and DataFrames of grouped data,
cogroups, table arguments, window specs, writers, merges and stream writers to
Python's garbage collector, so that their reference cycles are collected as
PySpark's
Key: SPARK-61971
URL: https://issues.apache.org/jira/browse/SPARK-61971
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
A regression of SPARK-61881 (#792), from a round-3 review against
pyspark-client 4.3.0: since the expressions of a UDF hold its function, rather
than its pickle, a reference cycle through that function and an object that
holds it is collected only if the object reports the function to Python's
garbage collector. Only Column and DataFrame did. So `h = Holder(); h.back =
<helper of a UDF of h>; del h; gc.collect()` leaked h for these helpers, where
PySpark collects all of them (they are plain objects that hold the Columns or
plans):
- GroupedData, of a groupBy of a UDF or of a DataFrame of a UDF;
- cogroups;
- asTable() of a DataFrame of a UDF;
- WindowSpec of partitionBy or orderBy of a UDF;
- df.write, df.writeTo, df.na and df.stat;
- mergeInto, and its clauses of a UDF condition;
- writeStream.
The same held for a cycle through the DataFrame of such a helper (df.back =
df.groupBy("id"), and the same with write, writeTo, na, stat and cogroup),
which PySpark collects. The helpers held a clone of the frame's plan, so the
frame stopped reporting the plan's objects too.
writeStream kept the functions of foreachBatch and foreach, where PySpark keeps
only their pickles. So `s.w = sdf.writeStream.foreachBatch(s); del s` never
freed s, which PySpark frees by its last reference.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]