Hyukjin Kwon created SPARK-61971:
------------------------------------

             Summary: Report the UDF functions and DataFrames of grouped data, 
cogroups, table arguments, window specs, writers, merges and stream writers to 
Python's garbage collector, so that their reference cycles are collected as 
PySpark's
                 Key: SPARK-61971
                 URL: https://issues.apache.org/jira/browse/SPARK-61971
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


A regression of SPARK-61881 (#792), from a round-3 review against 
pyspark-client 4.3.0: since the expressions of a UDF hold its function, rather 
than its pickle, a reference cycle through that function and an object that 
holds it is collected only if the object reports the function to Python's 
garbage collector. Only Column and DataFrame did. So `h = Holder(); h.back = 
<helper of a UDF of h>; del h; gc.collect()` leaked h for these helpers, where 
PySpark collects all of them (they are plain objects that hold the Columns or 
plans):
- GroupedData, of a groupBy of a UDF or of a DataFrame of a UDF;
- cogroups;
- asTable() of a DataFrame of a UDF;
- WindowSpec of partitionBy or orderBy of a UDF;
- df.write, df.writeTo, df.na and df.stat;
- mergeInto, and its clauses of a UDF condition;
- writeStream.

The same held for a cycle through the DataFrame of such a helper (df.back = 
df.groupBy("id"), and the same with write, writeTo, na, stat and cogroup), 
which PySpark collects. The helpers held a clone of the frame's plan, so the 
frame stopped reporting the plan's objects too.

writeStream kept the functions of foreachBatch and foreach, where PySpark keeps 
only their pickles. So `s.w = sdf.writeStream.foreachBatch(s); del s` never 
freed s, which PySpark frees by its last reference.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to