[ 
https://issues.apache.org/jira/browse/SPARK-61971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61971:
---------------------------------
    Fix Version/s: connect-rust-4.4.0

> Report the UDF functions and DataFrames of grouped data, cogroups, table 
> arguments, window specs, writers, merges and stream writers to Python's 
> garbage collector, so that their reference cycles are collected as PySpark's
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61971
>                 URL: https://issues.apache.org/jira/browse/SPARK-61971
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.4.0
>
>
> A regression of SPARK-61881 (#792), from a round-3 review against 
> pyspark-client 4.3.0: since the expressions of a UDF hold its function, 
> rather than its pickle, a reference cycle through that function and an object 
> that holds it is collected only if the object reports the function to 
> Python's garbage collector. Only Column and DataFrame did. So `h = Holder(); 
> h.back = <helper of a UDF of h>; del h; gc.collect()` leaked h for these 
> helpers, where PySpark collects all of them (they are plain objects that hold 
> the Columns or plans):
> - GroupedData, of a groupBy of a UDF or of a DataFrame of a UDF;
> - cogroups;
> - asTable() of a DataFrame of a UDF;
> - WindowSpec of partitionBy or orderBy of a UDF;
> - df.write, df.writeTo, df.na and df.stat;
> - mergeInto, and its clauses of a UDF condition;
> - writeStream.
> The same held for a cycle through the DataFrame of such a helper (df.back = 
> df.groupBy("id"), and the same with write, writeTo, na, stat and cogroup), 
> which PySpark collects. The helpers held a clone of the frame's plan, so the 
> frame stopped reporting the plan's objects too.
> writeStream kept the functions of foreachBatch and foreach, where PySpark 
> keeps only their pickles. So `s.w = sdf.writeStream.foreachBatch(s); del s` 
> never freed s, which PySpark frees by its last reference.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to