[
https://issues.apache.org/jira/browse/SPARK-61971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61971:
-----------------------------------
Labels: pull-request-available (was: )
> Report the UDF functions and DataFrames of grouped data, cogroups, table
> arguments, window specs, writers, merges and stream writers to Python's
> garbage collector, so that their reference cycles are collected as PySpark's
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61971
> URL: https://issues.apache.org/jira/browse/SPARK-61971
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> A regression of SPARK-61881 (#792), from a round-3 review against
> pyspark-client 4.3.0: since the expressions of a UDF hold its function,
> rather than its pickle, a reference cycle through that function and an object
> that holds it is collected only if the object reports the function to
> Python's garbage collector. Only Column and DataFrame did. So `h = Holder();
> h.back = <helper of a UDF of h>; del h; gc.collect()` leaked h for these
> helpers, where PySpark collects all of them (they are plain objects that hold
> the Columns or plans):
> - GroupedData, of a groupBy of a UDF or of a DataFrame of a UDF;
> - cogroups;
> - asTable() of a DataFrame of a UDF;
> - WindowSpec of partitionBy or orderBy of a UDF;
> - df.write, df.writeTo, df.na and df.stat;
> - mergeInto, and its clauses of a UDF condition;
> - writeStream.
> The same held for a cycle through the DataFrame of such a helper (df.back =
> df.groupBy("id"), and the same with write, writeTo, na, stat and cogroup),
> which PySpark collects. The helpers held a clone of the frame's plan, so the
> frame stopped reporting the plan's objects too.
> writeStream kept the functions of foreachBatch and foreach, where PySpark
> keeps only their pickles. So `s.w = sdf.writeStream.foreachBatch(s); del s`
> never freed s, which PySpark frees by its last reference.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]