Rakesh Kumar created SPARK-62108:
------------------------------------

             Summary: Grouped and cogrouped Arrow Python UDFs keep a task 
completion listener per group until the task ends
                 Key: SPARK-62108
                 URL: https://issues.apache.org/jira/browse/SPARK-62108
             Project: Spark
          Issue Type: Bug
          Components: SQL, PySpark
    Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0
            Reporter: Rakesh Kumar


Since Apache Spark 4.1.0 (SPARK-53562, SPARK-53609, SPARK-53610, SPARK-53611), 
the grouped and cogrouped Arrow Python runners write each group, and each side 
of each cogroup, as a separate Arrow stream through an {{ArrowWriterWrapper}}. 
The wrapper's constructor adds a task completion listener that closes it. 
{{TaskContextImpl}} keeps every listener until the task completes, and each one 
keeps its closure and its wrapper reachable. {{ArrowWriterWrapper.close()}} 
releases the Arrow buffers at the end of the group, but the listener, the 
closure and the wrapper stay on the heap, about 72 bytes per group plus a slot 
in the listener stack. The heap held by a task therefore grows linearly with 
its number of groups and is released only when the task ends.

Affected operators: {{applyInPandas}} and {{applyInArrow}} on grouped data, 
grouped aggregate pandas and Arrow UDFs, window pandas and Arrow UDFs, 
incremental Python aggregators ({{PythonIncrementalAggregateExec}}), and 
{{applyInPandas}} and {{applyInArrow}} on cogrouped data (two listeners per 
cogroup, including empty sides).

Reproduction: {{applyInArrow}} with an identity function over 10,000,000 rows 
in 1-row groups, 4 shuffle partitions (2,500,000 groups per task), 
{{local[4]}}, JDK 21, written to the {{noop}} sink, with {{jmap -histo:live}} 
of the JVM during the tasks (MB = 10^6 bytes):

{code:python}
from pyspark.sql import functions as F

spark.conf.set("spark.sql.shuffle.partitions", "4")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
df = spark.range(0, 10_000_000, 1, 4).select(F.col("id").alias("k"), 
F.col("id").alias("v"))
df.groupBy("k").applyInArrow(lambda t: t, "k long, v long") \
    .write.format("noop").mode("overwrite").save()
{code}

||Time into the job||60 s||120 s||180 s||
|Live ArrowWriterWrapper objects|2,925,928|5,997,525|9,117,860|
|Live task completion listeners|2,925,937|5,997,534|9,117,869|
|Live heap above the run with one listener per runner|233 MB|477 MB|705 MB|

At 180 s, the wrappers (364.7 MB), the listeners (145.9 MB) and their closures 
(145.9 MB) hold 656 MB. With one listener per runner instead of per group, the 
live heap does not grow during the job. With the default 1 GB driver memory 
(the driver JVM runs the tasks in local mode), the summed task GC time is 27.6 
s, against 5.1 s with one listener per runner, at about the same wall time 
(198.5 s and 199.7 s). With more groups per task, the heap keeps growing until 
the task ends.

Proposed fix: register one task completion listener per runner that closes the 
wrapper of the group being written, and let 
{{ArrowWriterWrapper.createAndStartArrowWriter}} close what it allocated if 
starting the stream fails. The data sent to the Python worker does not change.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to