Rakesh Kumar created SPARK-62108:
------------------------------------
Summary: Grouped and cogrouped Arrow Python UDFs keep a task
completion listener per group until the task ends
Key: SPARK-62108
URL: https://issues.apache.org/jira/browse/SPARK-62108
Project: Spark
Issue Type: Bug
Components: SQL, PySpark
Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0
Reporter: Rakesh Kumar
Since Apache Spark 4.1.0 (SPARK-53562, SPARK-53609, SPARK-53610, SPARK-53611),
the grouped and cogrouped Arrow Python runners write each group, and each side
of each cogroup, as a separate Arrow stream through an {{ArrowWriterWrapper}}.
The wrapper's constructor adds a task completion listener that closes it.
{{TaskContextImpl}} keeps every listener until the task completes, and each one
keeps its closure and its wrapper reachable. {{ArrowWriterWrapper.close()}}
releases the Arrow buffers at the end of the group, but the listener, the
closure and the wrapper stay on the heap, about 72 bytes per group plus a slot
in the listener stack. The heap held by a task therefore grows linearly with
its number of groups and is released only when the task ends.
Affected operators: {{applyInPandas}} and {{applyInArrow}} on grouped data,
grouped aggregate pandas and Arrow UDFs, window pandas and Arrow UDFs,
incremental Python aggregators ({{PythonIncrementalAggregateExec}}), and
{{applyInPandas}} and {{applyInArrow}} on cogrouped data (two listeners per
cogroup, including empty sides).
Reproduction: {{applyInArrow}} with an identity function over 10,000,000 rows
in 1-row groups, 4 shuffle partitions (2,500,000 groups per task),
{{local[4]}}, JDK 21, written to the {{noop}} sink, with {{jmap -histo:live}}
of the JVM during the tasks (MB = 10^6 bytes):
{code:python}
from pyspark.sql import functions as F
spark.conf.set("spark.sql.shuffle.partitions", "4")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
df = spark.range(0, 10_000_000, 1, 4).select(F.col("id").alias("k"),
F.col("id").alias("v"))
df.groupBy("k").applyInArrow(lambda t: t, "k long, v long") \
.write.format("noop").mode("overwrite").save()
{code}
||Time into the job||60 s||120 s||180 s||
|Live ArrowWriterWrapper objects|2,925,928|5,997,525|9,117,860|
|Live task completion listeners|2,925,937|5,997,534|9,117,869|
|Live heap above the run with one listener per runner|233 MB|477 MB|705 MB|
At 180 s, the wrappers (364.7 MB), the listeners (145.9 MB) and their closures
(145.9 MB) hold 656 MB. With one listener per runner instead of per group, the
live heap does not grow during the job. With the default 1 GB driver memory
(the driver JVM runs the tasks in local mode), the summed task GC time is 27.6
s, against 5.1 s with one listener per runner, at about the same wall time
(198.5 s and 199.7 s). With more groups per task, the heap keeps growing until
the task ends.
Proposed fix: register one task completion listener per runner that closes the
wrapper of the group being written, and let
{{ArrowWriterWrapper.createAndStartArrowWriter}} close what it allocated if
starting the stream fails. The data sent to the Python worker does not change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]