[
https://issues.apache.org/jira/browse/SPARK-62108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62108:
-----------------------------------
Labels: pull-request-available (was: )
> Grouped and cogrouped Arrow Python UDFs keep a task completion listener per
> group until the task ends
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-62108
> URL: https://issues.apache.org/jira/browse/SPARK-62108
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Reporter: Rakesh Kumar
> Priority: Major
> Labels: pull-request-available
>
> Since Apache Spark 4.1.0 (SPARK-53562, SPARK-53609, SPARK-53610,
> SPARK-53611), the grouped and cogrouped Arrow Python runners write each
> group, and each side of each cogroup, as a separate Arrow stream through an
> {{ArrowWriterWrapper}}. The wrapper's constructor adds a task completion
> listener that closes it. {{TaskContextImpl}} keeps every listener until the
> task completes, and each one keeps its closure and its wrapper reachable.
> {{ArrowWriterWrapper.close()}} releases the Arrow buffers at the end of the
> group, but the listener, the closure and the wrapper stay on the heap, about
> 72 bytes per group plus a slot in the listener stack. The heap held by a task
> therefore grows linearly with its number of groups and is released only when
> the task ends.
> Affected operators: {{applyInPandas}} and {{applyInArrow}} on grouped data,
> grouped aggregate pandas and Arrow UDFs, window pandas and Arrow UDFs,
> incremental Python aggregators ({{PythonIncrementalAggregateExec}}), and
> {{applyInPandas}} and {{applyInArrow}} on cogrouped data (two listeners per
> cogroup, including empty sides).
> Reproduction: {{applyInArrow}} with an identity function over 10,000,000 rows
> in 1-row groups, 4 shuffle partitions (2,500,000 groups per task),
> {{local[4]}}, JDK 21, written to the {{noop}} sink, with {{jmap -histo:live}}
> of the JVM during the tasks (MB = 10^6 bytes):
> {code:python}
> from pyspark.sql import functions as F
> spark.conf.set("spark.sql.shuffle.partitions", "4")
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> df = spark.range(0, 10_000_000, 1, 4).select(F.col("id").alias("k"),
> F.col("id").alias("v"))
> df.groupBy("k").applyInArrow(lambda t: t, "k long, v long") \
> .write.format("noop").mode("overwrite").save()
> {code}
> ||Time into the job||60 s||120 s||180 s||
> |Live ArrowWriterWrapper objects|2,925,928|5,997,525|9,117,860|
> |Live task completion listeners|2,925,937|5,997,534|9,117,869|
> |Live heap above the run with one listener per runner|233 MB|477 MB|705 MB|
> At 180 s, the wrappers (364.7 MB), the listeners (145.9 MB) and their
> closures (145.9 MB) hold 656 MB. With one listener per runner instead of per
> group, the live heap does not grow during the job. With the default 1 GB
> driver memory (the driver JVM runs the tasks in local mode), the summed task
> GC time is 27.6 s, against 5.1 s with one listener per runner, at about the
> same wall time (198.5 s and 199.7 s). With more groups per task, the heap
> keeps growing until the task ends.
> Proposed fix: register one task completion listener per runner that closes
> the wrapper of the group being written, and let
> {{ArrowWriterWrapper.createAndStartArrowWriter}} close what it allocated if
> starting the stream fails. The data sent to the Python worker does not change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]