[ 
https://issues.apache.org/jira/browse/SPARK-62108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62108:
-----------------------------------
    Labels: pull-request-available  (was: )

> Grouped and cogrouped Arrow Python UDFs keep a task completion listener per 
> group until the task ends
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62108
>                 URL: https://issues.apache.org/jira/browse/SPARK-62108
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, SQL
>    Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0
>            Reporter: Rakesh Kumar
>            Priority: Major
>              Labels: pull-request-available
>
> Since Apache Spark 4.1.0 (SPARK-53562, SPARK-53609, SPARK-53610, 
> SPARK-53611), the grouped and cogrouped Arrow Python runners write each 
> group, and each side of each cogroup, as a separate Arrow stream through an 
> {{ArrowWriterWrapper}}. The wrapper's constructor adds a task completion 
> listener that closes it. {{TaskContextImpl}} keeps every listener until the 
> task completes, and each one keeps its closure and its wrapper reachable. 
> {{ArrowWriterWrapper.close()}} releases the Arrow buffers at the end of the 
> group, but the listener, the closure and the wrapper stay on the heap, about 
> 72 bytes per group plus a slot in the listener stack. The heap held by a task 
> therefore grows linearly with its number of groups and is released only when 
> the task ends.
> Affected operators: {{applyInPandas}} and {{applyInArrow}} on grouped data, 
> grouped aggregate pandas and Arrow UDFs, window pandas and Arrow UDFs, 
> incremental Python aggregators ({{PythonIncrementalAggregateExec}}), and 
> {{applyInPandas}} and {{applyInArrow}} on cogrouped data (two listeners per 
> cogroup, including empty sides).
> Reproduction: {{applyInArrow}} with an identity function over 10,000,000 rows 
> in 1-row groups, 4 shuffle partitions (2,500,000 groups per task), 
> {{local[4]}}, JDK 21, written to the {{noop}} sink, with {{jmap -histo:live}} 
> of the JVM during the tasks (MB = 10^6 bytes):
> {code:python}
> from pyspark.sql import functions as F
> spark.conf.set("spark.sql.shuffle.partitions", "4")
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> df = spark.range(0, 10_000_000, 1, 4).select(F.col("id").alias("k"), 
> F.col("id").alias("v"))
> df.groupBy("k").applyInArrow(lambda t: t, "k long, v long") \
>     .write.format("noop").mode("overwrite").save()
> {code}
> ||Time into the job||60 s||120 s||180 s||
> |Live ArrowWriterWrapper objects|2,925,928|5,997,525|9,117,860|
> |Live task completion listeners|2,925,937|5,997,534|9,117,869|
> |Live heap above the run with one listener per runner|233 MB|477 MB|705 MB|
> At 180 s, the wrappers (364.7 MB), the listeners (145.9 MB) and their 
> closures (145.9 MB) hold 656 MB. With one listener per runner instead of per 
> group, the live heap does not grow during the job. With the default 1 GB 
> driver memory (the driver JVM runs the tasks in local mode), the summed task 
> GC time is 27.6 s, against 5.1 s with one listener per runner, at about the 
> same wall time (198.5 s and 199.7 s). With more groups per task, the heap 
> keeps growing until the task ends.
> Proposed fix: register one task completion listener per runner that closes 
> the wrapper of the group being written, and let 
> {{ArrowWriterWrapper.createAndStartArrowWriter}} close what it allocated if 
> starting the stream fails. The data sent to the Python worker does not change.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to