This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new df15e7ac35b4 [SPARK-60737][SQL][PYTHON] Close the Arrow input of 
Python UDFs when the task ends before the input is exhausted
df15e7ac35b4 is described below

commit df15e7ac35b4493bb3ed48aa70fe0797f62c4661
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 29 20:25:31 2026 -0700

    [SPARK-60737][SQL][PYTHON] Close the Arrow input of Python UDFs when the 
task ends before the input is exhausted
    
    ### What changes were proposed in this pull request?
    
    This PR closes the Arrow buffers of the input to Python workers when the 
task completes before all of the input has been written.
    
    1. `PythonArrowInput` adds a task completion listener in `newWriter` that 
closes `root` and `allocator`. The listener and `close()`, which runs once the 
input is exhausted, share `closeRootAndAllocator()`. It closes them only once, 
and only if `writer`, which creates them, has been created, so that the 
listener does not create the lazy `root` and `allocator` (SPARK-55285). The 
listener does not end `writer`, since the stream that it writes to may be 
closed already, and nothing sends i [...]
    2. `ArrowWriterWrapper.createAndStartArrowWriter`, which the grouped and 
cogrouped runners call for each group, creates the wrapper, which adds the task 
completion listener that closes its `root` and `allocator`, before anything 
that can fail. It used to allocate the vectors and write the schema first, so a 
failure there, e.g. an interrupt while writing the schema, left them open.
    
    ### Why are the changes needed?
    
    `PythonArrowInput` closes `root` and its child allocator of 
`ArrowUtils.rootAllocator` only at the end of the input. If the task completes 
before that, nothing closes them, and the buffers of `root` stay allocated for 
the lifetime of the executor: Arrow does not release buffers on GC, and the 
root allocator keeps a reference to each child allocator until it is closed. A 
task completes before the end of its input when it
    
    - fails, e.g. when the Python UDF raises an exception or reading the input 
fails,
    - is killed, or
    - stops reading the output early, e.g. with a `LIMIT` above a filter on the 
UDF result.
    
    `ArrowWriter.reset()` keeps the capacity of the vectors, so `root` holds 
the buffers of a batch from the first batch on. In local mode with 
`spark.sql.execution.arrow.maxRecordsPerBatch=100` and 1 KiB strings, each such 
task left 147,456 bytes allocated and its child allocator (`stdout writer for 
<python>`) open.
    
    This affects the runners that use `PythonArrowInput` directly: scalar 
pandas and Arrow UDFs (`ArrowEvalPythonExec`), Arrow Python UDTFs, 
`mapInPandas` and `mapInArrow`, `applyInPandasWithState`, and 
`transformWithState` in PySpark, as well as the partial aggregation of 
`PythonIncrementalAggregateExec` and the columnar input enabled by 
`spark.sql.execution.arrow.pythonUDF.columnarInput.enabled`. It is a regression 
from SPARK-44705 (Apache Spark 4.0.0): before it, the writer thread clos [...]
    
    The grouped and cogrouped runners already close the `ArrowWriterWrapper` of 
each group in a task completion listener, but `createAndStartArrowWriter` added 
the listener only after allocating the vectors and writing the schema. With 
`spark.python.udf.pipelined.enabled`, a grouped aggregate pandas UDF under a 
`LIMIT` sometimes left 49,152 bytes, when the writer was interrupted there at 
task completion.
    
    The new listener does not race with the writer, since task completion 
listeners run after the writer is done. The writer runs on the task thread, 
except in the pipelined mode, where the pre-completion listener added by 
SPARK-60113 stops the writer thread and waits for it to exit before any 
completion listener runs.
    
    SPARK-55351 also closed them in a task completion listener but was 
reverted. Unlike it, this PR keeps `root` and `allocator` lazy, and the 
listener does not end the writer.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, except that the Arrow memory of the input to Python workers is released 
when a task completes before the end of its input.
    
    ### How was this patch tested?
    
    - Added a test to `PythonUDFSuite`. It runs a scalar pandas UDF over 
100,000 rows of 1 KiB strings, more than the socket to the Python worker can 
buffer, in two jobs: one whose input fails after 1,000 rows, and one that stops 
reading the output early with a `LIMIT` above a filter on the UDF result. It 
runs both with and without `spark.python.udf.pipelined.enabled`, and checks 
that no allocator of the Arrow input is left open. Without this PR, it fails 
with two allocators left open, ea [...]
    - Added a test to `ArrowWriterSuite` in which writing the schema fails in 
`ArrowWriterWrapper.createAndStartArrowWriter`. Without this PR, the allocator 
stays open after the task completes.
    - `StatefulPythonRunnerDataSentSuite`, which drives `newWriter` directly, 
passes `TaskContext.empty()` instead of `null`, since the writer now adds a 
task completion listener to it.
    - Also ran all suites in `org.apache.spark.sql.execution.python`, 
`FlatMapGroupsInPandasWithStateSuite`, 
`FlatMapGroupsInPandasWithStateDistributionSuite`, and `ArrowConvertersSuite`.
    
    The `PythonUDFSuite` test needs a Python with pandas and PyArrow, and is 
skipped otherwise.
    
    ```
    PYSPARK_PYTHON=<python with pandas and pyarrow> build/sbt 'sql/testOnly 
org.apache.spark.sql.execution.python.PythonUDFSuite -- -z SPARK-60737'
    build/sbt 'sql/testOnly 
org.apache.spark.sql.execution.arrow.ArrowWriterSuite -- -z ArrowWriterWrapper'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59946 from dongjoon-hyun/SPARK-60737.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../sql/execution/arrow/ArrowWriterWrapper.scala   | 13 +++---
 .../sql/execution/python/PythonArrowInput.scala    | 21 +++++++++
 .../sql/execution/arrow/ArrowWriterSuite.scala     | 22 ++++++++-
 .../sql/execution/python/PythonUDFSuite.scala      | 52 +++++++++++++++++++++-
 .../StatefulPythonRunnerDataSentSuite.scala        |  4 +-
 5 files changed, 102 insertions(+), 10 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriterWrapper.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriterWrapper.scala
index 0008c1319ec9..4a729b3aed9b 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriterWrapper.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriterWrapper.scala
@@ -76,11 +76,14 @@ object ArrowWriterWrapper {
     val allocator = ArrowUtils.rootAllocator.newChildAllocator(
       s"stdout writer for $allocatorOwner", 0, Long.MaxValue)
     val root = VectorSchemaRoot.create(arrowSchema, allocator)
-    val arrowWriter = SparkArrowWriter.create(root)
-
-    val streamWriter = new ArrowStreamWriter(root, null, dataOut)
-    streamWriter.start()
+    // Create the wrapper, which adds the task completion listener that closes 
`root` and
+    // `allocator`, before anything that can fail: allocating the vectors, or 
writing the schema,
+    // e.g. when the writer is interrupted.
     // Unloader will be set by the caller after creation
-    ArrowWriterWrapper(streamWriter, arrowWriter, root, allocator, null, 
context)
+    val wrapper = ArrowWriterWrapper(null, null, root, allocator, null, 
context)
+    wrapper.arrowWriter = SparkArrowWriter.create(root)
+    wrapper.streamWriter = new ArrowStreamWriter(root, null, dataOut)
+    wrapper.streamWriter.start()
+    wrapper
   }
 }
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
index b0c457615382..596a330e2026 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
@@ -73,6 +73,9 @@ private[python] trait PythonArrowInput[IN] { self: 
BasePythonRunner[IN, _] =>
 
   protected var writer: ArrowStreamWriter = _
 
+  // Whether `root` and `allocator` have been closed.
+  private var rootAndAllocatorClosed = false
+
   protected def close(): Unit = {
     Utils.tryWithSafeFinally {
       // end writes footer to the output stream and doesn't clean any 
resources.
@@ -80,6 +83,19 @@ private[python] trait PythonArrowInput[IN] { self: 
BasePythonRunner[IN, _] =>
       // in the try block.
       writer.end()
     } {
+      closeRootAndAllocator()
+    }
+  }
+
+  /**
+   * Closes `root` and `allocator` if `writer` has been created, which creates 
them, and they are
+   * not closed yet. Besides `close()` at the end of the input, a task 
completion listener calls
+   * this, since the task can complete before the input is exhausted, e.g. 
when it fails, is
+   * killed, or stops reading the output early.
+   */
+  private def closeRootAndAllocator(): Unit = {
+    if (writer != null && !rootAndAllocatorClosed) {
+      rootAndAllocatorClosed = true
       root.close()
       allocator.close()
     }
@@ -91,6 +107,11 @@ private[python] trait PythonArrowInput[IN] { self: 
BasePythonRunner[IN, _] =>
       inputIterator: Iterator[IN],
       partitionIndex: Int,
       context: TaskContext): Writer = {
+    // Task completion listeners run after the writer is done: it runs on the 
task thread unless
+    // it is pipelined, and the pipelined writer is stopped and awaited before 
them (see
+    // `BasePythonRunner.startPipelinedWriter`). The listener does not end 
`writer`, since the
+    // stream that it writes to may be closed already, and nothing sends it to 
the worker anymore.
+    context.addTaskCompletionListener[Unit](_ => closeRootAndAllocator())
     new Writer(env, worker, inputIterator, partitionIndex, context) {
 
       protected override def writeCommand(dataOut: DataOutputStream): Unit = {
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
index 7f27c1d0e7aa..1932433d9234 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
@@ -17,13 +17,15 @@
 
 package org.apache.spark.sql.execution.arrow
 
+import java.io.{DataOutputStream, IOException, OutputStream}
+
 import scala.jdk.CollectionConverters._
 
 import org.apache.arrow.vector.{TimeStampVector, VectorSchemaRoot, 
ViewVarBinaryVector, ViewVarCharVector}
 import org.apache.arrow.vector.complex.ListViewVector
 import org.apache.arrow.vector.types.pojo.{ArrowType, FieldType}
 
-import org.apache.spark.{SparkArithmeticException, SparkFunSuite, 
SparkUnsupportedOperationException}
+import org.apache.spark.{SparkArithmeticException, SparkFunSuite, 
SparkUnsupportedOperationException, TaskContext}
 import org.apache.spark.sql.Row
 import org.apache.spark.sql.YearUDT
 import org.apache.spark.sql.catalyst.InternalRow
@@ -1300,4 +1302,22 @@ class ArrowWriterSuite extends SparkFunSuite {
 
     writer.root.close()
   }
+
+  test("SPARK-60737: ArrowWriterWrapper closes its allocator on task 
completion if starting the " +
+      "stream fails") {
+    val failingOut = new DataOutputStream(new OutputStream {
+      override def write(b: Int): Unit = throw new IOException("write failure")
+    })
+    val context = TaskContext.empty()
+    try {
+      intercept[IOException] {
+        ArrowWriterWrapper.createAndStartArrowWriter(new StructType().add("s", 
StringType),
+          "UTC", "ArrowWriterSuite", largeVarTypes = false, failingOut, 
context)
+      }
+    } finally {
+      context.markTaskCompleted(None)
+    }
+    assert(!ArrowUtils.rootAllocator.getChildAllocators.asScala
+      .exists(_.getName == "stdout writer for ArrowWriterSuite"))
+  }
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
index 9236ee9318d9..43dee0602039 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
@@ -18,12 +18,20 @@
 package org.apache.spark.sql.execution.python
 
 import scala.concurrent.duration._
+import scala.jdk.CollectionConverters._
 
-import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils, Row}
+import org.apache.arrow.memory.BufferAllocator
+
+import org.apache.spark.{SparkEnv, SparkException}
+import org.apache.spark.internal.config.Python.PYTHON_UDF_PIPELINED_EXECUTION
+import org.apache.spark.sql.{AnalysisException, DataFrame, 
IntegratedUDFTestUtils, Row}
+import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.functions.{aggregate, array, avg, col, count, lit, 
transform}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.LongType
+import org.apache.spark.sql.types.{LongType, StringType, StructType}
+import org.apache.spark.sql.util.ArrowUtils
+import org.apache.spark.unsafe.types.UTF8String
 
 class PythonUDFSuite extends SharedSparkSession {
   import testImplicits._
@@ -337,4 +345,44 @@ class PythonUDFSuite extends SharedSparkSession {
       }
     }
   }
+
+  test("SPARK-60737: Close the Arrow input of a pandas UDF when the task ends 
before the input") {
+    assume(shouldTestPandasUDFs)
+    registerTestUDF(TestScalarPandasUDF(name = "pandas_udf", returnType = 
Some(StringType)), spark)
+    val payload = UTF8String.fromString("x" * 1000)
+    // More input than the socket to the Python worker can buffer, so that the 
input is not
+    // exhausted when the task ends. Asking for the row at `failAt` throws.
+    def input(failAt: Int): DataFrame = {
+      val rdd = spark.sparkContext.parallelize(Seq(0), 1).mapPartitions { _ =>
+        Iterator.tabulate(100000) { i =>
+          if (i == failAt) throw new IllegalStateException("input failure")
+          InternalRow(payload)
+        }
+      }
+      spark.internalCreateDataFrame(rdd, new StructType().add("s", StringType))
+    }
+    // The open allocators of the Arrow input of Python workers.
+    def inputAllocators(): Set[BufferAllocator] = 
ArrowUtils.rootAllocator.getChildAllocators
+      .asScala.filter(_.getName.startsWith("stdout writer for")).toSet
+
+    Seq(false, true).foreach { pipelined =>
+      SparkEnv.get.conf.set(PYTHON_UDF_PIPELINED_EXECUTION, pipelined)
+      try {
+        withSQLConf(SQLConf.ARROW_EXECUTION_MAX_RECORDS_PER_BATCH.key -> 
"100") {
+          val allocatorsBefore = inputAllocators()
+          // The task fails.
+          val e = intercept[SparkException] {
+            input(failAt = 1000).selectExpr("pandas_udf(s)").collect()
+          }
+          assert(e.getMessage.contains("input failure"))
+          // The task stops reading the output early. The filter keeps the 
limit above the UDF.
+          assert(input(failAt = -1).selectExpr("pandas_udf(s) AS u").filter("u 
IS NOT NULL")
+            .limit(1).collect().length == 1)
+          assert((inputAllocators() -- allocatorsBefore).isEmpty)
+        }
+      } finally {
+        SparkEnv.get.conf.remove(PYTHON_UDF_PIPELINED_EXECUTION)
+      }
+    }
+  }
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/streaming/StatefulPythonRunnerDataSentSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/streaming/StatefulPythonRunnerDataSentSuite.scala
index 2f4499d19aa9..6ffb85cf7bc9 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/streaming/StatefulPythonRunnerDataSentSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/streaming/StatefulPythonRunnerDataSentSuite.scala
@@ -24,7 +24,7 @@ import scala.jdk.CollectionConverters._
 import org.apache.arrow.vector.ipc.WriteChannel
 import org.apache.arrow.vector.ipc.message.MessageSerializer
 
-import org.apache.spark.SparkEnv
+import org.apache.spark.{SparkEnv, TaskContext}
 import org.apache.spark.api.python.{BasePythonRunner, ChainedPythonFunctions, 
PythonBroadcast, PythonEvalType, SimplePythonFunction}
 import org.apache.spark.broadcast.Broadcast
 import org.apache.spark.sql.{QueryTest, Row}
@@ -60,7 +60,7 @@ class StatefulPythonRunnerDataSentSuite extends QueryTest 
with SharedSparkSessio
       MessageSerializer.serialize(
         new WriteChannel(Channels.newChannel(schemaMessage)), root.getSchema)
 
-      val inputWriter = newWriter(SparkEnv.get, null, input, 0, null)
+      val inputWriter = newWriter(SparkEnv.get, null, input, 0, 
TaskContext.empty())
       val buffer = new ByteArrayOutputStream()
       val sharedStream = new DataOutputStream(buffer)
       var hasInput = true


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to