This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new df15e7ac35b4 [SPARK-60737][SQL][PYTHON] Close the Arrow input of
Python UDFs when the task ends before the input is exhausted
df15e7ac35b4 is described below
commit df15e7ac35b4493bb3ed48aa70fe0797f62c4661
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 29 20:25:31 2026 -0700
[SPARK-60737][SQL][PYTHON] Close the Arrow input of Python UDFs when the
task ends before the input is exhausted
### What changes were proposed in this pull request?
This PR closes the Arrow buffers of the input to Python workers when the
task completes before all of the input has been written.
1. `PythonArrowInput` adds a task completion listener in `newWriter` that
closes `root` and `allocator`. The listener and `close()`, which runs once the
input is exhausted, share `closeRootAndAllocator()`. It closes them only once,
and only if `writer`, which creates them, has been created, so that the
listener does not create the lazy `root` and `allocator` (SPARK-55285). The
listener does not end `writer`, since the stream that it writes to may be
closed already, and nothing sends i [...]
2. `ArrowWriterWrapper.createAndStartArrowWriter`, which the grouped and
cogrouped runners call for each group, creates the wrapper, which adds the task
completion listener that closes its `root` and `allocator`, before anything
that can fail. It used to allocate the vectors and write the schema first, so a
failure there, e.g. an interrupt while writing the schema, left them open.
### Why are the changes needed?
`PythonArrowInput` closes `root` and its child allocator of
`ArrowUtils.rootAllocator` only at the end of the input. If the task completes
before that, nothing closes them, and the buffers of `root` stay allocated for
the lifetime of the executor: Arrow does not release buffers on GC, and the
root allocator keeps a reference to each child allocator until it is closed. A
task completes before the end of its input when it
- fails, e.g. when the Python UDF raises an exception or reading the input
fails,
- is killed, or
- stops reading the output early, e.g. with a `LIMIT` above a filter on the
UDF result.
`ArrowWriter.reset()` keeps the capacity of the vectors, so `root` holds
the buffers of a batch from the first batch on. In local mode with
`spark.sql.execution.arrow.maxRecordsPerBatch=100` and 1 KiB strings, each such
task left 147,456 bytes allocated and its child allocator (`stdout writer for
<python>`) open.
This affects the runners that use `PythonArrowInput` directly: scalar
pandas and Arrow UDFs (`ArrowEvalPythonExec`), Arrow Python UDTFs,
`mapInPandas` and `mapInArrow`, `applyInPandasWithState`, and
`transformWithState` in PySpark, as well as the partial aggregation of
`PythonIncrementalAggregateExec` and the columnar input enabled by
`spark.sql.execution.arrow.pythonUDF.columnarInput.enabled`. It is a regression
from SPARK-44705 (Apache Spark 4.0.0): before it, the writer thread clos [...]
The grouped and cogrouped runners already close the `ArrowWriterWrapper` of
each group in a task completion listener, but `createAndStartArrowWriter` added
the listener only after allocating the vectors and writing the schema. With
`spark.python.udf.pipelined.enabled`, a grouped aggregate pandas UDF under a
`LIMIT` sometimes left 49,152 bytes, when the writer was interrupted there at
task completion.
The new listener does not race with the writer, since task completion
listeners run after the writer is done. The writer runs on the task thread,
except in the pipelined mode, where the pre-completion listener added by
SPARK-60113 stops the writer thread and waits for it to exit before any
completion listener runs.
SPARK-55351 also closed them in a task completion listener but was
reverted. Unlike it, this PR keeps `root` and `allocator` lazy, and the
listener does not end the writer.
### Does this PR introduce _any_ user-facing change?
No, except that the Arrow memory of the input to Python workers is released
when a task completes before the end of its input.
### How was this patch tested?
- Added a test to `PythonUDFSuite`. It runs a scalar pandas UDF over
100,000 rows of 1 KiB strings, more than the socket to the Python worker can
buffer, in two jobs: one whose input fails after 1,000 rows, and one that stops
reading the output early with a `LIMIT` above a filter on the UDF result. It
runs both with and without `spark.python.udf.pipelined.enabled`, and checks
that no allocator of the Arrow input is left open. Without this PR, it fails
with two allocators left open, ea [...]
- Added a test to `ArrowWriterSuite` in which writing the schema fails in
`ArrowWriterWrapper.createAndStartArrowWriter`. Without this PR, the allocator
stays open after the task completes.
- `StatefulPythonRunnerDataSentSuite`, which drives `newWriter` directly,
passes `TaskContext.empty()` instead of `null`, since the writer now adds a
task completion listener to it.
- Also ran all suites in `org.apache.spark.sql.execution.python`,
`FlatMapGroupsInPandasWithStateSuite`,
`FlatMapGroupsInPandasWithStateDistributionSuite`, and `ArrowConvertersSuite`.
The `PythonUDFSuite` test needs a Python with pandas and PyArrow, and is
skipped otherwise.
```
PYSPARK_PYTHON=<python with pandas and pyarrow> build/sbt 'sql/testOnly
org.apache.spark.sql.execution.python.PythonUDFSuite -- -z SPARK-60737'
build/sbt 'sql/testOnly
org.apache.spark.sql.execution.arrow.ArrowWriterSuite -- -z ArrowWriterWrapper'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59946 from dongjoon-hyun/SPARK-60737.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../sql/execution/arrow/ArrowWriterWrapper.scala | 13 +++---
.../sql/execution/python/PythonArrowInput.scala | 21 +++++++++
.../sql/execution/arrow/ArrowWriterSuite.scala | 22 ++++++++-
.../sql/execution/python/PythonUDFSuite.scala | 52 +++++++++++++++++++++-
.../StatefulPythonRunnerDataSentSuite.scala | 4 +-
5 files changed, 102 insertions(+), 10 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriterWrapper.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriterWrapper.scala
index 0008c1319ec9..4a729b3aed9b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriterWrapper.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriterWrapper.scala
@@ -76,11 +76,14 @@ object ArrowWriterWrapper {
val allocator = ArrowUtils.rootAllocator.newChildAllocator(
s"stdout writer for $allocatorOwner", 0, Long.MaxValue)
val root = VectorSchemaRoot.create(arrowSchema, allocator)
- val arrowWriter = SparkArrowWriter.create(root)
-
- val streamWriter = new ArrowStreamWriter(root, null, dataOut)
- streamWriter.start()
+ // Create the wrapper, which adds the task completion listener that closes
`root` and
+ // `allocator`, before anything that can fail: allocating the vectors, or
writing the schema,
+ // e.g. when the writer is interrupted.
// Unloader will be set by the caller after creation
- ArrowWriterWrapper(streamWriter, arrowWriter, root, allocator, null,
context)
+ val wrapper = ArrowWriterWrapper(null, null, root, allocator, null,
context)
+ wrapper.arrowWriter = SparkArrowWriter.create(root)
+ wrapper.streamWriter = new ArrowStreamWriter(root, null, dataOut)
+ wrapper.streamWriter.start()
+ wrapper
}
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
index b0c457615382..596a330e2026 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
@@ -73,6 +73,9 @@ private[python] trait PythonArrowInput[IN] { self:
BasePythonRunner[IN, _] =>
protected var writer: ArrowStreamWriter = _
+ // Whether `root` and `allocator` have been closed.
+ private var rootAndAllocatorClosed = false
+
protected def close(): Unit = {
Utils.tryWithSafeFinally {
// end writes footer to the output stream and doesn't clean any
resources.
@@ -80,6 +83,19 @@ private[python] trait PythonArrowInput[IN] { self:
BasePythonRunner[IN, _] =>
// in the try block.
writer.end()
} {
+ closeRootAndAllocator()
+ }
+ }
+
+ /**
+ * Closes `root` and `allocator` if `writer` has been created, which creates
them, and they are
+ * not closed yet. Besides `close()` at the end of the input, a task
completion listener calls
+ * this, since the task can complete before the input is exhausted, e.g.
when it fails, is
+ * killed, or stops reading the output early.
+ */
+ private def closeRootAndAllocator(): Unit = {
+ if (writer != null && !rootAndAllocatorClosed) {
+ rootAndAllocatorClosed = true
root.close()
allocator.close()
}
@@ -91,6 +107,11 @@ private[python] trait PythonArrowInput[IN] { self:
BasePythonRunner[IN, _] =>
inputIterator: Iterator[IN],
partitionIndex: Int,
context: TaskContext): Writer = {
+ // Task completion listeners run after the writer is done: it runs on the
task thread unless
+ // it is pipelined, and the pipelined writer is stopped and awaited before
them (see
+ // `BasePythonRunner.startPipelinedWriter`). The listener does not end
`writer`, since the
+ // stream that it writes to may be closed already, and nothing sends it to
the worker anymore.
+ context.addTaskCompletionListener[Unit](_ => closeRootAndAllocator())
new Writer(env, worker, inputIterator, partitionIndex, context) {
protected override def writeCommand(dataOut: DataOutputStream): Unit = {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
index 7f27c1d0e7aa..1932433d9234 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
@@ -17,13 +17,15 @@
package org.apache.spark.sql.execution.arrow
+import java.io.{DataOutputStream, IOException, OutputStream}
+
import scala.jdk.CollectionConverters._
import org.apache.arrow.vector.{TimeStampVector, VectorSchemaRoot,
ViewVarBinaryVector, ViewVarCharVector}
import org.apache.arrow.vector.complex.ListViewVector
import org.apache.arrow.vector.types.pojo.{ArrowType, FieldType}
-import org.apache.spark.{SparkArithmeticException, SparkFunSuite,
SparkUnsupportedOperationException}
+import org.apache.spark.{SparkArithmeticException, SparkFunSuite,
SparkUnsupportedOperationException, TaskContext}
import org.apache.spark.sql.Row
import org.apache.spark.sql.YearUDT
import org.apache.spark.sql.catalyst.InternalRow
@@ -1300,4 +1302,22 @@ class ArrowWriterSuite extends SparkFunSuite {
writer.root.close()
}
+
+ test("SPARK-60737: ArrowWriterWrapper closes its allocator on task
completion if starting the " +
+ "stream fails") {
+ val failingOut = new DataOutputStream(new OutputStream {
+ override def write(b: Int): Unit = throw new IOException("write failure")
+ })
+ val context = TaskContext.empty()
+ try {
+ intercept[IOException] {
+ ArrowWriterWrapper.createAndStartArrowWriter(new StructType().add("s",
StringType),
+ "UTC", "ArrowWriterSuite", largeVarTypes = false, failingOut,
context)
+ }
+ } finally {
+ context.markTaskCompleted(None)
+ }
+ assert(!ArrowUtils.rootAllocator.getChildAllocators.asScala
+ .exists(_.getName == "stdout writer for ArrowWriterSuite"))
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
index 9236ee9318d9..43dee0602039 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
@@ -18,12 +18,20 @@
package org.apache.spark.sql.execution.python
import scala.concurrent.duration._
+import scala.jdk.CollectionConverters._
-import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils, Row}
+import org.apache.arrow.memory.BufferAllocator
+
+import org.apache.spark.{SparkEnv, SparkException}
+import org.apache.spark.internal.config.Python.PYTHON_UDF_PIPELINED_EXECUTION
+import org.apache.spark.sql.{AnalysisException, DataFrame,
IntegratedUDFTestUtils, Row}
+import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.functions.{aggregate, array, avg, col, count, lit,
transform}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.LongType
+import org.apache.spark.sql.types.{LongType, StringType, StructType}
+import org.apache.spark.sql.util.ArrowUtils
+import org.apache.spark.unsafe.types.UTF8String
class PythonUDFSuite extends SharedSparkSession {
import testImplicits._
@@ -337,4 +345,44 @@ class PythonUDFSuite extends SharedSparkSession {
}
}
}
+
+ test("SPARK-60737: Close the Arrow input of a pandas UDF when the task ends
before the input") {
+ assume(shouldTestPandasUDFs)
+ registerTestUDF(TestScalarPandasUDF(name = "pandas_udf", returnType =
Some(StringType)), spark)
+ val payload = UTF8String.fromString("x" * 1000)
+ // More input than the socket to the Python worker can buffer, so that the
input is not
+ // exhausted when the task ends. Asking for the row at `failAt` throws.
+ def input(failAt: Int): DataFrame = {
+ val rdd = spark.sparkContext.parallelize(Seq(0), 1).mapPartitions { _ =>
+ Iterator.tabulate(100000) { i =>
+ if (i == failAt) throw new IllegalStateException("input failure")
+ InternalRow(payload)
+ }
+ }
+ spark.internalCreateDataFrame(rdd, new StructType().add("s", StringType))
+ }
+ // The open allocators of the Arrow input of Python workers.
+ def inputAllocators(): Set[BufferAllocator] =
ArrowUtils.rootAllocator.getChildAllocators
+ .asScala.filter(_.getName.startsWith("stdout writer for")).toSet
+
+ Seq(false, true).foreach { pipelined =>
+ SparkEnv.get.conf.set(PYTHON_UDF_PIPELINED_EXECUTION, pipelined)
+ try {
+ withSQLConf(SQLConf.ARROW_EXECUTION_MAX_RECORDS_PER_BATCH.key ->
"100") {
+ val allocatorsBefore = inputAllocators()
+ // The task fails.
+ val e = intercept[SparkException] {
+ input(failAt = 1000).selectExpr("pandas_udf(s)").collect()
+ }
+ assert(e.getMessage.contains("input failure"))
+ // The task stops reading the output early. The filter keeps the
limit above the UDF.
+ assert(input(failAt = -1).selectExpr("pandas_udf(s) AS u").filter("u
IS NOT NULL")
+ .limit(1).collect().length == 1)
+ assert((inputAllocators() -- allocatorsBefore).isEmpty)
+ }
+ } finally {
+ SparkEnv.get.conf.remove(PYTHON_UDF_PIPELINED_EXECUTION)
+ }
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/streaming/StatefulPythonRunnerDataSentSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/streaming/StatefulPythonRunnerDataSentSuite.scala
index 2f4499d19aa9..6ffb85cf7bc9 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/streaming/StatefulPythonRunnerDataSentSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/streaming/StatefulPythonRunnerDataSentSuite.scala
@@ -24,7 +24,7 @@ import scala.jdk.CollectionConverters._
import org.apache.arrow.vector.ipc.WriteChannel
import org.apache.arrow.vector.ipc.message.MessageSerializer
-import org.apache.spark.SparkEnv
+import org.apache.spark.{SparkEnv, TaskContext}
import org.apache.spark.api.python.{BasePythonRunner, ChainedPythonFunctions,
PythonBroadcast, PythonEvalType, SimplePythonFunction}
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.sql.{QueryTest, Row}
@@ -60,7 +60,7 @@ class StatefulPythonRunnerDataSentSuite extends QueryTest
with SharedSparkSessio
MessageSerializer.serialize(
new WriteChannel(Channels.newChannel(schemaMessage)), root.getSchema)
- val inputWriter = newWriter(SparkEnv.get, null, input, 0, null)
+ val inputWriter = newWriter(SparkEnv.get, null, input, 0,
TaskContext.empty())
val buffer = new ByteArrayOutputStream()
val sharedStream = new DataOutputStream(buffer)
var hasInput = true
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]