This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 2c4ebfeb1828 [SPARK-60776][SQL] Fix Arrow memory leak when compressing
a record batch fails
2c4ebfeb1828 is described below
commit 2c4ebfeb1828e8c2d2ceabb0db82c83c04176257
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 31 15:30:23 2026 -0700
[SPARK-60776][SQL] Fix Arrow memory leak when compressing a record batch
fails
### What changes were proposed in this pull request?
This PR fixes an Arrow memory leak when compressing a record batch fails,
as a workaround until Arrow Java 20.0.0.
It adds `ArrowCompressionUtils.createVectorUnloader` and uses it in all
nine places that create a `VectorUnloader` with a codec:
`ArrowConverters.ArrowBatchIterator`, `ArrowCachedBatchSerializer` (3),
`PythonArrowInput` (2), `ColumnarArrowPythonInput` (2) and
`CoGroupedArrowPythonRunner`.
With a compressing codec, it returns a `ReleasingVectorUnloader`, which
extends Arrow's `VectorUnloader` and wraps the codec in a
`TrackingCompressionCodec`:
- `TrackingCompressionCodec` keeps the buffer that the batch holds for each
`compress` call: the input until the call returns, and the output after that.
- When `getRecordBatch` throws, `ReleasingVectorUnloader` closes these
buffers with `AutoCloseables.close`, which adds a failure of `close()` to the
original exception as a suppressed exception. When it returns, the batch owns
the buffers, so the list is only cleared.
With `NoCompressionCodec`, which cannot fail, `createVectorUnloader`
returns Arrow's own `VectorUnloader`, so the default configuration is unchanged.
Arrow Java 20.0.0 fixes this in `VectorUnloader` and
`AbstractCompressionCodec` (apache/arrow-java#1290). `ReleasingVectorUnloader`
must be removed with that upgrade, because it would release the buffers twice.
The new test then fails with `RefCnt has gone negative`, which I checked by
running it with the `VectorUnloader` and `AbstractCompressionCodec` changes of
apache/arrow-java#1290 on the test classpath.
Alternatives that were not taken:
- Wrap only the codec in `createCompressionCodec`: a codec does not know
where a batch ends, so it cannot tell the buffers of the failed batch from
those of the batches already returned.
- Copy `VectorUnloader.getRecordBatch` and `appendNodes` into Spark: it
brings Arrow's traversal and buffer layout checks into Spark.
### Why are the changes needed?
Arrow Java up to 19.0.0 does not release buffers when
`CompressionCodec.compress` throws inside `VectorUnloader.getRecordBatch`
(apache/arrow-java#1234):
- `appendNodes` retains each buffer before passing it to `compress`, and
`AbstractCompressionCodec.compress` closes its input only on success. So the
buffer being compressed keeps the extra reference.
- The buffers already compressed for the batch are only in a local list of
`getRecordBatch`, which nothing releases.
Closing the vectors and the allocator cannot release these buffers, so they
stay allocated until the JVM exits.
The realistic trigger is direct memory running out, because compression
allocates an additional direct buffer per input buffer. Arrow's Netty allocator
turns direct memory exhaustion into
`org.apache.arrow.memory.OutOfMemoryException`, which is not fatal to the
executor. So the executor keeps running with less direct memory for the next
tasks.
The task still fails with the original exception, because
`TaskContext.runTaskWithListeners` only adds a failure of a completion listener
as a suppressed exception. Closing the allocator reports the leak, as in this
example with a capped allocator.
```
org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of
size 256 (rounded from 196) due to memory limit. Current allocation: 65536
at
org.apache.arrow.vector.compression.AbstractCompressionCodec.compress(AbstractCompressionCodec.java:41)
at
org.apache.arrow.vector.VectorUnloader.appendNodes(VectorUnloader.java:131)
...
Suppressed: org.apache.spark.util.TaskCompletionListenerException:
Memory was leaked by query. Memory leaked: (16384)
```
On the zero-copy paths
(`ArrowCachedBatchSerializer.convertArrowBatchZeroCopy` and
`ColumnarArrowPythonInput.writeArrowDirect`), the buffers belong to the
allocator of the upstream vectors, so the leak is reported by the owner of
those vectors rather than by the code that failed.
This only happens when `spark.sql.execution.arrow.compression.codec` is
`zstd` or `lz4`; the default is `none`. By code reading, the affected paths
have existed since Apache Spark 4.1.0 for `ArrowConverters`, `PythonArrowInput`
and `CoGroupedArrowPythonRunner` (SPARK-54134, SPARK-54226), since Apache Spark
4.2.0 for `ColumnarArrowPythonInput` (SPARK-56350), and since Apache Spark
4.3.0 for `ArrowCachedBatchSerializer` (SPARK-57268).
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs with the newly added test case in
`ArrowCompressionUtilsSuite`. Its codec fails while the second batch compresses
its second buffer. The test checks that no buffer is released twice, that the
first batch keeps its buffers, and that closing the allocator finds no leak.
I also ran the affected paths in tasks with a temporary suite, which is not
part of this PR. It caps `ArrowUtils.rootAllocator` when the input rows run
out, so that the first allocation inside `compress` fails, and then measures
the memory left behind.
| Path | Before | After |
|---|---|---|
| `ArrowBatchIterator` | 16384 bytes | 0 bytes |
| `InternalRowToArrowCachedBatchIterator` | 16384 bytes | 0 bytes |
| `ColumnarBatchToArrowCachedBatchIterator` zero-copy, in the upstream
allocator | 8192 bytes | 0 bytes |
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59991 from dongjoon-hyun/SPARK-60776.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit e438d0cc23345d84601edfc40b7196549634dab8)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../execution/arrow/ArrowCompressionUtils.scala | 63 ++++++++++++++++++++++
.../sql/execution/arrow/ArrowConverters.scala | 2 +-
.../columnar/ArrowCachedBatchSerializer.scala | 8 +--
.../python/CoGroupedArrowPythonRunner.scala | 2 +-
.../python/ColumnarArrowPythonInput.scala | 7 +--
.../sql/execution/python/PythonArrowInput.scala | 4 +-
.../arrow/ArrowCompressionUtilsSuite.scala | 41 ++++++++++++++
7 files changed, 116 insertions(+), 11 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
index e71680751ab8..c6a57430ae48 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
@@ -17,10 +17,16 @@
package org.apache.spark.sql.execution.arrow
+import scala.collection.mutable.ArrayBuffer
+import scala.jdk.CollectionConverters._
+
import com.github.luben.zstd.Zstd
import org.apache.arrow.compression.{Lz4CompressionCodec, ZstdCompressionCodec}
import org.apache.arrow.memory.{ArrowBuf, BufferAllocator}
+import org.apache.arrow.util.AutoCloseables
+import org.apache.arrow.vector.{VectorSchemaRoot, VectorUnloader}
import org.apache.arrow.vector.compression.{CompressionCodec, CompressionUtil,
NoCompressionCodec}
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch
import org.apache.spark.SparkException
@@ -52,6 +58,23 @@ private[sql] object ArrowCompressionUtils {
s"Unsupported Arrow compression codec: $other. Supported values:
none, zstd, lz4")
}
}
+
+ /**
+ * Creates the [[VectorUnloader]] that turns `root` into record batches
compressed with `codec`,
+ * with null counts and aligned buffers.
+ *
+ * With a compressing codec, the unloader releases what it holds for a batch
when compressing
+ * the batch fails, e.g. on a direct memory OOM. Arrow Java before 20.0.0
leaks the buffer being
+ * compressed and the buffers already compressed for the batch
(apache/arrow-java#1234), and
+ * they stay allocated until the JVM exits.
+ */
+ def createVectorUnloader(root: VectorSchemaRoot, codec: CompressionCodec):
VectorUnloader = {
+ if (codec.getCodecType == CompressionUtil.CodecType.NO_COMPRESSION) {
+ new VectorUnloader(root, true, codec, true)
+ } else {
+ new ReleasingVectorUnloader(root, new TrackingCompressionCodec(codec))
+ }
+ }
}
/**
@@ -83,3 +106,43 @@ private[sql] class
BoundedZstdCompressionCodec(compressionLevel: Int)
compressedBuffer
}
}
+
+/**
+ * A [[VectorUnloader]] that releases the buffers it holds for a batch it
fails to unload. Remove
+ * once on Arrow 20.0.0, which releases them itself, so they would be released
twice
+ * (apache/arrow-java#1290).
+ */
+private class ReleasingVectorUnloader(root: VectorSchemaRoot, codec:
TrackingCompressionCodec)
+ extends VectorUnloader(root, true, codec, true) {
+
+ override def getRecordBatch(): ArrowRecordBatch = {
+ try {
+ super.getRecordBatch()
+ } catch {
+ case t: Throwable =>
+ AutoCloseables.close(t, codec.heldBuffers.asJava)
+ throw t
+ } finally {
+ // A returned batch owns the buffers now.
+ codec.heldBuffers.clear()
+ }
+ }
+}
+
+/** Delegates to `codec` and keeps, for each call, the buffer that the batch
holds afterwards. */
+private class TrackingCompressionCodec(codec: CompressionCodec) extends
CompressionCodec {
+ val heldBuffers = new ArrayBuffer[ArrowBuf]
+
+ override def compress(allocator: BufferAllocator, input: ArrowBuf): ArrowBuf
= {
+ // The unloader retained `input` for `codec`, which releases it only when
it succeeds.
+ heldBuffers += input
+ val output = codec.compress(allocator, input)
+ heldBuffers(heldBuffers.length - 1) = output
+ output
+ }
+
+ override def decompress(allocator: BufferAllocator, input: ArrowBuf):
ArrowBuf =
+ codec.decompress(allocator, input)
+
+ override def getCodecType: CompressionUtil.CodecType = codec.getCodecType
+}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
index a593f4962c62..e24048b3612d 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
@@ -99,7 +99,7 @@ private[sql] object ArrowConverters extends Logging {
// Create compression codec based on config
private val codec = ArrowCompressionUtils.createCompressionCodec(
SQLConf.get.arrowCompressionCodec, SQLConf.get.arrowZstdCompressionLevel)
- protected val unloader = new VectorUnloader(root, true, codec, true)
+ protected val unloader = ArrowCompressionUtils.createVectorUnloader(root,
codec)
protected val arrowWriter = ArrowWriter.create(root)
protected def maxBytesPerBatch: Long = -1L
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
index db91f91e2480..3a1e4829b2be 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
@@ -24,7 +24,7 @@ import scala.jdk.CollectionConverters._
import org.apache.arrow.flatbuf.{RecordBatch => FlatBufRecordBatch}
import org.apache.arrow.memory.BufferAllocator
-import org.apache.arrow.vector.{TypeLayout, VectorLoader, VectorSchemaRoot,
VectorUnloader}
+import org.apache.arrow.vector.{TypeLayout, VectorLoader, VectorSchemaRoot}
import org.apache.arrow.vector.compression.NoCompressionCodec
import org.apache.arrow.vector.ipc.{ReadChannel, WriteChannel}
import org.apache.arrow.vector.ipc.message.{ArrowBodyCompression,
ArrowFieldNode}
@@ -998,7 +998,7 @@ private class InternalRowToArrowCachedBatchIterator(
sparkSchema, timeZoneId, false, false, losslessInternalTypes = true)
private val root = VectorSchemaRoot.create(arrowSchema, allocator)
private val arrowWriter = ArrowWriter.create(root)
- private val unloader = new VectorUnloader(root, true, compressionCodec, true)
+ private val unloader = ArrowCompressionUtils.createVectorUnloader(root,
compressionCodec)
// Create statistics collectors for each column
private val statsCollectors: Array[ColumnStats] = schema.map { attr =>
@@ -1177,7 +1177,7 @@ private class ColumnarBatchToArrowCachedBatchIterator(
Utils.tryWithSafeFinally {
// Use VectorUnloader to create compressed RecordBatch
- val unloader = new VectorUnloader(root, true, compressionCodec, true)
+ val unloader = ArrowCompressionUtils.createVectorUnloader(root,
compressionCodec)
val recordBatch = unloader.getRecordBatch()
Utils.tryWithSafeFinally {
@@ -1204,7 +1204,7 @@ private class ColumnarBatchToArrowCachedBatchIterator(
// Convert columnar batch to rows, then to Arrow
val root = VectorSchemaRoot.create(arrowSchema, allocator)
val arrowWriter = ArrowWriter.create(root)
- val unloader = new VectorUnloader(root, true, compressionCodec, true)
+ val unloader = ArrowCompressionUtils.createVectorUnloader(root,
compressionCodec)
Utils.tryWithSafeFinally {
val rowIterator = batch.rowIterator().asScala
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala
index 0a917677a52e..1b3f0260de4e 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala
@@ -94,7 +94,7 @@ class CoGroupedArrowPythonRunner(
private def createUnloader(root: VectorSchemaRoot): VectorUnloader = {
val codec = ArrowCompressionUtils.createCompressionCodec(
compressionCodecName, SQLConf.get.arrowZstdCompressionLevel)
- new VectorUnloader(root, true, codec, true)
+ ArrowCompressionUtils.createVectorUnloader(root, codec)
}
protected def newWriter(
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowPythonInput.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowPythonInput.scala
index fbc21c0e2bf6..bee978f07fcf 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowPythonInput.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowPythonInput.scala
@@ -27,7 +27,7 @@ import org.apache.arrow.vector.ipc.WriteChannel
import org.apache.arrow.vector.ipc.message.MessageSerializer
import org.apache.spark.api.python.BasePythonRunner
-import org.apache.spark.sql.execution.arrow.ArrowWriter
+import org.apache.spark.sql.execution.arrow.{ArrowCompressionUtils,
ArrowWriter}
import org.apache.spark.sql.util.ArrowUtils
import org.apache.spark.sql.vectorized.{ArrowColumnVector, ColumnarBatch}
@@ -55,7 +55,8 @@ private[python] trait ColumnarArrowPythonInput
// Lazy-initialized ArrowWriter for the fallback (non-Arrow) path.
// Uses the pre-allocated `root` from PythonArrowInput.
private lazy val arrowWriter: ArrowWriter = ArrowWriter.create(root)
- private lazy val unloader: VectorUnloader = new VectorUnloader(root, true,
codec, true)
+ private lazy val unloader: VectorUnloader =
+ ArrowCompressionUtils.createVectorUnloader(root, codec)
protected def writeNextBatchToArrowStream(
root: VectorSchemaRoot,
@@ -119,7 +120,7 @@ private[python] trait ColumnarArrowPythonInput
val batchRoot = VectorSchemaRoot.of(selectedVectors: _*)
batchRoot.setRowCount(batch.numRows())
- val batchUnloader = new VectorUnloader(batchRoot, true, codec, true)
+ val batchUnloader = ArrowCompressionUtils.createVectorUnloader(batchRoot,
codec)
val recordBatch = batchUnloader.getRecordBatch()
try {
val writeChannel = new WriteChannel(Channels.newChannel(dataOut))
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
index 596a330e2026..6bc65a886abe 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
@@ -135,7 +135,7 @@ private[python] trait PythonArrowInput[IN] { self:
BasePythonRunner[IN, _] =>
private[python] trait BasicPythonArrowInput extends
PythonArrowInput[Iterator[InternalRow]] {
self: BasePythonRunner[Iterator[InternalRow], _] =>
protected lazy val arrowWriter: arrow.ArrowWriter = ArrowWriter.create(root)
- protected lazy val unloader = new VectorUnloader(root, true, codec, true)
+ protected lazy val unloader =
ArrowCompressionUtils.createVectorUnloader(root, codec)
protected val maxRecordsPerBatch: Int = {
val v = SQLConf.get.arrowMaxRecordsPerBatch
@@ -295,7 +295,7 @@ private[python] trait GroupedPythonArrowInput { self:
RowInputArrowPythonRunner
schema, timeZoneId, pythonExec,
largeVarTypes, dataOut, context)
// Set the unloader with compression after creating the writer
- writer.unloader = new VectorUnloader(writer.root, true,
self.codec, true)
+ writer.unloader =
ArrowCompressionUtils.createVectorUnloader(writer.root, self.codec)
nextBatchStart = inputIterator.next()
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
index cde828e1c743..f424aff29720 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
@@ -21,6 +21,7 @@ import java.io.ByteArrayOutputStream
import java.nio.{ByteBuffer, ByteOrder}
import java.nio.channels.Channels
+import scala.jdk.CollectionConverters._
import scala.util.Random
import org.apache.arrow.compression.ZstdCompressionCodec
@@ -260,4 +261,44 @@ class ArrowCompressionUtilsSuite extends SparkFunSuite {
allocator.close()
}
}
+
+ test("SPARK-60776: unloader releases the buffers of a batch it fails to
unload") {
+ var calls = 0
+ val codec = new BoundedZstdCompressionCodec(3) {
+ override protected def doCompress(allocator: BufferAllocator, input:
ArrowBuf): ArrowBuf = {
+ calls += 1
+ if (calls == 4) throw new IllegalStateException("compression failed")
+ super.doCompress(allocator, input)
+ }
+ }
+ val arrowSchema = ArrowUtils.toArrowSchema(
+ StructType(Seq(StructField("c", LongType))), "UTC", true, false)
+ val allocator =
+ ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName,
0, Long.MaxValue)
+ val root = VectorSchemaRoot.create(arrowSchema, allocator)
+ val writer = ArrowWriter.create(root)
+ val unloader = ArrowCompressionUtils.createVectorUnloader(root, codec)
+ Utils.tryWithSafeFinally {
+ writer.write(InternalRow(1L))
+ writer.finish()
+ // A batch compresses a validity buffer and a data buffer, so the first
batch succeeds and the
+ // second fails holding its compressed validity buffer and its retained
data buffer.
+ val batch = unloader.getRecordBatch()
+ Utils.tryWithSafeFinally {
+ val e = intercept[IllegalStateException](unloader.getRecordBatch())
+ assert(e.getMessage === "compression failed")
+ // A suppressed exception means that a buffer was released twice. If
this fails after an
+ // Arrow upgrade, Arrow releases these buffers itself and
ReleasingVectorUnloader must be
+ // removed.
+ assert(e.getSuppressed.isEmpty)
+ assert(batch.getBuffers.asScala.forall(_.refCnt() == 1))
+ } {
+ batch.close()
+ }
+ } {
+ // Closing the allocator fails if a buffer is left behind.
+ root.close()
+ allocator.close()
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]