This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 2c4ebfeb1828 [SPARK-60776][SQL] Fix Arrow memory leak when compressing 
a record batch fails
2c4ebfeb1828 is described below

commit 2c4ebfeb1828e8c2d2ceabb0db82c83c04176257
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 31 15:30:23 2026 -0700

    [SPARK-60776][SQL] Fix Arrow memory leak when compressing a record batch 
fails
    
    ### What changes were proposed in this pull request?
    
    This PR fixes an Arrow memory leak when compressing a record batch fails, 
as a workaround until Arrow Java 20.0.0.
    
    It adds `ArrowCompressionUtils.createVectorUnloader` and uses it in all 
nine places that create a `VectorUnloader` with a codec: 
`ArrowConverters.ArrowBatchIterator`, `ArrowCachedBatchSerializer` (3), 
`PythonArrowInput` (2), `ColumnarArrowPythonInput` (2) and 
`CoGroupedArrowPythonRunner`.
    
    With a compressing codec, it returns a `ReleasingVectorUnloader`, which 
extends Arrow's `VectorUnloader` and wraps the codec in a 
`TrackingCompressionCodec`:
    - `TrackingCompressionCodec` keeps the buffer that the batch holds for each 
`compress` call: the input until the call returns, and the output after that.
    - When `getRecordBatch` throws, `ReleasingVectorUnloader` closes these 
buffers with `AutoCloseables.close`, which adds a failure of `close()` to the 
original exception as a suppressed exception. When it returns, the batch owns 
the buffers, so the list is only cleared.
    
    With `NoCompressionCodec`, which cannot fail, `createVectorUnloader` 
returns Arrow's own `VectorUnloader`, so the default configuration is unchanged.
    
    Arrow Java 20.0.0 fixes this in `VectorUnloader` and 
`AbstractCompressionCodec` (apache/arrow-java#1290). `ReleasingVectorUnloader` 
must be removed with that upgrade, because it would release the buffers twice. 
The new test then fails with `RefCnt has gone negative`, which I checked by 
running it with the `VectorUnloader` and `AbstractCompressionCodec` changes of 
apache/arrow-java#1290 on the test classpath.
    
    Alternatives that were not taken:
    - Wrap only the codec in `createCompressionCodec`: a codec does not know 
where a batch ends, so it cannot tell the buffers of the failed batch from 
those of the batches already returned.
    - Copy `VectorUnloader.getRecordBatch` and `appendNodes` into Spark: it 
brings Arrow's traversal and buffer layout checks into Spark.
    
    ### Why are the changes needed?
    
    Arrow Java up to 19.0.0 does not release buffers when 
`CompressionCodec.compress` throws inside `VectorUnloader.getRecordBatch` 
(apache/arrow-java#1234):
    - `appendNodes` retains each buffer before passing it to `compress`, and 
`AbstractCompressionCodec.compress` closes its input only on success. So the 
buffer being compressed keeps the extra reference.
    - The buffers already compressed for the batch are only in a local list of 
`getRecordBatch`, which nothing releases.
    
    Closing the vectors and the allocator cannot release these buffers, so they 
stay allocated until the JVM exits.
    
    The realistic trigger is direct memory running out, because compression 
allocates an additional direct buffer per input buffer. Arrow's Netty allocator 
turns direct memory exhaustion into 
`org.apache.arrow.memory.OutOfMemoryException`, which is not fatal to the 
executor. So the executor keeps running with less direct memory for the next 
tasks.
    
    The task still fails with the original exception, because 
`TaskContext.runTaskWithListeners` only adds a failure of a completion listener 
as a suppressed exception. Closing the allocator reports the leak, as in this 
example with a capped allocator.
    
    ```
    org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of 
size 256 (rounded from 196) due to memory limit. Current allocation: 65536
            at 
org.apache.arrow.vector.compression.AbstractCompressionCodec.compress(AbstractCompressionCodec.java:41)
            at 
org.apache.arrow.vector.VectorUnloader.appendNodes(VectorUnloader.java:131)
            ...
            Suppressed: org.apache.spark.util.TaskCompletionListenerException: 
Memory was leaked by query. Memory leaked: (16384)
    ```
    
    On the zero-copy paths 
(`ArrowCachedBatchSerializer.convertArrowBatchZeroCopy` and 
`ColumnarArrowPythonInput.writeArrowDirect`), the buffers belong to the 
allocator of the upstream vectors, so the leak is reported by the owner of 
those vectors rather than by the code that failed.
    
    This only happens when `spark.sql.execution.arrow.compression.codec` is 
`zstd` or `lz4`; the default is `none`. By code reading, the affected paths 
have existed since Apache Spark 4.1.0 for `ArrowConverters`, `PythonArrowInput` 
and `CoGroupedArrowPythonRunner` (SPARK-54134, SPARK-54226), since Apache Spark 
4.2.0 for `ColumnarArrowPythonInput` (SPARK-56350), and since Apache Spark 
4.3.0 for `ArrowCachedBatchSerializer` (SPARK-57268).
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case in 
`ArrowCompressionUtilsSuite`. Its codec fails while the second batch compresses 
its second buffer. The test checks that no buffer is released twice, that the 
first batch keeps its buffers, and that closing the allocator finds no leak.
    
    I also ran the affected paths in tasks with a temporary suite, which is not 
part of this PR. It caps `ArrowUtils.rootAllocator` when the input rows run 
out, so that the first allocation inside `compress` fails, and then measures 
the memory left behind.
    
    | Path | Before | After |
    |---|---|---|
    | `ArrowBatchIterator` | 16384 bytes | 0 bytes |
    | `InternalRowToArrowCachedBatchIterator` | 16384 bytes | 0 bytes |
    | `ColumnarBatchToArrowCachedBatchIterator` zero-copy, in the upstream 
allocator | 8192 bytes | 0 bytes |
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59991 from dongjoon-hyun/SPARK-60776.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit e438d0cc23345d84601edfc40b7196549634dab8)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../execution/arrow/ArrowCompressionUtils.scala    | 63 ++++++++++++++++++++++
 .../sql/execution/arrow/ArrowConverters.scala      |  2 +-
 .../columnar/ArrowCachedBatchSerializer.scala      |  8 +--
 .../python/CoGroupedArrowPythonRunner.scala        |  2 +-
 .../python/ColumnarArrowPythonInput.scala          |  7 +--
 .../sql/execution/python/PythonArrowInput.scala    |  4 +-
 .../arrow/ArrowCompressionUtilsSuite.scala         | 41 ++++++++++++++
 7 files changed, 116 insertions(+), 11 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
index e71680751ab8..c6a57430ae48 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
@@ -17,10 +17,16 @@
 
 package org.apache.spark.sql.execution.arrow
 
+import scala.collection.mutable.ArrayBuffer
+import scala.jdk.CollectionConverters._
+
 import com.github.luben.zstd.Zstd
 import org.apache.arrow.compression.{Lz4CompressionCodec, ZstdCompressionCodec}
 import org.apache.arrow.memory.{ArrowBuf, BufferAllocator}
+import org.apache.arrow.util.AutoCloseables
+import org.apache.arrow.vector.{VectorSchemaRoot, VectorUnloader}
 import org.apache.arrow.vector.compression.{CompressionCodec, CompressionUtil, 
NoCompressionCodec}
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch
 
 import org.apache.spark.SparkException
 
@@ -52,6 +58,23 @@ private[sql] object ArrowCompressionUtils {
           s"Unsupported Arrow compression codec: $other. Supported values: 
none, zstd, lz4")
     }
   }
+
+  /**
+   * Creates the [[VectorUnloader]] that turns `root` into record batches 
compressed with `codec`,
+   * with null counts and aligned buffers.
+   *
+   * With a compressing codec, the unloader releases what it holds for a batch 
when compressing
+   * the batch fails, e.g. on a direct memory OOM. Arrow Java before 20.0.0 
leaks the buffer being
+   * compressed and the buffers already compressed for the batch 
(apache/arrow-java#1234), and
+   * they stay allocated until the JVM exits.
+   */
+  def createVectorUnloader(root: VectorSchemaRoot, codec: CompressionCodec): 
VectorUnloader = {
+    if (codec.getCodecType == CompressionUtil.CodecType.NO_COMPRESSION) {
+      new VectorUnloader(root, true, codec, true)
+    } else {
+      new ReleasingVectorUnloader(root, new TrackingCompressionCodec(codec))
+    }
+  }
 }
 
 /**
@@ -83,3 +106,43 @@ private[sql] class 
BoundedZstdCompressionCodec(compressionLevel: Int)
     compressedBuffer
   }
 }
+
+/**
+ * A [[VectorUnloader]] that releases the buffers it holds for a batch it 
fails to unload. Remove
+ * once on Arrow 20.0.0, which releases them itself, so they would be released 
twice
+ * (apache/arrow-java#1290).
+ */
+private class ReleasingVectorUnloader(root: VectorSchemaRoot, codec: 
TrackingCompressionCodec)
+  extends VectorUnloader(root, true, codec, true) {
+
+  override def getRecordBatch(): ArrowRecordBatch = {
+    try {
+      super.getRecordBatch()
+    } catch {
+      case t: Throwable =>
+        AutoCloseables.close(t, codec.heldBuffers.asJava)
+        throw t
+    } finally {
+      // A returned batch owns the buffers now.
+      codec.heldBuffers.clear()
+    }
+  }
+}
+
+/** Delegates to `codec` and keeps, for each call, the buffer that the batch 
holds afterwards. */
+private class TrackingCompressionCodec(codec: CompressionCodec) extends 
CompressionCodec {
+  val heldBuffers = new ArrayBuffer[ArrowBuf]
+
+  override def compress(allocator: BufferAllocator, input: ArrowBuf): ArrowBuf 
= {
+    // The unloader retained `input` for `codec`, which releases it only when 
it succeeds.
+    heldBuffers += input
+    val output = codec.compress(allocator, input)
+    heldBuffers(heldBuffers.length - 1) = output
+    output
+  }
+
+  override def decompress(allocator: BufferAllocator, input: ArrowBuf): 
ArrowBuf =
+    codec.decompress(allocator, input)
+
+  override def getCodecType: CompressionUtil.CodecType = codec.getCodecType
+}
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
index a593f4962c62..e24048b3612d 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
@@ -99,7 +99,7 @@ private[sql] object ArrowConverters extends Logging {
     // Create compression codec based on config
     private val codec = ArrowCompressionUtils.createCompressionCodec(
       SQLConf.get.arrowCompressionCodec, SQLConf.get.arrowZstdCompressionLevel)
-    protected val unloader = new VectorUnloader(root, true, codec, true)
+    protected val unloader = ArrowCompressionUtils.createVectorUnloader(root, 
codec)
     protected val arrowWriter = ArrowWriter.create(root)
     protected def maxBytesPerBatch: Long = -1L
 
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
index db91f91e2480..3a1e4829b2be 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
@@ -24,7 +24,7 @@ import scala.jdk.CollectionConverters._
 
 import org.apache.arrow.flatbuf.{RecordBatch => FlatBufRecordBatch}
 import org.apache.arrow.memory.BufferAllocator
-import org.apache.arrow.vector.{TypeLayout, VectorLoader, VectorSchemaRoot, 
VectorUnloader}
+import org.apache.arrow.vector.{TypeLayout, VectorLoader, VectorSchemaRoot}
 import org.apache.arrow.vector.compression.NoCompressionCodec
 import org.apache.arrow.vector.ipc.{ReadChannel, WriteChannel}
 import org.apache.arrow.vector.ipc.message.{ArrowBodyCompression, 
ArrowFieldNode}
@@ -998,7 +998,7 @@ private class InternalRowToArrowCachedBatchIterator(
     sparkSchema, timeZoneId, false, false, losslessInternalTypes = true)
   private val root = VectorSchemaRoot.create(arrowSchema, allocator)
   private val arrowWriter = ArrowWriter.create(root)
-  private val unloader = new VectorUnloader(root, true, compressionCodec, true)
+  private val unloader = ArrowCompressionUtils.createVectorUnloader(root, 
compressionCodec)
 
   // Create statistics collectors for each column
   private val statsCollectors: Array[ColumnStats] = schema.map { attr =>
@@ -1177,7 +1177,7 @@ private class ColumnarBatchToArrowCachedBatchIterator(
 
     Utils.tryWithSafeFinally {
       // Use VectorUnloader to create compressed RecordBatch
-      val unloader = new VectorUnloader(root, true, compressionCodec, true)
+      val unloader = ArrowCompressionUtils.createVectorUnloader(root, 
compressionCodec)
       val recordBatch = unloader.getRecordBatch()
 
       Utils.tryWithSafeFinally {
@@ -1204,7 +1204,7 @@ private class ColumnarBatchToArrowCachedBatchIterator(
     // Convert columnar batch to rows, then to Arrow
     val root = VectorSchemaRoot.create(arrowSchema, allocator)
     val arrowWriter = ArrowWriter.create(root)
-    val unloader = new VectorUnloader(root, true, compressionCodec, true)
+    val unloader = ArrowCompressionUtils.createVectorUnloader(root, 
compressionCodec)
 
     Utils.tryWithSafeFinally {
       val rowIterator = batch.rowIterator().asScala
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala
index 0a917677a52e..1b3f0260de4e 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala
@@ -94,7 +94,7 @@ class CoGroupedArrowPythonRunner(
   private def createUnloader(root: VectorSchemaRoot): VectorUnloader = {
     val codec = ArrowCompressionUtils.createCompressionCodec(
       compressionCodecName, SQLConf.get.arrowZstdCompressionLevel)
-    new VectorUnloader(root, true, codec, true)
+    ArrowCompressionUtils.createVectorUnloader(root, codec)
   }
 
   protected def newWriter(
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowPythonInput.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowPythonInput.scala
index fbc21c0e2bf6..bee978f07fcf 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowPythonInput.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowPythonInput.scala
@@ -27,7 +27,7 @@ import org.apache.arrow.vector.ipc.WriteChannel
 import org.apache.arrow.vector.ipc.message.MessageSerializer
 
 import org.apache.spark.api.python.BasePythonRunner
-import org.apache.spark.sql.execution.arrow.ArrowWriter
+import org.apache.spark.sql.execution.arrow.{ArrowCompressionUtils, 
ArrowWriter}
 import org.apache.spark.sql.util.ArrowUtils
 import org.apache.spark.sql.vectorized.{ArrowColumnVector, ColumnarBatch}
 
@@ -55,7 +55,8 @@ private[python] trait ColumnarArrowPythonInput
   // Lazy-initialized ArrowWriter for the fallback (non-Arrow) path.
   // Uses the pre-allocated `root` from PythonArrowInput.
   private lazy val arrowWriter: ArrowWriter = ArrowWriter.create(root)
-  private lazy val unloader: VectorUnloader = new VectorUnloader(root, true, 
codec, true)
+  private lazy val unloader: VectorUnloader =
+    ArrowCompressionUtils.createVectorUnloader(root, codec)
 
   protected def writeNextBatchToArrowStream(
       root: VectorSchemaRoot,
@@ -119,7 +120,7 @@ private[python] trait ColumnarArrowPythonInput
     val batchRoot = VectorSchemaRoot.of(selectedVectors: _*)
     batchRoot.setRowCount(batch.numRows())
 
-    val batchUnloader = new VectorUnloader(batchRoot, true, codec, true)
+    val batchUnloader = ArrowCompressionUtils.createVectorUnloader(batchRoot, 
codec)
     val recordBatch = batchUnloader.getRecordBatch()
     try {
       val writeChannel = new WriteChannel(Channels.newChannel(dataOut))
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
index 596a330e2026..6bc65a886abe 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala
@@ -135,7 +135,7 @@ private[python] trait PythonArrowInput[IN] { self: 
BasePythonRunner[IN, _] =>
 private[python] trait BasicPythonArrowInput extends 
PythonArrowInput[Iterator[InternalRow]] {
   self: BasePythonRunner[Iterator[InternalRow], _] =>
   protected lazy val arrowWriter: arrow.ArrowWriter = ArrowWriter.create(root)
-  protected lazy val unloader = new VectorUnloader(root, true, codec, true)
+  protected lazy val unloader = 
ArrowCompressionUtils.createVectorUnloader(root, codec)
 
   protected val maxRecordsPerBatch: Int = {
     val v = SQLConf.get.arrowMaxRecordsPerBatch
@@ -295,7 +295,7 @@ private[python] trait GroupedPythonArrowInput { self: 
RowInputArrowPythonRunner
               schema, timeZoneId, pythonExec,
               largeVarTypes, dataOut, context)
             // Set the unloader with compression after creating the writer
-            writer.unloader = new VectorUnloader(writer.root, true, 
self.codec, true)
+            writer.unloader = 
ArrowCompressionUtils.createVectorUnloader(writer.root, self.codec)
             nextBatchStart = inputIterator.next()
           }
         }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
index cde828e1c743..f424aff29720 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
@@ -21,6 +21,7 @@ import java.io.ByteArrayOutputStream
 import java.nio.{ByteBuffer, ByteOrder}
 import java.nio.channels.Channels
 
+import scala.jdk.CollectionConverters._
 import scala.util.Random
 
 import org.apache.arrow.compression.ZstdCompressionCodec
@@ -260,4 +261,44 @@ class ArrowCompressionUtilsSuite extends SparkFunSuite {
       allocator.close()
     }
   }
+
+  test("SPARK-60776: unloader releases the buffers of a batch it fails to 
unload") {
+    var calls = 0
+    val codec = new BoundedZstdCompressionCodec(3) {
+      override protected def doCompress(allocator: BufferAllocator, input: 
ArrowBuf): ArrowBuf = {
+        calls += 1
+        if (calls == 4) throw new IllegalStateException("compression failed")
+        super.doCompress(allocator, input)
+      }
+    }
+    val arrowSchema = ArrowUtils.toArrowSchema(
+      StructType(Seq(StructField("c", LongType))), "UTC", true, false)
+    val allocator =
+      ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName, 
0, Long.MaxValue)
+    val root = VectorSchemaRoot.create(arrowSchema, allocator)
+    val writer = ArrowWriter.create(root)
+    val unloader = ArrowCompressionUtils.createVectorUnloader(root, codec)
+    Utils.tryWithSafeFinally {
+      writer.write(InternalRow(1L))
+      writer.finish()
+      // A batch compresses a validity buffer and a data buffer, so the first 
batch succeeds and the
+      // second fails holding its compressed validity buffer and its retained 
data buffer.
+      val batch = unloader.getRecordBatch()
+      Utils.tryWithSafeFinally {
+        val e = intercept[IllegalStateException](unloader.getRecordBatch())
+        assert(e.getMessage === "compression failed")
+        // A suppressed exception means that a buffer was released twice. If 
this fails after an
+        // Arrow upgrade, Arrow releases these buffers itself and 
ReleasingVectorUnloader must be
+        // removed.
+        assert(e.getSuppressed.isEmpty)
+        assert(batch.getBuffers.asScala.forall(_.refCnt() == 1))
+      } {
+        batch.close()
+      }
+    } {
+      // Closing the allocator fails if a buffer is left behind.
+      root.close()
+      allocator.close()
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to