This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 26b58aa9c17a [SPARK-60779][SQL] Shrink zstd output buffers to reduce 
direct memory held by Arrow batches
26b58aa9c17a is described below

commit 26b58aa9c17acac056f1c3eab01f6396b14aea03
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Nov 1 11:22:48 2026 -0800

    [SPARK-60779][SQL] Shrink zstd output buffers to reduce direct memory held 
by Arrow batches
    
    ### What changes were proposed in this pull request?
    
    This PR makes `BoundedZstdCompressionCodec` copy a compressed output to a 
smaller buffer and release the buffer that zstd wrote into, when that at least 
halves the memory that the output takes and saves at least 4 KiB.
    
    The codec now overrides `compress`. After `super.compress` returns, it 
computes `roundedLength`, the size that the allocator would give the output. If 
`roundedLength <= capacity / 2` and `capacity - roundedLength >= 4 KiB`, it 
copies the output with its length prefix to a new buffer and closes the big one.
    
    - Empty and raw (incompressible) outputs are already in a buffer of their 
size, so they are not copied.
    - The 4 KiB floor keeps small buffers out of the copy. They would give back 
little memory but still pay for an allocation, a copy, and a release, and small 
batches are common with grouped and cogrouped Python UDFs, which write each 
group as its own batch.
    - The halving condition limits the extra memory while both buffers are 
alive. It matters from 16 MiB, where the allocator stops rounding up. Below 
that, an output that does not halve its buffer saves nothing.
    - If the new buffer cannot be allocated (`OutOfMemoryException`), the codec 
keeps the big buffer. The output is complete, so this only gives up the saving, 
and a batch that succeeded before does not fail now.
    - `createVectorUnloader` (SPARK-60776) wraps this codec, so 
`TrackingCompressionCodec` holds the new buffer. It assumes that a failed 
`compress` still holds its input, but the shrink runs after `super.compress` 
released the input. So the shrink must not fail with `OutOfMemoryException`, 
and a code comment explains this.
    
    All zstd write paths get this codec from 
`ArrowCompressionUtils.createCompressionCodec`: Python UDF input, `toPandas`, 
Spark Connect results, External UDF, and the Arrow cache. lz4 is not changed, 
because `Lz4CompressionCodec` already requests a buffer of the output size.
    
    The class documentation now limits its removal note to `doCompress`. Arrow 
20.0.0 will fix the overrun (apache/arrow-java#1119), but the main branch of 
Arrow Java does not shrink the output, so `compress` has to stay until it does.
    
    Alternatives that were not taken:
    
    - Shrink in `doCompress`: the SPARK-60707 test calls `doCompress` directly 
and checks for a write past the buffer that zstd wrote into, so it would check 
the copy instead. It would also need its own check for the raw fallback.
    - Copy at a lower threshold: with 25% saved as the threshold, the 
1,000,000-row UUID batch below would hold 22.20 MiB instead of 38.47 MiB, but 
its peak would be 56.67 MiB instead of 38.47 MiB.
    - No floor, or a higher one: without a floor, `getRecordBatch` and `close` 
of a 10-row BIGINT batch cost up to 31.8% more with 8 threads (about 1 us per 
batch). A 64 KiB floor would also skip 1,000-row batches, but the 10,000-row 
`lineitem` batch below would hold 0.93 MiB instead of 0.50 MiB.
    - A scratch buffer per batch or per task: it saved no CPU in the 
measurements and holds the same memory or more. `CompressionCodec` has no 
`close()`, and some paths create a codec per group or batch, so each path would 
need new code to release it.
    - A different `RoundingPolicy`: it applies to all Arrow allocations under 
the shared `ArrowUtils.rootAllocator`, and the buffer would still be 
`compressBound`-sized.
    
    ### Why are the changes needed?
    
    Arrow Java 19.0.0 leaves the zstd output in the buffer that it allocated 
before compressing, and [its main 
branch](https://github.com/apache/arrow-java/blob/67ef5abf308a7a158c4fa8eeebb72b87ca5a6f97/compression/src/main/java/org/apache/arrow/compression/ZstdCompressionCodec.java#L41-L55)
 still does:
    
    - `ZstdCompressionCodec.doCompress` requests `8 + compressBound(len)` 
bytes, and the allocator rounds that up to a power of two below 16 MiB.
    - `VectorUnloader` keeps the output of every vector buffer until the record 
batch is closed, that is, until it is serialized.
    
    So a batch holds about as much extra direct memory as its uncompressed 
vectors, whatever the compression ratio. For comparison, [Arrow 
C++](https://github.com/apache/arrow/blob/926885f1b1ede73badd24c2e6f63c9429441e99a/cpp/src/arrow/ipc/writer.cc#L210-L250)
 shrinks its output right after compressing.
    
    This is not a correctness issue, and the cost depends on the batch size:
    
    - A 10,000-row batch, the default 
`spark.sql.execution.arrow.maxRecordsPerBatch`, costs 0.1 to 2.6 MiB per task 
with narrow rows.
    - A 64 MB batch costs 64 to 88 MiB per task, or about 1.0 to 1.4 GiB with 
16 concurrent tasks. Batches reach that size with wide rows (the default 
`spark.sql.execution.arrow.maxBytesPerBatch` is 64MB) or with a larger 
`maxRecordsPerBatch`.
    
    Spark's `MemoryManager` does not track this memory, and when direct memory 
runs out, the task fails with `OutOfMemoryException`.
    
    This only happens when `spark.sql.execution.arrow.compression.codec` is 
`zstd`; the default is `none`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. The serialized bytes are the same, and only the direct memory in use 
changes.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases in 
`ArrowCompressionUtilsSuite`:
    
    - A much smaller output moves to a buffer of its size and the big buffer is 
released. This test fails without this change.
    - An output that would not at least halve its buffer is not copied, so the 
peak allocation does not grow. Its input is 20 MiB, where only the halving 
condition keeps the buffer.
    - An output that would save less than 4 KiB is not copied.
    - When there is no memory for the new buffer, the output stays in its 
buffer.
    
    I checked these tests by changing the code: each test fails when its branch 
is removed, and bringing back the SPARK-60707 overrun still fails the 
SPARK-60707 tests. `ArrowCompressionUtilsSuite`, 
`ArrowCachedBatchSerializerSuite`, `ArrowConvertersSuite`, and 
`ExecuteExternalUDFExecSuite` pass locally (152 tests).
    
    I also ran `createVectorUnloader` with this codec in a temporary script, 
which is not part of this PR. A batch that fails after an output was shrunk, 
and a batch that runs out of memory after a shrink gave up, both release each 
buffer once: there is no suppressed exception, the allocator goes back to the 
size of the vectors, and closing it succeeds.
    
    I measured the memory with a temporary benchmark, which is not part of this 
PR. It fills a root with `ArrowWriter` and calls 
`VectorUnloader.getRecordBatch()`, with an `AllocationListener` on the child 
allocator of the root to track allocations and releases. The environment is an 
Apple M3 Max, JDK 25.0.5, Arrow 19.0.0, and zstd level 3. For all measured 
batches, the serialized IPC bytes are the same as before.
    
    | Batch (MiB) | Vectors | Compressed | Held before | Held after | Peak 
before | Peak after |
    |---|---:|---:|---:|---:|---:|---:|
    | Sequential BIGINT, 10,000 rows | 0.13 | 0.02 | 0.13 | 0.03 | 0.13 | 0.16 |
    | `lineitem`, 10,000 rows | 2.63 | 0.37 | 2.63 | 0.50 | 2.63 | 1.00 |
    | `lineitem`, 1,000,000 rows | 178.00 | 37.95 | 178.70 | 52.75 | 178.70 | 
83.45 |
    | `lineitem`, 64 MB limit (389,390 rows) | 87.00 | 14.75 | 88.00 | 22.88 | 
88.00 | 38.88 |
    | Text, 64 MB limit (8,348 rows) | 64.06 | 13.97 | 64.28 | 16.03 | 64.28 | 
80.25 |
    | UUID strings, 1,000,000 rows | 68.00 | 20.98 | 38.59 | 38.47 | 38.59 | 
38.47 |
    | Random BIGINT, 1,000,000 rows (raw output) | 8.00 | 7.63 | 8.13 | 8.00 | 
8.13 | 8.00 |
    
    - `lineitem` is a generated TPC-H-like batch with 16 columns and 37 Arrow 
buffers. The text batch has about 8 KB of JSON-like text per row.
    - Held and peak are on top of the vectors. Held is what a batch holds until 
it is serialized, and peak is the highest point during `getRecordBatch`.
    - The peak goes up when one big buffer dominates a batch, as in the text 
batch, because both buffers are alive during the copy. The halving condition 
limits this to half of that buffer, and it lasts only for the copy.
    - The floor gives up less than 4 KiB per Arrow buffer, about 32 KiB for the 
10,000-row `lineitem` batch.
    - These numbers are Arrow allocator accounting, not RSS. A new buffer above 
4 MiB commits only the pages that zstd writes (checked on macOS), so the effect 
on RSS is smaller. I did not measure RSS or OOMs in a real executor, or 
anything on Linux.
    
    For the CPU cost, I compared the thread CPU time of `getRecordBatch` and 
`close` with and without the shrink. Each value is the median after/before 
ratio across rounds:
    
    | Batch | 1 thread | 8 threads | 8 threads with jemalloc |
    |---|---:|---:|---:|
    | Sequential BIGINT, 10 rows | 1.006 | 1.006 | 1.002 |
    | Sequential BIGINT, 100 rows | 1.004 | 1.002 | 0.997 |
    | `lineitem`, 10 rows | 1.001 | 1.008 | 0.999 |
    | `lineitem`, 100 rows | 1.000 | 1.003 | 1.002 |
    | `lineitem`, 1,000 rows | 1.003 | 1.008 | 1.006 |
    | `lineitem`, 10,000 rows | 1.003 | 1.006 | 1.009 |
    | `lineitem`, 1,000,000 rows | 1.003 | 0.998 | 1.007 |
    | Text, 64 MB limit | 1.010 | 1.002 | 1.007 |
    
    - Every median is within 1% of the codec without the shrink.
    - I also ran what `GroupedPythonArrowInput` does on the JVM for each group, 
from creating the writer to closing it, with a null output stream. The medians 
for 10 to 10,000 rows are between 0.991 and 1.007.
    - Other builds were running on the machine, so I compared thread CPU time, 
which does not show lock waits. I cannot rule out a small wall time increase 
with 8 threads.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59998 from dongjoon-hyun/SPARK-60779.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit aa6ea79e437426d7a0ec6bc75d8b9a710456894e)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../execution/arrow/ArrowCompressionUtils.scala    | 49 +++++++++++++--
 .../arrow/ArrowCompressionUtilsSuite.scala         | 73 +++++++++++++++++++++-
 2 files changed, 115 insertions(+), 7 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
index c6a57430ae48..ce995bb0bc0d 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
@@ -22,7 +22,7 @@ import scala.jdk.CollectionConverters._
 
 import com.github.luben.zstd.Zstd
 import org.apache.arrow.compression.{Lz4CompressionCodec, ZstdCompressionCodec}
-import org.apache.arrow.memory.{ArrowBuf, BufferAllocator}
+import org.apache.arrow.memory.{ArrowBuf, BufferAllocator, 
OutOfMemoryException}
 import org.apache.arrow.util.AutoCloseables
 import org.apache.arrow.vector.{VectorSchemaRoot, VectorUnloader}
 import org.apache.arrow.vector.compression.{CompressionCodec, CompressionUtil, 
NoCompressionCodec}
@@ -78,13 +78,22 @@ private[sql] object ArrowCompressionUtils {
 }
 
 /**
- * A [[ZstdCompressionCodec]] that never lets zstd write past its output 
buffer. arrow-java up to
- * 19.0 passes `compressBound(len) + 8` as the destination capacity while 
pointing zstd 8 bytes
- * into a buffer of that size, so zstd can overrun it by up to 8 bytes; here 
the capacity is
- * `compressBound(len)`. Remove once on Arrow 20.0.0 (apache/arrow-java#1119).
+ * A [[ZstdCompressionCodec]] that never lets zstd write past its output 
buffer and that does not
+ * keep that buffer for an output that needs much less memory.
+ *
+ * arrow-java up to 19.0 passes `compressBound(len) + 8` as the destination 
capacity while pointing
+ * zstd 8 bytes into a buffer of that size, so zstd can overrun it by up to 8 
bytes; here the
+ * capacity is `compressBound(len)`. Remove `doCompress` once on Arrow 20.0.0
+ * (apache/arrow-java#1119).
+ *
+ * arrow-java also leaves the output in that buffer, whose size the allocator 
rounds up to a power
+ * of two below 16 MiB, and `VectorUnloader` keeps the output of every vector 
buffer until the
+ * record batch is closed. So a batch would hold about as much direct memory 
as its uncompressed
+ * vectors, whatever the compression ratio.
  */
 private[sql] class BoundedZstdCompressionCodec(compressionLevel: Int)
   extends ZstdCompressionCodec(compressionLevel) {
+  import BoundedZstdCompressionCodec._
 
   override protected def doCompress(
       allocator: BufferAllocator,
@@ -105,6 +114,36 @@ private[sql] class 
BoundedZstdCompressionCodec(compressionLevel: Int)
     compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + 
bytesWritten)
     compressedBuffer
   }
+
+  override def compress(allocator: BufferAllocator, uncompressedBuffer: 
ArrowBuf): ArrowBuf = {
+    val compressed = super.compress(allocator, uncompressedBuffer)
+    val length = compressed.writerIndex()
+    val roundedLength = allocator.getRoundingPolicy.getRoundedSize(length)
+    // Requiring at least half of the buffer back also limits the extra memory 
that the copy needs
+    // while both buffers are alive. Empty and raw outputs are already in a 
buffer of their size.
+    if (roundedLength > compressed.capacity() / 2 ||
+        compressed.capacity() - roundedLength < MIN_SHRINK_SAVING_BYTES) {
+      compressed
+    } else {
+      try {
+        val shrunk = allocator.buffer(length)
+        shrunk.setBytes(0, compressed, 0, length)
+        shrunk.writerIndex(length)
+        compressed.close()
+        shrunk
+      } catch {
+        // The output is complete, so this only gives up the saving. Failing 
would also be wrong:
+        // the input is already released, and TrackingCompressionCodec would 
release it again.
+        case _: OutOfMemoryException => compressed
+      }
+    }
+  }
+}
+
+private object BoundedZstdCompressionCodec {
+  // A smaller saving does not pay for another allocation, copy and release, 
which a small batch,
+  // e.g. a group of a grouped Python UDF, would do for each of its buffers.
+  val MIN_SHRINK_SAVING_BYTES = 4 * 1024
 }
 
 /**
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
index f424aff29720..2bbdf770fea2 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
@@ -169,8 +169,9 @@ class ArrowCompressionUtilsSuite extends SparkFunSuite {
       }
       // Control: these inputs do trigger the overrun in Arrow's own codec, so 
the checks above
       // can fail. This depends on how zstd compresses literals. If it starts 
failing after an
-      // Arrow upgrade, Arrow has fixed the codec and 
BoundedZstdCompressionCodec can be removed;
-      // if it starts failing after a zstd-jni upgrade, pick inputs that still 
overrun instead.
+      // Arrow upgrade, Arrow has fixed the codec and 
BoundedZstdCompressionCodec.doCompress can be
+      // removed; if it starts failing after a zstd-jni upgrade, pick inputs 
that still overrun
+      // instead.
       assert(inputs.exists(writesPastEnd(stock.compressRaw, _)),
         s"no input of size $n made Arrow's ZstdCompressionCodec write past its 
output buffer")
     }
@@ -301,4 +302,72 @@ class ArrowCompressionUtilsSuite extends SparkFunSuite {
       allocator.close()
     }
   }
+
+  // Both an input of this size and zstd's output buffer for it take this much 
from the allocator.
+  private val inputSize = 100000
+  private val allocatedSize = 128 * 1024
+  private val compressibleInput = Array.tabulate[Byte](inputSize)(i => (i % 
7).toByte)
+
+  // Compresses `data` with the codec of the write paths the way 
VectorUnloader does, that is,
+  // while the caller keeps its own reference to the input, and checks the 
allocator and the output.
+  private def withZstdOutput(data: Array[Byte], memoryLimit: Long = 
Long.MaxValue)(
+      check: (BufferAllocator, ArrowBuf) => Unit): Unit = {
+    val allocator =
+      ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName, 
0, memoryLimit)
+    val codec = ArrowCompressionUtils.createCompressionCodec("zstd", 3)
+    Utils.tryWithSafeFinally {
+      val input = allocator.buffer(data.length)
+      Utils.tryWithSafeFinally {
+        input.setBytes(0, data)
+        input.writerIndex(data.length)
+        input.getReferenceManager.retain()
+        val output = codec.compress(allocator, input)
+        Utils.tryWithSafeFinally {
+          check(allocator, output)
+          assert(output.nioBuffer(0, 
8).order(ByteOrder.LITTLE_ENDIAN).getLong(0) === data.length)
+        } {
+          output.close()
+        }
+      } {
+        input.close()
+      }
+    } {
+      allocator.close()
+    }
+  }
+
+  test("SPARK-60779: zstd codec moves a much smaller output out of its 
compressBound buffer") {
+    withZstdOutput(compressibleInput) { (allocator, output) =>
+      assert(output.capacity() === 
allocator.getRoundingPolicy.getRoundedSize(output.writerIndex()))
+      assert(output.capacity() < allocatedSize / 2)
+      assert(allocator.getAllocatedMemory === allocatedSize + 
output.capacity())
+    }
+  }
+
+  test("SPARK-60779: zstd codec does not copy an output that would not at 
least halve its buffer") {
+    // The allocator does not round up from 16 MiB, so this output saves 
memory without halving
+    // its buffer. Below that, an output that does not halve its buffer saves 
nothing.
+    val size = 20 * 1024 * 1024
+    val input = new Random(3).nextBytes(size * 3 / 4) ++ new Array[Byte](size 
/ 4)
+    withZstdOutput(input) { (allocator, output) =>
+      val rounded = 
allocator.getRoundingPolicy.getRoundedSize(output.writerIndex())
+      assert(rounded > output.capacity() / 2)
+      assert(output.capacity() - rounded >= 
BoundedZstdCompressionCodec.MIN_SHRINK_SAVING_BYTES)
+      assert(allocator.getPeakMemoryAllocation === size + output.capacity())
+    }
+  }
+
+  test("SPARK-60779: zstd codec does not copy an output when that would save 
little memory") {
+    withZstdOutput(compressibleInput.take(800)) { (_, output) =>
+      // The buffer for this input is 1 KiB, so the output would halve it but 
save too little.
+      assert(output.writerIndex() <= 512)
+      assert(output.capacity() === 1024)
+    }
+  }
+
+  test("SPARK-60779: zstd codec keeps its output buffer when there is no 
memory to shrink it") {
+    withZstdOutput(compressibleInput, memoryLimit = 2 * allocatedSize) { (_, 
output) =>
+      assert(output.capacity() === allocatedSize)
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to