This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 26b58aa9c17a [SPARK-60779][SQL] Shrink zstd output buffers to reduce
direct memory held by Arrow batches
26b58aa9c17a is described below
commit 26b58aa9c17acac056f1c3eab01f6396b14aea03
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Nov 1 11:22:48 2026 -0800
[SPARK-60779][SQL] Shrink zstd output buffers to reduce direct memory held
by Arrow batches
### What changes were proposed in this pull request?
This PR makes `BoundedZstdCompressionCodec` copy a compressed output to a
smaller buffer and release the buffer that zstd wrote into, when that at least
halves the memory that the output takes and saves at least 4 KiB.
The codec now overrides `compress`. After `super.compress` returns, it
computes `roundedLength`, the size that the allocator would give the output. If
`roundedLength <= capacity / 2` and `capacity - roundedLength >= 4 KiB`, it
copies the output with its length prefix to a new buffer and closes the big one.
- Empty and raw (incompressible) outputs are already in a buffer of their
size, so they are not copied.
- The 4 KiB floor keeps small buffers out of the copy. They would give back
little memory but still pay for an allocation, a copy, and a release, and small
batches are common with grouped and cogrouped Python UDFs, which write each
group as its own batch.
- The halving condition limits the extra memory while both buffers are
alive. It matters from 16 MiB, where the allocator stops rounding up. Below
that, an output that does not halve its buffer saves nothing.
- If the new buffer cannot be allocated (`OutOfMemoryException`), the codec
keeps the big buffer. The output is complete, so this only gives up the saving,
and a batch that succeeded before does not fail now.
- `createVectorUnloader` (SPARK-60776) wraps this codec, so
`TrackingCompressionCodec` holds the new buffer. It assumes that a failed
`compress` still holds its input, but the shrink runs after `super.compress`
released the input. So the shrink must not fail with `OutOfMemoryException`,
and a code comment explains this.
All zstd write paths get this codec from
`ArrowCompressionUtils.createCompressionCodec`: Python UDF input, `toPandas`,
Spark Connect results, External UDF, and the Arrow cache. lz4 is not changed,
because `Lz4CompressionCodec` already requests a buffer of the output size.
The class documentation now limits its removal note to `doCompress`. Arrow
20.0.0 will fix the overrun (apache/arrow-java#1119), but the main branch of
Arrow Java does not shrink the output, so `compress` has to stay until it does.
Alternatives that were not taken:
- Shrink in `doCompress`: the SPARK-60707 test calls `doCompress` directly
and checks for a write past the buffer that zstd wrote into, so it would check
the copy instead. It would also need its own check for the raw fallback.
- Copy at a lower threshold: with 25% saved as the threshold, the
1,000,000-row UUID batch below would hold 22.20 MiB instead of 38.47 MiB, but
its peak would be 56.67 MiB instead of 38.47 MiB.
- No floor, or a higher one: without a floor, `getRecordBatch` and `close`
of a 10-row BIGINT batch cost up to 31.8% more with 8 threads (about 1 us per
batch). A 64 KiB floor would also skip 1,000-row batches, but the 10,000-row
`lineitem` batch below would hold 0.93 MiB instead of 0.50 MiB.
- A scratch buffer per batch or per task: it saved no CPU in the
measurements and holds the same memory or more. `CompressionCodec` has no
`close()`, and some paths create a codec per group or batch, so each path would
need new code to release it.
- A different `RoundingPolicy`: it applies to all Arrow allocations under
the shared `ArrowUtils.rootAllocator`, and the buffer would still be
`compressBound`-sized.
### Why are the changes needed?
Arrow Java 19.0.0 leaves the zstd output in the buffer that it allocated
before compressing, and [its main
branch](https://github.com/apache/arrow-java/blob/67ef5abf308a7a158c4fa8eeebb72b87ca5a6f97/compression/src/main/java/org/apache/arrow/compression/ZstdCompressionCodec.java#L41-L55)
still does:
- `ZstdCompressionCodec.doCompress` requests `8 + compressBound(len)`
bytes, and the allocator rounds that up to a power of two below 16 MiB.
- `VectorUnloader` keeps the output of every vector buffer until the record
batch is closed, that is, until it is serialized.
So a batch holds about as much extra direct memory as its uncompressed
vectors, whatever the compression ratio. For comparison, [Arrow
C++](https://github.com/apache/arrow/blob/926885f1b1ede73badd24c2e6f63c9429441e99a/cpp/src/arrow/ipc/writer.cc#L210-L250)
shrinks its output right after compressing.
This is not a correctness issue, and the cost depends on the batch size:
- A 10,000-row batch, the default
`spark.sql.execution.arrow.maxRecordsPerBatch`, costs 0.1 to 2.6 MiB per task
with narrow rows.
- A 64 MB batch costs 64 to 88 MiB per task, or about 1.0 to 1.4 GiB with
16 concurrent tasks. Batches reach that size with wide rows (the default
`spark.sql.execution.arrow.maxBytesPerBatch` is 64MB) or with a larger
`maxRecordsPerBatch`.
Spark's `MemoryManager` does not track this memory, and when direct memory
runs out, the task fails with `OutOfMemoryException`.
This only happens when `spark.sql.execution.arrow.compression.codec` is
`zstd`; the default is `none`.
### Does this PR introduce _any_ user-facing change?
No. The serialized bytes are the same, and only the direct memory in use
changes.
### How was this patch tested?
Pass the CIs with the newly added test cases in
`ArrowCompressionUtilsSuite`:
- A much smaller output moves to a buffer of its size and the big buffer is
released. This test fails without this change.
- An output that would not at least halve its buffer is not copied, so the
peak allocation does not grow. Its input is 20 MiB, where only the halving
condition keeps the buffer.
- An output that would save less than 4 KiB is not copied.
- When there is no memory for the new buffer, the output stays in its
buffer.
I checked these tests by changing the code: each test fails when its branch
is removed, and bringing back the SPARK-60707 overrun still fails the
SPARK-60707 tests. `ArrowCompressionUtilsSuite`,
`ArrowCachedBatchSerializerSuite`, `ArrowConvertersSuite`, and
`ExecuteExternalUDFExecSuite` pass locally (152 tests).
I also ran `createVectorUnloader` with this codec in a temporary script,
which is not part of this PR. A batch that fails after an output was shrunk,
and a batch that runs out of memory after a shrink gave up, both release each
buffer once: there is no suppressed exception, the allocator goes back to the
size of the vectors, and closing it succeeds.
I measured the memory with a temporary benchmark, which is not part of this
PR. It fills a root with `ArrowWriter` and calls
`VectorUnloader.getRecordBatch()`, with an `AllocationListener` on the child
allocator of the root to track allocations and releases. The environment is an
Apple M3 Max, JDK 25.0.5, Arrow 19.0.0, and zstd level 3. For all measured
batches, the serialized IPC bytes are the same as before.
| Batch (MiB) | Vectors | Compressed | Held before | Held after | Peak
before | Peak after |
|---|---:|---:|---:|---:|---:|---:|
| Sequential BIGINT, 10,000 rows | 0.13 | 0.02 | 0.13 | 0.03 | 0.13 | 0.16 |
| `lineitem`, 10,000 rows | 2.63 | 0.37 | 2.63 | 0.50 | 2.63 | 1.00 |
| `lineitem`, 1,000,000 rows | 178.00 | 37.95 | 178.70 | 52.75 | 178.70 |
83.45 |
| `lineitem`, 64 MB limit (389,390 rows) | 87.00 | 14.75 | 88.00 | 22.88 |
88.00 | 38.88 |
| Text, 64 MB limit (8,348 rows) | 64.06 | 13.97 | 64.28 | 16.03 | 64.28 |
80.25 |
| UUID strings, 1,000,000 rows | 68.00 | 20.98 | 38.59 | 38.47 | 38.59 |
38.47 |
| Random BIGINT, 1,000,000 rows (raw output) | 8.00 | 7.63 | 8.13 | 8.00 |
8.13 | 8.00 |
- `lineitem` is a generated TPC-H-like batch with 16 columns and 37 Arrow
buffers. The text batch has about 8 KB of JSON-like text per row.
- Held and peak are on top of the vectors. Held is what a batch holds until
it is serialized, and peak is the highest point during `getRecordBatch`.
- The peak goes up when one big buffer dominates a batch, as in the text
batch, because both buffers are alive during the copy. The halving condition
limits this to half of that buffer, and it lasts only for the copy.
- The floor gives up less than 4 KiB per Arrow buffer, about 32 KiB for the
10,000-row `lineitem` batch.
- These numbers are Arrow allocator accounting, not RSS. A new buffer above
4 MiB commits only the pages that zstd writes (checked on macOS), so the effect
on RSS is smaller. I did not measure RSS or OOMs in a real executor, or
anything on Linux.
For the CPU cost, I compared the thread CPU time of `getRecordBatch` and
`close` with and without the shrink. Each value is the median after/before
ratio across rounds:
| Batch | 1 thread | 8 threads | 8 threads with jemalloc |
|---|---:|---:|---:|
| Sequential BIGINT, 10 rows | 1.006 | 1.006 | 1.002 |
| Sequential BIGINT, 100 rows | 1.004 | 1.002 | 0.997 |
| `lineitem`, 10 rows | 1.001 | 1.008 | 0.999 |
| `lineitem`, 100 rows | 1.000 | 1.003 | 1.002 |
| `lineitem`, 1,000 rows | 1.003 | 1.008 | 1.006 |
| `lineitem`, 10,000 rows | 1.003 | 1.006 | 1.009 |
| `lineitem`, 1,000,000 rows | 1.003 | 0.998 | 1.007 |
| Text, 64 MB limit | 1.010 | 1.002 | 1.007 |
- Every median is within 1% of the codec without the shrink.
- I also ran what `GroupedPythonArrowInput` does on the JVM for each group,
from creating the writer to closing it, with a null output stream. The medians
for 10 to 10,000 rows are between 0.991 and 1.007.
- Other builds were running on the machine, so I compared thread CPU time,
which does not show lock waits. I cannot rule out a small wall time increase
with 8 threads.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59998 from dongjoon-hyun/SPARK-60779.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit aa6ea79e437426d7a0ec6bc75d8b9a710456894e)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../execution/arrow/ArrowCompressionUtils.scala | 49 +++++++++++++--
.../arrow/ArrowCompressionUtilsSuite.scala | 73 +++++++++++++++++++++-
2 files changed, 115 insertions(+), 7 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
index c6a57430ae48..ce995bb0bc0d 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
@@ -22,7 +22,7 @@ import scala.jdk.CollectionConverters._
import com.github.luben.zstd.Zstd
import org.apache.arrow.compression.{Lz4CompressionCodec, ZstdCompressionCodec}
-import org.apache.arrow.memory.{ArrowBuf, BufferAllocator}
+import org.apache.arrow.memory.{ArrowBuf, BufferAllocator,
OutOfMemoryException}
import org.apache.arrow.util.AutoCloseables
import org.apache.arrow.vector.{VectorSchemaRoot, VectorUnloader}
import org.apache.arrow.vector.compression.{CompressionCodec, CompressionUtil,
NoCompressionCodec}
@@ -78,13 +78,22 @@ private[sql] object ArrowCompressionUtils {
}
/**
- * A [[ZstdCompressionCodec]] that never lets zstd write past its output
buffer. arrow-java up to
- * 19.0 passes `compressBound(len) + 8` as the destination capacity while
pointing zstd 8 bytes
- * into a buffer of that size, so zstd can overrun it by up to 8 bytes; here
the capacity is
- * `compressBound(len)`. Remove once on Arrow 20.0.0 (apache/arrow-java#1119).
+ * A [[ZstdCompressionCodec]] that never lets zstd write past its output
buffer and that does not
+ * keep that buffer for an output that needs much less memory.
+ *
+ * arrow-java up to 19.0 passes `compressBound(len) + 8` as the destination
capacity while pointing
+ * zstd 8 bytes into a buffer of that size, so zstd can overrun it by up to 8
bytes; here the
+ * capacity is `compressBound(len)`. Remove `doCompress` once on Arrow 20.0.0
+ * (apache/arrow-java#1119).
+ *
+ * arrow-java also leaves the output in that buffer, whose size the allocator
rounds up to a power
+ * of two below 16 MiB, and `VectorUnloader` keeps the output of every vector
buffer until the
+ * record batch is closed. So a batch would hold about as much direct memory
as its uncompressed
+ * vectors, whatever the compression ratio.
*/
private[sql] class BoundedZstdCompressionCodec(compressionLevel: Int)
extends ZstdCompressionCodec(compressionLevel) {
+ import BoundedZstdCompressionCodec._
override protected def doCompress(
allocator: BufferAllocator,
@@ -105,6 +114,36 @@ private[sql] class
BoundedZstdCompressionCodec(compressionLevel: Int)
compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH +
bytesWritten)
compressedBuffer
}
+
+ override def compress(allocator: BufferAllocator, uncompressedBuffer:
ArrowBuf): ArrowBuf = {
+ val compressed = super.compress(allocator, uncompressedBuffer)
+ val length = compressed.writerIndex()
+ val roundedLength = allocator.getRoundingPolicy.getRoundedSize(length)
+ // Requiring at least half of the buffer back also limits the extra memory
that the copy needs
+ // while both buffers are alive. Empty and raw outputs are already in a
buffer of their size.
+ if (roundedLength > compressed.capacity() / 2 ||
+ compressed.capacity() - roundedLength < MIN_SHRINK_SAVING_BYTES) {
+ compressed
+ } else {
+ try {
+ val shrunk = allocator.buffer(length)
+ shrunk.setBytes(0, compressed, 0, length)
+ shrunk.writerIndex(length)
+ compressed.close()
+ shrunk
+ } catch {
+ // The output is complete, so this only gives up the saving. Failing
would also be wrong:
+ // the input is already released, and TrackingCompressionCodec would
release it again.
+ case _: OutOfMemoryException => compressed
+ }
+ }
+ }
+}
+
+private object BoundedZstdCompressionCodec {
+ // A smaller saving does not pay for another allocation, copy and release,
which a small batch,
+ // e.g. a group of a grouped Python UDF, would do for each of its buffers.
+ val MIN_SHRINK_SAVING_BYTES = 4 * 1024
}
/**
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
index f424aff29720..2bbdf770fea2 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
@@ -169,8 +169,9 @@ class ArrowCompressionUtilsSuite extends SparkFunSuite {
}
// Control: these inputs do trigger the overrun in Arrow's own codec, so
the checks above
// can fail. This depends on how zstd compresses literals. If it starts
failing after an
- // Arrow upgrade, Arrow has fixed the codec and
BoundedZstdCompressionCodec can be removed;
- // if it starts failing after a zstd-jni upgrade, pick inputs that still
overrun instead.
+ // Arrow upgrade, Arrow has fixed the codec and
BoundedZstdCompressionCodec.doCompress can be
+ // removed; if it starts failing after a zstd-jni upgrade, pick inputs
that still overrun
+ // instead.
assert(inputs.exists(writesPastEnd(stock.compressRaw, _)),
s"no input of size $n made Arrow's ZstdCompressionCodec write past its
output buffer")
}
@@ -301,4 +302,72 @@ class ArrowCompressionUtilsSuite extends SparkFunSuite {
allocator.close()
}
}
+
+ // Both an input of this size and zstd's output buffer for it take this much
from the allocator.
+ private val inputSize = 100000
+ private val allocatedSize = 128 * 1024
+ private val compressibleInput = Array.tabulate[Byte](inputSize)(i => (i %
7).toByte)
+
+ // Compresses `data` with the codec of the write paths the way
VectorUnloader does, that is,
+ // while the caller keeps its own reference to the input, and checks the
allocator and the output.
+ private def withZstdOutput(data: Array[Byte], memoryLimit: Long =
Long.MaxValue)(
+ check: (BufferAllocator, ArrowBuf) => Unit): Unit = {
+ val allocator =
+ ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName,
0, memoryLimit)
+ val codec = ArrowCompressionUtils.createCompressionCodec("zstd", 3)
+ Utils.tryWithSafeFinally {
+ val input = allocator.buffer(data.length)
+ Utils.tryWithSafeFinally {
+ input.setBytes(0, data)
+ input.writerIndex(data.length)
+ input.getReferenceManager.retain()
+ val output = codec.compress(allocator, input)
+ Utils.tryWithSafeFinally {
+ check(allocator, output)
+ assert(output.nioBuffer(0,
8).order(ByteOrder.LITTLE_ENDIAN).getLong(0) === data.length)
+ } {
+ output.close()
+ }
+ } {
+ input.close()
+ }
+ } {
+ allocator.close()
+ }
+ }
+
+ test("SPARK-60779: zstd codec moves a much smaller output out of its
compressBound buffer") {
+ withZstdOutput(compressibleInput) { (allocator, output) =>
+ assert(output.capacity() ===
allocator.getRoundingPolicy.getRoundedSize(output.writerIndex()))
+ assert(output.capacity() < allocatedSize / 2)
+ assert(allocator.getAllocatedMemory === allocatedSize +
output.capacity())
+ }
+ }
+
+ test("SPARK-60779: zstd codec does not copy an output that would not at
least halve its buffer") {
+ // The allocator does not round up from 16 MiB, so this output saves
memory without halving
+ // its buffer. Below that, an output that does not halve its buffer saves
nothing.
+ val size = 20 * 1024 * 1024
+ val input = new Random(3).nextBytes(size * 3 / 4) ++ new Array[Byte](size
/ 4)
+ withZstdOutput(input) { (allocator, output) =>
+ val rounded =
allocator.getRoundingPolicy.getRoundedSize(output.writerIndex())
+ assert(rounded > output.capacity() / 2)
+ assert(output.capacity() - rounded >=
BoundedZstdCompressionCodec.MIN_SHRINK_SAVING_BYTES)
+ assert(allocator.getPeakMemoryAllocation === size + output.capacity())
+ }
+ }
+
+ test("SPARK-60779: zstd codec does not copy an output when that would save
little memory") {
+ withZstdOutput(compressibleInput.take(800)) { (_, output) =>
+ // The buffer for this input is 1 KiB, so the output would halve it but
save too little.
+ assert(output.writerIndex() <= 512)
+ assert(output.capacity() === 1024)
+ }
+ }
+
+ test("SPARK-60779: zstd codec keeps its output buffer when there is no
memory to shrink it") {
+ withZstdOutput(compressibleInput, memoryLimit = 2 * allocatedSize) { (_,
output) =>
+ assert(output.capacity() === allocatedSize)
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]