This is an automated email from the ASF dual-hosted git repository.
peter-toth pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 20ea29cc86a6 [SPARK-60707][SQL] Avoid zstd writing past the output
buffer when compressing Arrow batches
20ea29cc86a6 is described below
commit 20ea29cc86a65dc3319bbbebaf8c69b8055ae516
Author: Andy Grove <[email protected]>
AuthorDate: Sat Oct 31 13:30:15 2026 +0100
[SPARK-60707][SQL] Avoid zstd writing past the output buffer when
compressing Arrow batches
### What changes were proposed in this pull request?
Adds `BoundedZstdCompressionCodec`, a subclass of Arrow's
`ZstdCompressionCodec` that overrides only `doCompress` so that zstd is given
`compressBound(len)` as its destination capacity, and uses it from
`ArrowCompressionUtils.createCompressionCodec`. That method creates the
write-side codec for the Arrow IPC batches written for Python and Spark Connect
when `spark.sql.execution.arrow.compression.codec=zstd`.
`ArrowCachedBatchSerializer.createCompressionCodec` was a second copy of
the same codec factory and a pass-through of the already lowercased
`spark.sql.execution.arrow.compression.codec`. It is removed, and its two
callers use `ArrowCompressionUtils.createCompressionCodec` directly, so the
workaround lives in one place. Its level test in
`ArrowCachedBatchSerializerSuite` duplicated the SPARK-57383 test in
`ArrowCompressionUtilsSuite` and is removed with it.
Decompression is inherited from Arrow unchanged. The class carries a note
that it can be removed once Spark is on Arrow 20.0.0.
### Why are the changes needed?
In arrow-java up to 19.0.0 (Spark ships 18.3.0 and 19.0.0 with this write
path), `ZstdCompressionCodec.doCompress` allocates `compressBound(len) + 8`
bytes, points zstd 8 bytes into that buffer (past the uncompressed-length
prefix), but still passes `compressBound(len) + 8` as the destination capacity.
zstd only guarantees that it stays within the capacity it is given, and on a
block that does not compress it can fill all of it, so it can write up to 8
bytes past the end of the buffer [...]
With `arrow-memory-netty` the overrun can also hit the next buffer of the
same batch: its length prefix gets overwritten, and reading the batch then
fails with `Error decompressing: Unknown frame descriptor`. The request lands
in that range for buffers of 433 to 440 bytes, for example 55 rows of a BIGINT
column that does not compress.
This is fixed upstream in apache/arrow-java#1119 (Arrow 20.0.0). Arrow
20.0.0 is not released yet, and the maintenance branches cannot take a major
Arrow bump, so this works around it on the Spark side.
Only sessions that opt in to zstd are affected; the default is `none`.
### Does this PR introduce _any_ user-facing change?
No. With zstd enabled, compressed output is the same; only the capacity
handed to zstd changes, which removes the out-of-bounds write.
### How was this patch tested?
Added to `ArrowCompressionUtilsSuite`:
- A test that allocates the codec's output from an allocator that puts a
sentinel guard directly behind each buffer, compresses random inputs with about
2% of the bytes zeroed (which makes zstd fill its whole destination for inputs
of roughly 280 to 1000 bytes), and asserts the guard is intact. It checks 100
inputs each at 280, 500 and 700 bytes, and a few other sizes from 1 byte to 256
KB as smoke checks. For each of the three sizes, a control asserts that Arrow's
stock `ZstdCompress [...]
- A test that writes batches of three BIGINT columns the way
`ArrowConverters` does, with one reused root and `VectorUnloader`, and asserts
that no data buffer's length prefix was overwritten by the previous buffer.
With Arrow's stock `ZstdCompressionCodec` in the factory it fails; with
`BoundedZstdCompressionCodec` it passes.
- A round-trip test for compressible and incompressible buffers through
`createCompressionCodec("zstd", ...)`. It asserts that the factory returns
`BoundedZstdCompressionCodec` and which path (zstd frame or raw) each case
takes.
Ran `ArrowCompressionUtilsSuite` and `ArrowCachedBatchSerializerSuite`
locally on this tree (83 tests, all passing).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Sonnet 5.5)
Closes #59922 from andygrove/SPARK-60707.
Authored-by: Andy Grove <[email protected]>
Signed-off-by: Peter Toth <[email protected]>
---
.../execution/arrow/ArrowCompressionUtils.scala | 39 ++++-
.../columnar/ArrowCachedBatchSerializer.scala | 30 +---
.../arrow/ArrowCompressionUtilsSuite.scala | 177 ++++++++++++++++++++-
.../columnar/ArrowCachedBatchSerializerSuite.scala | 43 +----
4 files changed, 219 insertions(+), 70 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
index 19b84d1cd5d8..e71680751ab8 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
@@ -17,8 +17,10 @@
package org.apache.spark.sql.execution.arrow
+import com.github.luben.zstd.Zstd
import org.apache.arrow.compression.{Lz4CompressionCodec, ZstdCompressionCodec}
-import org.apache.arrow.vector.compression.{CompressionCodec,
NoCompressionCodec}
+import org.apache.arrow.memory.{ArrowBuf, BufferAllocator}
+import org.apache.arrow.vector.compression.{CompressionCodec, CompressionUtil,
NoCompressionCodec}
import org.apache.spark.SparkException
@@ -34,13 +36,16 @@ private[sql] object ArrowCompressionUtils {
* carry a compression level, so that factory overload always builds a codec
at the default
* level, silently dropping the configured one. The level only matters on
the write side; the
* read side looks up the codec by the type recorded in the IPC message, so
it is unaffected.
+ *
+ * For zstd it must also be a [[BoundedZstdCompressionCodec]], which neither
factory overload
+ * returns, not even `createCodec(codecType, compressionLevel)`, which does
keep the level.
*/
def createCompressionCodec(
codecName: String,
zstdCompressionLevel: Int): CompressionCodec = {
codecName match {
case "none" => NoCompressionCodec.INSTANCE
- case "zstd" => new ZstdCompressionCodec(zstdCompressionLevel)
+ case "zstd" => new BoundedZstdCompressionCodec(zstdCompressionLevel)
case "lz4" => new Lz4CompressionCodec()
case other =>
throw SparkException.internalError(
@@ -48,3 +53,33 @@ private[sql] object ArrowCompressionUtils {
}
}
}
+
+/**
+ * A [[ZstdCompressionCodec]] that never lets zstd write past its output
buffer. arrow-java up to
+ * 19.0 passes `compressBound(len) + 8` as the destination capacity while
pointing zstd 8 bytes
+ * into a buffer of that size, so zstd can overrun it by up to 8 bytes; here
the capacity is
+ * `compressBound(len)`. Remove once on Arrow 20.0.0 (apache/arrow-java#1119).
+ */
+private[sql] class BoundedZstdCompressionCodec(compressionLevel: Int)
+ extends ZstdCompressionCodec(compressionLevel) {
+
+ override protected def doCompress(
+ allocator: BufferAllocator,
+ uncompressedBuffer: ArrowBuf): ArrowBuf = {
+ val srcSize = uncompressedBuffer.writerIndex()
+ val maxSize = Zstd.compressBound(srcSize)
+ val compressedBuffer =
allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + maxSize)
+ val bytesWritten = Zstd.compressUnsafe(
+ compressedBuffer.memoryAddress() +
CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH,
+ maxSize,
+ uncompressedBuffer.memoryAddress(),
+ srcSize,
+ compressionLevel)
+ if (Zstd.isError(bytesWritten)) {
+ compressedBuffer.close()
+ throw SparkException.internalError("Error compressing: " +
Zstd.getErrorName(bytesWritten))
+ }
+ compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH +
bytesWritten)
+ compressedBuffer
+ }
+}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
index baba2c1365c8..db91f91e2480 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
@@ -22,17 +22,16 @@ import java.nio.channels.Channels
import scala.jdk.CollectionConverters._
-import org.apache.arrow.compression.{Lz4CompressionCodec, ZstdCompressionCodec}
import org.apache.arrow.flatbuf.{RecordBatch => FlatBufRecordBatch}
import org.apache.arrow.memory.BufferAllocator
import org.apache.arrow.vector.{TypeLayout, VectorLoader, VectorSchemaRoot,
VectorUnloader}
-import org.apache.arrow.vector.compression.{CompressionCodec,
NoCompressionCodec}
+import org.apache.arrow.vector.compression.NoCompressionCodec
import org.apache.arrow.vector.ipc.{ReadChannel, WriteChannel}
import org.apache.arrow.vector.ipc.message.{ArrowBodyCompression,
ArrowFieldNode}
import org.apache.arrow.vector.ipc.message.{ArrowRecordBatch,
MessageSerializer}
import org.apache.arrow.vector.types.pojo.Field
-import org.apache.spark.{SparkException, TaskContext}
+import org.apache.spark.TaskContext
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.Attribute
@@ -40,7 +39,7 @@ import
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
import org.apache.spark.sql.catalyst.types.DataTypeUtils
import org.apache.spark.sql.columnar.{CachedBatch,
SimpleMetricsCachedBatchSerializer}
import org.apache.spark.sql.errors.ExecutionErrors
-import org.apache.spark.sql.execution.arrow.ArrowWriter
+import org.apache.spark.sql.execution.arrow.{ArrowCompressionUtils,
ArrowWriter}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.sql.util.ArrowUtils
@@ -263,25 +262,6 @@ private object ArrowCachedBatchSerializer {
}
}
- // scalastyle:off caselocale
- def createCompressionCodec(
- codecName: String,
- compressionLevel: Int): CompressionCodec = {
- codecName.toLowerCase match {
- case "none" => NoCompressionCodec.INSTANCE
- // The codec instance must be constructed directly so that
compressionLevel is honored:
- // CompressionCodec.Factory.createCodec(codecType) ignores the level and
builds a codec at
- // the default level. The level only matters on the write side; the read
side looks up the
- // codec by the type recorded in the IPC message.
- case "zstd" => new ZstdCompressionCodec(compressionLevel)
- case "lz4" => new Lz4CompressionCodec()
- case other =>
- throw SparkException.internalError(
- s"Unsupported Arrow compression codec: $other. Supported values:
none, zstd, lz4")
- }
- }
- // scalastyle:on caselocale
-
def serializeBatch(batch: ArrowRecordBatch): Array[Byte] = {
val out = new ByteArrayOutputStream()
val writeChannel = new WriteChannel(Channels.newChannel(out))
@@ -1005,7 +985,7 @@ private class InternalRowToArrowCachedBatchIterator(
compressionCodecName: String,
compressionLevel: Int) extends Iterator[ArrowCachedBatch] {
- private val compressionCodec =
ArrowCachedBatchSerializer.createCompressionCodec(
+ private val compressionCodec = ArrowCompressionUtils.createCompressionCodec(
compressionCodecName,
compressionLevel)
@@ -1112,7 +1092,7 @@ private class ColumnarBatchToArrowCachedBatchIterator(
compressionCodecName: String,
compressionLevel: Int) extends Iterator[ArrowCachedBatch] {
- private val compressionCodec =
ArrowCachedBatchSerializer.createCompressionCodec(
+ private val compressionCodec = ArrowCompressionUtils.createCompressionCodec(
compressionCodecName,
compressionLevel)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
index 50a980dcf37a..cde828e1c743 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
@@ -18,16 +18,23 @@
package org.apache.spark.sql.execution.arrow
import java.io.ByteArrayOutputStream
+import java.nio.{ByteBuffer, ByteOrder}
import java.nio.channels.Channels
+import scala.util.Random
+
+import org.apache.arrow.compression.ZstdCompressionCodec
+import org.apache.arrow.memory.{ArrowBuf, BufferAllocator, RootAllocator}
import org.apache.arrow.vector.{VarCharVector, VectorSchemaRoot,
VectorUnloader}
import org.apache.arrow.vector.compression.NoCompressionCodec
import org.apache.arrow.vector.ipc.WriteChannel
import org.apache.arrow.vector.ipc.message.MessageSerializer
import org.apache.spark.{SparkException, SparkFunSuite}
-import org.apache.spark.sql.types.{StringType, StructField, StructType}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.types.{LongType, StringType, StructField,
StructType}
import org.apache.spark.sql.util.ArrowUtils
+import org.apache.spark.util.Utils
class ArrowCompressionUtilsSuite extends SparkFunSuite {
@@ -85,4 +92,172 @@ class ArrowCompressionUtilsSuite extends SparkFunSuite {
}
assert(e.getMessage.contains("Unsupported Arrow compression codec:
snappy"))
}
+
+ // Expose the protected `doCompress` so that a test can look at the raw
output buffer.
+ private class RawBoundedZstdCompressionCodec extends
BoundedZstdCompressionCodec(3) {
+ def compressRaw(allocator: BufferAllocator, in: ArrowBuf): ArrowBuf =
doCompress(allocator, in)
+ }
+
+ // Arrow's own codec, which up to arrow-java 19.0 gives zstd a destination
capacity that is 8
+ // bytes too large.
+ private class RawStockZstdCompressionCodec extends ZstdCompressionCodec(3) {
+ def compressRaw(allocator: BufferAllocator, in: ArrowBuf): ArrowBuf =
doCompress(allocator, in)
+ }
+
+ // Random bytes with about 2% of them zeroed. Pure random bytes only make
zstd fill its whole
+ // destination when one byte value happens to be frequent enough (more than
about n / 128 + 4
+ // occurrences) for zstd to try Huffman-coding the literals and give up.
Zeroing some bytes
+ // makes that the usual case instead of a property of the seed.
+ private def noisyBytes(random: Random, n: Int): Array[Byte] = {
+ val bytes = random.nextBytes(n)
+ (0 until n).foreach { i => if (random.nextInt(50) == 0) bytes(i) = 0 }
+ bytes
+ }
+
+ // Runs `compressRaw` into a buffer that is followed by sentinel bytes, and
returns whether the
+ // sentinel was overwritten, i.e. whether zstd wrote past the end of the
buffer.
+ private def writesPastEnd(
+ compressRaw: (BufferAllocator, ArrowBuf) => ArrowBuf,
+ data: Array[Byte]): Boolean = {
+ val guardSize = 64
+ val guardByte = 0x5A.toByte
+ var requestedSize = 0L
+ val guardedAllocator = new RootAllocator(Long.MaxValue) {
+ override def buffer(size: Long): ArrowBuf = {
+ requestedSize = size
+ val buf = super.buffer(size + guardSize)
+ buf.setBytes(size, Array.fill(guardSize)(guardByte))
+ buf
+ }
+ }
+ val inputAllocator =
+ ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName,
0, Long.MaxValue)
+ Utils.tryWithSafeFinally {
+ val input = inputAllocator.buffer(data.length)
+ val out = try {
+ input.setBytes(0, data)
+ input.writerIndex(data.length)
+ compressRaw(guardedAllocator, input)
+ } finally {
+ input.close()
+ }
+ try {
+ val guard = new Array[Byte](guardSize)
+ out.getBytes(requestedSize, guard)
+ !guard.forall(_ == guardByte)
+ } finally {
+ out.close()
+ }
+ } {
+
Utils.tryWithSafeFinally(guardedAllocator.close())(inputAllocator.close())
+ }
+ }
+
+ test("SPARK-60707: zstd compression does not write past the end of its
output buffer") {
+ val bounded = new RawBoundedZstdCompressionCodec
+ val stock = new RawStockZstdCompressionCodec
+ val random = new Random(0)
+ // Of these inputs, only those of roughly 280 to 1000 bytes make zstd fill
its whole
+ // destination; smaller and larger ones stop short of the end. The sizes
in that range are
+ // checked with many inputs each, and the other sizes only as smoke checks.
+ Seq(280, 500, 700).foreach { n =>
+ val inputs = Seq.fill(100)(noisyBytes(random, n))
+ inputs.foreach { data =>
+ assert(!writesPastEnd(bounded.compressRaw, data),
+ s"zstd wrote past the end of its output buffer (input size $n)")
+ }
+ // Control: these inputs do trigger the overrun in Arrow's own codec, so
the checks above
+ // can fail. This depends on how zstd compresses literals. If it starts
failing after an
+ // Arrow upgrade, Arrow has fixed the codec and
BoundedZstdCompressionCodec can be removed;
+ // if it starts failing after a zstd-jni upgrade, pick inputs that still
overrun instead.
+ assert(inputs.exists(writesPastEnd(stock.compressRaw, _)),
+ s"no input of size $n made Arrow's ZstdCompressionCodec write past its
output buffer")
+ }
+ Seq(1, 7, 100, 4096, 65536, 262144).foreach { n =>
+ assert(!writesPastEnd(bounded.compressRaw, noisyBytes(random, n)),
+ s"zstd wrote past the end of its output buffer (input size $n)")
+ }
+ }
+
+ test("SPARK-60707: zstd codec round-trips compressible and incompressible
buffers") {
+ val allocator =
+ ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName,
0, Long.MaxValue)
+ val codec = ArrowCompressionUtils.createCompressionCodec("zstd", 3)
+ val random = new Random(1)
+ Utils.tryWithSafeFinally {
+ // The write paths get the bounded codec, not Arrow's own.
+ assert(codec.isInstanceOf[BoundedZstdCompressionCodec])
+ Seq(1, 100, 4096, 100000).foreach { n =>
+ val compressible = Array.tabulate[Byte](n)(i => (i % 7).toByte)
+ val incompressible = random.nextBytes(n)
+ Seq(compressible -> (n >= 100), incompressible -> false).foreach {
+ case (expected, expectZstdFrame) =>
+ val input = allocator.buffer(n)
+ input.setBytes(0, expected)
+ input.writerIndex(n)
+ val compressed = codec.compress(allocator, input)
+ // The buffer starts with the uncompressed length, or -1 if Arrow
kept the input as is
+ // because zstd did not make it smaller. Check which path each
case takes.
+ val prefix = compressed.nioBuffer(0,
8).order(ByteOrder.LITTLE_ENDIAN).getLong(0)
+ assert(prefix === (if (expectZstdFrame) n.toLong else -1L))
+ val decompressed = codec.decompress(allocator, compressed)
+ try {
+ val actual = new Array[Byte](n)
+ decompressed.getBytes(0, actual)
+ assert(actual.sameElements(expected))
+ } finally {
+ decompressed.close()
+ }
+ }
+ }
+ } {
+ allocator.close()
+ }
+ }
+
+ test("SPARK-60707: batches written through VectorUnloader keep their length
prefixes") {
+ // Writes batches the way ArrowConverters does, with one reused root, and
checks the length
+ // prefix of every data buffer. An overrun of one buffer can overwrite the
prefix of the next
+ // buffer in the same batch, which makes the batch fail to decompress when
it is read. How
+ // often that happens depends on the memory allocator's layout, so this
cannot replace the
+ // guard test above, but it covers the path that readers see.
+ val rows = 55
+ val columns = 3
+ val sparkSchema = StructType(
+ (0 until columns).map(i => StructField(s"c$i", LongType, nullable =
false)))
+ val arrowSchema = ArrowUtils.toArrowSchema(sparkSchema, "UTC", true, false)
+ val allocator =
+ ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName,
0, Long.MaxValue)
+ val root = VectorSchemaRoot.create(arrowSchema, allocator)
+ val writer = ArrowWriter.create(root)
+ val unloader = new VectorUnloader(
+ root, true, ArrowCompressionUtils.createCompressionCodec("zstd", 3),
true)
+ val random = new Random(2)
+ Utils.tryWithSafeFinally {
+ (0 until 300).foreach { _ =>
+ val values = ByteBuffer.wrap(noisyBytes(random, rows * columns * 8))
+ .order(ByteOrder.LITTLE_ENDIAN).asLongBuffer()
+ (0 until rows).foreach { _ =>
+ writer.write(InternalRow.fromSeq(Seq.fill(columns)(values.get())))
+ }
+ writer.finish()
+ val batch = unloader.getRecordBatch()
+ try {
+ // The buffers of a column are its validity buffer and then its data
buffer.
+ (0 until columns).foreach { c =>
+ val dataBuffer = batch.getBuffers.get(2 * c + 1)
+ val prefix = dataBuffer.nioBuffer(0,
8).order(ByteOrder.LITTLE_ENDIAN).getLong(0)
+ assert(prefix === -1L || prefix === rows * 8L,
+ s"the length prefix of column $c was overwritten")
+ }
+ } finally {
+ batch.close()
+ }
+ writer.reset()
+ }
+ } {
+ root.close()
+ allocator.close()
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializerSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializerSuite.scala
index 347c6b42beaa..7e560cc5a839 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializerSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializerSuite.scala
@@ -24,7 +24,7 @@ import org.apache.arrow.vector.{
BigIntVector, BitVector, DateDayVector, DecimalVector,
Float4Vector, Float8Vector, IntVector, LargeVarCharVector, SmallIntVector,
TimeNanoVector, TimeStampMicroTZVector, TimeStampMicroVector, TinyIntVector,
- VarBinaryVector, VarCharVector, VectorSchemaRoot, VectorUnloader}
+ VarBinaryVector, VarCharVector, VectorSchemaRoot}
import org.apache.arrow.vector.complex.{MapVector, StructVector}
import org.apache.arrow.vector.types.pojo.{ArrowType, Field, FieldType}
@@ -1987,47 +1987,6 @@ class ArrowCachedBatchSerializerSuite extends QueryTest
with SharedSparkSession
}
}
- test("zstd compression level is honored by createCompressionCodec") {
- // Regression test: createCompressionCodec used to rebuild the codec
through the
- // single-argument factory overload, which silently dropped the level, so
every configured
- // level compressed at the zstd default. Compress the same batch at an
ultra-fast negative
- // level and at a high level and assert the high level yields a strictly
smaller payload.
- def compressedSize(level: Int): Int = {
- val schema = Seq(AttributeReference("str_col", StringType)())
- val sparkSchema = StructType(schema.map(a => StructField(a.name,
a.dataType)))
- val arrowSchema = ArrowUtils.toArrowSchema(sparkSchema, "UTC", false,
false)
- val root = VectorSchemaRoot.create(arrowSchema, ArrowUtils.rootAllocator)
- try {
- root.allocateNew()
- val strVector = root.getVector("str_col").asInstanceOf[VarCharVector]
- // Compressible but non-trivial corpus: shared structure with per-row
variation, so
- // different zstd levels produce measurably different output sizes.
- (0 until 2000).foreach { i =>
- val value =
- s"[email protected],record-${i % 97},payload-${i *
2654435761L}".getBytes("UTF-8")
- strVector.setSafe(i, value, 0, value.length)
- }
- root.setRowCount(2000)
- val codec = ArrowCachedBatchSerializer.createCompressionCodec("zstd",
level)
- val recordBatch = new VectorUnloader(root, true, codec,
true).getRecordBatch()
- try {
- ArrowCachedBatchSerializer.serializeBatch(recordBatch).length
- } finally {
- recordBatch.close()
- }
- } finally {
- root.close()
- }
- }
-
- val fastSize = compressedSize(-5)
- val highSize = compressedSize(19)
- assert(highSize < fastSize,
- s"zstd level 19 should compress smaller than level -5, " +
- s"got level 19 -> $highSize bytes vs level -5 -> $fastSize bytes; " +
- "equal sizes mean the configured level is being ignored")
- }
-
test("collectStatistics returns null bounds when all Float/Double values are
NaN") {
// When every non-null value in a Float or Double column is NaN,
calculateMinMaxFloat/Double
// finds no valid (non-NaN) values. hasValue stays false -> returns (null,
null) -> null bounds.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]