This is an automated email from the ASF dual-hosted git repository.

peter-toth pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 9e9e79713236 [SPARK-60707][SQL] Avoid zstd writing past the output 
buffer when compressing Arrow batches
9e9e79713236 is described below

commit 9e9e79713236e014236c65e4b117907960413153
Author: Andy Grove <[email protected]>
AuthorDate: Sat Oct 31 13:30:15 2026 +0100

    [SPARK-60707][SQL] Avoid zstd writing past the output buffer when 
compressing Arrow batches
    
    ### What changes were proposed in this pull request?
    
    Adds `BoundedZstdCompressionCodec`, a subclass of Arrow's 
`ZstdCompressionCodec` that overrides only `doCompress` so that zstd is given 
`compressBound(len)` as its destination capacity, and uses it from 
`ArrowCompressionUtils.createCompressionCodec`. That method creates the 
write-side codec for the Arrow IPC batches written for Python and Spark Connect 
when `spark.sql.execution.arrow.compression.codec=zstd`.
    
    `ArrowCachedBatchSerializer.createCompressionCodec` was a second copy of 
the same codec factory and a pass-through of the already lowercased 
`spark.sql.execution.arrow.compression.codec`. It is removed, and its two 
callers use `ArrowCompressionUtils.createCompressionCodec` directly, so the 
workaround lives in one place. Its level test in 
`ArrowCachedBatchSerializerSuite` duplicated the SPARK-57383 test in 
`ArrowCompressionUtilsSuite` and is removed with it.
    
    Decompression is inherited from Arrow unchanged. The class carries a note 
that it can be removed once Spark is on Arrow 20.0.0.
    
    ### Why are the changes needed?
    
    In arrow-java up to 19.0.0 (Spark ships 18.3.0 and 19.0.0 with this write 
path), `ZstdCompressionCodec.doCompress` allocates `compressBound(len) + 8` 
bytes, points zstd 8 bytes into that buffer (past the uncompressed-length 
prefix), but still passes `compressBound(len) + 8` as the destination capacity. 
zstd only guarantees that it stays within the capacity it is given, and on a 
block that does not compress it can fill all of it, so it can write up to 8 
bytes past the end of the buffer [...]
    
    With `arrow-memory-netty` the overrun can also hit the next buffer of the 
same batch: its length prefix gets overwritten, and reading the batch then 
fails with `Error decompressing: Unknown frame descriptor`. The request lands 
in that range for buffers of 433 to 440 bytes, for example 55 rows of a BIGINT 
column that does not compress.
    
    This is fixed upstream in apache/arrow-java#1119 (Arrow 20.0.0). Arrow 
20.0.0 is not released yet, and the maintenance branches cannot take a major 
Arrow bump, so this works around it on the Spark side.
    
    Only sessions that opt in to zstd are affected; the default is `none`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. With zstd enabled, compressed output is the same; only the capacity 
handed to zstd changes, which removes the out-of-bounds write.
    
    ### How was this patch tested?
    
    Added to `ArrowCompressionUtilsSuite`:
    
    - A test that allocates the codec's output from an allocator that puts a 
sentinel guard directly behind each buffer, compresses random inputs with about 
2% of the bytes zeroed (which makes zstd fill its whole destination for inputs 
of roughly 280 to 1000 bytes), and asserts the guard is intact. It checks 100 
inputs each at 280, 500 and 700 bytes, and a few other sizes from 1 byte to 256 
KB as smoke checks. For each of the three sizes, a control asserts that Arrow's 
stock `ZstdCompress [...]
    - A test that writes batches of three BIGINT columns the way 
`ArrowConverters` does, with one reused root and `VectorUnloader`, and asserts 
that no data buffer's length prefix was overwritten by the previous buffer. 
With Arrow's stock `ZstdCompressionCodec` in the factory it fails; with 
`BoundedZstdCompressionCodec` it passes.
    - A round-trip test for compressible and incompressible buffers through 
`createCompressionCodec("zstd", ...)`. It asserts that the factory returns 
`BoundedZstdCompressionCodec` and which path (zstd frame or raw) each case 
takes.
    
    Ran `ArrowCompressionUtilsSuite` and `ArrowCachedBatchSerializerSuite` 
locally on this tree (83 tests, all passing).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Sonnet 5.5)
    
    Closes #59922 from andygrove/SPARK-60707.
    
    Authored-by: Andy Grove <[email protected]>
    Signed-off-by: Peter Toth <[email protected]>
    (cherry picked from commit 20ea29cc86a65dc3319bbbebaf8c69b8055ae516)
    Signed-off-by: Peter Toth <[email protected]>
---
 .../execution/arrow/ArrowCompressionUtils.scala    |  39 ++++-
 .../columnar/ArrowCachedBatchSerializer.scala      |  30 +---
 .../arrow/ArrowCompressionUtilsSuite.scala         | 177 ++++++++++++++++++++-
 .../columnar/ArrowCachedBatchSerializerSuite.scala |  43 +----
 4 files changed, 219 insertions(+), 70 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
index 19b84d1cd5d8..e71680751ab8 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
@@ -17,8 +17,10 @@
 
 package org.apache.spark.sql.execution.arrow
 
+import com.github.luben.zstd.Zstd
 import org.apache.arrow.compression.{Lz4CompressionCodec, ZstdCompressionCodec}
-import org.apache.arrow.vector.compression.{CompressionCodec, 
NoCompressionCodec}
+import org.apache.arrow.memory.{ArrowBuf, BufferAllocator}
+import org.apache.arrow.vector.compression.{CompressionCodec, CompressionUtil, 
NoCompressionCodec}
 
 import org.apache.spark.SparkException
 
@@ -34,13 +36,16 @@ private[sql] object ArrowCompressionUtils {
    * carry a compression level, so that factory overload always builds a codec 
at the default
    * level, silently dropping the configured one. The level only matters on 
the write side; the
    * read side looks up the codec by the type recorded in the IPC message, so 
it is unaffected.
+   *
+   * For zstd it must also be a [[BoundedZstdCompressionCodec]], which neither 
factory overload
+   * returns, not even `createCodec(codecType, compressionLevel)`, which does 
keep the level.
    */
   def createCompressionCodec(
       codecName: String,
       zstdCompressionLevel: Int): CompressionCodec = {
     codecName match {
       case "none" => NoCompressionCodec.INSTANCE
-      case "zstd" => new ZstdCompressionCodec(zstdCompressionLevel)
+      case "zstd" => new BoundedZstdCompressionCodec(zstdCompressionLevel)
       case "lz4" => new Lz4CompressionCodec()
       case other =>
         throw SparkException.internalError(
@@ -48,3 +53,33 @@ private[sql] object ArrowCompressionUtils {
     }
   }
 }
+
+/**
+ * A [[ZstdCompressionCodec]] that never lets zstd write past its output 
buffer. arrow-java up to
+ * 19.0 passes `compressBound(len) + 8` as the destination capacity while 
pointing zstd 8 bytes
+ * into a buffer of that size, so zstd can overrun it by up to 8 bytes; here 
the capacity is
+ * `compressBound(len)`. Remove once on Arrow 20.0.0 (apache/arrow-java#1119).
+ */
+private[sql] class BoundedZstdCompressionCodec(compressionLevel: Int)
+  extends ZstdCompressionCodec(compressionLevel) {
+
+  override protected def doCompress(
+      allocator: BufferAllocator,
+      uncompressedBuffer: ArrowBuf): ArrowBuf = {
+    val srcSize = uncompressedBuffer.writerIndex()
+    val maxSize = Zstd.compressBound(srcSize)
+    val compressedBuffer = 
allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + maxSize)
+    val bytesWritten = Zstd.compressUnsafe(
+      compressedBuffer.memoryAddress() + 
CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH,
+      maxSize,
+      uncompressedBuffer.memoryAddress(),
+      srcSize,
+      compressionLevel)
+    if (Zstd.isError(bytesWritten)) {
+      compressedBuffer.close()
+      throw SparkException.internalError("Error compressing: " + 
Zstd.getErrorName(bytesWritten))
+    }
+    compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + 
bytesWritten)
+    compressedBuffer
+  }
+}
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
index baba2c1365c8..db91f91e2480 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala
@@ -22,17 +22,16 @@ import java.nio.channels.Channels
 
 import scala.jdk.CollectionConverters._
 
-import org.apache.arrow.compression.{Lz4CompressionCodec, ZstdCompressionCodec}
 import org.apache.arrow.flatbuf.{RecordBatch => FlatBufRecordBatch}
 import org.apache.arrow.memory.BufferAllocator
 import org.apache.arrow.vector.{TypeLayout, VectorLoader, VectorSchemaRoot, 
VectorUnloader}
-import org.apache.arrow.vector.compression.{CompressionCodec, 
NoCompressionCodec}
+import org.apache.arrow.vector.compression.NoCompressionCodec
 import org.apache.arrow.vector.ipc.{ReadChannel, WriteChannel}
 import org.apache.arrow.vector.ipc.message.{ArrowBodyCompression, 
ArrowFieldNode}
 import org.apache.arrow.vector.ipc.message.{ArrowRecordBatch, 
MessageSerializer}
 import org.apache.arrow.vector.types.pojo.Field
 
-import org.apache.spark.{SparkException, TaskContext}
+import org.apache.spark.TaskContext
 import org.apache.spark.rdd.RDD
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.Attribute
@@ -40,7 +39,7 @@ import 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
 import org.apache.spark.sql.catalyst.types.DataTypeUtils
 import org.apache.spark.sql.columnar.{CachedBatch, 
SimpleMetricsCachedBatchSerializer}
 import org.apache.spark.sql.errors.ExecutionErrors
-import org.apache.spark.sql.execution.arrow.ArrowWriter
+import org.apache.spark.sql.execution.arrow.{ArrowCompressionUtils, 
ArrowWriter}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.util.ArrowUtils
@@ -263,25 +262,6 @@ private object ArrowCachedBatchSerializer {
     }
   }
 
-  // scalastyle:off caselocale
-  def createCompressionCodec(
-      codecName: String,
-      compressionLevel: Int): CompressionCodec = {
-    codecName.toLowerCase match {
-      case "none" => NoCompressionCodec.INSTANCE
-      // The codec instance must be constructed directly so that 
compressionLevel is honored:
-      // CompressionCodec.Factory.createCodec(codecType) ignores the level and 
builds a codec at
-      // the default level. The level only matters on the write side; the read 
side looks up the
-      // codec by the type recorded in the IPC message.
-      case "zstd" => new ZstdCompressionCodec(compressionLevel)
-      case "lz4" => new Lz4CompressionCodec()
-      case other =>
-        throw SparkException.internalError(
-          s"Unsupported Arrow compression codec: $other. Supported values: 
none, zstd, lz4")
-    }
-  }
-  // scalastyle:on caselocale
-
   def serializeBatch(batch: ArrowRecordBatch): Array[Byte] = {
     val out = new ByteArrayOutputStream()
     val writeChannel = new WriteChannel(Channels.newChannel(out))
@@ -1005,7 +985,7 @@ private class InternalRowToArrowCachedBatchIterator(
     compressionCodecName: String,
     compressionLevel: Int) extends Iterator[ArrowCachedBatch] {
 
-  private val compressionCodec = 
ArrowCachedBatchSerializer.createCompressionCodec(
+  private val compressionCodec = ArrowCompressionUtils.createCompressionCodec(
     compressionCodecName,
     compressionLevel)
 
@@ -1112,7 +1092,7 @@ private class ColumnarBatchToArrowCachedBatchIterator(
     compressionCodecName: String,
     compressionLevel: Int) extends Iterator[ArrowCachedBatch] {
 
-  private val compressionCodec = 
ArrowCachedBatchSerializer.createCompressionCodec(
+  private val compressionCodec = ArrowCompressionUtils.createCompressionCodec(
     compressionCodecName,
     compressionLevel)
 
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
index 50a980dcf37a..cde828e1c743 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
@@ -18,16 +18,23 @@
 package org.apache.spark.sql.execution.arrow
 
 import java.io.ByteArrayOutputStream
+import java.nio.{ByteBuffer, ByteOrder}
 import java.nio.channels.Channels
 
+import scala.util.Random
+
+import org.apache.arrow.compression.ZstdCompressionCodec
+import org.apache.arrow.memory.{ArrowBuf, BufferAllocator, RootAllocator}
 import org.apache.arrow.vector.{VarCharVector, VectorSchemaRoot, 
VectorUnloader}
 import org.apache.arrow.vector.compression.NoCompressionCodec
 import org.apache.arrow.vector.ipc.WriteChannel
 import org.apache.arrow.vector.ipc.message.MessageSerializer
 
 import org.apache.spark.{SparkException, SparkFunSuite}
-import org.apache.spark.sql.types.{StringType, StructField, StructType}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.types.{LongType, StringType, StructField, 
StructType}
 import org.apache.spark.sql.util.ArrowUtils
+import org.apache.spark.util.Utils
 
 class ArrowCompressionUtilsSuite extends SparkFunSuite {
 
@@ -85,4 +92,172 @@ class ArrowCompressionUtilsSuite extends SparkFunSuite {
     }
     assert(e.getMessage.contains("Unsupported Arrow compression codec: 
snappy"))
   }
+
+  // Expose the protected `doCompress` so that a test can look at the raw 
output buffer.
+  private class RawBoundedZstdCompressionCodec extends 
BoundedZstdCompressionCodec(3) {
+    def compressRaw(allocator: BufferAllocator, in: ArrowBuf): ArrowBuf = 
doCompress(allocator, in)
+  }
+
+  // Arrow's own codec, which up to arrow-java 19.0 gives zstd a destination 
capacity that is 8
+  // bytes too large.
+  private class RawStockZstdCompressionCodec extends ZstdCompressionCodec(3) {
+    def compressRaw(allocator: BufferAllocator, in: ArrowBuf): ArrowBuf = 
doCompress(allocator, in)
+  }
+
+  // Random bytes with about 2% of them zeroed. Pure random bytes only make 
zstd fill its whole
+  // destination when one byte value happens to be frequent enough (more than 
about n / 128 + 4
+  // occurrences) for zstd to try Huffman-coding the literals and give up. 
Zeroing some bytes
+  // makes that the usual case instead of a property of the seed.
+  private def noisyBytes(random: Random, n: Int): Array[Byte] = {
+    val bytes = random.nextBytes(n)
+    (0 until n).foreach { i => if (random.nextInt(50) == 0) bytes(i) = 0 }
+    bytes
+  }
+
+  // Runs `compressRaw` into a buffer that is followed by sentinel bytes, and 
returns whether the
+  // sentinel was overwritten, i.e. whether zstd wrote past the end of the 
buffer.
+  private def writesPastEnd(
+      compressRaw: (BufferAllocator, ArrowBuf) => ArrowBuf,
+      data: Array[Byte]): Boolean = {
+    val guardSize = 64
+    val guardByte = 0x5A.toByte
+    var requestedSize = 0L
+    val guardedAllocator = new RootAllocator(Long.MaxValue) {
+      override def buffer(size: Long): ArrowBuf = {
+        requestedSize = size
+        val buf = super.buffer(size + guardSize)
+        buf.setBytes(size, Array.fill(guardSize)(guardByte))
+        buf
+      }
+    }
+    val inputAllocator =
+      ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName, 
0, Long.MaxValue)
+    Utils.tryWithSafeFinally {
+      val input = inputAllocator.buffer(data.length)
+      val out = try {
+        input.setBytes(0, data)
+        input.writerIndex(data.length)
+        compressRaw(guardedAllocator, input)
+      } finally {
+        input.close()
+      }
+      try {
+        val guard = new Array[Byte](guardSize)
+        out.getBytes(requestedSize, guard)
+        !guard.forall(_ == guardByte)
+      } finally {
+        out.close()
+      }
+    } {
+      
Utils.tryWithSafeFinally(guardedAllocator.close())(inputAllocator.close())
+    }
+  }
+
+  test("SPARK-60707: zstd compression does not write past the end of its 
output buffer") {
+    val bounded = new RawBoundedZstdCompressionCodec
+    val stock = new RawStockZstdCompressionCodec
+    val random = new Random(0)
+    // Of these inputs, only those of roughly 280 to 1000 bytes make zstd fill 
its whole
+    // destination; smaller and larger ones stop short of the end. The sizes 
in that range are
+    // checked with many inputs each, and the other sizes only as smoke checks.
+    Seq(280, 500, 700).foreach { n =>
+      val inputs = Seq.fill(100)(noisyBytes(random, n))
+      inputs.foreach { data =>
+        assert(!writesPastEnd(bounded.compressRaw, data),
+          s"zstd wrote past the end of its output buffer (input size $n)")
+      }
+      // Control: these inputs do trigger the overrun in Arrow's own codec, so 
the checks above
+      // can fail. This depends on how zstd compresses literals. If it starts 
failing after an
+      // Arrow upgrade, Arrow has fixed the codec and 
BoundedZstdCompressionCodec can be removed;
+      // if it starts failing after a zstd-jni upgrade, pick inputs that still 
overrun instead.
+      assert(inputs.exists(writesPastEnd(stock.compressRaw, _)),
+        s"no input of size $n made Arrow's ZstdCompressionCodec write past its 
output buffer")
+    }
+    Seq(1, 7, 100, 4096, 65536, 262144).foreach { n =>
+      assert(!writesPastEnd(bounded.compressRaw, noisyBytes(random, n)),
+        s"zstd wrote past the end of its output buffer (input size $n)")
+    }
+  }
+
+  test("SPARK-60707: zstd codec round-trips compressible and incompressible 
buffers") {
+    val allocator =
+      ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName, 
0, Long.MaxValue)
+    val codec = ArrowCompressionUtils.createCompressionCodec("zstd", 3)
+    val random = new Random(1)
+    Utils.tryWithSafeFinally {
+      // The write paths get the bounded codec, not Arrow's own.
+      assert(codec.isInstanceOf[BoundedZstdCompressionCodec])
+      Seq(1, 100, 4096, 100000).foreach { n =>
+        val compressible = Array.tabulate[Byte](n)(i => (i % 7).toByte)
+        val incompressible = random.nextBytes(n)
+        Seq(compressible -> (n >= 100), incompressible -> false).foreach {
+          case (expected, expectZstdFrame) =>
+            val input = allocator.buffer(n)
+            input.setBytes(0, expected)
+            input.writerIndex(n)
+            val compressed = codec.compress(allocator, input)
+            // The buffer starts with the uncompressed length, or -1 if Arrow 
kept the input as is
+            // because zstd did not make it smaller. Check which path each 
case takes.
+            val prefix = compressed.nioBuffer(0, 
8).order(ByteOrder.LITTLE_ENDIAN).getLong(0)
+            assert(prefix === (if (expectZstdFrame) n.toLong else -1L))
+            val decompressed = codec.decompress(allocator, compressed)
+            try {
+              val actual = new Array[Byte](n)
+              decompressed.getBytes(0, actual)
+              assert(actual.sameElements(expected))
+            } finally {
+              decompressed.close()
+            }
+        }
+      }
+    } {
+      allocator.close()
+    }
+  }
+
+  test("SPARK-60707: batches written through VectorUnloader keep their length 
prefixes") {
+    // Writes batches the way ArrowConverters does, with one reused root, and 
checks the length
+    // prefix of every data buffer. An overrun of one buffer can overwrite the 
prefix of the next
+    // buffer in the same batch, which makes the batch fail to decompress when 
it is read. How
+    // often that happens depends on the memory allocator's layout, so this 
cannot replace the
+    // guard test above, but it covers the path that readers see.
+    val rows = 55
+    val columns = 3
+    val sparkSchema = StructType(
+      (0 until columns).map(i => StructField(s"c$i", LongType, nullable = 
false)))
+    val arrowSchema = ArrowUtils.toArrowSchema(sparkSchema, "UTC", true, false)
+    val allocator =
+      ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName, 
0, Long.MaxValue)
+    val root = VectorSchemaRoot.create(arrowSchema, allocator)
+    val writer = ArrowWriter.create(root)
+    val unloader = new VectorUnloader(
+      root, true, ArrowCompressionUtils.createCompressionCodec("zstd", 3), 
true)
+    val random = new Random(2)
+    Utils.tryWithSafeFinally {
+      (0 until 300).foreach { _ =>
+        val values = ByteBuffer.wrap(noisyBytes(random, rows * columns * 8))
+          .order(ByteOrder.LITTLE_ENDIAN).asLongBuffer()
+        (0 until rows).foreach { _ =>
+          writer.write(InternalRow.fromSeq(Seq.fill(columns)(values.get())))
+        }
+        writer.finish()
+        val batch = unloader.getRecordBatch()
+        try {
+          // The buffers of a column are its validity buffer and then its data 
buffer.
+          (0 until columns).foreach { c =>
+            val dataBuffer = batch.getBuffers.get(2 * c + 1)
+            val prefix = dataBuffer.nioBuffer(0, 
8).order(ByteOrder.LITTLE_ENDIAN).getLong(0)
+            assert(prefix === -1L || prefix === rows * 8L,
+              s"the length prefix of column $c was overwritten")
+          }
+        } finally {
+          batch.close()
+        }
+        writer.reset()
+      }
+    } {
+      root.close()
+      allocator.close()
+    }
+  }
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializerSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializerSuite.scala
index 347c6b42beaa..7e560cc5a839 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializerSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializerSuite.scala
@@ -24,7 +24,7 @@ import org.apache.arrow.vector.{
   BigIntVector, BitVector, DateDayVector, DecimalVector,
   Float4Vector, Float8Vector, IntVector, LargeVarCharVector, SmallIntVector,
   TimeNanoVector, TimeStampMicroTZVector, TimeStampMicroVector, TinyIntVector,
-  VarBinaryVector, VarCharVector, VectorSchemaRoot, VectorUnloader}
+  VarBinaryVector, VarCharVector, VectorSchemaRoot}
 import org.apache.arrow.vector.complex.{MapVector, StructVector}
 import org.apache.arrow.vector.types.pojo.{ArrowType, Field, FieldType}
 
@@ -1987,47 +1987,6 @@ class ArrowCachedBatchSerializerSuite extends QueryTest 
with SharedSparkSession
     }
   }
 
-  test("zstd compression level is honored by createCompressionCodec") {
-    // Regression test: createCompressionCodec used to rebuild the codec 
through the
-    // single-argument factory overload, which silently dropped the level, so 
every configured
-    // level compressed at the zstd default. Compress the same batch at an 
ultra-fast negative
-    // level and at a high level and assert the high level yields a strictly 
smaller payload.
-    def compressedSize(level: Int): Int = {
-      val schema = Seq(AttributeReference("str_col", StringType)())
-      val sparkSchema = StructType(schema.map(a => StructField(a.name, 
a.dataType)))
-      val arrowSchema = ArrowUtils.toArrowSchema(sparkSchema, "UTC", false, 
false)
-      val root = VectorSchemaRoot.create(arrowSchema, ArrowUtils.rootAllocator)
-      try {
-        root.allocateNew()
-        val strVector = root.getVector("str_col").asInstanceOf[VarCharVector]
-        // Compressible but non-trivial corpus: shared structure with per-row 
variation, so
-        // different zstd levels produce measurably different output sizes.
-        (0 until 2000).foreach { i =>
-          val value =
-            s"[email protected],record-${i % 97},payload-${i * 
2654435761L}".getBytes("UTF-8")
-          strVector.setSafe(i, value, 0, value.length)
-        }
-        root.setRowCount(2000)
-        val codec = ArrowCachedBatchSerializer.createCompressionCodec("zstd", 
level)
-        val recordBatch = new VectorUnloader(root, true, codec, 
true).getRecordBatch()
-        try {
-          ArrowCachedBatchSerializer.serializeBatch(recordBatch).length
-        } finally {
-          recordBatch.close()
-        }
-      } finally {
-        root.close()
-      }
-    }
-
-    val fastSize = compressedSize(-5)
-    val highSize = compressedSize(19)
-    assert(highSize < fastSize,
-      s"zstd level 19 should compress smaller than level -5, " +
-        s"got level 19 -> $highSize bytes vs level -5 -> $fastSize bytes; " +
-        "equal sizes mean the configured level is being ignored")
-  }
-
   test("collectStatistics returns null bounds when all Float/Double values are 
NaN") {
     // When every non-null value in a Float or Double column is NaN, 
calculateMinMaxFloat/Double
     // finds no valid (non-NaN) values. hasValue stays false -> returns (null, 
null) -> null bounds.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to