This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 7a02b6b616c2 [SPARK-60816][SQL][PYTHON] Write empty Arrow buffers 
uncompressed so that PyArrow can read LZ4 batches
7a02b6b616c2 is described below

commit 7a02b6b616c23f38b5422ed3ff1459fe8cb7b0b6
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 00:34:32 2026 -0800

    [SPARK-60816][SQL][PYTHON] Write empty Arrow buffers uncompressed so that 
PyArrow can read LZ4 batches
    
    ### What changes were proposed in this pull request?
    
    This PR adds `UncompressedEmptyBuffers`, a stackable trait for Arrow's 
`CompressionCodec`, and mixes it into both codecs that 
`ArrowCompressionUtils.createCompressionCodec` creates for 
`spark.sql.execution.arrow.compression.codec`: `BoundedZstdCompressionCodec` 
and `Lz4CompressionCodec`.
    
    For an empty input buffer (`writerIndex == 0`), the trait writes the 8-byte 
length prefix `-1` (`CompressionUtil.NO_COMPRESSION_LENGTH`), which means that 
the body is stored uncompressed. All other buffers go to the codec unchanged.
    
    Like Arrow's `AbstractCompressionCodec.compress`, the trait closes the 
input buffer only on success, because `TrackingCompressionCodec` (SPARK-60776) 
releases the input when `compress` fails.
    
    This is the same output as the upstream fix, apache/arrow-java#1248, which 
is only available in the unreleased Arrow 20.0.0. The trait has a note to 
remove it once Spark is on that version.
    
    ### Why are the changes needed?
    
    With `spark.sql.execution.arrow.compression.codec=lz4`, PyArrow fails to 
read a batch if a string or binary column in the batch has an empty data buffer.
    
    ```python
    spark.conf.set("spark.sql.execution.arrow.compression.codec", "lz4")
    spark.range(3).selectExpr("CAST(NULL AS STRING) AS s").toArrow()
    ```
    
    ```
    OSError: Lz4 compressed input contains less than one frame
    ```
    
    Arrow Java up to 
[v19.0.0](https://github.com/apache/arrow-java/releases/tag/v19.0.0) 
(2026-03-16) writes an empty buffer as the length prefix `0` with no bytes 
after it (apache/arrow-java#1196). The Arrow Java reader treats that as an 
empty buffer, but Arrow C++ passes the zero-byte body to the codec, and the LZ4 
frame decoder rejects it because it contains no frame. The same code path is in 
Arrow C++ 18.0.0, the minimum PyArrow version of PySpark.
    
    A data buffer is empty when all values of a batch are null or `""`, 
including nested fields, and in 0-row batches. So the following fail with `lz4`:
    
    - `toArrow` and `toPandas`
    - Arrow-based Python UDFs and functions such as `pandas_udf`, `arrow_udf`, 
`useArrow=True`, `mapInArrow`, `mapInPandas`, and `applyInPandas`
    - Cogrouped `applyInPandas` with a key on only one side, even without 
nulls, because the other side is sent as a 0-row batch
    - Spark Connect `collect`, `toArrow`, and `toPandas`, including an empty 
result that has a string column, even without nulls
    
    Other buffers also get the `0` prefix when they are empty, but Arrow C++ 
does not decompress them in that case, so they do not fail.
    
    The `-1` prefix is part of the Arrow IPC format, and Arrow Java already 
writes it for a buffer that does not compress, so readers already handle it. 
This PR applies it to zstd as well because a `0` prefix with no bytes is not a 
valid zstd frame either, even though Arrow C++ accepts it, and Arrow 20.0.0 
writes `-1` for all codecs.
    
    Arrow 20.0.0 is not released yet, and a Spark-side fix can also be 
backported to the maintenance branches, which are on Arrow 18.3.0 and 19.0.0.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The reads above no longer fail when 
`spark.sql.execution.arrow.compression.codec` is `lz4`. For example, the above 
`toArrow` call now returns a table whose `s` column has three nulls. This 
failure exists since 
[v4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11), 
which added the configuration. The default value is `none`, which is not 
affected.
    
    With `zstd`, and in `ArrowCachedBatchSerializer`, which is read back by 
Arrow Java, the only change is the prefix of the empty buffers. The results are 
the same.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases.
    
    - A new test in `ArrowCompressionUtilsSuite` checks that `lz4` and `zstd` 
write an empty buffer as an 8-byte `-1` prefix and that the input buffer is 
released. Without this fix, it fails with `0 did not equal -1`.
    - Another new test in the same suite makes the allocation of the prefix 
fail and checks that `createVectorUnloader` releases the empty buffer exactly 
once. It also passes without this fix. It fails with `RefCnt has gone negative` 
if the trait closes the input buffer on failure as well.
    - `test_toArrow_with_compression_codec_empty_data_buffers` in 
`test_arrow.py` reads all-null strings with PyArrow for each codec. It also 
runs over Spark Connect through `ArrowParityTests`. Without this fix, its `lz4` 
case fails with the above error in both modes.
    
    I also ran the following locally with PyArrow 25.0.0.
    
    - `ArrowCompressionUtilsSuite`, `ArrowCachedBatchSerializerSuite`, 
`ArrowConvertersSuite`, and `ExecuteExternalUDFExecSuite`
    - The existing compression tests in `test_arrow.py`, 
`test_pandas_udf_scalar.py`, `test_pandas_grouped_map.py`, and 
`test_pandas_udf_grouped_agg.py`
    - An ad-hoc script that runs the failing paths listed above and their 
controls with all three codecs (78 cases), and another that reads cached 
batches from `ArrowCachedBatchSerializer` through `collect`, `toArrow`, and 
`pandas_udf` (48 cases)
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60034 from dongjoon-hyun/SPARK-60816.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 57f7ad61f7664d772f83f6843a2a20958a5f49b4)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 python/pyspark/sql/tests/arrow/test_arrow.py       |  9 ++++
 .../execution/arrow/ArrowCompressionUtils.scala    | 30 ++++++++++++-
 .../arrow/ArrowCompressionUtilsSuite.scala         | 49 +++++++++++++++++++++-
 3 files changed, 84 insertions(+), 4 deletions(-)

diff --git a/python/pyspark/sql/tests/arrow/test_arrow.py 
b/python/pyspark/sql/tests/arrow/test_arrow.py
index df4239651d34..006bd9169d0b 100644
--- a/python/pyspark/sql/tests/arrow/test_arrow.py
+++ b/python/pyspark/sql/tests/arrow/test_arrow.py
@@ -1894,6 +1894,15 @@ class ArrowTestsMixin:
                     self.assertEqual(t.num_rows, 10000)
                     self.assertEqual(t.column_names, ["id", "str_col", 
"mod_col"])
 
+    def test_toArrow_with_compression_codec_empty_data_buffers(self):
+        # SPARK-60816: All strings are null, so the data buffer of the column 
is empty.
+        df = self.spark.range(3).selectExpr("CAST(NULL AS STRING) AS s")
+
+        for codec in ["none", "zstd", "lz4"]:
+            with self.subTest(compressionCodec=codec):
+                with 
self.sql_conf({"spark.sql.execution.arrow.compression.codec": codec}):
+                    self.assertEqual(df.toArrow().to_pydict(), {"s": [None] * 
3})
+
     def test_toPandas_double_nested_array_empty_outer(self):
         schema = StructType([StructField("data", 
ArrayType(ArrayType(StringType())))])
         df = self.spark.createDataFrame([Row(data=[])], schema=schema)
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
index ce995bb0bc0d..8c87dfc117b6 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
@@ -51,8 +51,9 @@ private[sql] object ArrowCompressionUtils {
       zstdCompressionLevel: Int): CompressionCodec = {
     codecName match {
       case "none" => NoCompressionCodec.INSTANCE
-      case "zstd" => new BoundedZstdCompressionCodec(zstdCompressionLevel)
-      case "lz4" => new Lz4CompressionCodec()
+      case "zstd" =>
+        new BoundedZstdCompressionCodec(zstdCompressionLevel) with 
UncompressedEmptyBuffers
+      case "lz4" => new Lz4CompressionCodec with UncompressedEmptyBuffers
       case other =>
         throw SparkException.internalError(
           s"Unsupported Arrow compression codec: $other. Supported values: 
none, zstd, lz4")
@@ -146,6 +147,31 @@ private object BoundedZstdCompressionCodec {
   val MIN_SHRINK_SAVING_BYTES = 4 * 1024
 }
 
+/**
+ * A [[CompressionCodec]] mixin that gives an empty buffer the `-1` (stored 
uncompressed) length
+ * prefix. arrow-java up to 19.0 writes a `0` prefix and no compressed data, 
so Arrow C++ (and
+ * PyArrow) decompresses a zero-byte body, which LZ4 rejects with "Lz4 
compressed input contains
+ * less than one frame". Remove once on Arrow 20.0.0 (apache/arrow-java#1248).
+ */
+private[sql] trait UncompressedEmptyBuffers extends CompressionCodec {
+
+  abstract override def compress(
+      allocator: BufferAllocator,
+      uncompressedBuffer: ArrowBuf): ArrowBuf = {
+    if (uncompressedBuffer.writerIndex() == 0L) {
+      val compressedBuffer = 
allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)
+      compressedBuffer.setLong(0, CompressionUtil.NO_COMPRESSION_LENGTH)
+      compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)
+      // Like `super.compress`, release the input only on success. 
TrackingCompressionCodec relies
+      // on this to release it on failure.
+      uncompressedBuffer.close()
+      compressedBuffer
+    } else {
+      super.compress(allocator, uncompressedBuffer)
+    }
+  }
+}
+
 /**
  * A [[VectorUnloader]] that releases the buffers it holds for a batch it 
fails to unload. Remove
  * once on Arrow 20.0.0, which releases them itself, so they would be released 
twice
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
index 2bbdf770fea2..7737ccc35053 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
@@ -25,9 +25,9 @@ import scala.jdk.CollectionConverters._
 import scala.util.Random
 
 import org.apache.arrow.compression.ZstdCompressionCodec
-import org.apache.arrow.memory.{ArrowBuf, BufferAllocator, RootAllocator}
+import org.apache.arrow.memory.{ArrowBuf, BufferAllocator, 
OutOfMemoryException, RootAllocator}
 import org.apache.arrow.vector.{VarCharVector, VectorSchemaRoot, 
VectorUnloader}
-import org.apache.arrow.vector.compression.NoCompressionCodec
+import org.apache.arrow.vector.compression.{CompressionUtil, 
NoCompressionCodec}
 import org.apache.arrow.vector.ipc.WriteChannel
 import org.apache.arrow.vector.ipc.message.MessageSerializer
 
@@ -370,4 +370,49 @@ class ArrowCompressionUtilsSuite extends SparkFunSuite {
       assert(output.capacity() === allocatedSize)
     }
   }
+
+  test("SPARK-60816: empty buffers are written with the uncompressed length 
prefix") {
+    // For an empty buffer, arrow-java up to 19.0 writes a `0` length prefix 
and no compressed data,
+    // which Arrow C++ (PyArrow) fails to decompress for LZ4. Readers already 
see the `-1` prefix
+    // for buffers that do not compress.
+    val allocator =
+      ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName, 
0, Long.MaxValue)
+    Utils.tryWithSafeFinally {
+      Seq("lz4", "zstd").foreach { codecName =>
+        val codec = ArrowCompressionUtils.createCompressionCodec(codecName, 3)
+        // A newly allocated buffer has writerIndex 0, so `compress` sees an 
empty buffer.
+        val compressed = codec.compress(allocator, allocator.buffer(16))
+        try {
+          assert(compressed.writerIndex() === 
CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)
+          assert(compressed.getLong(0) === 
CompressionUtil.NO_COMPRESSION_LENGTH, codecName)
+        } finally {
+          compressed.close()
+        }
+      }
+    } {
+      // Fails if `compress` leaked its input buffer.
+      allocator.close()
+    }
+  }
+
+  test("SPARK-60816: unloader releases an empty buffer once if its prefix 
allocation fails") {
+    val arrowSchema = ArrowUtils.toArrowSchema(
+      StructType(Seq(StructField("c", LongType))), "UTC", true, false)
+    val allocator =
+      ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName, 
0, Long.MaxValue)
+    val root = VectorSchemaRoot.create(arrowSchema, allocator)
+    val codec = ArrowCompressionUtils.createCompressionCodec("lz4", 3)
+    val unloader = ArrowCompressionUtils.createVectorUnloader(root, codec)
+    Utils.tryWithSafeFinally {
+      root.allocateNew()
+      // With no rows the first buffer is empty, so its length prefix is the 
first allocation.
+      allocator.setLimit(0)
+      intercept[OutOfMemoryException](unloader.getRecordBatch())
+    } {
+      // Closing the vector fails if the buffer was released twice, and 
closing the allocator fails
+      // if it was not released.
+      root.close()
+      allocator.close()
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to