This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 7a02b6b616c2 [SPARK-60816][SQL][PYTHON] Write empty Arrow buffers
uncompressed so that PyArrow can read LZ4 batches
7a02b6b616c2 is described below
commit 7a02b6b616c23f38b5422ed3ff1459fe8cb7b0b6
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 00:34:32 2026 -0800
[SPARK-60816][SQL][PYTHON] Write empty Arrow buffers uncompressed so that
PyArrow can read LZ4 batches
### What changes were proposed in this pull request?
This PR adds `UncompressedEmptyBuffers`, a stackable trait for Arrow's
`CompressionCodec`, and mixes it into both codecs that
`ArrowCompressionUtils.createCompressionCodec` creates for
`spark.sql.execution.arrow.compression.codec`: `BoundedZstdCompressionCodec`
and `Lz4CompressionCodec`.
For an empty input buffer (`writerIndex == 0`), the trait writes the 8-byte
length prefix `-1` (`CompressionUtil.NO_COMPRESSION_LENGTH`), which means that
the body is stored uncompressed. All other buffers go to the codec unchanged.
Like Arrow's `AbstractCompressionCodec.compress`, the trait closes the
input buffer only on success, because `TrackingCompressionCodec` (SPARK-60776)
releases the input when `compress` fails.
This is the same output as the upstream fix, apache/arrow-java#1248, which
is only available in the unreleased Arrow 20.0.0. The trait has a note to
remove it once Spark is on that version.
### Why are the changes needed?
With `spark.sql.execution.arrow.compression.codec=lz4`, PyArrow fails to
read a batch if a string or binary column in the batch has an empty data buffer.
```python
spark.conf.set("spark.sql.execution.arrow.compression.codec", "lz4")
spark.range(3).selectExpr("CAST(NULL AS STRING) AS s").toArrow()
```
```
OSError: Lz4 compressed input contains less than one frame
```
Arrow Java up to
[v19.0.0](https://github.com/apache/arrow-java/releases/tag/v19.0.0)
(2026-03-16) writes an empty buffer as the length prefix `0` with no bytes
after it (apache/arrow-java#1196). The Arrow Java reader treats that as an
empty buffer, but Arrow C++ passes the zero-byte body to the codec, and the LZ4
frame decoder rejects it because it contains no frame. The same code path is in
Arrow C++ 18.0.0, the minimum PyArrow version of PySpark.
A data buffer is empty when all values of a batch are null or `""`,
including nested fields, and in 0-row batches. So the following fail with `lz4`:
- `toArrow` and `toPandas`
- Arrow-based Python UDFs and functions such as `pandas_udf`, `arrow_udf`,
`useArrow=True`, `mapInArrow`, `mapInPandas`, and `applyInPandas`
- Cogrouped `applyInPandas` with a key on only one side, even without
nulls, because the other side is sent as a 0-row batch
- Spark Connect `collect`, `toArrow`, and `toPandas`, including an empty
result that has a string column, even without nulls
Other buffers also get the `0` prefix when they are empty, but Arrow C++
does not decompress them in that case, so they do not fail.
The `-1` prefix is part of the Arrow IPC format, and Arrow Java already
writes it for a buffer that does not compress, so readers already handle it.
This PR applies it to zstd as well because a `0` prefix with no bytes is not a
valid zstd frame either, even though Arrow C++ accepts it, and Arrow 20.0.0
writes `-1` for all codecs.
Arrow 20.0.0 is not released yet, and a Spark-side fix can also be
backported to the maintenance branches, which are on Arrow 18.3.0 and 19.0.0.
### Does this PR introduce _any_ user-facing change?
Yes. The reads above no longer fail when
`spark.sql.execution.arrow.compression.codec` is `lz4`. For example, the above
`toArrow` call now returns a table whose `s` column has three nulls. This
failure exists since
[v4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11),
which added the configuration. The default value is `none`, which is not
affected.
With `zstd`, and in `ArrowCachedBatchSerializer`, which is read back by
Arrow Java, the only change is the prefix of the empty buffers. The results are
the same.
### How was this patch tested?
Pass the CIs with the newly added test cases.
- A new test in `ArrowCompressionUtilsSuite` checks that `lz4` and `zstd`
write an empty buffer as an 8-byte `-1` prefix and that the input buffer is
released. Without this fix, it fails with `0 did not equal -1`.
- Another new test in the same suite makes the allocation of the prefix
fail and checks that `createVectorUnloader` releases the empty buffer exactly
once. It also passes without this fix. It fails with `RefCnt has gone negative`
if the trait closes the input buffer on failure as well.
- `test_toArrow_with_compression_codec_empty_data_buffers` in
`test_arrow.py` reads all-null strings with PyArrow for each codec. It also
runs over Spark Connect through `ArrowParityTests`. Without this fix, its `lz4`
case fails with the above error in both modes.
I also ran the following locally with PyArrow 25.0.0.
- `ArrowCompressionUtilsSuite`, `ArrowCachedBatchSerializerSuite`,
`ArrowConvertersSuite`, and `ExecuteExternalUDFExecSuite`
- The existing compression tests in `test_arrow.py`,
`test_pandas_udf_scalar.py`, `test_pandas_grouped_map.py`, and
`test_pandas_udf_grouped_agg.py`
- An ad-hoc script that runs the failing paths listed above and their
controls with all three codecs (78 cases), and another that reads cached
batches from `ArrowCachedBatchSerializer` through `collect`, `toArrow`, and
`pandas_udf` (48 cases)
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60034 from dongjoon-hyun/SPARK-60816.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 57f7ad61f7664d772f83f6843a2a20958a5f49b4)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/sql/tests/arrow/test_arrow.py | 9 ++++
.../execution/arrow/ArrowCompressionUtils.scala | 30 ++++++++++++-
.../arrow/ArrowCompressionUtilsSuite.scala | 49 +++++++++++++++++++++-
3 files changed, 84 insertions(+), 4 deletions(-)
diff --git a/python/pyspark/sql/tests/arrow/test_arrow.py
b/python/pyspark/sql/tests/arrow/test_arrow.py
index df4239651d34..006bd9169d0b 100644
--- a/python/pyspark/sql/tests/arrow/test_arrow.py
+++ b/python/pyspark/sql/tests/arrow/test_arrow.py
@@ -1894,6 +1894,15 @@ class ArrowTestsMixin:
self.assertEqual(t.num_rows, 10000)
self.assertEqual(t.column_names, ["id", "str_col",
"mod_col"])
+ def test_toArrow_with_compression_codec_empty_data_buffers(self):
+ # SPARK-60816: All strings are null, so the data buffer of the column
is empty.
+ df = self.spark.range(3).selectExpr("CAST(NULL AS STRING) AS s")
+
+ for codec in ["none", "zstd", "lz4"]:
+ with self.subTest(compressionCodec=codec):
+ with
self.sql_conf({"spark.sql.execution.arrow.compression.codec": codec}):
+ self.assertEqual(df.toArrow().to_pydict(), {"s": [None] *
3})
+
def test_toPandas_double_nested_array_empty_outer(self):
schema = StructType([StructField("data",
ArrayType(ArrayType(StringType())))])
df = self.spark.createDataFrame([Row(data=[])], schema=schema)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
index ce995bb0bc0d..8c87dfc117b6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtils.scala
@@ -51,8 +51,9 @@ private[sql] object ArrowCompressionUtils {
zstdCompressionLevel: Int): CompressionCodec = {
codecName match {
case "none" => NoCompressionCodec.INSTANCE
- case "zstd" => new BoundedZstdCompressionCodec(zstdCompressionLevel)
- case "lz4" => new Lz4CompressionCodec()
+ case "zstd" =>
+ new BoundedZstdCompressionCodec(zstdCompressionLevel) with
UncompressedEmptyBuffers
+ case "lz4" => new Lz4CompressionCodec with UncompressedEmptyBuffers
case other =>
throw SparkException.internalError(
s"Unsupported Arrow compression codec: $other. Supported values:
none, zstd, lz4")
@@ -146,6 +147,31 @@ private object BoundedZstdCompressionCodec {
val MIN_SHRINK_SAVING_BYTES = 4 * 1024
}
+/**
+ * A [[CompressionCodec]] mixin that gives an empty buffer the `-1` (stored
uncompressed) length
+ * prefix. arrow-java up to 19.0 writes a `0` prefix and no compressed data,
so Arrow C++ (and
+ * PyArrow) decompresses a zero-byte body, which LZ4 rejects with "Lz4
compressed input contains
+ * less than one frame". Remove once on Arrow 20.0.0 (apache/arrow-java#1248).
+ */
+private[sql] trait UncompressedEmptyBuffers extends CompressionCodec {
+
+ abstract override def compress(
+ allocator: BufferAllocator,
+ uncompressedBuffer: ArrowBuf): ArrowBuf = {
+ if (uncompressedBuffer.writerIndex() == 0L) {
+ val compressedBuffer =
allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)
+ compressedBuffer.setLong(0, CompressionUtil.NO_COMPRESSION_LENGTH)
+ compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)
+ // Like `super.compress`, release the input only on success.
TrackingCompressionCodec relies
+ // on this to release it on failure.
+ uncompressedBuffer.close()
+ compressedBuffer
+ } else {
+ super.compress(allocator, uncompressedBuffer)
+ }
+ }
+}
+
/**
* A [[VectorUnloader]] that releases the buffers it holds for a batch it
fails to unload. Remove
* once on Arrow 20.0.0, which releases them itself, so they would be released
twice
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
index 2bbdf770fea2..7737ccc35053 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowCompressionUtilsSuite.scala
@@ -25,9 +25,9 @@ import scala.jdk.CollectionConverters._
import scala.util.Random
import org.apache.arrow.compression.ZstdCompressionCodec
-import org.apache.arrow.memory.{ArrowBuf, BufferAllocator, RootAllocator}
+import org.apache.arrow.memory.{ArrowBuf, BufferAllocator,
OutOfMemoryException, RootAllocator}
import org.apache.arrow.vector.{VarCharVector, VectorSchemaRoot,
VectorUnloader}
-import org.apache.arrow.vector.compression.NoCompressionCodec
+import org.apache.arrow.vector.compression.{CompressionUtil,
NoCompressionCodec}
import org.apache.arrow.vector.ipc.WriteChannel
import org.apache.arrow.vector.ipc.message.MessageSerializer
@@ -370,4 +370,49 @@ class ArrowCompressionUtilsSuite extends SparkFunSuite {
assert(output.capacity() === allocatedSize)
}
}
+
+ test("SPARK-60816: empty buffers are written with the uncompressed length
prefix") {
+ // For an empty buffer, arrow-java up to 19.0 writes a `0` length prefix
and no compressed data,
+ // which Arrow C++ (PyArrow) fails to decompress for LZ4. Readers already
see the `-1` prefix
+ // for buffers that do not compress.
+ val allocator =
+ ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName,
0, Long.MaxValue)
+ Utils.tryWithSafeFinally {
+ Seq("lz4", "zstd").foreach { codecName =>
+ val codec = ArrowCompressionUtils.createCompressionCodec(codecName, 3)
+ // A newly allocated buffer has writerIndex 0, so `compress` sees an
empty buffer.
+ val compressed = codec.compress(allocator, allocator.buffer(16))
+ try {
+ assert(compressed.writerIndex() ===
CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)
+ assert(compressed.getLong(0) ===
CompressionUtil.NO_COMPRESSION_LENGTH, codecName)
+ } finally {
+ compressed.close()
+ }
+ }
+ } {
+ // Fails if `compress` leaked its input buffer.
+ allocator.close()
+ }
+ }
+
+ test("SPARK-60816: unloader releases an empty buffer once if its prefix
allocation fails") {
+ val arrowSchema = ArrowUtils.toArrowSchema(
+ StructType(Seq(StructField("c", LongType))), "UTC", true, false)
+ val allocator =
+ ArrowUtils.rootAllocator.newChildAllocator(this.getClass.getSimpleName,
0, Long.MaxValue)
+ val root = VectorSchemaRoot.create(arrowSchema, allocator)
+ val codec = ArrowCompressionUtils.createCompressionCodec("lz4", 3)
+ val unloader = ArrowCompressionUtils.createVectorUnloader(root, codec)
+ Utils.tryWithSafeFinally {
+ root.allocateNew()
+ // With no rows the first buffer is empty, so its length prefix is the
first allocation.
+ allocator.setLimit(0)
+ intercept[OutOfMemoryException](unloader.getRecordBatch())
+ } {
+ // Closing the vector fails if the buffer was released twice, and
closing the allocator fails
+ // if it was not released.
+ root.close()
+ allocator.close()
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]