This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new ff70618e296b [SPARK-57475][CORE] Validate length before allocation in
`Encoders.BitmapArrays.decode`
ff70618e296b is described below
commit ff70618e296b54a85a170508096104ea015dfd1e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Jun 15 15:14:52 2026 -0700
[SPARK-57475][CORE] Validate length before allocation in
`Encoders.BitmapArrays.decode`
### What changes were proposed in this pull request?
`Encoders.BitmapArrays.decode()` now validates the count prefix before
allocating the array via `Objects.checkFromIndexSize(0, numBitmaps,
buf.readableBytes() / 8)`, which requires `0 <= numBitmaps <=
buf.readableBytes() / 8` and throws `IndexOutOfBoundsException` otherwise. The
divisor `8` is the minimum on-wire size of one element, since an empty
`RoaringBitmap` serializes to 8 bytes (a 4-byte cookie followed by a 4-byte
size).
### Why are the changes needed?
To make it robust.
Previously, `decode()` allocated `new RoaringBitmap[numBitmaps]` without
checking. So,
- A negative value throws an opaque `NegativeArraySizeException`.
- An oversized value can trigger `OutOfMemoryError` on a corrupt or hostile
frame.
The `numBitmaps` count comes from a `MergeStatuses` reply that the driver
decodes in
`ExternalBlockStoreClient.finalizeShuffleMerge`. The server that produces
this reply is
pluggable via `spark.shuffle.push.server.mergedShuffleFileManagerImpl`, so
the frame may be
encoded by a third-party implementation rather than Spark's own `Encoders`.
Since the driver
decodes bytes it did not encode, a version mismatch, an implementation bug,
or a truncated
frame can reach this allocation. Validating first fails fast with a clear
error, consistent
with the other array decoders in `Encoders`.
https://github.com/apache/spark/blob/0caf79030573233c6db526a4289499f8756adeaf/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java#L545-L554
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs with newly added test cases.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #56523 from dongjoon-hyun/SPARK-57475.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 2f4d32a324deec8e3841e0bdff4959a620d4d7ed)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit e9286c59819c5244dcef35f41319036de2cdb560)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 7cca9423a5bf101e1bfd0aec08661d1e39f83cd1)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 3b781eb793ef3fa53575c94a393c3713acb76818)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../java/org/apache/spark/network/protocol/Encoders.java | 3 +++
.../org/apache/spark/network/protocol/EncodersSuite.java | 14 ++++++++++++++
2 files changed, 17 insertions(+)
diff --git
a/common/network-common/src/main/java/org/apache/spark/network/protocol/Encoders.java
b/common/network-common/src/main/java/org/apache/spark/network/protocol/Encoders.java
index a91200e825d2..c4e94847ea0a 100644
---
a/common/network-common/src/main/java/org/apache/spark/network/protocol/Encoders.java
+++
b/common/network-common/src/main/java/org/apache/spark/network/protocol/Encoders.java
@@ -213,6 +213,9 @@ public class Encoders {
public static RoaringBitmap[] decode(ByteBuf buf) {
int numBitmaps = buf.readInt();
+ // The divisor 8 is the minimum on-wire size of one element, since an
empty RoaringBitmap
+ // serializes to 8 bytes (a 4-byte cookie followed by a 4-byte size).
+ Objects.checkFromIndexSize(0, numBitmaps, buf.readableBytes() / 8);
RoaringBitmap[] bitmaps = new RoaringBitmap[numBitmaps];
for (int i = 0; i < bitmaps.length; i ++) {
bitmaps[i] = Bitmaps.decode(buf);
diff --git
a/common/network-common/src/test/java/org/apache/spark/network/protocol/EncodersSuite.java
b/common/network-common/src/test/java/org/apache/spark/network/protocol/EncodersSuite.java
index 24b766639210..91c95a3c76e0 100644
---
a/common/network-common/src/test/java/org/apache/spark/network/protocol/EncodersSuite.java
+++
b/common/network-common/src/test/java/org/apache/spark/network/protocol/EncodersSuite.java
@@ -89,6 +89,20 @@ public class EncodersSuite {
assertArrayEquals(bitmaps, decodedBitmaps);
}
+ @Test
+ public void testBitmapArraysDecodeShouldFailWhenLengthIsNegative() {
+ ByteBuf buf = Unpooled.buffer();
+ buf.writeInt(-1);
+ assertThrows(IndexOutOfBoundsException.class, () ->
Encoders.BitmapArrays.decode(buf));
+ }
+
+ @Test
+ public void testBitmapArraysDecodeShouldFailWhenLengthExceedsReadableBytes()
{
+ ByteBuf buf = Unpooled.buffer();
+ buf.writeInt(Integer.MAX_VALUE);
+ assertThrows(IndexOutOfBoundsException.class, () ->
Encoders.BitmapArrays.decode(buf));
+ }
+
@Test
public void testByteArraysEncodeDecode() {
byte[] arr = new byte[] { 1, 2, 3, 4, 5 };
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
