This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new a763fc28d0d1 [SPARK-57507][SQL] Fix `UTF8String.reverse` reading past
the end of a truncated trailing UTF-8 sequence
a763fc28d0d1 is described below
commit a763fc28d0d17f732d12bacadbf8489770bc3474
Author: YangJie <[email protected]>
AuthorDate: Fri Jun 19 16:31:07 2026 +0800
[SPARK-57507][SQL] Fix `UTF8String.reverse` reading past the end of a
truncated trailing UTF-8 sequence
### What changes were proposed in this pull request?
`UTF8String.reverse()` reverses a string one UTF-8 character at a time,
using `numBytesForFirstByte(getByte(i))` to determine the width of the
character at byte position `i`. The per-character copy length was clamped to
`numBytes` (the total length) instead of `numBytes - i` (the bytes that
actually remain). When the last bytes of the string are a truncated multi-byte
sequence (a leader byte whose declared width exceeds the remaining bytes),
`copyMemory` read past the end of the strin [...]
### Why are the changes needed?
`UTF8String` can hold malformed UTF-8 (for example, bytes produced by
binary coercion or truncated input). For such a string ending in an incomplete
multi-byte sequence, `reverse()` performed an out-of-bounds read and produced a
wrong result. This PR clamps the per-character copy length to the bytes that
actually remain, so the read stays in bounds. Well-formed UTF-8 is unaffected,
since a complete sequence never exceeds the remaining bytes.
Note: other helpers that take a character width from `numBytesForFirstByte`
without clamping (`codePointFrom`, and `trimLeft`/`trimRight` via
`copyUTF8String`) can over-read on the same truncated input. This PR is scoped
to `reverse()`; the siblings are tracked in SPARK-57520.
### Does this PR introduce _any_ user-facing change?
Yes, it fixes incorrect results. The SQL `reverse()` function on a string
value that contains malformed UTF-8 ending in a truncated multi-byte sequence
no longer reads past the end of the value; only previously-incorrect results
change.
### How was this patch tested?
Added cases to `UTF8StringSuite#reverse()` for truncated trailing 2-, 3-,
and 4-byte leaders, and for a complete multi-byte character followed by an
orphan leader. Each uses a sliced backing array with a trailing sentinel byte
so the previous over-read produces a deterministically wrong value; the cases
fail on the old code and pass with the fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #56569 from LuciferYang/utf8string-reverse-truncated-fix.
Authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
(cherry picked from commit a240b1e9c4e3877897c51aa3968fb9535cba08c8)
Signed-off-by: yangjie01 <[email protected]>
---
.../org/apache/spark/unsafe/types/UTF8String.java | 2 +-
.../apache/spark/unsafe/types/UTF8StringSuite.java | 24 ++++++++++++++++++++++
2 files changed, 25 insertions(+), 1 deletion(-)
diff --git
a/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
b/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
index 463a97b55844..2b840413672a 100644
--- a/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
+++ b/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
@@ -1157,7 +1157,7 @@ public final class UTF8String implements
Comparable<UTF8String>, Externalizable,
int i = 0; // position in byte
while (i < numBytes) {
- int len = Math.min(numBytesForFirstByte(getByte(i)), numBytes);
+ int len = Math.min(numBytesForFirstByte(getByte(i)), numBytes - i);
int targetOffset = Math.max(result.length - i - len, 0);
copyMemory(this.base, this.offset + i, result,
BYTE_ARRAY_OFFSET + targetOffset, len);
diff --git
a/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java
b/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java
index c4a66fdffdd4..14a3d4cb603a 100644
---
a/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java
+++
b/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java
@@ -326,6 +326,30 @@ public class UTF8StringSuite {
assertEquals(EMPTY_UTF8, EMPTY_UTF8.reverse());
assertEquals(fromString("者行孙"), fromString("孙行者").reverse());
assertEquals(fromString("者行孙 olleh"), fromString("hello 孙行者").reverse());
+ // Malformed UTF-8: a truncated trailing multi-byte sequence must be
reversed as orphan
+ // bytes without reading past the end of the string. The backing arrays
carry an extra
+ // trailing byte so a regression that over-reads would produce a
deterministically wrong
+ // result rather than reading uninitialized memory.
+ // 'A' followed by an incomplete 2-byte leader (0xCE).
+ byte[] truncated2 = new byte[]{0x41, (byte) 0xCE, 0x42};
+ assertEquals(
+ fromBytes(new byte[]{(byte) 0xCE, 0x41}),
+ fromBytes(truncated2, 0, 2).reverse());
+ // 'A' followed by an incomplete 3-byte leader (0xE4 0xB8).
+ byte[] truncated3 = new byte[]{0x41, (byte) 0xE4, (byte) 0xB8, 0x42};
+ assertEquals(
+ fromBytes(new byte[]{(byte) 0xE4, (byte) 0xB8, 0x41}),
+ fromBytes(truncated3, 0, 3).reverse());
+ // 'A' followed by an incomplete 4-byte leader (0xF0 0x90).
+ byte[] truncated4 = new byte[]{0x41, (byte) 0xF0, (byte) 0x90, 0x42};
+ assertEquals(
+ fromBytes(new byte[]{(byte) 0xF0, (byte) 0x90, 0x41}),
+ fromBytes(truncated4, 0, 3).reverse());
+ // A complete 3-byte character (U+4E16) followed by an incomplete 2-byte
leader (0xCE).
+ byte[] truncatedMid = new byte[]{(byte) 0xE4, (byte) 0xB8, (byte) 0x96,
(byte) 0xCE, 0x42};
+ assertEquals(
+ fromBytes(new byte[]{(byte) 0xCE, (byte) 0xE4, (byte) 0xB8, (byte)
0x96}),
+ fromBytes(truncatedMid, 0, 4).reverse());
}
@Test
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
