plusplusjiajia opened a new pull request, #9329:
URL: https://github.com/apache/paimon/pull/9329
### Purpose
`SUBSTRING` and `TRIM` operate below the character level, so they split
characters outside the BMP in half.
`SubstringTransform` decodes to a Java `String`, whose `length()` counts
UTF-16 code units, so a supplementary character takes two positions.
`TrimTransform` goes through `StringUtils.ltrim`/`rtrim`, which
remove one `char` at a time, so two supplementary characters sharing a
high surrogate match on that half.
SUBSTRING('😀abc', 2, 2) -> "?a" expected "ab"
TRIM(BOTH '😀' FROM '😁x😁') -> "?x😁" expected "😁x😁"
Both now stay inside `BinaryString`, which counts characters as Spark's
`UTF8String` does. This adds the `trimLeft`/`trimRight`/`trim(BinaryString)`
overloads Spark has and this port was missing, so
`TrimTransform` also stops decoding and re-encoding every row. Comparing
whole characters is enough because UTF-8 is self-synchronizing: a character's
byte sequence never occurs inside another character's.
`BinaryString.substring` clamps a negative begin instead of throwing, so
`checkArgument` now also rejects a non-positive begin. That only changes the
exception type — such inputs already threw
`StringIndexOutOfBoundsException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
