jiwen624 opened a new pull request, #57858:
URL: https://github.com/apache/spark/pull/57858
### What changes were proposed in this pull request?
Mask.transformInput iterated the input by UTF-16 code unit (String.map).
Each half of a surrogate pair is categorized as Character.SURROGATE, so
supplementary characters were classified as "other" and emitted twice; the
replacement character was read with charAt(0), taking only the high surrogate.
This iterates by code point and reads the replacement with codePointAt(0).
The classification logic is unchanged, and doGenCode calls the same static
Mask.transformInput, so both the interpreted and codegen paths are fixed.
### Why are the changes needed?
mask is documented for "creating copies of tables with sensitive information
removed", but it mishandles characters outside the BMP:
```SQL
SELECT mask('A𝐀 1𝟎'); -- 'A𝐀 1𝟎' unchanged, expected 'XX
nn'
SELECT mask('🙂', 'Y', 'y', 'n', '*'); -- '**', expected '*'
SELECT mask('ABC', '🙂'); -- '???', expected '🙂🙂🙂'
```
The first is the most serious: supplementary letters and digits (U+1D400, or
living scripts such as Adlam at U+1E900) are UPPERCASE_LETTER /
LOWERCASE_LETTER / DECIMAL_DIGIT_NUMBER at the code-point level, so they should
be masked but pass through verbatim — the function silently fails to redact.
The expression is also internally inconsistent: checkInputDataTypes requires
the replacement to be exactly one character using UTF8String.numChars, which
counts code points, so '🙂' passes validation and is then truncated during
evaluation.
The char-wise iteration was inherited from Hive's GenericUDFMask, but Spark
did not port Hive's argument handling — Hive takes charAt(0) with no arity
check at all. Elsewhere Spark is code-point based (length, substring, instr,
locate, position), and this class of fix has been applied repeatedly
(SPARK-55453, SPARK-57747, SPARK-57932, SPARK-57506, SPARK-57507). mask is the
outlier.
### Does this PR introduce _any_ user-facing change?
Yes. mask now treats a supplementary character as a single character. Input
consisting only of BMP characters is unaffected. Data containing
supplementary-plane letters or digits will now be masked where it was
previously returned unchanged.
### How was this patch tested?
New test cases added
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
