sepuri sai krishna created FLINK-40350:
------------------------------------------
Summary: LPAD/RPAD split supplementary-plane characters into
unpaired surrogates
Key: FLINK-40350
URL: https://issues.apache.org/jira/browse/FLINK-40350
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner, Table SQL / Runtime
Affects Versions: 2.3.0, 2.2.0, 2.1.0, 2.0.0
Reporter: sepuri sai krishna
h3. Problem
{{LPAD}} and {{RPAD}} measure length in UTF-16 code units instead of
characters. When the requested length falls in the middle of a
supplementary-plane character, the function splits the surrogate pair and
returns a string containing an unpaired surrogate, which is not valid Unicode.
In the examples below, {{E}} stands for the single-character string U+1F600
GRINNING FACE, which UTF-16 encodes as the surrogate pair U+D83D U+DE00. It is
written as {{E}} rather than inline so that this description stays within the
Basic Multilingual Plane.
|| Expression || Returned, as UTF-16 code units || Note ||
| {{LPAD(E, 1, 'x')}} | {{U+D83D}} | unpaired high surrogate: the first half of
{{E}} |
| {{RPAD(E, 1, 'x')}} | {{U+D83D}} | unpaired high surrogate: the first half of
{{E}} |
| {{RPAD('a', 4, E)}} | {{U+0061 U+D83D U+DE00 U+D83D}} | {{'a'}}, then an
intact {{E}}, then a trailing unpaired high surrogate |
The third case is the least obvious: the padding is copied one code unit at a
time, so the final repetition of the pad string can stop between the two halves
of its character.
The documentation states that the length is measured in characters:
{quote}Returns a new string from string1 left-padded with string2 to a length
of integer *characters*.{quote}
{{E}} is one character, so {{LPAD(E, 1, 'x')}} should return {{E}} unchanged
rather than half of it.
h3. Impact
Silent corruption of string data. The result is invalid UTF-16 and does not
round-trip: encoding it yields a replacement character or malformed bytes, and
comparisons and downstream string functions then operate on a value the query
never produced. No exception is thrown and nothing is logged.
Affects any query where {{LPAD}} or {{RPAD}} truncates, or where the pad string
contains supplementary-plane characters - emoji, historic scripts, and less
common CJK ideographs.
h3. Affects
Present in every branch checked from release-1.19 through release-2.3, and on
master.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
