[
https://issues.apache.org/jira/browse/FLINK-40265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108109#comment-18108109
]
Natea Eshetu Beshada commented on FLINK-40265:
----------------------------------------------
[~bruceqiu] ive assigned it to you
> BinaryStringData.compareTo uses inconsistent UTF-16 and UTF-8 ordering for
> SMP characters
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-40265
> URL: https://issues.apache.org/jira/browse/FLINK-40265
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
> Reporter: Shuo Cheng
> Assignee: Qiu Yanjun
> Priority: Major
>
> {{BinaryStringData.compareTo}} uses two different comparison paths:
> * If both values have a non-null {{javaObject}}, it delegates to
> {{String.compareTo}}, which compares UTF-16 code units.
> * Otherwise, it materializes both values and compares their UTF-8 bytes as
> unsigned bytes.
> These orders are not equivalent for comparisons between an SMP character and
> BMP characters in the range U+E000-U+FFFF. As a result, the comparison result
> depends on whether a {{BinaryStringData}} is Java-backed or binary-backed and
> can change after a serializer round trip.
> For example, compare GRINNING FACE (😀, U+1F600) with FULLWIDTH EXCLAMATION
> MARK (！, U+FF01):
> * UTF-16: D83D DE00 < FF01, so 😀 < ！
> * UTF-8: F0 9F 98 80 > EF BC 81, so 😀 > ！
> h3. Reproduce
> {code:java}
> @Test
> void compareToIsConsistentAfterSerialization() throws IOException {
> BinaryStringData grinningFace = BinaryStringData.fromString("😀");
> BinaryStringData fullWidthExclamationMark =
> BinaryStringData.fromString("!");
> DataOutputSerializer output = new DataOutputSerializer(16);
> StringDataSerializer.INSTANCE.serialize(grinningFace, output);
> BinaryStringData deserializedGrinningFace =
> (BinaryStringData)
> StringDataSerializer.INSTANCE.deserialize(
> new
> DataInputDeserializer(output.getCopyOfBuffer()));
> assertThat(deserializedGrinningFace).isEqualTo(grinningFace);
>
> assertThat(deserializedGrinningFace.compareTo(fullWidthExclamationMark)).isPositive();
> assertThat(grinningFace.compareTo(fullWidthExclamationMark))
>
> .isEqualTo(deserializedGrinningFace.compareTo(fullWidthExclamationMark));
> }
> {code}
> h3. Actual behavior
> After serialization:
> * {{grinningFace.compareTo(fullWidthExclamationMark)}} is negative because
> both values retain Java objects and use {{String.compareTo}}.
> * {{deserializedGrinningFace.compareTo(fullWidthExclamationMark)}} is
> positive because the deserialized value is binary-backed and comparison uses
> UTF-8 bytes.
> The final assertion fails.
> h3. Expected behavior
> {{BinaryStringData.compareTo}} should produce the same ordering regardless of
> the backing representation and before/after serialization. The class Javadoc
> and {{SortUtil.putStringNormalizedKey}} already describe/use UTF-8 byte
> ordering, so comparison should follow one canonical ordering.
> h3. Impact
> This violates the {{Comparable}} contract. The original and deserialized
> values compare equal to each other, but have opposite ordering relative to
> the same third value. It can cause representation-dependent results in
> generated string comparisons, sorting, MIN/MAX, or other operations using
> {{compareTo}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
