[
https://issues.apache.org/jira/browse/FLINK-40265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuo Cheng updated FLINK-40265:
-------------------------------
Description:
h3. Description
{{BinaryStringData.compareTo}} uses two different comparison paths:
* If both values have a non-null {{javaObject}}, it delegates to
{{String.compareTo}}, which compares UTF-16 code units.
* Otherwise, it materializes both values and compares their UTF-8 bytes as
unsigned bytes.
These orders are not equivalent for comparisons between an SMP character and
BMP characters in the range U+E000-U+FFFF. As a result, the comparison result
depends on whether a {{BinaryStringData}} is Java-backed or binary-backed and
can change after a serializer round trip.
For example, compare GRINNING FACE (😀, U+1F600) with FULLWIDTH EXCLAMATION MARK
(！, U+FF01):
* UTF-16: D83D DE00 < FF01, so 😀 < ！
* UTF-8: F0 9F 98 80 > EF BC 81, so 😀 > ！
h3. Reproduce
{code:java}
@Test
void compareToIsConsistentAfterSerialization() throws IOException {
BinaryStringData grinningFace = BinaryStringData.fromString("😀");
BinaryStringData fullWidthExclamationMark =
BinaryStringData.fromString("!");
DataOutputSerializer output = new DataOutputSerializer(16);
StringDataSerializer.INSTANCE.serialize(grinningFace, output);
BinaryStringData deserializedGrinningFace =
(BinaryStringData)
StringDataSerializer.INSTANCE.deserialize(
new
DataInputDeserializer(output.getCopyOfBuffer()));
assertThat(deserializedGrinningFace).isEqualTo(grinningFace);
assertThat(deserializedGrinningFace.compareTo(fullWidthExclamationMark)).isPositive();
assertThat(grinningFace.compareTo(fullWidthExclamationMark))
.isEqualTo(deserializedGrinningFace.compareTo(fullWidthExclamationMark));
}
{code}
h3. Actual behavior
After serialization:
* {{grinningFace.compareTo(fullWidthExclamationMark)}} is negative because both
values retain Java objects and use {{String.compareTo}}.
* {{deserializedGrinningFace.compareTo(fullWidthExclamationMark)}} is positive
because the deserialized value is binary-backed and comparison uses UTF-8 bytes.
The final assertion fails.
h3. Expected behavior
{{BinaryStringData.compareTo}} should produce the same ordering regardless of
the backing representation and before/after serialization. The class Javadoc
and {{SortUtil.putStringNormalizedKey}} already describe/use UTF-8 byte
ordering, so comparison should follow one canonical ordering.
h3. Impact
This violates the {{Comparable}} contract. The original and deserialized values
compare equal to each other, but have opposite ordering relative to the same
third value. It can cause representation-dependent results in generated string
comparisons, sorting, MIN/MAX, or other operations using {{compareTo}}.
was:
h3. Description
{{BinaryStringData.compareTo}} uses two different comparison paths:
* If both values have a non-null {{javaObject}}, it delegates to
{{String.compareTo}}, which compares UTF-16 code units.
* Otherwise, it materializes both values and compares their UTF-8 bytes as
unsigned bytes.
These orders are not equivalent for comparisons between an SMP character and
BMP characters in the range U+E000-U+FFFF. As a result, the comparison result
depends on whether a {{BinaryStringData}} is Java-backed or binary-backed and
can change after a serializer round trip.
For example, compare GRINNING FACE (😀, U+1F600) with FULLWIDTH EXCLAMATION MARK
(！, U+FF01):
* UTF-16: D83D DE00 < FF01, so 😀 < ！
* UTF-8: F0 9F 98 80 > EF BC 81, so 😀 > ！
h3. Reproduce
{code:java}
@Test
void compareToIsConsistentAfterSerialization() throws IOException {
BinaryStringData grinningFace = BinaryStringData.fromString("😀");
BinaryStringData fullWidthExclamationMark =
BinaryStringData.fromString("！");
DataOutputSerializer output = new DataOutputSerializer(16);
StringDataSerializer.INSTANCE.serialize(grinningFace, output);
BinaryStringData deserializedGrinningFace =
(BinaryStringData)
StringDataSerializer.INSTANCE.deserialize(
new
DataInputDeserializer(output.getCopyOfBuffer()));
assertThat(deserializedGrinningFace).isEqualTo(grinningFace);
assertThat(deserializedGrinningFace.compareTo(fullWidthExclamationMark)).isPositive();
assertThat(grinningFace.compareTo(fullWidthExclamationMark))
.isEqualTo(deserializedGrinningFace.compareTo(fullWidthExclamationMark));
}
{code}
h3. Actual behavior
After serialization:
* {{grinningFace.compareTo(fullWidthExclamationMark)}} is negative because both
values retain Java objects and use {{String.compareTo}}.
* {{deserializedGrinningFace.compareTo(fullWidthExclamationMark)}} is positive
because the deserialized value is binary-backed and comparison uses UTF-8 bytes.
The final assertion fails.
h3. Expected behavior
{{BinaryStringData.compareTo}} should produce the same ordering regardless of
the backing representation and before/after serialization. The class Javadoc
and {{SortUtil.putStringNormalizedKey}} already describe/use UTF-8 byte
ordering, so comparison should follow one canonical ordering.
h3. Impact
This violates the {{Comparable}} contract. The original and deserialized values
compare equal to each other, but have opposite ordering relative to the same
third value. It can cause representation-dependent results in generated string
comparisons, sorting, MIN/MAX, or other operations using {{compareTo}}.
> BinaryStringData.compareTo uses inconsistent UTF-16 and UTF-8 ordering for
> SMP characters
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-40265
> URL: https://issues.apache.org/jira/browse/FLINK-40265
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
> Reporter: Shuo Cheng
> Priority: Major
>
> h3. Description
> {{BinaryStringData.compareTo}} uses two different comparison paths:
> * If both values have a non-null {{javaObject}}, it delegates to
> {{String.compareTo}}, which compares UTF-16 code units.
> * Otherwise, it materializes both values and compares their UTF-8 bytes as
> unsigned bytes.
> These orders are not equivalent for comparisons between an SMP character and
> BMP characters in the range U+E000-U+FFFF. As a result, the comparison result
> depends on whether a {{BinaryStringData}} is Java-backed or binary-backed and
> can change after a serializer round trip.
> For example, compare GRINNING FACE (😀, U+1F600) with FULLWIDTH EXCLAMATION
> MARK (！, U+FF01):
> * UTF-16: D83D DE00 < FF01, so 😀 < ！
> * UTF-8: F0 9F 98 80 > EF BC 81, so 😀 > ！
> h3. Reproduce
> {code:java}
> @Test
> void compareToIsConsistentAfterSerialization() throws IOException {
> BinaryStringData grinningFace = BinaryStringData.fromString("😀");
> BinaryStringData fullWidthExclamationMark =
> BinaryStringData.fromString("!");
> DataOutputSerializer output = new DataOutputSerializer(16);
> StringDataSerializer.INSTANCE.serialize(grinningFace, output);
> BinaryStringData deserializedGrinningFace =
> (BinaryStringData)
> StringDataSerializer.INSTANCE.deserialize(
> new
> DataInputDeserializer(output.getCopyOfBuffer()));
> assertThat(deserializedGrinningFace).isEqualTo(grinningFace);
>
> assertThat(deserializedGrinningFace.compareTo(fullWidthExclamationMark)).isPositive();
> assertThat(grinningFace.compareTo(fullWidthExclamationMark))
>
> .isEqualTo(deserializedGrinningFace.compareTo(fullWidthExclamationMark));
> }
> {code}
> h3. Actual behavior
> After serialization:
> * {{grinningFace.compareTo(fullWidthExclamationMark)}} is negative because
> both values retain Java objects and use {{String.compareTo}}.
> * {{deserializedGrinningFace.compareTo(fullWidthExclamationMark)}} is
> positive because the deserialized value is binary-backed and comparison uses
> UTF-8 bytes.
> The final assertion fails.
> h3. Expected behavior
> {{BinaryStringData.compareTo}} should produce the same ordering regardless of
> the backing representation and before/after serialization. The class Javadoc
> and {{SortUtil.putStringNormalizedKey}} already describe/use UTF-8 byte
> ordering, so comparison should follow one canonical ordering.
> h3. Impact
> This violates the {{Comparable}} contract. The original and deserialized
> values compare equal to each other, but have opposite ordering relative to
> the same third value. It can cause representation-dependent results in
> generated string comparisons, sorting, MIN/MAX, or other operations using
> {{compareTo}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
