Jackie-Jiang commented on PR #8878:
URL: https://github.com/apache/pinot/pull/8878#issuecomment-1152604460
> > Is the UTF-8 normalized if it is always encoded from a String object?
>
> yes, but not if it came from bytes (not sure if this is possible)
Currently they are always encoded from String. But seems like even java
string is not always normalized:
```
public static void main(String[] args) {
String e1 = "\u00e9";
String e2 = "\u0065\u0301";
System.out.println(e1);
System.out.println(e2);
System.out.println(e1.equals(e2));
System.out.println(e1.compareTo(e2));
System.out.println(Arrays.toString(e1.getBytes(UTF_8)));
System.out.println(Arrays.toString(e2.getBytes(UTF_8)));
System.out.println(new String(e1.getBytes(UTF_8), UTF_8).equals(new
String(e2.getBytes(UTF_8), UTF_8)));
System.out.println(new String(e1.getBytes(UTF_8), UTF_8).compareTo(new
String(e2.getBytes(UTF_8), UTF_8)));
}
é
é
false
132
[-61, -87]
[101, -52, -127]
false
132
```
The problem exists with the current implementation, and we should probably
use
[normalizer](https://docs.oracle.com/javase/7/docs/api/java/text/Normalizer.html)
to normalize it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
