mkaravel commented on code in PR #46700:
URL: https://github.com/apache/spark/pull/46700#discussion_r1628799978
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -296,6 +344,45 @@ public static String toLowerCase(final String target,
final int collationId) {
return UCharacter.toLowerCase(locale, target);
}
+ /**
+ * Converts a single code point to lowercase using ICU rules, with special
handling for
+ * one-to-many case mappings (i.e. characters that map to multiple
characters in lowercase).
+ *
+ * @param codePoint The code point to convert to lowercase.
+ * @param sb The StringBuilder to append the lowercase character to.
+ */
+ private static void lowercaseCodePoint(final int codePoint, final
StringBuilder sb) {
+ if (codePoint == 0x0130) {
+ // Latin capital letter I with dot above is mapped to 2 lowercase
characters.
+ sb.appendCodePoint(0x0069);
+ sb.appendCodePoint(0x0307);
+ }
+ else if (codePoint == 0x03C2) {
+ // Greek final and non-final capital letter sigma should be mapped the
same.
+ sb.appendCodePoint(0x03C3);
+ }
+ else {
+ // All other characters should follow context-unaware ICU single-code
point case mapping.
+ sb.appendCodePoint(UCharacter.toLowerCase(codePoint));
Review Comment:
I believe this does not map illegal UTF8 byte sequences to U+FFFD.
Do we plan to make this change as part of this PR, or in a follow up one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
