mkaravel commented on code in PR #47771:
URL: https://github.com/apache/spark/pull/47771#discussion_r1729505414
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
Review Comment:
The source of truth for these is this file:
https://unicode.org/Public/UNIDATA/SpecialCasing.txt
From what I have seen the one-to-many mappings in Unicode have been
extremely stable for a while.
I agree with adding a golden file that would go over all possible title
casings and record the one-to-many mappings. This will allow us to check if in
the future, if a new Unicode version is used whether we have changes or not. I
suggest to do this as a follow up PR to this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
