mkaravel commented on code in PR #47771:
URL: https://github.com/apache/spark/pull/47771#discussion_r1729507317
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -482,6 +483,13 @@ private static void appendLowercaseCodePoint(final int
codePoint, final StringBu
private static final int CODE_POINT_COMBINING_DOT = 0x307;
private static final int CODE_POINT_COMBINED_LOWERCASE_I_DOT =
CODE_POINT_LOWERCASE_I << 16 | CODE_POINT_COMBINING_DOT;
+ private static final int ASCII_SPACE_CODEPOINT = 32;
+ private static final int CAPITAL_SIGMA = 0x03A3;
+ private static final int SMALL_NON_FINAL_SIGMA = 0x03C3;
+ private static final int SMALL_FINAL_SIGMA = 0x03C2;
+ private static final int CAPITAL_I_WITH_DOT_ABOVE = 0x0130;
+ private static final int LOWERCASE_I = 0x0069;
Review Comment:
```suggestion
private static final int ASCII_SMALL_I = 0x0069;
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -482,6 +483,13 @@ private static void appendLowercaseCodePoint(final int
codePoint, final StringBu
private static final int CODE_POINT_COMBINING_DOT = 0x307;
private static final int CODE_POINT_COMBINED_LOWERCASE_I_DOT =
CODE_POINT_LOWERCASE_I << 16 | CODE_POINT_COMBINING_DOT;
+ private static final int ASCII_SPACE_CODEPOINT = 32;
Review Comment:
```suggestion
private static final int ASCII_SPACE = 0x0020;
```
In some of the names with use the word "CODEPOINT" or "CODE_POINT" and in
some others we don't. In my comments I have opted for not using the additional
word(s). I think the main here is to be consistent. One possibility would be to
create a separate class `SpecialCodePointConstants`, define the constants in
this class (using the simpler names) and then access the constants through that
class. For example: `SpecialCodePointConstants.ASCII_SPACE`.
Also along the lines of the previous comment, use the hex 4-digit
representation for the ASCII space character.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
Review Comment:
I think using the term "new behavior" here is misleading moving forward. In
1-2 years from now "new" will have a vague and unrealistic meaning, which can
be confusing. I think we need to rephrase this sentence to something like:
"Title-casting a string using ICU case mappings".
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
Review Comment:
```suggestion
```
Please remove this empty line.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
+ // We will actually call 'codePointFrom()' 2 times for each character in
the worst case (once
+ // here, and once in 'followedByCasedLetter'). Example of a string where
we call it 2 times
+ // for almost every character is 'ΣΣΣΣΣ' (a string consisting only of
Greek capital sigma)
+ // and 'Σ`````' (a string consisting of a Greek capital sigma, followed
by case-ignorable
+ // characters).
+ int codepoint = target.codePointFrom(offset);
+ // Appending the correctly cased character onto 'sb'.
+ appendTitleCasedCodepoint(sb, codepoint, newWord, precededByCasedLetter,
target, offset);
+ // Updating 'newWord', 'precededByCasedLetter' and 'offset' to be ready
for the next character
+ // that we will process.
+ newWord = (codepoint == ASCII_SPACE_CODEPOINT);
+ if(!UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
+ precededByCasedLetter = UCharacter.hasBinaryProperty(codepoint,
UProperty.CASED);
+ }
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ }
+ return sb.build();
+ }
+
+ private static void appendTitleCasedCodepoint(
+ UTF8StringBuilder sb,
+ int codepoint,
+ boolean isAfterAsciiSpace,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ if(isAfterAsciiSpace) {
+ // Titlecasing a character if it is in the beginning of a new word.
+ appendCodepointToTitleCase(sb, codepoint);
+ return;
+ }
+ if(codepoint == CAPITAL_SIGMA) {
+ // Handling capital Greek letter sigma ('Σ').
+ appendLowerCasedGreekCapitalSigma(sb, precededByCasedLetter, target,
offset);
+ return;
+ }
+ // If it's not the beginning of a word, or a capital Greek letter sigma
('Σ'), we lowercase the
+ // character. We specially handle 'CAPITAL_I_WITH_DOT_ABOVE'.
+ if (codepoint == CAPITAL_I_WITH_DOT_ABOVE) {
+ sb.appendCodePoint(LOWERCASE_I);
+ sb.appendCodePoint(DOT_ABOVE);
+ return;
+ }
+ sb.appendCodePoint(UCharacter.toLowerCase(codepoint));
+ }
+
+ private static void appendLowerCasedGreekCapitalSigma(
+ UTF8StringBuilder sb,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ int codepoint;
+ if (!followedByCasedLetter(target,offset) && precededByCasedLetter) {
+ codepoint = SMALL_FINAL_SIGMA;
+ }
+ else {
+ codepoint = SMALL_NON_FINAL_SIGMA;
+ }
+ sb.appendCodePoint(codepoint);
+ }
+
+ /**
+ * Checks if the character beginning at 'offset'(in 'targets' byte array) is
followed by a cased
+ * letter.
+ */
+ private static boolean followedByCasedLetter(UTF8String target, int offset) {
+ // Moving the offset one character forward, so we could start the linear
search from there.
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ int len = target.numBytes();
+
+ while(offset < len) {
+ int codepoint = target.codePointFrom(offset);
+
+ if(UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ continue;
+ }
+ return UCharacter.hasBinaryProperty(codepoint, UProperty.CASED);
+ }
+ return false;
+ }
+
+ /**
+ * Appends titlecase of a single character to a 'StringBuilder' using the
ICU root locale rules.
+ */
+ private static void appendCodepointToTitleCase(UTF8StringBuilder sb, int
codepoint) {
+ String toTitleCase =
codepointOneToManyTitleCasePrecalculation.get(codepoint);
+ if(toTitleCase == null) {
+ sb.appendCodePoint(UCharacter.toTitleCase(codepoint));
+ }
+ else {
Review Comment:
```suggestion
} else {
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
Review Comment:
Same below.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -482,6 +483,13 @@ private static void appendLowercaseCodePoint(final int
codePoint, final StringBu
private static final int CODE_POINT_COMBINING_DOT = 0x307;
Review Comment:
```suggestion
private static final int CODE_POINT_COMBINING_DOT = 0x0307;
```
I think it it creates less of a cognitive overhead if we use the same format
for all code points: `0xDDDD`.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -482,6 +483,13 @@ private static void appendLowercaseCodePoint(final int
codePoint, final StringBu
private static final int CODE_POINT_COMBINING_DOT = 0x307;
private static final int CODE_POINT_COMBINED_LOWERCASE_I_DOT =
CODE_POINT_LOWERCASE_I << 16 | CODE_POINT_COMBINING_DOT;
+ private static final int ASCII_SPACE_CODEPOINT = 32;
+ private static final int CAPITAL_SIGMA = 0x03A3;
Review Comment:
```suggestion
private static final int GREEK_CAPITAL_SIGMA = 0x03A3;
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -482,6 +483,13 @@ private static void appendLowercaseCodePoint(final int
codePoint, final StringBu
private static final int CODE_POINT_COMBINING_DOT = 0x307;
private static final int CODE_POINT_COMBINED_LOWERCASE_I_DOT =
CODE_POINT_LOWERCASE_I << 16 | CODE_POINT_COMBINING_DOT;
+ private static final int ASCII_SPACE_CODEPOINT = 32;
+ private static final int CAPITAL_SIGMA = 0x03A3;
+ private static final int SMALL_NON_FINAL_SIGMA = 0x03C3;
+ private static final int SMALL_FINAL_SIGMA = 0x03C2;
Review Comment:
```suggestion
private static final int GREEK_FINAL_SIGMA = 0x03C2;
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -482,6 +483,13 @@ private static void appendLowercaseCodePoint(final int
codePoint, final StringBu
private static final int CODE_POINT_COMBINING_DOT = 0x307;
private static final int CODE_POINT_COMBINED_LOWERCASE_I_DOT =
CODE_POINT_LOWERCASE_I << 16 | CODE_POINT_COMBINING_DOT;
+ private static final int ASCII_SPACE_CODEPOINT = 32;
+ private static final int CAPITAL_SIGMA = 0x03A3;
+ private static final int SMALL_NON_FINAL_SIGMA = 0x03C3;
Review Comment:
```suggestion
private static final int GREEK_SMALL_SIGMA = 0x03C3;
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
Review Comment:
```suggestion
private static final HashMap<Integer, String>
codepointOneToManyTitleCaseLookupTable =
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
+ // We will actually call 'codePointFrom()' 2 times for each character in
the worst case (once
+ // here, and once in 'followedByCasedLetter'). Example of a string where
we call it 2 times
+ // for almost every character is 'ΣΣΣΣΣ' (a string consisting only of
Greek capital sigma)
+ // and 'Σ`````' (a string consisting of a Greek capital sigma, followed
by case-ignorable
+ // characters).
+ int codepoint = target.codePointFrom(offset);
+ // Appending the correctly cased character onto 'sb'.
+ appendTitleCasedCodepoint(sb, codepoint, newWord, precededByCasedLetter,
target, offset);
+ // Updating 'newWord', 'precededByCasedLetter' and 'offset' to be ready
for the next character
+ // that we will process.
+ newWord = (codepoint == ASCII_SPACE_CODEPOINT);
+ if(!UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
+ precededByCasedLetter = UCharacter.hasBinaryProperty(codepoint,
UProperty.CASED);
+ }
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ }
+ return sb.build();
+ }
+
+ private static void appendTitleCasedCodepoint(
+ UTF8StringBuilder sb,
+ int codepoint,
+ boolean isAfterAsciiSpace,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ if(isAfterAsciiSpace) {
+ // Titlecasing a character if it is in the beginning of a new word.
+ appendCodepointToTitleCase(sb, codepoint);
+ return;
+ }
+ if(codepoint == CAPITAL_SIGMA) {
+ // Handling capital Greek letter sigma ('Σ').
+ appendLowerCasedGreekCapitalSigma(sb, precededByCasedLetter, target,
offset);
+ return;
+ }
+ // If it's not the beginning of a word, or a capital Greek letter sigma
('Σ'), we lowercase the
+ // character. We specially handle 'CAPITAL_I_WITH_DOT_ABOVE'.
+ if (codepoint == CAPITAL_I_WITH_DOT_ABOVE) {
+ sb.appendCodePoint(LOWERCASE_I);
+ sb.appendCodePoint(DOT_ABOVE);
+ return;
+ }
+ sb.appendCodePoint(UCharacter.toLowerCase(codepoint));
+ }
+
+ private static void appendLowerCasedGreekCapitalSigma(
+ UTF8StringBuilder sb,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ int codepoint;
+ if (!followedByCasedLetter(target,offset) && precededByCasedLetter) {
+ codepoint = SMALL_FINAL_SIGMA;
+ }
+ else {
+ codepoint = SMALL_NON_FINAL_SIGMA;
+ }
+ sb.appendCodePoint(codepoint);
+ }
+
+ /**
+ * Checks if the character beginning at 'offset'(in 'targets' byte array) is
followed by a cased
+ * letter.
+ */
+ private static boolean followedByCasedLetter(UTF8String target, int offset) {
+ // Moving the offset one character forward, so we could start the linear
search from there.
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ int len = target.numBytes();
+
+ while(offset < len) {
+ int codepoint = target.codePointFrom(offset);
+
+ if(UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
Review Comment:
```suggestion
if (UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE))
{
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
+ // We will actually call 'codePointFrom()' 2 times for each character in
the worst case (once
+ // here, and once in 'followedByCasedLetter'). Example of a string where
we call it 2 times
+ // for almost every character is 'ΣΣΣΣΣ' (a string consisting only of
Greek capital sigma)
+ // and 'Σ`````' (a string consisting of a Greek capital sigma, followed
by case-ignorable
+ // characters).
+ int codepoint = target.codePointFrom(offset);
+ // Appending the correctly cased character onto 'sb'.
+ appendTitleCasedCodepoint(sb, codepoint, newWord, precededByCasedLetter,
target, offset);
+ // Updating 'newWord', 'precededByCasedLetter' and 'offset' to be ready
for the next character
+ // that we will process.
+ newWord = (codepoint == ASCII_SPACE_CODEPOINT);
+ if(!UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
+ precededByCasedLetter = UCharacter.hasBinaryProperty(codepoint,
UProperty.CASED);
+ }
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ }
+ return sb.build();
+ }
+
+ private static void appendTitleCasedCodepoint(
+ UTF8StringBuilder sb,
+ int codepoint,
+ boolean isAfterAsciiSpace,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ if(isAfterAsciiSpace) {
+ // Titlecasing a character if it is in the beginning of a new word.
+ appendCodepointToTitleCase(sb, codepoint);
+ return;
+ }
+ if(codepoint == CAPITAL_SIGMA) {
+ // Handling capital Greek letter sigma ('Σ').
+ appendLowerCasedGreekCapitalSigma(sb, precededByCasedLetter, target,
offset);
+ return;
+ }
+ // If it's not the beginning of a word, or a capital Greek letter sigma
('Σ'), we lowercase the
+ // character. We specially handle 'CAPITAL_I_WITH_DOT_ABOVE'.
+ if (codepoint == CAPITAL_I_WITH_DOT_ABOVE) {
+ sb.appendCodePoint(LOWERCASE_I);
+ sb.appendCodePoint(DOT_ABOVE);
+ return;
+ }
+ sb.appendCodePoint(UCharacter.toLowerCase(codepoint));
+ }
+
+ private static void appendLowerCasedGreekCapitalSigma(
+ UTF8StringBuilder sb,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ int codepoint;
+ if (!followedByCasedLetter(target,offset) && precededByCasedLetter) {
+ codepoint = SMALL_FINAL_SIGMA;
+ }
+ else {
+ codepoint = SMALL_NON_FINAL_SIGMA;
+ }
+ sb.appendCodePoint(codepoint);
+ }
+
+ /**
+ * Checks if the character beginning at 'offset'(in 'targets' byte array) is
followed by a cased
+ * letter.
+ */
+ private static boolean followedByCasedLetter(UTF8String target, int offset) {
+ // Moving the offset one character forward, so we could start the linear
search from there.
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ int len = target.numBytes();
+
+ while(offset < len) {
Review Comment:
```suggestion
while (offset < len) {
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
+ // We will actually call 'codePointFrom()' 2 times for each character in
the worst case (once
+ // here, and once in 'followedByCasedLetter'). Example of a string where
we call it 2 times
+ // for almost every character is 'ΣΣΣΣΣ' (a string consisting only of
Greek capital sigma)
+ // and 'Σ`````' (a string consisting of a Greek capital sigma, followed
by case-ignorable
+ // characters).
+ int codepoint = target.codePointFrom(offset);
+ // Appending the correctly cased character onto 'sb'.
+ appendTitleCasedCodepoint(sb, codepoint, newWord, precededByCasedLetter,
target, offset);
+ // Updating 'newWord', 'precededByCasedLetter' and 'offset' to be ready
for the next character
+ // that we will process.
+ newWord = (codepoint == ASCII_SPACE_CODEPOINT);
+ if(!UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
+ precededByCasedLetter = UCharacter.hasBinaryProperty(codepoint,
UProperty.CASED);
+ }
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ }
+ return sb.build();
+ }
+
+ private static void appendTitleCasedCodepoint(
+ UTF8StringBuilder sb,
+ int codepoint,
+ boolean isAfterAsciiSpace,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ if(isAfterAsciiSpace) {
Review Comment:
```suggestion
if (isAfterAsciiSpace) {
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
Review Comment:
```suggestion
while (offset < len) {
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
+ // We will actually call 'codePointFrom()' 2 times for each character in
the worst case (once
+ // here, and once in 'followedByCasedLetter'). Example of a string where
we call it 2 times
+ // for almost every character is 'ΣΣΣΣΣ' (a string consisting only of
Greek capital sigma)
+ // and 'Σ`````' (a string consisting of a Greek capital sigma, followed
by case-ignorable
+ // characters).
+ int codepoint = target.codePointFrom(offset);
+ // Appending the correctly cased character onto 'sb'.
+ appendTitleCasedCodepoint(sb, codepoint, newWord, precededByCasedLetter,
target, offset);
+ // Updating 'newWord', 'precededByCasedLetter' and 'offset' to be ready
for the next character
+ // that we will process.
+ newWord = (codepoint == ASCII_SPACE_CODEPOINT);
+ if(!UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
+ precededByCasedLetter = UCharacter.hasBinaryProperty(codepoint,
UProperty.CASED);
+ }
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ }
+ return sb.build();
+ }
+
+ private static void appendTitleCasedCodepoint(
+ UTF8StringBuilder sb,
+ int codepoint,
+ boolean isAfterAsciiSpace,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ if(isAfterAsciiSpace) {
+ // Titlecasing a character if it is in the beginning of a new word.
+ appendCodepointToTitleCase(sb, codepoint);
+ return;
+ }
+ if(codepoint == CAPITAL_SIGMA) {
+ // Handling capital Greek letter sigma ('Σ').
+ appendLowerCasedGreekCapitalSigma(sb, precededByCasedLetter, target,
offset);
+ return;
+ }
+ // If it's not the beginning of a word, or a capital Greek letter sigma
('Σ'), we lowercase the
+ // character. We specially handle 'CAPITAL_I_WITH_DOT_ABOVE'.
+ if (codepoint == CAPITAL_I_WITH_DOT_ABOVE) {
+ sb.appendCodePoint(LOWERCASE_I);
+ sb.appendCodePoint(DOT_ABOVE);
+ return;
+ }
+ sb.appendCodePoint(UCharacter.toLowerCase(codepoint));
+ }
+
+ private static void appendLowerCasedGreekCapitalSigma(
+ UTF8StringBuilder sb,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ int codepoint;
+ if (!followedByCasedLetter(target,offset) && precededByCasedLetter) {
+ codepoint = SMALL_FINAL_SIGMA;
+ }
+ else {
+ codepoint = SMALL_NON_FINAL_SIGMA;
+ }
Review Comment:
```suggestion
int codepoint;
if (!followedByCasedLetter(target, offset) && precededByCasedLetter) {
codepoint = SMALL_FINAL_SIGMA;
} else {
codepoint = SMALL_NON_FINAL_SIGMA;
}
```
Coding style. Please notice the space I added after the comma when calling
`followedByCasedLetter`.
##########
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java:
##########
@@ -1042,23 +1069,24 @@ public void testInitCap() throws SparkException {
assertInitCap("𝔸", "UTF8_LCASE", "𝔸");
assertInitCap("𝔸", "UNICODE", "𝔸");
assertInitCap("𝔸", "UNICODE_CI", "𝔸");
- assertInitCap("𐐅", "UTF8_BINARY", "𐐭");
+ assertInitCap("𐐅", "UTF8_BINARY", "\uD801\uDC05","𐐭");
Review Comment:
```suggestion
assertInitCap("𐐅", "UTF8_BINARY", "\uD801\uDC05", "𐐭");
```
##########
common/unsafe/src/main/java/org/apache/spark/unsafe/UTF8StringBuilder.java:
##########
@@ -93,7 +93,38 @@ public void appendBytes(Object base, long offset, int
length) {
cursor += length;
}
+ public void appendByte(byte singleByte) {
Review Comment:
Does this need to be a public method? Looks like we only use it below.
In a different direction: below we grow the buffer with either 1, 2, 3, or 4
bytes. In each case we call the `grow` method 1, 2, 3, or 4 times respectively.
Does it make sense to have four different overloads each taking as inputs 1, 2,
3, or 4 bytes and do the growing once with the correct value?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
+ // We will actually call 'codePointFrom()' 2 times for each character in
the worst case (once
+ // here, and once in 'followedByCasedLetter'). Example of a string where
we call it 2 times
+ // for almost every character is 'ΣΣΣΣΣ' (a string consisting only of
Greek capital sigma)
+ // and 'Σ`````' (a string consisting of a Greek capital sigma, followed
by case-ignorable
+ // characters).
+ int codepoint = target.codePointFrom(offset);
+ // Appending the correctly cased character onto 'sb'.
+ appendTitleCasedCodepoint(sb, codepoint, newWord, precededByCasedLetter,
target, offset);
+ // Updating 'newWord', 'precededByCasedLetter' and 'offset' to be ready
for the next character
+ // that we will process.
+ newWord = (codepoint == ASCII_SPACE_CODEPOINT);
+ if(!UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
+ precededByCasedLetter = UCharacter.hasBinaryProperty(codepoint,
UProperty.CASED);
+ }
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ }
+ return sb.build();
+ }
+
+ private static void appendTitleCasedCodepoint(
+ UTF8StringBuilder sb,
+ int codepoint,
+ boolean isAfterAsciiSpace,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ if(isAfterAsciiSpace) {
+ // Titlecasing a character if it is in the beginning of a new word.
+ appendCodepointToTitleCase(sb, codepoint);
+ return;
+ }
+ if(codepoint == CAPITAL_SIGMA) {
+ // Handling capital Greek letter sigma ('Σ').
+ appendLowerCasedGreekCapitalSigma(sb, precededByCasedLetter, target,
offset);
+ return;
+ }
+ // If it's not the beginning of a word, or a capital Greek letter sigma
('Σ'), we lowercase the
+ // character. We specially handle 'CAPITAL_I_WITH_DOT_ABOVE'.
+ if (codepoint == CAPITAL_I_WITH_DOT_ABOVE) {
+ sb.appendCodePoint(LOWERCASE_I);
+ sb.appendCodePoint(DOT_ABOVE);
+ return;
+ }
+ sb.appendCodePoint(UCharacter.toLowerCase(codepoint));
+ }
+
+ private static void appendLowerCasedGreekCapitalSigma(
+ UTF8StringBuilder sb,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ int codepoint;
+ if (!followedByCasedLetter(target,offset) && precededByCasedLetter) {
+ codepoint = SMALL_FINAL_SIGMA;
+ }
+ else {
+ codepoint = SMALL_NON_FINAL_SIGMA;
+ }
+ sb.appendCodePoint(codepoint);
+ }
+
+ /**
+ * Checks if the character beginning at 'offset'(in 'targets' byte array) is
followed by a cased
+ * letter.
+ */
+ private static boolean followedByCasedLetter(UTF8String target, int offset) {
+ // Moving the offset one character forward, so we could start the linear
search from there.
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ int len = target.numBytes();
+
+ while(offset < len) {
+ int codepoint = target.codePointFrom(offset);
+
+ if(UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ continue;
+ }
+ return UCharacter.hasBinaryProperty(codepoint, UProperty.CASED);
+ }
+ return false;
+ }
+
+ /**
+ * Appends titlecase of a single character to a 'StringBuilder' using the
ICU root locale rules.
+ */
+ private static void appendCodepointToTitleCase(UTF8StringBuilder sb, int
codepoint) {
+ String toTitleCase =
codepointOneToManyTitleCasePrecalculation.get(codepoint);
+ if(toTitleCase == null) {
Review Comment:
```suggestion
if (toTitleCase == null) {
```
##########
common/unsafe/src/main/java/org/apache/spark/unsafe/UTF8StringBuilder.java:
##########
@@ -93,7 +93,38 @@ public void appendBytes(Object base, long offset, int
length) {
cursor += length;
}
+ public void appendByte(byte singleByte) {
+ grow(1);
+ buffer[cursor - Platform.BYTE_ARRAY_OFFSET] = singleByte;
+ cursor++;
+ }
+
public UTF8String build() {
return UTF8String.fromBytes(buffer, 0, totalSize());
}
+
+ public void appendCodePoint(int codePoint) {
+ if (codePoint <= 0x7F) {
+ appendByte((byte) codePoint);
+ }
+ else if (codePoint <= 0x7FF) {
+ appendByte((byte) (0xC0 | (codePoint >> 6)));
+ appendByte((byte) (0x80 | (codePoint & 0x3F)));
+ }
+ else if (codePoint <= 0xFFFF) {
+ appendByte((byte) (0xE0 | (codePoint >> 12)));
+ appendByte((byte) (0x80 | ((codePoint >> 6) & 0x3F)));
+ appendByte((byte) (0x80 | (codePoint & 0x3F)));
+ }
+ else if (codePoint <= 0x10FFFF) {
+ appendByte((byte) (0xF0 | (codePoint >> 18)));
+ appendByte((byte) (0x80 | ((codePoint >> 12) & 0x3F)));
+ appendByte((byte) (0x80 | ((codePoint >> 6) & 0x3F)));
+ appendByte((byte) (0x80 | (codePoint & 0x3F)));
+ }
+ else {
+ throw new IllegalArgumentException("Invalid Unicode codePoint: " +
codePoint);
+ }
Review Comment:
```suggestion
if (codePoint <= 0x7F) {
appendByte((byte) codePoint);
} else if (codePoint <= 0x7FF) {
appendByte((byte) (0xC0 | (codePoint >> 6)));
appendByte((byte) (0x80 | (codePoint & 0x3F)));
} else if (codePoint <= 0xFFFF) {
appendByte((byte) (0xE0 | (codePoint >> 12)));
appendByte((byte) (0x80 | ((codePoint >> 6) & 0x3F)));
appendByte((byte) (0x80 | (codePoint & 0x3F)));
} else if (codePoint <= 0x10FFFF) {
appendByte((byte) (0xF0 | (codePoint >> 18)));
appendByte((byte) (0x80 | ((codePoint >> 12) & 0x3F)));
appendByte((byte) (0x80 | ((codePoint >> 6) & 0x3F)));
appendByte((byte) (0x80 | (codePoint & 0x3F)));
} else {
throw new IllegalArgumentException("Invalid Unicode codePoint: " +
codePoint);
}
```
Is there possibly some Java functionality for doing this, or ICU
functionality for doing this, so we don't have to do it ourselves here?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
+ // We will actually call 'codePointFrom()' 2 times for each character in
the worst case (once
+ // here, and once in 'followedByCasedLetter'). Example of a string where
we call it 2 times
+ // for almost every character is 'ΣΣΣΣΣ' (a string consisting only of
Greek capital sigma)
+ // and 'Σ`````' (a string consisting of a Greek capital sigma, followed
by case-ignorable
+ // characters).
+ int codepoint = target.codePointFrom(offset);
+ // Appending the correctly cased character onto 'sb'.
+ appendTitleCasedCodepoint(sb, codepoint, newWord, precededByCasedLetter,
target, offset);
+ // Updating 'newWord', 'precededByCasedLetter' and 'offset' to be ready
for the next character
+ // that we will process.
+ newWord = (codepoint == ASCII_SPACE_CODEPOINT);
+ if(!UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
+ precededByCasedLetter = UCharacter.hasBinaryProperty(codepoint,
UProperty.CASED);
+ }
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ }
+ return sb.build();
+ }
+
+ private static void appendTitleCasedCodepoint(
+ UTF8StringBuilder sb,
+ int codepoint,
+ boolean isAfterAsciiSpace,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ if(isAfterAsciiSpace) {
+ // Titlecasing a character if it is in the beginning of a new word.
+ appendCodepointToTitleCase(sb, codepoint);
+ return;
+ }
+ if(codepoint == CAPITAL_SIGMA) {
Review Comment:
```suggestion
if (codepoint == CAPITAL_SIGMA) {
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
Review Comment:
I believe our coding style asks to have a space between `while` and the
parenthesis.
##########
common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java:
##########
@@ -1362,4 +1364,18 @@ public void toBinaryString() {
UTF8String.fromString("111111111111111111111111111111111111111111111111111111111111111"),
UTF8String.toBinaryString(Long.MAX_VALUE));
}
+
+ @Test
+ public void testAppendCodepointToUTF8StringBuilder() {
Review Comment:
Could you please add a comment regarding what this test is actually testing?
##########
common/unsafe/src/main/java/org/apache/spark/unsafe/UTF8StringBuilder.java:
##########
@@ -93,7 +93,38 @@ public void appendBytes(Object base, long offset, int
length) {
cursor += length;
}
+ public void appendByte(byte singleByte) {
+ grow(1);
+ buffer[cursor - Platform.BYTE_ARRAY_OFFSET] = singleByte;
+ cursor++;
Review Comment:
```suggestion
++cursor;
```
##########
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java:
##########
@@ -1000,20 +1017,30 @@ public void testInitCap() throws SparkException {
assertInitCap("ÄBĆΔE", "UTF8_LCASE", "Äbćδe");
assertInitCap("ÄBĆΔE", "UNICODE", "Äbćδe");
assertInitCap("ÄBĆΔE", "UNICODE_CI", "Äbćδe");
+ // Case-variable character length
+ assertInitCap("İo", "UTF8_BINARY", "İo","I\u0307o");
Review Comment:
```suggestion
assertInitCap("İo", "UTF8_BINARY", "İo", "I\u0307o");
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -550,6 +558,197 @@ public static UTF8String toTitleCase(final UTF8String
target, final int collatio
BreakIterator.getWordInstance(locale)));
}
+ /**
+ * This 'HashMap' is introduced as a performance speedup. Since titlecasing
a codepoint can result
+ * in more than a single codepoint, for correctness, we would use
'UCharacter.toTitleCase(String)'
+ * which returns a 'String'. If we use 'UCharacter.toTitleCase(int)' (the
version of the same
+ * function which converts a single codepoint to its titlecase codepoint),
it would be faster than
+ * the previously mentioned version, but the problem here is that we don't
handle when titlecasing
+ * a codepoint yields more than 1 codepoint. Since there are only 48
codepoints that are mapped to
+ * more than 1 codepoint when titlecased, they are precalculated here, so
that the faster function
+ * for titlecasing could be used in combination with this 'HashMap' in the
method
+ * 'appendCodepointToTitleCase'.
+ */
+ private static final HashMap<Integer, String>
codepointOneToManyTitleCasePrecalculation =
+ new HashMap<>(){{
+ put(223, "Ss");
+ put(329, "ʼN");
+ put(496, "J̌");
+ put(912, "Ϊ́");
+ put(944, "Ϋ́");
+ put(1415, "Եւ");
+ put(7830, "H̱");
+ put(7831, "T̈");
+ put(7832, "W̊");
+ put(7833, "Y̊");
+ put(7834, "Aʾ");
+ put(8016, "Υ̓");
+ put(8018, "Υ̓̀");
+ put(8020, "Υ̓́");
+ put(8022, "Υ̓͂");
+ put(8114, "Ὰͅ");
+ put(8116, "Άͅ");
+ put(8118, "Α͂");
+ put(8119, "ᾼ͂");
+ put(8130, "Ὴͅ");
+ put(8132, "Ήͅ");
+ put(8134, "Η͂");
+ put(8135, "ῌ͂");
+ put(8146, "Ϊ̀");
+ put(8147, "Ϊ́");
+ put(8150, "Ι͂");
+ put(8151, "Ϊ͂");
+ put(8162, "Ϋ̀");
+ put(8163, "Ϋ́");
+ put(8164, "Ρ̓");
+ put(8166, "Υ͂");
+ put(8167, "Ϋ͂");
+ put(8178, "Ὼͅ");
+ put(8180, "Ώͅ");
+ put(8182, "Ω͂");
+ put(8183, "ῼ͂");
+ put(64256, "Ff");
+ put(64257, "Fi");
+ put(64258, "Fl");
+ put(64259, "Ffi");
+ put(64260, "Ffl");
+ put(64261, "St");
+ put(64262, "St");
+ put(64275, "Մն");
+ put(64276, "Մե");
+ put(64277, "Մի");
+ put(64278, "Վն");
+ put(64279, "Մխ");
+ }};
+
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ // In the default UTF8String implementation, `toLowerCase` method
implicitly does UTF8String
+ // validation (replacing invalid UTF-8 byte sequences with Unicode
replacement character
+ // U+FFFD), but now we have to do the validation manually.
+ target = target.makeValid();
+
+ // Building the title cased target with 'sb'.
+ UTF8StringBuilder sb = new UTF8StringBuilder();
+
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next.
+ int offset = 0;
+ int len = target.numBytes();
+
+ while(offset < len) {
+ // We will actually call 'codePointFrom()' 2 times for each character in
the worst case (once
+ // here, and once in 'followedByCasedLetter'). Example of a string where
we call it 2 times
+ // for almost every character is 'ΣΣΣΣΣ' (a string consisting only of
Greek capital sigma)
+ // and 'Σ`````' (a string consisting of a Greek capital sigma, followed
by case-ignorable
+ // characters).
+ int codepoint = target.codePointFrom(offset);
+ // Appending the correctly cased character onto 'sb'.
+ appendTitleCasedCodepoint(sb, codepoint, newWord, precededByCasedLetter,
target, offset);
+ // Updating 'newWord', 'precededByCasedLetter' and 'offset' to be ready
for the next character
+ // that we will process.
+ newWord = (codepoint == ASCII_SPACE_CODEPOINT);
+ if(!UCharacter.hasBinaryProperty(codepoint, UProperty.CASE_IGNORABLE)) {
+ precededByCasedLetter = UCharacter.hasBinaryProperty(codepoint,
UProperty.CASED);
+ }
+ offset += UTF8String.numBytesForFirstByte(target.getByte(offset));
+ }
+ return sb.build();
+ }
+
+ private static void appendTitleCasedCodepoint(
+ UTF8StringBuilder sb,
+ int codepoint,
+ boolean isAfterAsciiSpace,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ if(isAfterAsciiSpace) {
+ // Titlecasing a character if it is in the beginning of a new word.
+ appendCodepointToTitleCase(sb, codepoint);
+ return;
+ }
+ if(codepoint == CAPITAL_SIGMA) {
+ // Handling capital Greek letter sigma ('Σ').
+ appendLowerCasedGreekCapitalSigma(sb, precededByCasedLetter, target,
offset);
+ return;
+ }
+ // If it's not the beginning of a word, or a capital Greek letter sigma
('Σ'), we lowercase the
+ // character. We specially handle 'CAPITAL_I_WITH_DOT_ABOVE'.
+ if (codepoint == CAPITAL_I_WITH_DOT_ABOVE) {
+ sb.appendCodePoint(LOWERCASE_I);
+ sb.appendCodePoint(DOT_ABOVE);
+ return;
+ }
+ sb.appendCodePoint(UCharacter.toLowerCase(codepoint));
+ }
+
+ private static void appendLowerCasedGreekCapitalSigma(
+ UTF8StringBuilder sb,
+ boolean precededByCasedLetter,
+ UTF8String target,
+ int offset) {
+ int codepoint;
+ if (!followedByCasedLetter(target,offset) && precededByCasedLetter) {
+ codepoint = SMALL_FINAL_SIGMA;
+ }
+ else {
+ codepoint = SMALL_NON_FINAL_SIGMA;
+ }
Review Comment:
Alternatively:
```java
int codepoint = (!followedByCasedLetter(target, offset) &&
precededByCasedLetter) ? GREEK_FINAL_SIGMA : GREEK_SMALL_SIGMA;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
