viktorluc-db commented on code in PR #47771:
URL: https://github.com/apache/spark/pull/47771#discussion_r1719599069
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -1267,6 +1268,128 @@ public static UTF8String[] icuSplitSQL(final UTF8String
string, final UTF8String
return strings.toArray(new UTF8String[0]);
}
+ /**
+ * Title casing a string according to a new behaviour. Iterates over the
string and title cases
+ * the first character in each word, and lowercases every other character.
Handles lowercasing
+ * capital Greek letter sigma ('Σ') separately, taking into account if it
should be a small final
+ * Greek sigma ('ς') or small non-final Greek sigma ('σ'). Words are
separated by ASCII
+ * space(\u0020).
+ *
+ * @param target UTF8String to be title cased
+ * @return title cased target
+ */
+ public static UTF8String toTitleCaseICU(UTF8String target) {
+
+ Iterator<Integer> codepointIterator = target.codePointIterator(
+ CodePointIteratorType.CODE_POINT_ITERATOR_MAKE_VALID);
+
+ // Building the title cased target with 'sb'.
+ StringBuilder sb = new StringBuilder();
+ // 'newWord' is true if the current character is the beginning of a word,
false otherwise.
+ boolean newWord = true;
+ // We are maintaining if the current character is preceded by a cased
letter.
+ // This is used when lowercasing capital Greek letter sigma ('Σ'), to
figure out if it should be
+ // lowercased into σ or ς.
+ boolean precededByCasedLetter = false;
+
+ // 'offset' is a byte offset in target's byte array pointing to the
beginning of the character
+ // that we need to process next(this is only actually used in
appendLowerCasedGreekCapitalSigma)
+ int offset = 0;
Review Comment:
Using an iterator and an offset both at the same time here is unnecessary, i
agree, will fix that.
Regarding the performance note, yes, we are reading 2 times every character.
For example in this string 'ΣΣΣΣΣ' (a string consisting only of Greek capital
sigma).
Regarding the first point which proposes a solution where we have a fixed
size buffer of size M into which we store the read codepoints in order not to
read them again, it would still have a problem with strings like 'Σ`````' (a
string consisting of a Greek capital sigma, followed by case-ignorable
characters) since it would still read N+N-M (where N is the size of the string)
in the best case. This number of reads would be okay if N=M, but i don't think
that instantiating a integer buffer with the size of the input string would be
suitable here? I think that if the number of reads is to be lowered, probably
some other approach would need to be implemented.
Besides this, i think that instantiating single character java Strings and
making calls with them to the ICU(which was done in this PR) is a bigger
performance issue than this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
