nikolamand-db commented on code in PR #45816:
URL: https://github.com/apache/spark/pull/45816#discussion_r1549998126
##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -447,6 +447,37 @@ private UTF8String toUpperCaseSlow() {
return fromString(toString().toUpperCase());
}
+ /**
+ * Optimized lowercase comparison for UTF8_BINARY_LCASE collation
+ */
+ public int compareLowercase(UTF8String other) {
+ int curr;
+ for (curr = 0; curr < numBytes && curr < other.numBytes; curr++) {
+ byte left = getByte(curr);
+ byte right = other.getByte(curr);
+ if (numBytesForFirstByte(left) != 1 || numBytesForFirstByte(right) != 1)
{
+ return compareLowercaseSuffixSlow(other, curr);
+ }
+ int lowerLeft = Character.toLowerCase(left);
+ int lowerRight = Character.toLowerCase(right);
+ if (lowerLeft > 127 || lowerRight > 127) {
Review Comment:
The idea is that if we remain in ASCII space, we don't need to worry about
locales and length of character representation in UTF8 encoding; if we step
outside this set of characters, we may encounter issues with both.
Ignoring locales (which `Character.ToLowerCase` does) means that we will
break compliance with `UTF8String.toLowerCase` which is locale-dependent. Issue
with varying length of character encoding in bytes introduces major performance
penalty as we then need to deal with byte buffers to store incoming converted
data. Although this method should have same asymptotic complexity, it will be
several times slower than simply comparing char-by-char. Since current naive
implementation is not that much bad (~7x slower than UTF8_BINARY), we woudn't
gain much in terms of performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
