mkaravel commented on code in PR #46511:
URL: https://github.com/apache/spark/pull/46511#discussion_r1603716669
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
Review Comment:
Let's specify that this is 0-based.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
Review Comment:
```suggestion
* @return the position of the first occurrence of pattern in target, if
not found, `MATCH_NOT_FOUND` returned.
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ for (int i = startPos; i <= target.numChars(); ++i) {
+ if (lowercaseMatchFrom(target, lowercasePattern, i)) {
+ return i;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns whether the target string ends with the specified suffix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the suffix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same suffix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return whether the target string ends with the specified suffix in
lowercase
+ */
+ public static boolean lowercaseMatchUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ return lowercaseMatchLengthUntil(target, lowercasePattern, endPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that ends with
+ * the specified suffix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the suffix is already lowercased. The method only
+ * considers the part of target string that ends at the specified position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ assert endPos <= target.numChars();
+ for (int len = 0; len <= endPos; ++len) {
+ if (target.substring(endPos - len,
endPos).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
Review Comment:
Same thing here for creating a JIRA for using the pattern-length heuristic.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
Review Comment:
You probably mean end position here. Also let's specify that `startPos` is
0-based.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
Review Comment:
Also that `startPos` refers to UTF8 characters not bytes.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ for (int i = startPos; i <= target.numChars(); ++i) {
+ if (lowercaseMatchFrom(target, lowercasePattern, i)) {
+ return i;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns whether the target string ends with the specified suffix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the suffix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same suffix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
Review Comment:
We also need to explain/specify that `endPos` refers to UTF8 characters.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
Review Comment:
Let's also say that if a match is not found, then `MATCH_NOT_FOUND` is
returned.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
Review Comment:
nit: Please try to use the full extent of the 100 characters per line for
the comments.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
Review Comment:
Shouldn't we mention that we are interested in a match after `startPos`?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ for (int i = startPos; i <= target.numChars(); ++i) {
+ if (lowercaseMatchFrom(target, lowercasePattern, i)) {
+ return i;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns whether the target string ends with the specified suffix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the suffix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same suffix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
Review Comment:
Let's specify that this is 0-based.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ for (int i = startPos; i <= target.numChars(); ++i) {
Review Comment:
Assert that startPos is non-negative?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ for (int i = startPos; i <= target.numChars(); ++i) {
+ if (lowercaseMatchFrom(target, lowercasePattern, i)) {
+ return i;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns whether the target string ends with the specified suffix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the suffix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same suffix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return whether the target string ends with the specified suffix in
lowercase
+ */
+ public static boolean lowercaseMatchUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ return lowercaseMatchLengthUntil(target, lowercasePattern, endPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that ends with
+ * the specified suffix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the suffix is already lowercased. The method only
+ * considers the part of target string that ends at the specified position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
Review Comment:
```suggestion
* @param endPos the 0-based end position for searching (in the target
string)
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
Review Comment:
Could we please create a ticket to re-implement this using the heuristic of
starting with a length that is equal to the pattern, which also means that we
probably need to pass this length as an argument since the length of the
pattern and the length of the lowercase pattern may be different. Actually,
even using the length of the lowercase pattern should be okay since this is
meant to act as a starting point for the search.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ for (int i = startPos; i <= target.numChars(); ++i) {
+ if (lowercaseMatchFrom(target, lowercasePattern, i)) {
+ return i;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns whether the target string ends with the specified suffix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the suffix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same suffix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return whether the target string ends with the specified suffix in
lowercase
+ */
+ public static boolean lowercaseMatchUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ return lowercaseMatchLengthUntil(target, lowercasePattern, endPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that ends with
+ * the specified suffix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the suffix is already lowercased. The method only
+ * considers the part of target string that ends at the specified position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ assert endPos <= target.numChars();
+ for (int len = 0; len <= endPos; ++len) {
+ if (target.substring(endPos - len,
endPos).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the last occurrence of the pattern string
+ * in the target string until the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return the position of the last occurrence of pattern in target, if not
found, -1 returned.
Review Comment:
```suggestion
* @return the position of the last occurrence of pattern in target, if
not found, `MATCH_NOT_FOUND` returned.
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ for (int i = startPos; i <= target.numChars(); ++i) {
+ if (lowercaseMatchFrom(target, lowercasePattern, i)) {
+ return i;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns whether the target string ends with the specified suffix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the suffix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same suffix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return whether the target string ends with the specified suffix in
lowercase
+ */
+ public static boolean lowercaseMatchUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ return lowercaseMatchLengthUntil(target, lowercasePattern, endPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that ends with
+ * the specified suffix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the suffix is already lowercased. The method only
+ * considers the part of target string that ends at the specified position.
Review Comment:
```suggestion
* The method assumes that the suffix is already lowercased. The method
only
* considers the part of target string that ends at the specified
(non-inclusive) position (that is the method does not look at UTF8 characters
of the target string at or after position `endPos`).
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
Review Comment:
Specify that "position" refers to UTF8 characters not bytes?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ for (int i = startPos; i <= target.numChars(); ++i) {
+ if (lowercaseMatchFrom(target, lowercasePattern, i)) {
+ return i;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns whether the target string ends with the specified suffix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the suffix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same suffix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return whether the target string ends with the specified suffix in
lowercase
+ */
+ public static boolean lowercaseMatchUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ return lowercaseMatchLengthUntil(target, lowercasePattern, endPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that ends with
+ * the specified suffix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the suffix is already lowercased. The method only
+ * considers the part of target string that ends at the specified position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ assert endPos <= target.numChars();
+ for (int len = 0; len <= endPos; ++len) {
+ if (target.substring(endPos - len,
endPos).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the last occurrence of the pattern string
+ * in the target string until the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return the position of the last occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseRFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ for (int i = endPos; i >= 0; --i) {
Review Comment:
Assert that endPos is non-negative?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ for (int i = startPos; i <= target.numChars(); ++i) {
+ if (lowercaseMatchFrom(target, lowercasePattern, i)) {
+ return i;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns whether the target string ends with the specified suffix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the suffix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same suffix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return whether the target string ends with the specified suffix in
lowercase
+ */
+ public static boolean lowercaseMatchUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ return lowercaseMatchLengthUntil(target, lowercasePattern, endPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that ends with
+ * the specified suffix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the suffix is already lowercased. The method only
+ * considers the part of target string that ends at the specified position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ assert endPos <= target.numChars();
+ for (int len = 0; len <= endPos; ++len) {
+ if (target.substring(endPos - len,
endPos).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the last occurrence of the pattern string
+ * in the target string until the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return the position of the last occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseRFind(
Review Comment:
Maybe rename this to `lowercaseReverseFind`?
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -183,6 +320,22 @@ public static int findInSet(final UTF8String match, final
UTF8String set, int co
return 0;
}
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation.
+ *
+ * @param target the string to be searched in
+ * @param pattern the string to be searched for
+ * @param start the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
Review Comment:
```suggestion
* @return the position of the first occurrence of pattern in target, if
not found `MATCH_NOT_FOUND` is returned.
```
##########
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java:
##########
@@ -23,7 +23,7 @@
import static org.junit.jupiter.api.Assertions.*;
-
+// checkstyle.off: AvoidEscapedUnicodeCharacters
Review Comment:
Nice!
##########
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java:
##########
@@ -279,20 +320,44 @@ public void testEndsWith() throws SparkException {
assertEndsWith("ab世De", "AB世dE", "UNICODE_CI", true);
assertEndsWith("äbćδe", "ÄbćδE", "UNICODE_CI", true);
assertEndsWith("äbćδe", "ÄBcΔÉ", "UNICODE_CI", false);
- // Case-variable character length
- assertEndsWith("The İo", "i̇o", "UNICODE_CI", true);
- assertEndsWith("The i̇o", "İo", "UNICODE_CI", true);
- assertEndsWith("the İodine", "the i̇odine", "UTF8_BINARY_LCASE", true);
- assertEndsWith("the i̇odine", "the İodine", "UTF8_BINARY_LCASE", true);
- assertEndsWith("The İodiNe", "i̇oDine", "UTF8_BINARY_LCASE", true);
- assertEndsWith("The İodiNe", "i̇oDin", "UTF8_BINARY_LCASE", false);
// Characters with the same binary lowercase representation
assertEndsWith("The Kelvin", "Kelvin", "UTF8_BINARY_LCASE", true);
assertEndsWith("The Kelvin", "Kelvin", "UTF8_BINARY_LCASE", true);
assertEndsWith("The KKelvin", "KKelvin", "UTF8_BINARY_LCASE", true);
assertEndsWith("The 2 Kelvin", "2 Kelvin", "UTF8_BINARY_LCASE", true);
assertEndsWith("The 2 Kelvin", "2 Kelvin", "UTF8_BINARY_LCASE", true);
assertEndsWith("The KKelvin", "KKelvin,", "UTF8_BINARY_LCASE", false);
+ // Case-variable character length
+ assertEndsWith("i̇", "\u0307", "UNICODE_CI", false);
+ assertEndsWith("i̇", "İ", "UNICODE_CI", true);
+ assertEndsWith("İ", "i", "UNICODE_CI", false);
+ assertEndsWith("İİİ", "i̇i̇", "UNICODE_CI", true);
+ assertEndsWith("İİİ", "ii̇", "UNICODE_CI", false);
+ assertEndsWith("İi̇İ", "İi̇", "UNICODE_CI", true);
+ assertEndsWith("i̇İi̇i̇", "\u0307İi̇İ", "UNICODE_CI", false);
+ assertEndsWith("the i̇o", "io", "UNICODE_CI", false);
+ assertEndsWith("the i̇o", "Io", "UNICODE_CI", false);
+ assertEndsWith("the i̇o", "i̇o", "UNICODE_CI", true);
+ assertEndsWith("the i̇o", "İo", "UNICODE_CI", true);
+ assertEndsWith("the İo", "io", "UNICODE_CI", false);
+ assertEndsWith("the İo", "Io", "UNICODE_CI", false);
+ assertEndsWith("the İo", "i̇o", "UNICODE_CI", true);
+ assertEndsWith("the İo", "İo", "UNICODE_CI", true);
+ assertEndsWith("i̇", "\u0307", "UTF8_BINARY_LCASE", true); // != UNICODE_CI
Review Comment:
Can we also add?
```
assertEndsWith("İ", "\u0307", "UTF8_BINARY_LCASE", false);
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
Review Comment:
```suggestion
if (target.substring(startPos, startPos + len,
/*IsEphemeral=*/true).toLowerCase().equals(lowercasePattern)) {
```
There is no need to make a copy here.
##########
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java:
##########
@@ -279,20 +320,44 @@ public void testEndsWith() throws SparkException {
assertEndsWith("ab世De", "AB世dE", "UNICODE_CI", true);
assertEndsWith("äbćδe", "ÄbćδE", "UNICODE_CI", true);
assertEndsWith("äbćδe", "ÄBcΔÉ", "UNICODE_CI", false);
- // Case-variable character length
- assertEndsWith("The İo", "i̇o", "UNICODE_CI", true);
- assertEndsWith("The i̇o", "İo", "UNICODE_CI", true);
- assertEndsWith("the İodine", "the i̇odine", "UTF8_BINARY_LCASE", true);
- assertEndsWith("the i̇odine", "the İodine", "UTF8_BINARY_LCASE", true);
- assertEndsWith("The İodiNe", "i̇oDine", "UTF8_BINARY_LCASE", true);
- assertEndsWith("The İodiNe", "i̇oDin", "UTF8_BINARY_LCASE", false);
// Characters with the same binary lowercase representation
assertEndsWith("The Kelvin", "Kelvin", "UTF8_BINARY_LCASE", true);
assertEndsWith("The Kelvin", "Kelvin", "UTF8_BINARY_LCASE", true);
assertEndsWith("The KKelvin", "KKelvin", "UTF8_BINARY_LCASE", true);
assertEndsWith("The 2 Kelvin", "2 Kelvin", "UTF8_BINARY_LCASE", true);
assertEndsWith("The 2 Kelvin", "2 Kelvin", "UTF8_BINARY_LCASE", true);
assertEndsWith("The KKelvin", "KKelvin,", "UTF8_BINARY_LCASE", false);
+ // Case-variable character length
+ assertEndsWith("i̇", "\u0307", "UNICODE_CI", false);
+ assertEndsWith("i̇", "İ", "UNICODE_CI", true);
+ assertEndsWith("İ", "i", "UNICODE_CI", false);
+ assertEndsWith("İİİ", "i̇i̇", "UNICODE_CI", true);
+ assertEndsWith("İİİ", "ii̇", "UNICODE_CI", false);
+ assertEndsWith("İi̇İ", "İi̇", "UNICODE_CI", true);
+ assertEndsWith("i̇İi̇i̇", "\u0307İi̇İ", "UNICODE_CI", false);
+ assertEndsWith("the i̇o", "io", "UNICODE_CI", false);
+ assertEndsWith("the i̇o", "Io", "UNICODE_CI", false);
+ assertEndsWith("the i̇o", "i̇o", "UNICODE_CI", true);
+ assertEndsWith("the i̇o", "İo", "UNICODE_CI", true);
+ assertEndsWith("the İo", "io", "UNICODE_CI", false);
+ assertEndsWith("the İo", "Io", "UNICODE_CI", false);
+ assertEndsWith("the İo", "i̇o", "UNICODE_CI", true);
+ assertEndsWith("the İo", "İo", "UNICODE_CI", true);
+ assertEndsWith("i̇", "\u0307", "UTF8_BINARY_LCASE", true); // != UNICODE_CI
+ assertEndsWith("i̇", "İ", "UTF8_BINARY_LCASE", true);
+ assertEndsWith("İ", "i", "UTF8_BINARY_LCASE", false);
+ assertEndsWith("İİİ", "i̇i̇", "UTF8_BINARY_LCASE", true);
+ assertEndsWith("İİİ", "ii̇", "UTF8_BINARY_LCASE", false);
Review Comment:
Also add?
```
assertEndsWith("İİİ", "\u0307i̇", "UTF8_BINARY_LCASE", false);
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return whether the target string starts with the specified prefix in
UTF8_BINARY_LCASE
+ */
+ public static boolean lowercaseMatchFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ return lowercaseMatchLengthFrom(target, lowercasePattern, startPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that starts with
+ * the specified prefix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the prefix is already lowercased. The method only
+ * considers the part of target string that starts from the specified
position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthFrom(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ assert startPos >= 0;
+ for (int len = 0; len <= target.numChars() - startPos; ++len) {
+ if (target.substring(startPos, startPos +
len).toLowerCase().equals(lowercasePattern)) {
+ return len;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the position of the first occurrence of the pattern string
+ * in the target string from the specified position (0-based index),
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the pattern string is already lowercased prior to method call.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
+ * @return the position of the first occurrence of pattern in target, if not
found, -1 returned.
+ */
+ public static int lowercaseFind(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int startPos) {
+ for (int i = startPos; i <= target.numChars(); ++i) {
+ if (lowercaseMatchFrom(target, lowercasePattern, i)) {
+ return i;
+ }
+ }
+ return MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns whether the target string ends with the specified suffix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the suffix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same suffix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return whether the target string ends with the specified suffix in
lowercase
+ */
+ public static boolean lowercaseMatchUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ return lowercaseMatchLengthUntil(target, lowercasePattern, endPos) !=
MATCH_NOT_FOUND;
+ }
+
+ /**
+ * Returns the length of the substring of the target string that ends with
+ * the specified suffix, with respect to the UTF8_BINARY_LCASE collation.
+ * The method assumes that the suffix is already lowercased. The method only
+ * considers the part of target string that ends at the specified position.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param endPos the end position for searching (in the target string)
+ * @return length of the target substring that ends with the specified
suffix in lowercase
+ */
+ public static int lowercaseMatchLengthUntil(
+ final UTF8String target,
+ final UTF8String lowercasePattern,
+ int endPos) {
+ assert endPos <= target.numChars();
+ for (int len = 0; len <= endPos; ++len) {
+ if (target.substring(endPos - len,
endPos).toLowerCase().equals(lowercasePattern)) {
Review Comment:
```suggestion
if (target.substring(endPos - len, endPos,
/*isEphemeral=*/true).toLowerCase().equals(lowercasePattern)) {
```
There is no need to make a copy.
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
Review Comment:
Same for other comments below.
##########
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java:
##########
@@ -709,12 +774,24 @@ public void testLocate() throws SparkException {
assertLocate("大千", "test大千世界大千世界", 9, "UNICODE_CI", 9);
assertLocate("大千", "大千世界大千世界", 1, "UNICODE_CI", 1);
// Case-variable character length
+ assertLocate("\u0307", "i̇", 1, "UTF8_BINARY", 2);
Review Comment:
```suggestion
assertLocate("\u0307", "i̇", 1, "UTF8_BINARY_LCASE", 2);
```
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:
##########
@@ -34,6 +34,143 @@
* Utility class for collation-aware UTF8String operations.
*/
public class CollationAwareUTF8String {
+
+ /**
+ * The constant value to indicate that the match is not found
+ * when searching for a pattern string in a target string.
+ */
+ private static final int MATCH_NOT_FOUND = -1;
+
+ /**
+ * Returns whether the target string starts with the specified prefix,
+ * with respect to the UTF8_BINARY_LCASE collation. The method assumes
+ * that the prefix is already lowercased prior to method call to avoid the
+ * overhead of calling .toLowerCase() multiple times on the same prefix
string.
+ *
+ * @param target the string to be searched in
+ * @param lowercasePattern the string to be searched for
+ * @param startPos the start position for searching (in the target string)
Review Comment:
We also need to specify that `startPos` refers to UTF8 characters not bytes.
##########
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java:
##########
@@ -709,12 +774,24 @@ public void testLocate() throws SparkException {
assertLocate("大千", "test大千世界大千世界", 9, "UNICODE_CI", 9);
assertLocate("大千", "大千世界大千世界", 1, "UNICODE_CI", 1);
// Case-variable character length
+ assertLocate("\u0307", "i̇", 1, "UTF8_BINARY", 2);
+ assertLocate("\u0307", "İ", 1, "UTF8_BINARY_LCASE", 0); // != UTF8_BINARY
Review Comment:
I think the comment is wrong here.
##########
common/unsafe/src/test/java/org/apache/spark/unsafe/types/CollationSupportSuite.java:
##########
@@ -709,12 +774,24 @@ public void testLocate() throws SparkException {
assertLocate("大千", "test大千世界大千世界", 9, "UNICODE_CI", 9);
assertLocate("大千", "大千世界大千世界", 1, "UNICODE_CI", 1);
// Case-variable character length
+ assertLocate("\u0307", "i̇", 1, "UTF8_BINARY", 2);
Review Comment:
I am not sure what the intent of this test case is here. If it is for
UTF8_BINARY_LCASE then we have it again below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
