uros-db commented on code in PR #45856:
URL: https://github.com/apache/spark/pull/45856#discussion_r1565835313
##########
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:
##########
@@ -143,6 +143,41 @@ public static boolean execICU(final UTF8String l, final
UTF8String r,
* Collation-aware regexp expressions.
*/
+ public static class StringSplit {
+ public static UTF8String[] exec(final UTF8String string, final UTF8String
regex,
+ final int limit, final int collationId) {
+ CollationFactory.Collation collation =
CollationFactory.fetchCollation(collationId);
+ if (collation.supportsBinaryEquality) {
+ return execBinary(string, regex, limit);
+ } else {
+ return execLowercase(string, regex, limit);
+ }
+ }
+ public static String genCode(final String string, final String regex,
final String limit,
+ final int collationId) {
+ CollationFactory.Collation collation =
CollationFactory.fetchCollation(collationId);
+ String expr = "CollationSupport.StringSplit.exec";
+ if (collation.supportsBinaryEquality) {
+ return String.format(expr + "Binary(%s, %s, %s)", string, regex,
limit);
+ } else {
+ return String.format(expr + "Lowercase(%s, %s, %s)", string, regex,
limit);
+ }
+ }
+ public static UTF8String[] execBinary(final UTF8String string, final
UTF8String regex,
+ final int limit) {
+ return string.split(regex, limit);
+ }
+ public static UTF8String[] execLowercase(final UTF8String string, final
UTF8String regex,
+ final int limit) {
+ if (string.numBytes() != 0 && regex.numBytes() == 0) {
+ return string.split(regex, limit);
+ } else {
+ // ui flags toggle unicode case-insensitive matching
+ return string.split(UTF8String.fromString("(?ui)" + regex.toString()),
limit);
Review Comment:
I wonder if `UTF8String.concat(UTF8String.fromString("(?ui)"), regex)` would
be better
it does allocate a new byte array either way, but I think we should do it
instead
maybe even add something like `CollationAwareUTF8String.getLowercaseRegex`
that does that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
