Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-17 Thread via GitHub
mkaravel commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1605449462 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -118,76 +119,433 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-17 Thread via GitHub
mkaravel commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r160575 ## connector/connect/common/src/main/protobuf/spark/connect/types.proto: ## @@ -101,7 +101,7 @@ message DataType { message String { uint32

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-17 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1604917892 ## connector/connect/common/src/main/protobuf/spark/connect/types.proto: ## @@ -101,7 +101,7 @@ message DataType { message String { uint32

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-17 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1604916810 ## common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala: ## @@ -152,4 +231,168 @@ class CollationFactorySuite extends AnyFunSuite

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-17 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1604679971 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-17 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1604678110 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -245,29 +599,26 @@ public static StringSearch getStringSearch( *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-17 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1604672125 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-17 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1604667645 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-17 Thread via GitHub
nikolamand-db commented on PR #46180: URL: https://github.com/apache/spark/pull/46180#issuecomment-2116855540 @mkaravel @dbatomic please review again, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-15 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1601383798 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +118,490 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-15 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1601366046 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-15 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1601363774 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-15 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1601334716 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -245,29 +599,26 @@ public static StringSearch getStringSearch(

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-15 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1601309336 ## connector/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala: ## @@ -80,7 +81,7 @@ object DataTypeProtoConverter {

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-15 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1601306785 ## connector/connect/common/src/main/protobuf/spark/connect/types.proto: ## @@ -101,7 +101,7 @@ message DataType { message String { uint32

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-15 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1601298545 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -772,12 +772,17 @@ object SQLConf { " produced by a builtin function

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-09 Thread via GitHub
mihailom-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1595382023 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-09 Thread via GitHub
mihailom-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1595376570 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-09 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1595124256 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-09 Thread via GitHub
uros-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1595090780 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-09 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1595078750 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-08 Thread via GitHub
mkaravel commented on PR #46180: URL: https://github.com/apache/spark/pull/46180#issuecomment-2101717141 How do we name a trailing-space-insensitive collation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-08 Thread via GitHub
mkaravel commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1594835020 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-08 Thread via GitHub
mkaravel commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1594834488 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-08 Thread via GitHub
mkaravel commented on PR #46180: URL: https://github.com/apache/spark/pull/46180#issuecomment-2101713318 > > User can use collation specifiers in any order except of locale which is mandatory and must go first. There is a one-to-one mapping between collation ids and collation names defined

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-08 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1594153009 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-08 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1593910153 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-08 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1593900492 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-08 Thread via GitHub
uros-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1593724507 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-08 Thread via GitHub
uros-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1593716451 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,445 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1593130617 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1593127364 ## common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala: ## @@ -30,31 +33,101 @@ import

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
stefankandic commented on PR #46180: URL: https://github.com/apache/spark/pull/46180#issuecomment-2098821075 will we have to do the same for pyspark - as `StringType` there only supports 4 initial collations? -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1592505295 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1592489108 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1592463454 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1592461113 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1592445524 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1592442858 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -217,17 +581,12 @@ public static StringSearch getStringSearch( *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1592441389 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1592440194 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1592436706 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,438 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-05-07 Thread via GitHub
dbatomic commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1592428205 ## common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala: ## @@ -30,31 +33,101 @@ import

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-04-26 Thread via GitHub
bart-samwel commented on PR #46180: URL: https://github.com/apache/spark/pull/46180#issuecomment-2079429882 > User can use collation specifiers in any order except of locale which is mandatory and must go first. There is a one-to-one mapping between collation ids and collation names

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-04-25 Thread via GitHub
uros-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1579147729 ## common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala: ## @@ -152,4 +219,147 @@ class CollationFactorySuite extends AnyFunSuite

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-04-25 Thread via GitHub
nikolamand-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1579098122 ## common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala: ## @@ -152,4 +219,147 @@ class CollationFactorySuite extends

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-04-25 Thread via GitHub
mihailom-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1579089752 ## common/unsafe/src/test/scala/org/apache/spark/unsafe/types/CollationFactorySuite.scala: ## @@ -152,4 +219,147 @@ class CollationFactorySuite extends AnyFunSuite

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-04-25 Thread via GitHub
mihailom-db commented on code in PR #46180: URL: https://github.com/apache/spark/pull/46180#discussion_r1579005770 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -117,76 +119,422 @@ public Collation( } /** - *

Re: [PR] [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers [spark]

2024-04-24 Thread via GitHub
nikolamand-db commented on PR #46180: URL: https://github.com/apache/spark/pull/46180#issuecomment-2075286793 Please review collation team @dbatomic @stefankandic @uros-db @mihailom-db @stevomitric. -- This is an automated message from the Apache Git Service. To respond to the message,