dongjoon-hyun commented on code in PR #59887:
URL: https://github.com/apache/spark/pull/59887#discussion_r4406696002
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala:
##########
@@ -587,4 +587,107 @@ class HiveOrcSourceSuite extends OrcSuite with
TestHiveSingleton {
}
}
}
+
+ test("SPARK-60642: Hive ORC writes and reads collated strings") {
+ // The Hive type parser rejects `string collate <name>`, which
`catalogString` produces for a
+ // collated string, so the serializer must not put a collation in the type
string.
+ val collations = Seq("UTF8_BINARY", "UTF8_LCASE", "UNICODE_CI")
+ withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+ collations.foreach { collation =>
+ withTable("t") {
+ sql(s"CREATE TABLE t (i INT, c STRING COLLATE $collation) USING ORC")
+ sql("INSERT INTO t SELECT 1, 'a'")
+ checkAnswer(spark.table("t"), Row(1, "a"))
+ // The collation is kept by the catalog, not by the file.
+ assert(spark.table("t").schema("c").dataType ===
StringType(collation))
+ checkAnswer(
+ sql("SELECT i FROM t WHERE c = 'A'"),
+ if (collation == "UTF8_BINARY") Nil else Seq(Row(1)))
+ }
+ }
+ }
+ }
+
+ test("SPARK-60642: Hive ORC writes and reads collated strings in nested
types") {
+ withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+ withTable("t") {
+ sql(
+ """CREATE TABLE t (
+ | a ARRAY<STRING COLLATE UNICODE_CI>,
+ | s STRUCT<f: STRING COLLATE UTF8_LCASE, g: ARRAY<STRING COLLATE
UNICODE_CI>>,
+ | m MAP<STRING, STRING COLLATE UNICODE_CI>,
+ | b MAP<STRING COLLATE UTF8_BINARY, ARRAY<STRING COLLATE
UNICODE_CI>>
Review Comment:
Thank you for the review, @spark-test.
You are right: `STRING COLLATE UTF8_BINARY` in the DDL reaches the writer as
the default `StringType`, so the nested-types test does not exercise the
`MapType` key arm. In 8d63a49a03b, the DataFrame-write test has a `MAP<STRING
COLLATE UTF8_BINARY, STRING COLLATE UNICODE_CI>` column built from
`StringType("UTF8_BINARY")`, and the expected file schema now includes `m
MAP<STRING, STRING>`.
One correction to the summary: `SPARK-58814: parallel schema merging ...`
lives in `OrcSourceSuite`, which `HiveOrcSourceSuite` does not extend, so it
does not run on the Hive ORC writer. The `CHAR(n)`/`VARCHAR(n)` length is
preserved by `StringHelper.removeCollation` itself, and the new collated
`CHAR`/`VARCHAR` test checks the values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
