spark-test commented on code in PR #59887:
URL: https://github.com/apache/spark/pull/59887#discussion_r4406533420
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala:
##########
@@ -587,4 +587,107 @@ class HiveOrcSourceSuite extends OrcSuite with
TestHiveSingleton {
}
}
}
+
+ test("SPARK-60642: Hive ORC writes and reads collated strings") {
+ // The Hive type parser rejects `string collate <name>`, which
`catalogString` produces for a
+ // collated string, so the serializer must not put a collation in the type
string.
+ val collations = Seq("UTF8_BINARY", "UTF8_LCASE", "UNICODE_CI")
+ withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+ collations.foreach { collation =>
+ withTable("t") {
+ sql(s"CREATE TABLE t (i INT, c STRING COLLATE $collation) USING ORC")
+ sql("INSERT INTO t SELECT 1, 'a'")
+ checkAnswer(spark.table("t"), Row(1, "a"))
+ // The collation is kept by the catalog, not by the file.
+ assert(spark.table("t").schema("c").dataType ===
StringType(collation))
+ checkAnswer(
+ sql("SELECT i FROM t WHERE c = 'A'"),
+ if (collation == "UTF8_BINARY") Nil else Seq(Row(1)))
+ }
+ }
+ }
+ }
+
+ test("SPARK-60642: Hive ORC writes and reads collated strings in nested
types") {
+ withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+ withTable("t") {
+ sql(
+ """CREATE TABLE t (
+ | a ARRAY<STRING COLLATE UNICODE_CI>,
+ | s STRUCT<f: STRING COLLATE UTF8_LCASE, g: ARRAY<STRING COLLATE
UNICODE_CI>>,
+ | m MAP<STRING, STRING COLLATE UNICODE_CI>,
+ | b MAP<STRING COLLATE UTF8_BINARY, ARRAY<STRING COLLATE
UNICODE_CI>>
Review Comment:
**Nit (P3):** The UTF8_BINARY collation on this key is gone before the
write. `b` also has a non-UTF8_BINARY path (`b.value.element`), so
`StructField.dataTypeJsonValue` serializes the whole field type through
`removeCollations` and records only the UNICODE_CI path in `__COLLATIONS`. When
`HiveExternalCatalog` restores the schema, the key comes back as the plain
`StringType`, whose `catalogString` is `string`. So the INSERT hands
`OrcSerializer` an uncollated key, and nothing in the suite writes a key type
that needs rewriting (the nanosecond-timestamp tests only use map values). The
production `MapType` arm does recurse into keys today. But if that regressed, a
DataFrame write with a `MapType(StringType("UTF8_BINARY"), ...)` key, or a
non-binary key under `spark.sql.collation.allowInMapKeys=true`, would fail in
Hive's type parser again while every test still passed.
One way to cover it: add a map column keyed by `StringType("UTF8_BINARY")`
to the DataFrame-write test's schema, and `MAP<STRING, ...>` to its expected
file schema. That route keeps the explicit instance all the way to the
serializer, the same way the test's top-level `b` column does.
**Verification:**
- **Regression:** If the Hive ORC serializer passes map key types through
unchanged, the strengthened map-key coverage fails with Hive's type-parser
error. The current implementation passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
