This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ca1514a2b4e7 [SPARK-60642][SQL] Support collated strings in the Hive
ORC data source writer
ca1514a2b4e7 is described below
commit ca1514a2b4e72f9529871f34d09881cd4921df82
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 28 19:20:34 2026 -0700
[SPARK-60642][SQL] Support collated strings in the Hive ORC data source
writer
### What changes were proposed in this pull request?
This PR aims to remove the collations from the Hive type strings that
`OrcSerializer` of the Hive ORC data source (`spark.sql.orc.impl=hive`) builds.
`OrcSerializer` builds both the `columns.types` property of `OrcSerde` and
the struct `ObjectInspector` from `catalogString`. A collated string prints as
`string collate <name>`, and `CHAR`/`VARCHAR` with a collation print as
`char(n) collate <name>` and `varchar(n) collate <name>`. Hive's
`TypeInfoUtils` rejects these type strings.
`OrcSerializer.toHiveCompatibleDataType` already converts the Spark types
that Hive cannot express (nanosecond timestamps) and recurses into `STRUCT`,
`ARRAY`, and `MAP`. This PR adds a `StringType` case that calls
`StringHelper.removeCollation`, so a collated `STRING`, and the one nested in
`STRUCT`, `ARRAY`, and `MAP`, becomes `string`, and a collated
`CHAR(n)`/`VARCHAR(n)` keeps only its length. ORC files only store `string` for
these columns, and the collation is kept by the catal [...]
The read path of this data source does not build any type string, so it
needs no change: `OrcFileOperator.readSchema` reads the file schema,
`setRequiredColumns` uses the column names and ordinals, and `unwrapOrcStructs`
unwraps by `ObjectInspector`.
### Why are the changes needed?
Writing a collated string through the Hive ORC data source fails in the
task.
```sql
SET spark.sql.orc.impl=hive;
CREATE TABLE t (i INT, c STRING COLLATE UNICODE_CI) USING ORC;
INSERT INTO t SELECT 1, 'a';
```
```
java.lang.IllegalArgumentException: Error: type expected at the position 4
of 'int:string collate UNICODE_CI' but 'string collate UNICODE_CI' is found.
```
The same failure happens for `df.write.orc(...)` with a collated column,
for `STORED AS ORC` tables with `spark.sql.hive.convertMetastoreOrc=true`, and
for collated columns nested in `STRUCT`, `ARRAY`, and `MAP`. Collated
`CHAR`/`VARCHAR` fail with `Error: ',', ':', or ';' expected at position 7 from
'char(3) collate UNICODE_CI:...'`.
An explicit `UTF8_BINARY` collation, such as `StringType("UTF8_BINARY")`,
is also affected, because it prints as `string collate UTF8_BINARY`, unlike the
default `StringType`.
### Does this PR introduce _any_ user-facing change?
Yes. Writing collated strings through the Hive ORC data source works now.
The failure exists since Apache Spark
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19),
which introduced the collations.
### How was this patch tested?
Pass the CIs with the newly added test cases in `HiveOrcSourceSuite`. All
of them fail without this fix.
- a top-level column with `UTF8_BINARY`, `UTF8_LCASE`, and `UNICODE_CI`,
including the catalog collation and a collation-aware predicate
- collated strings nested in `STRUCT`, `ARRAY`, and `MAP`
- `DataFrame` write and read, including `StringType("UTF8_BINARY")` (also
as a `MAP` key) and the plain `string` in the ORC file schema
- collated `CHAR`/`VARCHAR` by default and with
`spark.sql.preserveCharVarcharTypeInfo` and
`spark.sql.charVarchar.standardSemantics.enabled`
- a `STORED AS ORC` table with `spark.sql.hive.convertMetastoreOrc=true`
```
build/sbt -Phive 'hive/testOnly
org.apache.spark.sql.hive.orc.HiveOrcSourceSuite
org.apache.spark.sql.hive.orc.HiveOrcQuerySuite
org.apache.spark.sql.hive.orc.HiveOrcPartitionDiscoverySuite
org.apache.spark.sql.hive.orc.OrcHadoopFsRelationSuite'
build/sbt -Phive hive/scalastyle hive/Test/scalastyle
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
Closes #59887 from dongjoon-hyun/SPARK-60642.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../apache/spark/sql/hive/orc/OrcFileFormat.scala | 3 +
.../spark/sql/hive/orc/HiveOrcSourceSuite.scala | 106 +++++++++++++++++++++
2 files changed, 109 insertions(+)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
index 255f2fdc5b2e..0e75329400f7 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
@@ -270,6 +270,9 @@ private[orc] class OrcSerializer(dataSchema: StructType,
conf: Configuration)
dataType match {
case _: TimestampNTZNanosType | _: TimestampLTZNanosType =>
TimestampType
+ // Hive types have no collations, and its type parser rejects `string
collate <name>`.
+ case s: StringType =>
+ StringHelper.removeCollation(s)
case StructType(fields) =>
StructType(fields.map(f => f.copy(dataType =
toHiveCompatibleDataType(f.dataType))))
case ArrayType(elementType, containsNull) =>
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
index 3f6825cbf39a..ca897c7b9f37 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
@@ -587,4 +587,110 @@ class HiveOrcSourceSuite extends OrcSuite with
TestHiveSingleton {
}
}
}
+
+ test("SPARK-60642: Hive ORC writes and reads collated strings") {
+ // The Hive type parser rejects `string collate <name>`, which
`catalogString` produces for a
+ // collated string, so the serializer must not put a collation in the type
string.
+ val collations = Seq("UTF8_BINARY", "UTF8_LCASE", "UNICODE_CI")
+ withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+ collations.foreach { collation =>
+ withTable("t") {
+ sql(s"CREATE TABLE t (i INT, c STRING COLLATE $collation) USING ORC")
+ sql("INSERT INTO t SELECT 1, 'a'")
+ checkAnswer(spark.table("t"), Row(1, "a"))
+ // The collation is kept by the catalog, not by the file.
+ assert(spark.table("t").schema("c").dataType ===
StringType(collation))
+ checkAnswer(
+ sql("SELECT i FROM t WHERE c = 'A'"),
+ if (collation == "UTF8_BINARY") Nil else Seq(Row(1)))
+ }
+ }
+ }
+ }
+
+ test("SPARK-60642: Hive ORC writes and reads collated strings in nested
types") {
+ withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+ withTable("t") {
+ sql(
+ """CREATE TABLE t (
+ | a ARRAY<STRING COLLATE UNICODE_CI>,
+ | s STRUCT<f: STRING COLLATE UTF8_LCASE, g: ARRAY<STRING COLLATE
UNICODE_CI>>,
+ | m MAP<STRING, STRING COLLATE UNICODE_CI>,
+ | b MAP<STRING COLLATE UTF8_BINARY, ARRAY<STRING COLLATE
UNICODE_CI>>
+ |) USING ORC""".stripMargin)
+ sql(
+ """INSERT INTO t SELECT
+ | array('a'),
+ | named_struct('f', 'b', 'g', array('c')),
+ | map('k', 'd'),
+ | map('k', array('e'))""".stripMargin)
+ checkAnswer(
+ spark.table("t"),
+ Row(Seq("a"), Row("b", Seq("c")), Map("k" -> "d"), Map("k" ->
Seq("e"))))
+ checkAnswer(sql("SELECT a[0], s.f, s.g[0], m['k'], b['k'][0] FROM t"),
+ Row("a", "b", "c", "d", "e"))
+ }
+ }
+ }
+
+ test("SPARK-60642: Hive ORC data source writes collated strings through
DataFrame") {
+ withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ // `StringType("UTF8_BINARY")` prints as `string collate UTF8_BINARY`
unlike `StringType`.
+ // A DDL `COLLATE UTF8_BINARY` becomes the default `StringType`, so
only this test covers
+ // an explicit `UTF8_BINARY` map key.
+ val schema = new StructType()
+ .add("b", StringType("UTF8_BINARY"))
+ .add("c", StringType("UNICODE_CI"))
+ .add("a", ArrayType(StringType("UTF8_LCASE")))
+ .add("s", new StructType().add("f", StringType("UNICODE_CI")))
+ .add("m", MapType(StringType("UTF8_BINARY"),
StringType("UNICODE_CI")))
+ val rows = Seq(Row("a", "b", Seq("c"), Row("d"), Map("k" -> "v")))
+ val df = spark.createDataFrame(spark.sparkContext.parallelize(rows),
schema)
+ df.write.orc(path)
+ // The file stores plain strings.
+ assert(spark.read.orc(path).schema === StructType.fromDDL(
+ "b STRING, c STRING, a ARRAY<STRING>, s STRUCT<f: STRING>, m
MAP<STRING, STRING>"))
+ // A read with a collated schema decodes the same values.
+ checkAnswer(spark.read.schema(df.schema).orc(path), df)
+ }
+ }
+ }
+
+ test("SPARK-60642: Hive ORC writes and reads collated CHAR and VARCHAR") {
+ // CHAR and VARCHAR with a collation print as `char(3) collate <name>`
under the last two
+ // confs, and as `string collate <name>` by default.
+ Seq(
+ Seq.empty[(String, String)],
+ Seq(SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "true"),
+ Seq(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true")).foreach {
confs =>
+ withSQLConf((SQLConf.ORC_IMPLEMENTATION.key -> "hive") +: confs: _*) {
+ withTable("t") {
+ sql(
+ """CREATE TABLE t (
+ | c CHAR(3) COLLATE UNICODE_CI,
+ | v VARCHAR(3) COLLATE UTF8_LCASE,
+ | b CHAR(3) COLLATE UTF8_BINARY,
+ | a ARRAY<VARCHAR(3) COLLATE UNICODE_CI>
+ |) USING ORC""".stripMargin)
+ sql("INSERT INTO t SELECT 'a', 'b', 'c', array('d')")
+ checkAnswer(spark.table("t"), Row("a ", "b", "c ", Seq("d")))
+ }
+ }
+ }
+ }
+
+ test("SPARK-60642: Hive ORC writes collated strings through a converted Hive
serde table") {
+ withSQLConf(
+ SQLConf.ORC_IMPLEMENTATION.key -> "hive",
+ HiveUtils.CONVERT_METASTORE_ORC.key -> "true") {
+ withTable("t") {
+ sql("CREATE TABLE t (i INT, c STRING COLLATE UNICODE_CI) STORED AS
ORC")
+ sql("INSERT INTO t SELECT 1, 'a'")
+ checkAnswer(spark.table("t"), Row(1, "a"))
+ checkAnswer(sql("SELECT i FROM t WHERE c = 'A'"), Row(1))
+ }
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
