dongjoon-hyun opened a new pull request, #59887:
URL: https://github.com/apache/spark/pull/59887
### What changes were proposed in this pull request?
This PR aims to remove the collations from the Hive type strings that
`OrcSerializer` of the Hive ORC data source (`spark.sql.orc.impl=hive`) builds.
`OrcSerializer` builds both the `columns.types` property of `OrcSerde` and
the struct `ObjectInspector` from `catalogString`. A collated string prints as
`string collate <name>`, and `CHAR`/`VARCHAR` with a collation print as
`char(n) collate <name>` and `varchar(n) collate <name>`. Hive's
`TypeInfoUtils` rejects these type strings.
`OrcSerializer.toHiveCompatibleDataType` already converts the Spark types
that Hive cannot express (nanosecond timestamps) and recurses into `STRUCT`,
`ARRAY`, and `MAP`. This PR adds a `StringType` case that calls
`StringHelper.removeCollation`, so a collated `STRING`, and the one nested in
`STRUCT`, `ARRAY`, and `MAP`, becomes `string`, and a collated
`CHAR(n)`/`VARCHAR(n)` keeps only its length. ORC files only store `string` for
these columns, and the collation is kept by the catalog or by the
user-specified schema, as in the native ORC data source.
The read path of this data source does not build any type string, so it
needs no change: `OrcFileOperator.readSchema` reads the file schema,
`setRequiredColumns` uses the column names and ordinals, and `unwrapOrcStructs`
unwraps by `ObjectInspector`.
### Why are the changes needed?
Writing a collated string through the Hive ORC data source fails in the task.
```sql
SET spark.sql.orc.impl=hive;
CREATE TABLE t (i INT, c STRING COLLATE UNICODE_CI) USING ORC;
INSERT INTO t SELECT 1, 'a';
```
```
java.lang.IllegalArgumentException: Error: type expected at the position 4
of 'int:string collate UNICODE_CI' but 'string collate UNICODE_CI' is found.
```
The same failure happens for `df.write.orc(...)` with a collated column, for
`STORED AS ORC` tables with `spark.sql.hive.convertMetastoreOrc=true`, and for
collated columns nested in `STRUCT`, `ARRAY`, and `MAP`. Collated
`CHAR`/`VARCHAR` fail with `Error: ',', ':', or ';' expected at position 7 from
'char(3) collate UNICODE_CI:...'`.
An explicit `UTF8_BINARY` collation, such as `StringType("UTF8_BINARY")`, is
also affected, because it prints as `string collate UTF8_BINARY`, unlike the
default `StringType`.
### Does this PR introduce _any_ user-facing change?
Yes. Writing collated strings through the Hive ORC data source works now.
The failure exists since Apache Spark
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19),
which introduced the collations.
### How was this patch tested?
Pass the CIs with the newly added test cases in `HiveOrcSourceSuite`. All of
them fail without this fix.
- a top-level column with `UTF8_BINARY`, `UTF8_LCASE`, and `UNICODE_CI`,
including the catalog collation and a collation-aware predicate
- collated strings nested in `STRUCT`, `ARRAY`, and `MAP`
- `DataFrame` write and read, including `StringType("UTF8_BINARY")` and the
plain `string` in the ORC file schema
- collated `CHAR`/`VARCHAR` by default and with
`spark.sql.preserveCharVarcharTypeInfo` and
`spark.sql.charVarchar.standardSemantics.enabled`
- a `STORED AS ORC` table with `spark.sql.hive.convertMetastoreOrc=true`
```
build/sbt -Phive 'hive/testOnly
org.apache.spark.sql.hive.orc.HiveOrcSourceSuite
org.apache.spark.sql.hive.orc.HiveOrcQuerySuite
org.apache.spark.sql.hive.orc.HiveOrcPartitionDiscoverySuite
org.apache.spark.sql.hive.orc.OrcHadoopFsRelationSuite'
build/sbt -Phive hive/scalastyle hive/Test/scalastyle
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
