dongjoon-hyun opened a new pull request, #59799:
URL: https://github.com/apache/spark/pull/59799
[SPARK-60565][SQL] Fix reading and writing collated strings in Hive serde
tables
### What changes were proposed in this pull request?
This PR aims to remove the collations from the Hive column types in
`HiveClientImpl.toHiveColumn`, which converts Spark columns into Hive columns.
- Collated `STRING` types, including the nested ones in `ARRAY`, `STRUCT`,
and `MAP`, become the regular `string` type via
`SchemaUtils.replaceCollatedStringWithString`, like `HiveExternalCatalog` does
before calling Hive metastore (SPARK-47873).
- The raw `CHAR`/`VARCHAR` type string in the field metadata, e.g. `char(3)
collate UTF8_LCASE`, loses its ` collate <name>` clauses. The type string is
not parsed again, because the parsing depends on the session configurations.
For example, `struct<a:char(3),b:timestamp>` would become
`struct<a:char(3),b:timestamp_ntz>` with
`spark.sql.timestampType=TIMESTAMP_NTZ`.
### Why are the changes needed?
Hive has no collations. `HiveExternalCatalog` removes the collations before
calling Hive metastore, so a Hive serde table with collated string columns is
created with `string` columns in Hive metastore, and Spark restores the
collations from the table properties. However, `InsertIntoHiveTable`,
`InsertIntoHiveDirCommand`, and `HiveTableScanExec` build the Hive table from
the Spark schema via `HiveClientImpl.toHiveTable`, and `toHiveColumn` uses
`catalogString`, e.g. `string collate UNICODE_CI`. Hive SerDe fails to parse
such column types (`columns.types`), so Spark fails to read and write the table.
```sql
CREATE TABLE t (i INT, c STRING COLLATE UNICODE_CI) STORED AS TEXTFILE;
INSERT INTO t VALUES (1, 'a');
```
```
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID
0) (localhost executor driver): java.lang.IllegalArgumentException: Error: type
expected at the position 4 of 'int:string collate UNICODE_CI' but 'string
collate UNICODE_CI' is found.
```
Hive SerDe fails in the same way with
- `SELECT`, e.g. of the rows that Hive wrote,
- `CREATE TABLE ... AS SELECT` and `DataFrameWriter.saveAsTable` with
`format("hive")`, which also drop the new table,
- `INSERT OVERWRITE [LOCAL] DIRECTORY` with Hive SerDe, e.g. `ROW FORMAT
DELIMITED` or `STORED AS TEXTFILE`.
It affects the `TEXTFILE`, `SEQUENCEFILE`, `RCFILE`, and `AVRO` tables, and
the `ORC` and `PARQUET` tables when `spark.sql.hive.convertMetastoreOrc` and
`spark.sql.hive.convertMetastoreParquet` are `false`. An explicit `COLLATE
UTF8_BINARY` also fails because its `catalogString` is `string collate
UTF8_BINARY`. This has been the behavior since such tables can be created
(SPARK-47873, Apache Spark 4.0.0).
For collated `CHAR`/`VARCHAR` types (SPARK-54870), the raw type string in
the field metadata keeps the collations even after `HiveExternalCatalog`
removes them from the data types. So the Hive metastore calls fail too, e.g.
`ALTER TABLE ... ADD COLUMNS` on a table with a `CHAR(3) COLLATE UTF8_LCASE`
column, and `CREATE TABLE ... STORED AS PARQUET AS SELECT * FROM` such a table.
### Does this PR introduce _any_ user-facing change?
Yes. Spark reads and writes Hive serde tables with collated string columns,
and `ALTER TABLE ... ADD COLUMNS` works on tables with collated
`CHAR`/`VARCHAR` columns. Spark keeps the collations of the columns as before,
and Hive reads the tables with the regular string types.
Hive ORC and Parquet tables that Spark converts to the native data sources
(`spark.sql.hive.convertMetastoreOrc` and
`spark.sql.hive.convertMetastoreParquet`, both `true` by default) don't use
Hive SerDe, so this PR doesn't change them.
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `HiveSerDeReadWriteSuite`: reads and writes collated `STRING`, `ARRAY`,
`STRUCT`, `MAP`, `CHAR`, and `VARCHAR` columns via Hive and Spark, and CTAS for
all Hive serde formats.
- `InsertSuite`: `INSERT OVERWRITE LOCAL DIRECTORY` with collated strings.
- `AlterTableAddColumnsSuite`: `ALTER TABLE ... ADD COLUMNS` on a table with
collated `CHAR` and `VARCHAR` columns.
- `HiveClientImplSuite`: `toHiveColumn` removes the collations, and keeps
`timestamp` with `spark.sql.timestampType=TIMESTAMP_NTZ`.
The new tests in `HiveSerDeReadWriteSuite` and `InsertSuite` fail without
this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
