wayne-x-wang opened a new pull request, #60204:
URL: https://github.com/apache/spark/pull/60204
### What changes were proposed in this pull request?
Remove collation declarations when converting Spark string types to ORC
schema strings in `OrcUtils.getOrcSchemaString`, using
`StringHelper.removeCollation`.
Add regression tests for catalog table reads and explicit-schema file reads.
The tests cover V1/V2 scans, vectorized and non-vectorized readers, strings
nested in structs, arrays and map values, and collation-aware predicates.
### Why are the changes needed?
Reading an ORC table with a `STRING COLLATE UTF8_LCASE` column fails because
Spark passes a schema containing `string collate UTF8_LCASE` to ORC's type
parser.
Both vectorized and non-vectorized native ORC readers encounter this error.
The conversion removes collation declarations from the ORC schema string
while retaining Spark's logical collation information. Using
`StringHelper.removeCollation` also preserves CHAR/VARCHAR lengths.
### Does this PR introduce _any_ user-facing change?
Yes. Affected ORC tables can be read successfully.
For a UTF8_LCASE column containing 'a', 'A' and 'b', reading the column
returns all three values, and filtering with `k = 'a'` returns both 'a' and 'A'.
### How was this patch tested?
The original table-read regression failed with both vectorized settings
before the production fix and passed after it.
Local validation:
- Related ORC source, ORC query and collation suites passed: 352 tests, with
4 ignored.
- After adding explicit V1/V2 scan assertions, all 8 targeted regression
tests passed.
- SQL production and test scalastyle checks passed.
Commands:
```bash
build/sbt \
'sql/testOnly *OrcSourceV1Suite *OrcSourceV2Suite -- -z SPARK-60123'
build/sbt \
'sql/testOnly *OrcSourceV1Suite *OrcSourceV2Suite *OrcV1QuerySuite
*OrcV2QuerySuite *CollationSuite'
build/sbt sql/scalastyle sql/Test/scalastyle
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6 Astra)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
