comphead opened a new pull request, #5602:
URL: https://github.com/apache/datafusion-comet/pull/5602
## Rationale for this change
The Parquet schema adapter matched field names case-insensitively with
`eq_ignore_ascii_case`, which folds only ASCII `A-Z`/`a-z`. Spark's
`ParquetReadSupport.clipParquetGroupFields` folds with
`name.toLowerCase(Locale.ROOT)` (full Unicode) for both matching and duplicate
detection. Under Comet's default (`spark.sql.caseSensitive=false`), a column
differing only by non-ASCII case (file `MÜNCHEN`, query `münchen`) silently
read back as `null` while Spark resolved it.
The nested-struct path (`parquet_support.rs`) and the projection path
(`parquet_exec.rs`) already used `to_lowercase()`, so `schema_adapter.rs` was
the lone ASCII-only outlier, making top-level and nested resolution
inconsistent.
## What changes are included in this PR?
In `native/core/src/parquet/schema_adapter.rs`:
- Add a `names_match(a, b, case_sensitive)` helper: exact compare when
case-sensitive, otherwise `to_lowercase()` compare. `str::to_lowercase` is the
Unicode-aware equivalent of Java's `toLowerCase(Locale.ROOT)`.
- Route all 7 `eq_ignore_ascii_case` sites through it: name match in
`remap_physical_schema`, `check_column_duplicate`, the
`wrap_all_type_mismatches` field/index lookups, and the missing-column check in
`replace_missing_with_defaults`.
- Fix a latent bug: the `generateFakeColumnName` block (unmatched field-id
guard) folded case even in case-sensitive mode; it now respects
`case_sensitive`.
- Collapse the redundant `if case_sensitive { … } else { … }` branches.
No behavior change in case-sensitive mode. Nested-struct name folding was
already Unicode-aware, so this only aligns top-level matching with it and with
Spark.
## How are these changes tested?
Rust unit tests in `schema_adapter.rs`:
- `parquet_case_insensitive_unicode_name_match` — file `MÜNCHEN` resolves to
`münchen`.
- `parquet_duplicate_fields_case_insensitive_unicode` — `Ω`/`ω` fold to a
duplicate-field error.
- `parquet_case_insensitive_unicode_nested_struct_field` — nested `GRÜN`
resolves to `grün` (carries a `NULL`).
- `parquet_case_insensitive_unicode_top_level_and_nested_struct` —
`CAFÉ`/`RÉSUMÉ` resolve to `café`/`résumé`.
Scala end-to-end test in `CometNativeReaderSuite` (`native reader
case-insensitive resolution for top-level and nested struct fields`): ASCII and
non-ASCII names, at top level and inside a struct, asserting a
`CometNativeScanExec` and matching Spark under `spark.sql.caseSensitive=false`
(fields resolve) and `true` (read back `null`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
