peterxcli opened a new issue, #5495: URL: https://github.com/apache/datafusion-comet/issues/5495
### What is the problem the feature request solves? Spark and Comet do not currently use the same case-insensitive comparison for Parquet field names: - Spark resolves case-insensitive names with Java `String.equalsIgnoreCase`, which includes Unicode one-character case mappings. - Comet's native Parquet schema adapter uses Rust `eq_ignore_ascii_case`, which is ASCII-only ([current matcher](https://github.com/apache/datafusion-comet/blob/23bfd83b8ffc1071b487a90d8dff26019bb2f1bf/native/core/src/parquet/schema_adapter.rs#L546-L578)). This can turn a present physical Parquet column into a missing native column. For example, with `spark.sql.caseSensitive=false`, Spark resolves `É`/`é` and `Σ`/`σ`. Java also treats ASCII `k`/`s` as equal to the Kelvin sign `K` and long s `ſ`. A native scan that misses the physical field may incorrectly return SQL NULL instead of the stored value. PR #5407 exposes this existing mismatch for projected Variant columns, which previously stayed on Spark. It therefore adds a conservative planning fallback for affected direct Variant names ([fallback gate](https://github.com/apache/datafusion-comet/blob/23bfd83b8ffc1071b487a90d8dff26019bb2f1bf/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala#L970-L984), [regression](https://github.com/apache/datafusion-comet/blob/23bfd83b8ffc1071b487a90d8dff26019bb2f1bf/spark/src/test/scala/org/apache/comet/exec/CometNativeReaderSuite.scala#L73-L101)). Because the physical footer name is unavailable at that planning boundary, the fallback also catches some exact-name scans. ### Describe the potential solution Make case-insensitive field resolution throughout the native Parquet schema adapter match Spark's resolver, including: 1. required/logical/physical field matching; 2. duplicate-name detection; 3. physical column-index remapping; and 4. missing-column/default-value detection. The implementation should use one shared comparison contract rather than fixing only the Variant cast path. After native matching has Spark parity, remove #5407's conservative Scala fallback. Definition of done: - `É`/`é`, `Σ`/`σ`, `K`/`k`, and `ſ`/`s` resolve to the same physical columns as Spark. - Exact-name, ordinary ASCII-case, and case-sensitive scans retain their current behavior. - Ambiguous/duplicate names fail or fall back consistently with Spark rather than selecting an arbitrary field. - The behavior is covered for Variant and at least one existing non-Variant native type. - The fallback in `CometScanRule` is removed. ### Additional context - Review reproduction: https://github.com/apache/datafusion-comet/pull/5407#discussion_r3869971570 - Native Variant roadmap: #5438 - This concerns Parquet schema field names. It is separate from #5474, which tracks object-key ordering inside encoded Variant values. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
