andygrove commented on issue #5495: URL: https://github.com/apache/datafusion-comet/issues/5495#issuecomment-5482707655
I reproduced this on main (eacaa9cab) with a plain bigint column, so it's live today and not Variant-specific: write Parquet with a top-level column `É`, declare the table as `é`, set `spark.sql.caseSensitive=false`, and the native scan silently returns all NULLs where Spark returns the data. Same for `é`→`É`, `Ä`→`ä`, `Д`→`д`, `Σ`→`σ`, and `K` (U+212A)→`k`. ASCII is fine in both directions. Two corrections to the description. First, the `ſ`/`s` item in the definition of done looks wrong to me — Spark's Parquet clipping doesn't use `equalsIgnoreCase`. `ParquetReadSupport.clipParquetGroupFields` groups on `getName.toLowerCase(Locale.ROOT)` and looks up the same way; only the analyzer-level resolver uses `equalsIgnoreCase`. Since `"ſ".toLowerCase(ROOT)` is `ſ`, Spark returns NULL there too and Comet already agrees (I got NULL from both). Wouldn't Rust's `to_lowercase()` be the contract we want, fixing `É`/`é`, `Σ`/`σ` and Kelvin-`K`/`k` without introducing a new divergence on `ſ`/`s`? Second, nested struct fields don't appear to be affected — `struct<É:long>` read as `struct<é:long>` came back correct on the native scan, even with the declared fields ordered differently from the file. That would confine the fix to the top-level `eq_ignore_ascii_case` sites in `schema_adapter.rs`. One note on scope: `spark.comet.scan.impl` is gone on main so `native_datafusion` is the only Parquet path, and #5407 is still open, so nothing is shielding this today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
