dwsmith1983 opened a new issue, #5540: URL: https://github.com/apache/datafusion-comet/issues/5540
### Describe the bug The native Iceberg scan builds its schema adapter with no JVM case tables and case sensitivity hardcoded to false (iceberg_scan.rs:226), so case-insensitive name matching falls back to Rust's str::to_lowercase. Rust ships newer Unicode data than the JDK, so codepoints that gained lowercase mappings in Unicode 14+ fold together in Rust while Java keeps them distinct. A table with two such columns is perfectly legal to Spark, but Comet's adapter folds them into a duplicate and the query dies with _LEGACY_ERROR_TEMP_2093 Found duplicate field(s) ... in case-insensitive mode, an error Spark itself would never raise. The same divergence class was fixed for the parquet and Delta scan paths in #5365 by shipping the running JVM's case data to native (JvmCaseTables); the Iceberg path is the remaining consumer of the fallback (there's a code comment marking it). ### Steps to reproduce Columns are U+10570 (VITHKUQI CAPITAL LETTER A) and U+10597 (its lowercase), a Unicode-14 case pair that JDK 17 treats as distinct. They may render as boxes depending on your font; copy-paste still works, or use the constructed variant below. ```sql CREATE TABLE hadoop_catalog.db.uni_case (`U+10570` INT, `U+10597` INT) USING iceberg; INSERT INTO hadoop_catalog.db.uni_case VALUES (1, 2); SELECT `U+10570`, `U+10597` FROM hadoop_catalog.db.uni_case; ``` Font-independent version (spark-shell): ```scala val upper = new String(Character.toChars(0x10570)) val lower = new String(Character.toChars(0x10597)) spark.sql(s"CREATE TABLE hadoop_catalog.db.uni_case (`$upper` INT, `$lower` INT) USING iceberg") spark.sql(s"INSERT INTO hadoop_catalog.db.uni_case VALUES (1, 2)") spark.sql(s"SELECT `$upper`, `$lower` FROM hadoop_catalog.db.uni_case").collect() ``` ### Expected behavior The query returns [1,2] under Comet exactly as under stock Spark. The adapter should also honor spark.sql.caseSensitive rather than hardcoding insensitive matching. ### Additional context Fix direction: thread the same case tables NativeScanCommon carries through the Iceberg scan config, and pass the real case-sensitivity flag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
