andygrove opened a new issue, #7293: URL: https://github.com/apache/datafusion-comet/issues/7293
### What is the problem the feature request solves? Comet 1.2.0 is branched and 1.3.0 is due 2026-11-20. This epic collects the Iceberg work that should land in that window. It has two goals: 1. Turn native Iceberg writes (`spark.comet.write.iceberg.enabled`) back on by default on `main`, so they run as the default for the whole 1.3.0 cycle. The criteria are in #5644 and the plan to restore the default after the branch cut is in #6962. 2. Fix the native Iceberg scan bugs that return wrong rows or values, because a faster write path does not help if reads are wrong. ### Describe the potential solution **Enable native Iceberg writes by default** (#5644, #5649) - [ ] Open a PR that reverts the default from #6962 back to `true`, with docs and the upgrade guide updated - [ ] #5644: classify the remaining fallback reasons in the Iceberg CI write report (input not in Arrow format, writes with no recorded reason, "IcebergWrite is not supported", "could not determine the table format-version", Spark-planned `AppendData` and streaming writes). Check whether `INSERT ... VALUES` falling back through `LocalTableScanExec` accounts for most of the first bucket - [ ] #5644: assert manifest metrics parity per data type on the pinned Iceberg versions (1.5.2, 1.8.1, 1.10.0, 1.11.0) - [ ] #5643: a keep-or-lift decision for each remaining eligibility restriction - [ ] #6996: fall back for brotli writes that no JVM reader can read - [ ] #7242: NaN manifest bounds on Parquet 60. This must be fixed before the DataFusion 56 upgrade (#6410) lands - [ ] #7008: `write.parquet.compression-codec=none` is accepted natively but rejected by iceberg-java - [ ] #7028: bump iceberg-rust and narrow the nested schema evolution fallback Writer coverage and performance, if time allows: - [ ] #7049: native `AssertNotNull` so writes into NOT NULL columns do not fall back - [ ] #6955, #6954, #6959: lift fallbacks on settings iceberg-java ignores, translate the Parquet settings that have a parquet-rs equivalent, and support `uuid` and `unknown` columns - [ ] #6772: hand written Parquet footers to the JVM instead of reading them back, which matters on object stores **Native Iceberg scan correctness** Wrong results: - [ ] #7136: drops -0.0 rows for floating-point predicates - [ ] #7137: drops NULL rows for `NOT (col <=> value)` - [ ] #7138: drops rows from data files that store a narrower type than the column - [ ] #7129: applies equality deletes to the wrong column when column names differ only in case - [ ] #6822: wrong values for a column renamed after a `VERSION AS OF` snapshot - [ ] #6790, #6794, #7212: NULLs where Spark reads values, for files read through a name mapping or without field ids ### Additional context This epic lists the work and does not decide it. Items may move to a later milestone, and the writer coverage items are optional. Related epics: #5649 (native Iceberg writes), #7040 and #7051 (nested types in the native scans and writers), and #6911 (end-to-end Iceberg performance). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
