andygrove opened a new issue, #6669: URL: https://github.com/apache/datafusion-comet/issues/6669
### What is the problem the feature request solves? With #6668, the native Iceberg writer takes appends and overwrites to format-version 3 tables, but declines any write whose write schema includes the row lineage columns `_row_id` and `_last_updated_sequence_number`. From Iceberg 1.10 that is every copy-on-write `DELETE`, `UPDATE` and `MERGE` and every `rewrite_data_files` on a v3 table, so all of them stay on iceberg-java. ### Describe the potential solution Write the two lineage columns natively, with the values iceberg-java writes: - **Copy-on-write DML (Spark 4.x).** `dropNonDataColumns` projects the `ReplaceData` row down to the write schema's data columns. It would instead carry `_row_id` and `_last_updated_sequence_number` from the row's metadata columns. iceberg-java's `ExtractRowLineage` copies both from the metadata row and writes nulls for a row that arrives without one (an inserted row). Spark's metadata column flags already null `_last_updated_sequence_number` on updated rows and keep their `_row_id`. - **`rewrite_data_files`.** The rewrite table's schema includes the two columns, so their values arrive as ordinary input columns. - **Field ids.** The columns carry reserved field ids (`Integer.MAX_VALUE - 107` and `- 108`), which the field-id decoration and the iceberg-rust schema need to accept. - **Manifest metrics.** The `DataFile` metrics for the two columns must match iceberg-java's after `rebuildDataFilesWithJavaMetrics`. - **Gate.** Relax `requireNoMetadataColumns` for exactly these two columns, and keep declining any other metadata column. Tests should compare `_row_id` and `_last_updated_sequence_number` read back after native and iceberg-java writes on Iceberg 1.10 and 1.11, including a `MERGE` that inserts rows and a compaction that follows an update. ### Additional context Part of #5649. Follows #6668. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
