lawofcycles opened a new issue, #15394: URL: https://github.com/apache/iceberg/issues/15394
### Feature Request / Improvement Currently, `IncrementalChangelogScan` throws `UnsupportedOperationException` when any snapshot in the scan range contains delete manifests. https://github.com/apache/iceberg/blob/f1499bde7830a8f003995311cd2b3a3503214a81/core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java#L108-L111 This prevents changelog scans from working with merge-on-read tables. In Spark, `create_changelog_view` cannot be used for any table written with MOR. ### Background The initial implementation of `IncrementalChangelogScan` in #5382 was scoped to CoW tables, with the intent to add delete file support in a follow-up PR. This feature was also requested in #8975. ### Proposal Support delete files (position deletes, equality deletes, and deletion vectors) in changelog scans. The existing `DeleteFileIndex` / `DeleteLoader` infrastructure can already identify which rows are deleted by a given delete file, so the core building blocks are in place. #### Phase 1. Delete file support for all format versions For each snapshot in the scan range that contains delete manifests, the scan would 1. Identify newly added delete files alongside newly added/removed data files 2. Use the existing delete file merging infrastructure to determine which rows in existing data files are affected 3. Emit `DELETE` for rows removed by delete files, `INSERT` for rows in newly added data files UPDATE detection (`UPDATE_BEFORE`/`UPDATE_AFTER` pairs) can rely on identifier columns, which is the same approach `CreateChangelogViewProcedure` already uses for carryover removal and update image computation. #### Phase 2. Row lineage optimization for v3 tables For v3 tables with row lineage, the changelog scan can be more efficient. - `_last_updated_sequence_number` allows filtering to only rows changed within the scan range, avoiding full data file reads - `_row_id` enables reliable matching of the same logical row across data files for UPDATE detection, without depending on identifier columns Note that row lineage does not track changes made via equality deletes ([spec](https://iceberg.apache.org/spec/#row-lineage)), so Phase 1 remains necessary as the general path. ### Scope - Phase 1 covers v2 tables (position deletes, equality deletes) and v3 tables (deletion vectors, legacy position deletes from upgraded tables) - Phase 2 is an optimization for v3 tables with row lineage ### Query engine Spark ### Willingness to contribute - [ ] I can contribute this improvement/feature independently - [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
