yihua opened a new pull request, #953: URL: https://github.com/apache/incubator-xtable/pull/953
## *Important Read* - *Please ensure the GitHub issue is mentioned at the beginning of the PR* ## What is the purpose of the pull request Issue #722, #348. Proof of concept for Iceberg V3 deletion vectors under the Hudi pluggable table format, opened as a draft to capture the current state of the work and its breakdown. Stacked on #894, so this PR shows that PR's commits until it merges; review #894 first. Under this branch a Hudi merge-on-read table with `hoodie.table.format=ICEBERG` produces a valid Iceberg V3 snapshot on every deltacommit, with updates and deletes represented as deletion vectors plus new data files, so any Iceberg V3 engine reads the table between Hudi compactions with no translation job. It depends on the Hudi write mode proposed in apache/hudi#20167 (updates as positional deletes and inserts) and the design in apache/hudi RFC-115; the branch pins a Hudi build carrying those changes. The commits on top of #894 are ordered so they can be split into standalone PRs: 1. Fix zero-length data files registered by replace commits (xtable-core only, mergeable on its own). 2. Fix Iceberg snapshot rollback to revert to the parent snapshot (plugin module). 3. Support configurable Iceberg format version on table creation (default stays 2). 4. Upgrade Iceberg to 1.11.0 (pom bump and the manifest-reader deprecation shim; the build needs JDK 17). 5. Never expire Iceberg snapshots by age under the pluggable table format; expiry follows Hudi archival instead. 6. Represent merge-on-read deltacommits as Iceberg V3 deletion vectors: the RowDelta and deletion-vector writer, the plugin extractor, and coverage for rollback, repeated updates of the same keys, clustering and cleaning, partitioned tables and the Spark write client. 7. Delete deletion-vector files when the snapshots referencing them expire. ## Brief change log - Deletion-vector writer and extractor for merge-on-read deltacommits in the pluggable table format module - Configurable Iceberg format version, snapshot rollback to the parent snapshot, no age-based expiry under the pluggable format - Iceberg 1.11.0 upgrade - Replace-commit data file size fix in xtable-core - Puffin file cleanup at archival - Integration tests for the above ## Verify this pull request This change added tests and can be verified as follows: - Pluggable table format module integration tests: deletion vectors after updates and deletes, rollback, repeated updates of the same keys, deletion vectors dropped after compaction and clustering, partitioned tables, the Spark write client, Puffin cleanup at expiry. - xtable-core Iceberg target and Hudi source tests for the replace-commit file size fix and the format version option. - Not yet green against upstream main: the branch conflicts with main in `pom.xml` and `BaseFileUpdatesExtractor` and needs a rebase once #894 lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
