yihua opened a new pull request, #954: URL: https://github.com/apache/incubator-xtable/pull/954
## *Important Read* Issue: #722. Stacked on #894, review that first; this PR shows the parent's commits until it merges. Design: apache/hudi RFC-114 (apache/hudi#20171); companion Hudi writer PR: apache/hudi#20167. Draft to capture the current state of the proof of concept. ## What is the purpose of the pull request Makes merge-on-read tables work under the Iceberg pluggable table format. With the Hudi writer routing updates as positional deletes plus inserts, every deltacommit becomes one Iceberg V3 snapshot: new data files plus one Puffin deletion vector per touched base file, merged with the file's previous vector. Compaction and clustering publish as file rewrites that drop the vectors of the files they replace, rollback reverts to the parent snapshot, and snapshot expiry moves from age-based on every commit (which breaks the plugin's timeline fencing on any table older than the retention window) to Hudi archival, which also deletes superseded Puffin files. ## Brief change log - `IcebergDataFileUpdatesSync.applyRowDelta` writes deletion vectors with `BaseDVFileWriter`, loading each data file's previous vector, and commits `addRows`, `addDeletes` and `removeDeletes` in one `RowDelta` with `validateFromSnapshot`. - `HudiPositionalDeleteExtractor` reads positions and the base file instant from Hudi log delete blocks and resolves the data file through the file-system view. - `IcebergRollbackExecutor` reverts to the parent snapshot. - Replace commits register data files with their real size. - Configurable Iceberg format version on table creation (default stays 2); the plugin creates tables at version 3. - `TargetTable.NO_METADATA_EXPIRY` disables age-based expiry; the plugin uses it, and archival deletes `.puffin` files. - Iceberg 1.11.0 (build and tests on JDK 17). The pom pins a local Hudi build (`1.2.0-iceberg-dv-SNAPSHOT`) carrying the writer changes, so CI cannot resolve it until a Hudi release ships them. The upstreaming order in RFC-114 Appendix D splits this into mergeable pieces. ## Verify this pull request This change added tests and can be verified as follows: - `ITIcebergMergeOnReadDeletionVectors`: inserts, repeated updates of the same keys with vector supersede, deletes, compaction, rollback, partitioned tables, the Spark write client; rows asserted through Spark on Iceberg and through Hudi. - `ITIcebergMorDvTableServices`: clustering, cleaning, archival deleting superseded Puffin files. - `TestIcebergConversionTargetRetention`, `TestIcebergFormatVersion`, `TestIcebergTableFormatWiring`. - Module suite green after rebasing on main: 33 tests, 1 pre-existing skip, with `-Dmaven.build.cache.enabled=false` (the build cache extension otherwise skips test executions). Known gaps to resolve before this leaves draft status: 1. `HudiPositionalDeleteExtractor` runs for every deltacommit and fails the instant on any log block that is not a positional delete, so a plain merge-on-read table under the Iceberg format (without the Hudi write mode) now fails its first upsert instead of publishing base files only; this needs a gate on the writer mode or a documented decision. 2. Puffin deletion at archival is safe only because archival expires by snapshot id (reachable-file cleanup); the time-based expiry path would delete a Puffin file that still carries a live vector for another data file. 3. A rolled-back snapshot stays in the table's snapshot list and is never expired, since the archiver expires only instants it finds in the archived timeline. 4. `TargetTable.NO_METADATA_EXPIRY` is honored by the Iceberg target only; the Delta and Hudi targets pass the negative duration into their retention configs. 5. `TestBaseFileUpdatesExtractor` fails against the pinned Hudi build because of the Hudi-side file-system view change, which is flagged in the Hudi PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
