yihua opened a new pull request, #894: URL: https://github.com/apache/incubator-xtable/pull/894
## *Important Read* Issue: #722. Companion Hudi PR: https://github.com/apache/hudi/pull/13216 (merged, ships in Hudi 1.1+). This carries the commits from #723 plus one commit that makes the new module's integration tests run against the pluggable table format. #723 remains the pull request of record; this exists so CI runs on the fixed branch, since I cannot push to that branch. Happy to fold the fix into #723 instead. The added commit: - The failsafe plugin sets `hoodie.table.format=ICEBERG` for the module and the Hudi test harness applies it to every table it creates, along with the table version and metadata-table setting a pluggable format needs. `ITIcebergTableFormat` and `ITIcebergVariousActions` created their tables through overloads that do not set the format, so they ran against the native Hudi format and the Iceberg assertions failed on a table that was never written. - `ITIcebergTableFormat` normalizes the local-timestamp columns per format the way `ITConversionController` already does, rather than comparing a Hudi timestamp against raw Iceberg micros (#672). - Drops an assertion in `testsForClustering` expecting the first commit to be archived, which the equivalent test in `xtable-core` does not assert, and disables `testsForSavepointRestore` for the reason given below. - Removes two empty placeholder test classes. ## What is the purpose of the pull request Apache Hudi 1.1 introduced a pluggable table format SPI (`HoodieTableFormat`, RFC-93). This pull request implements that SPI for Iceberg, so a Hudi writer maintains an Iceberg metadata tree in the table directory as it commits. The result is a table that presents as a standard Iceberg table to any engine while Hudi writes and operates it, with no separate sync job and no second copy of the data. The new module `xtable-hudi-support/xtable-iceberg-pluggable-tf` registers `IcebergTableFormat` through `META-INF/services`. A table opts in with `hoodie.table.format=ICEBERG` in `hoodie.properties`; tables without it are unaffected. The design reuses the Hudi-to-Iceberg translation already in `xtable-core`. Each Hudi lifecycle event (commit, clean, archive, rollback, savepoint) turns the Hudi action into an `IncrementalTableChanges` and pushes it through the existing `TableFormatSync` into `IcebergConversionTarget`. What is new is that this runs inside the Hudi commit rather than in an out-of-band sync, and that Iceberg becomes the authority on which commits are visible: `IcebergActiveTimeline` treats only instants recorded in an Iceberg snapshot summary as completed and downgrades the rest to inflight. Known limitations of this first change, each intended as a follow-up: - `IcebergBackedTableMetadata` extends `FileSystemBackedTableMetadata`, so enabling this format replaces the Hudi metadata table with filesystem listing. Metadata-table indexes (record-level index, column and partition stats) are not available to a table using the Iceberg format. - Merge-on-read updates are not represented. `HudiDataFileExtractor` skips log files, so a log block never becomes an Iceberg data file. Copy-on-write is the supported table type. - Restore is not supported. A savepoint changes no data, so no Iceberg snapshot records it, and `IcebergActiveTimeline` therefore reports the completed savepoint instant as inflight. - `IcebergActiveTimeline` does not override `reload()`, so a caller that reloads the active timeline gets the native timeline rather than the one reconstructed from Iceberg. - A snapshot written by a writer other than this format has no XTable metadata in its summary and is not tolerated when the timeline is reconstructed. - Tables using this format are created at Iceberg format version 2. There is no way to request version 3 yet. ## Brief change log New module `xtable-iceberg-pluggable-tf`: - `IcebergTableFormat`, the `HoodieTableFormat` implementation, with its service registration. - `IcebergTimelineFactory` and `IcebergActiveTimeline`, which reconstruct the Hudi active timeline from Iceberg snapshots on top of the v2 timeline. - `IcebergMetadataFactory` and `IcebergBackedTableMetadata`, supplying file listings in place of the Hudi metadata table. - `IcebergTimelineArchiver` and `IcebergRollbackExecutor`, mapping Hudi archival and rollback onto Iceberg snapshot expiry and rollback, stopping at savepoints and refusing to roll back past Iceberg's recorded tip. `xtable-core`: - `HudiIncrementalTableChangeExtractor`, computing `IncrementalTableChanges` for a single completed instant from its commit metadata. - `HudiDataFileExtractor.getDiffForCommit` and `getDiffForReplaceCommit`, which derive the file diff from the commit metadata of the commit being written rather than by comparing two committed snapshots, plus a constructor taking a `FileSystemViewManager`. - `HudiTableExtractor` overload building `InternalTable` from commit metadata. - `IcebergConversionTarget.expireSnapshotIds` and `rollbackToSnapshotId`. - `ConversionTargetFactory` skips a registered `ConversionTarget` whose engine library is absent from the classpath instead of failing the service-loader iteration, and identifies the Delta kernel target by class name. The new module carries Hudi and Iceberg but not Delta. `xtable-api`: - `InternalTable.latestTableOperationId` and `TableSyncMetadata.latestTableOperationId`, the serialized Hudi instant recorded in the Iceberg snapshot summary. This is what `IcebergActiveTimeline`, `IcebergTimelineArchiver`, and `IcebergRollbackExecutor` read back. The existing `TableSyncMetadata.of` overloads are retained. Test harness: - `TestJavaHudiTable` accepts table-level properties, so a test can create a table with a pluggable format, and `TestAbstractHudiTable` no longer pins the table version and metadata-table setting unconditionally. ## Verify this pull request This change added tests and can be verified as follows: - `ITIcebergPluggableFormatSync` verifies end to end that a Hudi commit on a table configured with `hoodie.table.format=ICEBERG` produces an Iceberg snapshot whose row count matches. - `ITIcebergTableFormat` runs the Hudi-to-Iceberg sync matrix (partitioned and unpartitioned, incremental and full sync modes, concurrent writes, time travel, out-of-order commits, metadata retention) against the pluggable format, reading the same base path as both a Hudi table and an Iceberg table and comparing the datasets. - `ITIcebergVariousActions` covers insert, upsert, delete, and clustering flows on a table using the format, and exercises incremental change extraction against each. - `TestIcebergTableFormatDiscovery` verifies the table format is resolved through the service loader and selected by `hoodie.table.format`. The module's integration tests run with `hoodie.table.format=ICEBERG` set by the failsafe plugin, so every table they create uses the pluggable format. Unit-level coverage of `IcebergActiveTimeline` and `IcebergBackedTableMetadata` is left to a follow-up; both are exercised through the integration tests. `ITIcebergVariousActions.testsForSavepointRestore` is disabled. A savepoint changes no data, so no Iceberg snapshot records it, and the reconstructed timeline reports the completed savepoint instant as inflight. Representing savepoints in Iceberg is a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
