yihua opened a new pull request, #955: URL: https://github.com/apache/incubator-xtable/pull/955
## *Important Read* Issue: #722. Proof of concept, stacked on #954 (and therefore on #894); this PR shows both parents' commits until they merge. Draft to capture the current state; not intended for merge in this shape. ## What is the purpose of the pull request Lets a user keep an existing Iceberg pipeline and add only `spark.plugins=org.apache.spark.HudiIcebergPlugin`: writes, row-level commands, maintenance procedures and streaming commits go through Hudi, and the pluggable table format publishes each Hudi instant as an Iceberg snapshot in the same table. ## Brief change log New module `xtable-hudi-support/xtable-iceberg-spark-plugin` (Spark 3.4, Iceberg 1.11): - `HudiIcebergPlugin` rewrites `spark.sql.catalog.<name>` to `HudiSparkCatalog extends SparkCatalog` and appends `HudiIcebergExtensions`. - `HudiSparkTable` returns a `V1Write` backed by `HoodieSparkSqlWriter`, with Hudi configuration derived from the Iceberg table (identifier fields select upsert, partition transforms map to Hudi partitioning, `write.merge.mode=merge-on-read` selects merge-on-read with deletion vectors). - Parser and resolution rules claim `UPDATE`, `DELETE` and `MERGE` on managed tables; `rewrite_data_files` and `rewrite_position_delete_files` run Hudi clustering and compaction; `expire_snapshots` is a no-op. - Adopt and eject through the `hudi.managed` table property; Structured Streaming micro-batches commit through Hudi. - Plugin-side format config (`IcebergFormatConfig`) threaded through the pluggable table format. Known data-correctness gaps to resolve before this leaves proof-of-concept status: 1. A managed table with an unsupported partition transform silently falls back to Iceberg's writer, so its writes become foreign snapshots the Hudi timeline skips. 2. Adopt creates `.hoodie` only, without bootstrapping Hudi metadata from the Iceberg table, so row-level operations on natively written rows tag nothing (a delete is dropped, an update inserts a duplicate). 3. The streaming epoch commit does not record the epoch id, so a driver failure between the Hudi commit and Spark's commit log replays the micro-batch on key-less tables. 4. DataFrame `overwrite(predicate)` maps to Hudi `insert_overwrite`, which replaces every partition present in the input rather than only the predicate's; `TRUNCATE TABLE` is a silent no-op; rollback fails when the current snapshot is foreign. 5. `INSERT INTO` on a keyed table is an upsert with commit-time ordering, and duplicate source matches in `MERGE` pick one row instead of failing. ## Verify this pull request This change added tests and can be verified as follows: - `ITHudiIcebergSparkPlugin`, 15 tests: DDL, inserts, DataFrame appends, dynamic and static partition overwrite, upsert through identifier fields, `UPDATE`, `DELETE`, `MERGE`, clustering and compaction procedures, adopt, eject, native write, re-adopt, merge-on-read deletion vectors, streaming micro-batches; reads verified through the plugin catalog and through a plain `SparkCatalog` over the same warehouse. - Pluggable table format module suite green alongside (33 tests, 1 pre-existing skip), JDK 17, build cache disabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
