yihua opened a new issue, #20168: URL: https://github.com/apache/hudi/issues/20168
RFC-93 lets a Hudi writer publish Iceberg metadata for every commit through the `HoodieTableFormat` SPI, but only copy-on-write commits become visible to Iceberg readers. A merge-on-read deltacommit writes log files, which have no Iceberg representation, so an Iceberg engine reading the table sees stale data until the next compaction. Compaction and merge-on-read rollback also complete without notifying the table format, so the plugin cannot keep Iceberg metadata consistent with the Hudi timeline. Proposal: add a merge-on-read write mode in which an update becomes a positional delete against the old file group plus an insert into a new file slice, so every deltacommit maps to an Iceberg V3 snapshot of data files plus deletion vectors; fire the table format completion hook on compaction, log compaction and rollback; and let the file-system view tolerate data files in the table directory that Hudi did not name. The design is in RFC-114, and the writer changes come in a companion PR. The Iceberg side lives in Apache XTable, on top of apache/incubator-xtable#894. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
