yihua opened a new pull request, #20167: URL: https://github.com/apache/hudi/pull/20167
### Describe the issue this Pull Request addresses issue: #16073 Opened as a draft to capture the current state of the work. The design is written up in RFC-115 (link to follow). Table formats built on deletion vectors, such as Iceberg V3, can only represent a merge-on-read table exactly if every live record sits in a base file and log files carry positional deletes alone. Hudi's upsert appends the new version of an updated record to the current file group's log, which such formats cannot express between compactions. ### Summary and Changelog - New write config `hoodie.write.updates.as.deletes.and.inserts` (default false). On a merge-on-read table, an upsert routes each tagged update as a positional delete against the record's current file group plus an untagged insert that the insert partitioner routes to a fresh file slice. Log files then hold positional deletes only. - Multi-tagged records resolve to their latest location, so a key that already moved once is tombstoned in the right file group on the next update. - Small-file routing is kept out of this mode so an insert can never land in the file group that was just tombstoned within the same commit. - Fail-fast guards: event-time ordering, log compaction, and a tagged update without a valid record position are rejected while the mode is on. - `HoodieTableFormat` is now notified on compaction completion and on merge-on-read rollback, so a pluggable table format can drop the deletion vectors of rewritten or rolled-back base files. Table format settings persist in the table config, and the file-system view ignores data files Hudi did not name. - Tests: `TestHoodieJavaClientUpdatesAsDeletesAndInserts` (decomposition, repeated updates of the same keys, default behavior unchanged, compaction after decomposed updates, the two rejections), `TestSparkUpdatesAsDeletesAndInserts` through the Spark write client, and `TestHoodieJavaClientMergeOnReadForTestFormat` for the table-format hooks. ### Impact No behavior change unless the new config is enabled. Pluggable table format implementations now receive compaction and rollback notifications. ### Risk Level low: the write mode is opt-in and guarded; the table-format notifications are no-ops for the native format. ### Documentation Update The new config carries its description in `HoodieWriteConfig`. A website page will be needed once the feature is finalized. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
