raghav-reglobe commented on issue #2201: URL: https://github.com/apache/iceberg-rust/issues/2201#issuecomment-4886855382
Thanks for laying that out so clearly — the split works well for us, and we're glad to stay out of the CoW critical path. We'll take you up on the MoR lane. That's where our production need actually lives (V3 merge-on-read tables with deletion vectors), and it's the half our RowDelta lineage in #2678 was built for: MATCHED rows become deletion vectors against the scanned file/position rather than file rewrites, NOT MATCHED / updated rows append to new files, and the commit is a single `RowDelta` snapshot carrying both the appends and the DVs. We recently added end-to-end tests on that branch covering exactly this combined commit shape (data + same-sequence DVs in one atomic snapshot), so the commit side is proven — what we'd be contributing is the execution-node layer on top. Two design questions so we align with your plan shape rather than invent a parallel one: 1. Would you prefer MoR as a mode of `IcebergMergeWriteExec` (branching at the write stage, sharing `IcebergMergeExec` classification) or as sibling exec nodes? The classification output looks mode-agnostic to us as long as MATCHED rows carry `_file` + row position through the join. 2. Related — does the scan-side plumbing in your PoC already thread a `_pos` metadata column alongside `_file`? CoW only needs `_file`, but DV construction needs the row's position within it, so that may be the first shared piece to agree on. We'll rebase onto your incremental PRs as they land, and happy to review the CoW series from the RowDelta angle since the remove-data-files path lives in the same lineage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
