Hi EJ, Thanks for initiating the discussion and +1 on the direction. A few notes on your questions.
(1) Yes, it is worth doing. The cost is that it adds driver-side I/O during the commit: one read per data file in scope, while the rest of the job waits. That is probably fine for DV bitmaps, which are small, but it grows with the number of files in scope, and the value-comparison tier would mean reading data files rather than bitmaps. So I would suggest gating the new granularity under a table property. (2) Xiening suggested the retry path — I'd lean the other way and do it in the validation path, so we don't have to abandon the attempt and loop back. Validation already runs after a refresh() on every attempt, so the union gets recomputed against fresh state anyway. Also, most of the read-merge-write code already exists and already runs in the commit path on the driver: MergingSnapshotProducer.mergeDVs() into DVUtil.mergeAndWriteDVsIfRequired. (3) Three pieces of prior art: https://github.com/apache/iceberg/pull/17754 — does this for pure delete commits. https://github.com/trinodb/trino/pull/30860 — skipped the coarse delete check outright for v3 UPDATE/MERGE https://github.com/apache/iceberg/issues/14613 — earlier row-level concurrency discussion, closed as stale. Huaxin On Tue, Sep 8, 2026 at 2:47 PM Xiening Dai <[email protected]> wrote: > Hi EJ, > > I think in general this is a good idea. A finer-granularity conflict > resolution is going to help the overall commit performance. Whether or not > it will make a significant difference would depend on the user scenario. > > For your 2nd question, I think you always need to re-try commit until > either 1) there's real conflict you cannot resolve and transaction has to > fail, or 2) there's no more version conflict and you commit successfully. > > > On 2026/09/08 20:44:57 EJ Song wrote: > > Hi all, > > > > I'd like to gauge appetite for making concurrent UPDATE/MERGE on V3 > > deletion vectors fail less often when the operations touched disjoint > rows. > > > > Background: with one DV per data file, validateAddedDVs fails the commit > > when a concurrent op added a DV for a file this op also DV'd, at file > > granularity, regardless of whether the deleted positions actually > overlap. > > That ValidationException is not retryable (the commit loop is onlyRetryOn > > CommitFailedException), so the whole operation is recomputed and re-run. > > For two operations that modified disjoint rows in the same file, that is > > wasted work on a spurious conflict. Delta Lake's row-level concurrency > > already reduces this to row granularity on DV-enabled tables; I would > like > > to pursue the same for Iceberg V3 DVs. > > > > Core idea: refine validateAddedDVs from file granularity to position > > granularity. When a concurrent DV exists for a file this op also DV'd, > > compare the newly-deleted positions: > > > > (dv_concurrent intersect dv_self) minus dv_base == empty > > > > If disjoint, union the two into a single DV and proceed; if they overlap, > > fail as today. This reuses existing DV primitives (DVUtil.readDV, > > PositionDeleteIndex.merge, BaseDVFileWriter); the new code is a > > bitmap-overlap test plus wiring a cross-operation union into the > validation > > path. It does not change isolation semantics, which stay in > > validateNoConflictingDeletes and validateNoConflictingData. > > > > I wrote up the full design (soundness conditions, a sequenced set of > > follow-up refinements, non-goals including compaction-vs-DML which is not > > reconciled today, and configuration) as a GitHub issue: > > https://github.com/apache/iceberg/issues/18020 > > > > Questions: > > > > 1. Is the file-level false-failure in validateAddedDVs worth refining to > > position granularity? > > 2. Should the disjoint-case union happen in the validation path (union in > > place), or via a retryable commit that re-reads and unions the base DV on > > each refreshed apply? (Details in the issue.) > > 3. Any prior art or earlier design decision I have missed? > > > > Thanks, > > EJ > > >
