Hi EJ, I think in general this is a good idea. A finer-granularity conflict resolution is going to help the overall commit performance. Whether or not it will make a significant difference would depend on the user scenario.
For your 2nd question, I think you always need to re-try commit until either 1) there's real conflict you cannot resolve and transaction has to fail, or 2) there's no more version conflict and you commit successfully. On 2026/09/08 20:44:57 EJ Song wrote: > Hi all, > > I'd like to gauge appetite for making concurrent UPDATE/MERGE on V3 > deletion vectors fail less often when the operations touched disjoint rows. > > Background: with one DV per data file, validateAddedDVs fails the commit > when a concurrent op added a DV for a file this op also DV'd, at file > granularity, regardless of whether the deleted positions actually overlap. > That ValidationException is not retryable (the commit loop is onlyRetryOn > CommitFailedException), so the whole operation is recomputed and re-run. > For two operations that modified disjoint rows in the same file, that is > wasted work on a spurious conflict. Delta Lake's row-level concurrency > already reduces this to row granularity on DV-enabled tables; I would like > to pursue the same for Iceberg V3 DVs. > > Core idea: refine validateAddedDVs from file granularity to position > granularity. When a concurrent DV exists for a file this op also DV'd, > compare the newly-deleted positions: > > (dv_concurrent intersect dv_self) minus dv_base == empty > > If disjoint, union the two into a single DV and proceed; if they overlap, > fail as today. This reuses existing DV primitives (DVUtil.readDV, > PositionDeleteIndex.merge, BaseDVFileWriter); the new code is a > bitmap-overlap test plus wiring a cross-operation union into the validation > path. It does not change isolation semantics, which stay in > validateNoConflictingDeletes and validateNoConflictingData. > > I wrote up the full design (soundness conditions, a sequenced set of > follow-up refinements, non-goals including compaction-vs-DML which is not > reconciled today, and configuration) as a GitHub issue: > https://github.com/apache/iceberg/issues/18020 > > Questions: > > 1. Is the file-level false-failure in validateAddedDVs worth refining to > position granularity? > 2. Should the disjoint-case union happen in the validation path (union in > place), or via a retryable commit that re-reads and unions the base DV on > each refreshed apply? (Details in the issue.) > 3. Any prior art or earlier design decision I have missed? > > Thanks, > EJ >
