Thanks both, and thanks for the support on the direction. On magnitude, agreed it depends on the workload. The design keeps the added cost targeted: the finer-grained check only does work for a data file where both the committing operation and a concurrent commit added a DV, and that overlap is determined from manifest metadata (partition + column bounds, then the referenced data file) before anything is read. Commits that touch disjoint files fall through unchanged, so the cost is only paid in the case that fails outright today.
On the retry question (option 1 vs 2), I'd do the union in the validation path (option 1), and I think that already satisfies the "retry until resolved or fail" model. validateAddedDVs runs inside the existing commit loop after refresh() on every attempt, so on a genuine version conflict the loop retries and the union is recomputed against the freshly refreshed state; a disjoint overlap unions in place and the commit proceeds, and a true position overlap throws a ValidationException and fails. The outer retry on CommitFailedException stays as-is for real version conflicts; we just resolve the DV overlap in validate() rather than making the overlap itself a separate retryable loop-back, which is what avoids recomputing the whole operation on a spurious conflict. For concrete grounding: the pure-delete case (concurrent DELETEs producing DVs for the same file, which are commutative) is already implemented in #17754, which merges the concurrent DVs unconditionally. The extension here is the position-overlap check that lets the same merge apply to operations that also rewrite rows (UPDATE / MERGE), where the union is only safe when the newly deleted positions are disjoint. On gating the added granularity under a table property, I think the DV-bitmap tier may not need one, since the cost is already bounded by the layering above: an operation gate, then a conflicting-file metadata gate, then a single small roaring-bitmap read only on genuine overlap (disjoint-file commits read nothing). The tier that would actually read data files (value-exact comparison, to resolve overlaps a bitmap can't decide) is separate and only invoked when necessary; that's where per-commit data reads grow, and where a table property to gate it makes sense. It can be introduced together with that tier. Thanks EJ On Thu, Sep 10, 2026 at 5:07 PM huaxin gao <[email protected]> wrote: > Hi EJ, > > Thanks for initiating the discussion and +1 on the direction. A few notes > on your questions. > > (1) Yes, it is worth doing. The cost is that it adds driver-side I/O during > the commit: one read per data file in scope, while the rest of the job > waits. That is probably fine for DV bitmaps, which are small, but it grows > with the number of files in scope, and the value-comparison tier would mean > reading data files rather than bitmaps. So I would suggest gating the new > granularity under a table property. > > (2) Xiening suggested the retry path — I'd lean the other way and do it in > the validation path, so we don't have to abandon the attempt and loop back. > Validation already runs after a refresh() on every attempt, so the union > gets recomputed against fresh state anyway. > > Also, most of the read-merge-write code already exists and already runs in > the commit path on the driver: MergingSnapshotProducer.mergeDVs() into > DVUtil.mergeAndWriteDVsIfRequired. > > (3) Three pieces of prior art: > https://github.com/apache/iceberg/pull/17754 — does this for pure delete > commits. > https://github.com/trinodb/trino/pull/30860 — skipped the coarse delete > check > outright for v3 UPDATE/MERGE > https://github.com/apache/iceberg/issues/14613 — earlier row-level > concurrency > discussion, closed as stale. > > Huaxin > > On Tue, Sep 8, 2026 at 2:47 PM Xiening Dai <[email protected]> wrote: > >> Hi EJ, >> >> I think in general this is a good idea. A finer-granularity conflict >> resolution is going to help the overall commit performance. Whether or not >> it will make a significant difference would depend on the user scenario. >> >> For your 2nd question, I think you always need to re-try commit until >> either 1) there's real conflict you cannot resolve and transaction has to >> fail, or 2) there's no more version conflict and you commit successfully. >> >> >> On 2026/09/08 20:44:57 EJ Song wrote: >> > Hi all, >> > >> > I'd like to gauge appetite for making concurrent UPDATE/MERGE on V3 >> > deletion vectors fail less often when the operations touched disjoint >> rows. >> > >> > Background: with one DV per data file, validateAddedDVs fails the commit >> > when a concurrent op added a DV for a file this op also DV'd, at file >> > granularity, regardless of whether the deleted positions actually >> overlap. >> > That ValidationException is not retryable (the commit loop is >> onlyRetryOn >> > CommitFailedException), so the whole operation is recomputed and re-run. >> > For two operations that modified disjoint rows in the same file, that is >> > wasted work on a spurious conflict. Delta Lake's row-level concurrency >> > already reduces this to row granularity on DV-enabled tables; I would >> like >> > to pursue the same for Iceberg V3 DVs. >> > >> > Core idea: refine validateAddedDVs from file granularity to position >> > granularity. When a concurrent DV exists for a file this op also DV'd, >> > compare the newly-deleted positions: >> > >> > (dv_concurrent intersect dv_self) minus dv_base == empty >> > >> > If disjoint, union the two into a single DV and proceed; if they >> overlap, >> > fail as today. This reuses existing DV primitives (DVUtil.readDV, >> > PositionDeleteIndex.merge, BaseDVFileWriter); the new code is a >> > bitmap-overlap test plus wiring a cross-operation union into the >> validation >> > path. It does not change isolation semantics, which stay in >> > validateNoConflictingDeletes and validateNoConflictingData. >> > >> > I wrote up the full design (soundness conditions, a sequenced set of >> > follow-up refinements, non-goals including compaction-vs-DML which is >> not >> > reconciled today, and configuration) as a GitHub issue: >> > https://github.com/apache/iceberg/issues/18020 >> > >> > Questions: >> > >> > 1. Is the file-level false-failure in validateAddedDVs worth refining to >> > position granularity? >> > 2. Should the disjoint-case union happen in the validation path (union >> in >> > place), or via a retryable commit that re-reads and unions the base DV >> on >> > each refreshed apply? (Details in the issue.) >> > 3. Any prior art or earlier design decision I have missed? >> > >> > Thanks, >> > EJ >> > >> >
