Hi all,
> work done by AWS on this that resulted in the fine-grained commits proposal.
> The idea there was to send the changes that the client is making to the
> content-metadata tree to the service.
Agree that this is the way to go and I propose to start with this and
then see if and what else is needed.
Robert
On Thu, Jul 10, 2025 at 5:24 AM Eric Maynard <eric.w.mayn...@gmail.com> wrote:
>
> Hey Ryan, that's exactly the intent. If the Iceberg REST protocol itself
> can provide a way for clients to express which commits (or which aspects of
> commits) can be reconciled with which other commits then I think that would
> be ideal. Beyond reconciliation, I think fine-grained commits are just a
> good idea. I'm keen to explore this approach further on the Iceberg side.
>
> As for Polaris itself, my intent was to explore a simple way that clients
> could label two commits as "mutually reconcilable", and to consent to the
> risk associated with blind deconfliction. This would move us away from the
> current state ("blind rejection", I suppose) and might help incrementalize
> the move towards richer deconfliction logic. But if Iceberg itself can
> alternatively help us get there with fine-grained commits just as easily,
> my vote would be to pursue that instead.
>
> --EM
>
>
>
> On Wed, Jul 9, 2025 at 2:07 PM Ryan Blue <rdb...@gmail.com> wrote:
>
> > I think there's a better way to achieve the results that this proposal is
> > aiming for. If I understand correctly, this is trying to create a way for
> > Polaris to be able to apply changes from one commit on top of another
> > without sending a commit conflict back to the client. Parallel appends are
> > the primary example of this being safe, but not something that the service
> > can handle.
> >
> > I would recommend taking a look at the work done by AWS on this that
> > resulted in the fine-grained commits proposal. The idea there was to send
> > the changes that the client is making to the content-metadata tree to the
> > service. That would also pass finer-grained validations/assertions that
> > enable the service to know when changes can be safely applied, even when
> > another writer has committed first. I think that's a better strategy and
> > would love to see it in the Iceberg REST protocol.
> >
> > Taking that approach, this would allow clients to have a way to tell the
> > service _what_ conflicts, not just tell the service what won't and let the
> > service blindly apply changes. That reduces risk (no need to apply changes
> > blindly) while also making this more valuable because it doesn't require
> > user involvement across jobs and would be useful to all REST service
> > implementations.
> >
> > Ryan
> >
> > On Tue, Jul 8, 2025 at 5:10 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > The general feature of allowing clients to declare that they take
> > > responsibility for overwriting each other's change is fine from my POV.
> > As
> > > previous messages point out there are use cases for it.
> > >
> > > However, the term "deconfliction" does not sound perfect to me in this
> > > case. From my POV deconfliction means some deliberate action on the
> > > metadata taken by the Polaris Server in order to resolve / reconcile
> > > conflicting changes submitted by clients.
> > >
> > > Do you think we could treat these use cases as different features for the
> > > sake of clarity? I can see some users being potentially confused by this
> > > (as I was).
> > >
> > > For example, to implement preemptive conflict resolution (one party wins)
> > > we probably need to define means for clients / engines to declare
> > relative
> > > priorities and elect to participate in this feature (the proposed
> > headers).
> > > However, append/append resolution may not necessarily fall under the same
> > > election flag. That is, clients allowing (future) append/append
> > resolution
> > > may not want to allow preemptive resolution. WDYT?
> > >
> > > I'm +1 to evolving Polartis to be able to properly support real conflict
> > > resolution (starting with append/append conflicts).
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Tue, Jul 8, 2025 at 12:58 PM Eric Maynard <eric.w.mayn...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > What Russel wrote in (2) is correct -- this will essentially be a
> > > > best-effort deconfliction, but my hope is that we can expand the
> > > supported
> > > > cases over time. It may be helpful to make a distinction between
> > commits
> > > > which are "eligible for deconfliction" from commits that "the server
> > > knows
> > > > how to successfully deconflict". The proposal pertains to the former,
> > but
> > > > I'm hoping the draft implementation I publish soon will help clarify
> > the
> > > > latter. My plan is to focus on append-only commits first.
> > > >
> > > > Dmitri, as with all commits the server will write the final
> > metadata.json
> > > > content. As for how it does so, it will vary a bit based on the nature
> > of
> > > > the conflicting commits, but the general idea is that the commits have
> > > > consented to deconfliction by the server. So in Robert's example both
> > > > snapshots should be preserved, but the commits have given the server
> > > > license to decide which of those should be the *latest* snapshot after
> > > > deconfliction.
> > > >
> > > > --EM
> > > >
> > > > On Tue, Jul 8, 2025 at 9:13 AM Russell Spitzer <
> > > russell.spit...@gmail.com>
> > > > wrote:
> > > >
> > > > > For me there are two big use cases for this:
> > > > >
> > > > > 1. Simple overwrite.
> > > > > I may have several jobs, for example one that does TTL , the command
> > it
> > > > > runs is idempotent and always deletes all rows / files before a
> > certain
> > > > > point. The other is a merge/update command. In this situation I don't
> > > > even
> > > > > need to reconcile, I always want the merge to succeed because it is
> > > more
> > > > > expensive while my delete is very cheap and not that important. (A
> > > > > subsequent TTL delete run will remove the records so as long as I
> > don't
> > > > > have constant conflicts everything is fine.)
> > > > >
> > > > > For example I may have a very complicated and long GDPR job that
> > > > modifies a
> > > > > lot of data and then a TTL job that removes rows before a certain
> > point
> > > > in
> > > > > time (as well as appends and other things going on). The TTL Job can
> > > > always
> > > > > be sacrificed for the GDPR job.
> > > > >
> > > > > 2. Actual Deconfliction
> > > > > In this case I just want to do a cheap very of the deconfliction that
> > > > would
> > > > > eventually happen on the client side, i.e. without any filters. In
> > case
> > > > of
> > > > > conflict I open up my previous manifest list and the one in the new
> > > > > TableUpdate and create a new Update that contains a combined manifest
> > > > list
> > > > > with the added entries from the conflicting manifest list in my
> > > current.
> > > > > This is more complicated to perform but we can do it relatively
> > easily
> > > > for
> > > > > jobs that just modify the manifest list itself. For more complicated
> > > > > situations (a manifest which is being removed has already been
> > removed)
> > > > we
> > > > > would have to actually dig into manifests or just fail. I would
> > > probably
> > > > > start by only tackling the case in which the manifest level
> > > modifications
> > > > > can be isolated and allowing the user to pass along the tag basically
> > > > > stating that the changes cannot conflict.
> > > > >
> > > > > Being able to start doing this kind of work on the Catalog side is
> > > going
> > > > to
> > > > > be important once we ever get Fine Grained Commits into the Iceberg
> > > Rest
> > > > > spec so experimenting with the ideas now I think is important.
> > > > >
> > > > > On Tue, Jul 8, 2025 at 9:21 AM Dmitri Bourlatchkov <di...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Eric,
> > > > > >
> > > > > > Just to clarify: Are you proposing Polaris Server to reconcile
> > > metadata
> > > > > > changes from two conflicting commit requests or are you proposing
> > the
> > > > > > clients to do that?
> > > > > >
> > > > > > In other words, who will make the final metadata JSON? Clients (and
> > > the
> > > > > > server merely commits it) or the Polaris Server itself (modifying
> > > what
> > > > > the
> > > > > > clients submitted)?
> > > > > >
> > > > > > Sorry, I'm a bit lost on that point based on the current doc text.
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Mon, Jul 7, 2025 at 6:33 PM Eric Maynard <
> > > eric.w.mayn...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Wanted to share this short design doc
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1tkqBOYtkcA7fbDmhIAE6_6Jmus5WwP6vS6jA_JHp4Ms
> > > > > > > >
> > > > > > > for
> > > > > > > a simple method of allowing conflicting commits to both be
> > > committed.
> > > > > If
> > > > > > > implemented, this would allow e.g. two writers doing append-only
> > > > > > operations
> > > > > > > to a table in Polaris to always succeed.
> > > > > > >
> > > > > > > If you're interested, please take a look. In the meantime, I'll
> > be
> > > > > > > preparing a small draft PR to serve as a reference
> > implementation.
> > > > > > >
> > > > > > > --EM
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
