Hi Alex,

> leave room for 3 values: STAGED, LIVE, DELETED [...]

This is a nice idea! I think it is a promising avenue to explore for
supporting reliable location overlap checks and credential refresh for
staged creates.

Cheers,
Dmitri.

On Tue, Nov 17, 2026 at 7:16 AM Alex Dutra via dev <[email protected]>
wrote:

> Hi all,
>
> A correction to my previous message, and a few suggestions:
>
> 1) Spark can ask the catalog to purge. Since Iceberg 1.12, Spark has a
> catalog option, rest-catalog-purge, off by default. When set to true,
> Spark deletes no files itself. Without the option, which includes all
> earlier Iceberg versions, my previous point stands though: Spark drops
> without purge, then deletes the files itself. Recommending
> rest-catalog-purge=true for catalogs with soft delete enabled seems
> worth documenting in [5544].
>
> 2) But catalogs can also ask clients not to delete files. Iceberg
> Java's S3FileIO has a property, s3.delete-enabled. When a catalog sets
> this property to false, S3FileIO silently skips deletes.
> Interestingly, Lakekeeper does this by default for warehouses with
> soft deletion enabled. Should Polaris do something similar? This
> property has a few drawbacks, the biggest one being that it disables
> all client-side deletes, including on live tables (i.e., when running
> expire_snapshots or remove_orphan_files). Therefore, it only makes
> sense to send that property **if the catalog handles all file
> deletions server-side**, including snapshot expirations, which Polaris
> doesn't do today. (Lakekeeper does that in its commercial offering
> only, afaict; for the open-source edition, the documentation tells
> users to override s3.delete-enabled=true in their Spark configuration
> for maintenance jobs.) All of this to say, pushing
> s3.delete-enabled=false could be an opt-in later, but for that Polaris
> should first implement a proper snapshot expiry service.
>
> 3) I would like to introduce soft-creations too, for staged table
> creations. IOW, the table would be in 3 possible states: STAGED, LIVE,
> DELETED. Today a staged table isn't persisted until the transaction
> commits, so Polaris has nothing to hold its locations or to authorize
> against in the meantime. A persisted STAGED entity would:
>
> - hold its locations and reserve its name, like a soft-deleted table;
> - let remote signing authorize against the entity instead of a token;
> - let Polaris clean up abandoned creates.
>
> It would require keeping the entity invisible to list and load until
> it commits, and an expiry service for abandoned entries, much like the
> hold for soft-deleted tables in [5544]. It also changes concurrent
> creates of the same name to fail at staging time rather than at commit
> time, which I believe is a net improvement.
>
> I'm not suggesting we do all this for Phase 1, only that the state
> model should leave room for 3 values: STAGED, LIVE, DELETED.
>
> WDYT?
>
> Thanks,
> Alex
>
> [5544]: https://github.com/apache/polaris/pull/5544
>
> On Fri, Nov 13, 2026 at 5:54 PM Alex Dutra <[email protected]> wrote:
> >
> > Hi Prithvi, hi all,
> >
> > Two points from the remote signing work that relate to this proposal.
> >
> > 1. Spark's DROP TABLE ... PURGE is a drop without purge.
> >
> > Spark doesn't ask the catalog to purge:
> >
> > - It loads the table and calls the catalog's drop with purge=false.
> > - It then deletes the table's files itself, through the table's FileIO.
> >
> > With Spark, the catalog never sees a purge. On a catalog with
> > soft-delete enabled, a Spark DROP TABLE ... PURGE would:
> >
> > - put the table on hold, with its name reserved, even though the user
> > asked for a purge;
> > - delete its files right away from the client.
> >
> > The hold would then keep a table whose files are gone, which matters
> > once recovery is added. With vended credentials, Polaris can't see or
> > prevent those deletes. This might be worth documenting.
> >
> > 2. Soft-delete would help remote signing.
> >
> > I'm reviving S3 remote signing. Polaris authorizes each sign request
> > against the table, so it does see those deletes. Today, after a drop,
> > the table and its grants are gone. My current approach records the
> > principal's DROP_TABLE_WITH_PURGE decision in the remote signing token
> > when the table is loaded. After the drop, it then allows only reads
> > and deletes within the table's locations.
> >
> > In [5544], a soft-deleted table keeps its row, id and grants. So for
> > those tables Polaris could:
> >
> > - authorize the deletes against the table itself when they happen,
> > instead of relying on a decision recorded at load time;
> > - choose a policy for the hold: either allow the deletes, since the
> > client asked for a purge, or refuse them, so the table stays
> > recoverable. Only remote signing makes the second option possible.
> >
> > Two questions:
> >
> > - Which policy makes sense for deletes during the hold?
> > - The resolver hides dropped entities. Would an option to resolve a
> > soft-deleted table, for this purpose only, be acceptable?
> >
> > Neither point should block Phase 1. I just want to make sure the two
> > features fit together.
> >
> > Thanks,
> > Alex
> >
> > [5544]: https://github.com/apache/polaris/pull/5544
> >
> >
> > On Thu, Nov 12, 2026 at 12:44 AM Dmitri Bourlatchkov via dev
> > <[email protected]> wrote:
> > >
> > > Hi Prithvi and all,
> > >
> > > I did another review round in GH.
> > >
> > > Re-posting here the key comment: I'm very hesitant about
> > > expiring soft-deleted entities during REST requests against other
> entities.
> > > I think the resulting behaviours are too "magical" from the end user's
> > > perspective.
> > >
> > > I believe the expiry use case belongs more naturally in the Admin Tool
> > > given the current codebase. Eventually it could also be handled by the
> > > delegation service [3990].
> > >
> > > WDYT?
> > >
> > > Thanks,
> > > Dmitri.
> > > [3990] https://github.com/apache/polaris/pull/3990
> > >
> > > On Mon, Oct 26, 2026 at 4:34 PM Prithvi S <[email protected]
> >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Quick follow-up on the cleanup gating for Phase 1:
> > > > holding the identifier until the entity cleanup and its
> > > > child tasks finish, then releasing it.
> > > >
> > > > While implementing this, I found that the coupling is
> > > > in the metastore manager. dropEntityIfExists removes
> > > > the entity (freeing the name) and schedules the cleanup
> > > > task in the same transaction, by design.
> > > >
> > > > To keep the name reserved until cleanup completes, I'd
> > > > need to separate these operations: keep the dropped
> > > > entity, schedule cleanup, and remove the entity only
> > > > after the cleanup task and its child tasks have finished.
> > > >
> > > > We don't currently have a way to detect when a task
> > > > tree has completed. Tasks are fire-and-forget and
> > > > self-remove on success, so I'd also need to add a small
> > > > completion check.
> > > >
> > > > This touches PolarisMetaStoreManager, which has both
> > > > JDBC (atomic and transactional) and NoSQL
> > > > implementations. I wanted to check the direction before
> > > > coding, in line with our interface-change guidance.
> > > >
> > > > Does keeping the entity reserved and gating its removal
> > > > on cleanup completion sound right, or would you prefer
> > > > a different approach, such as a dedicated purge task
> > > > that handles the final entity removal?
> > > >
> > > > Thanks,
> > > > Prithvi
> > > >
> > > > >
> > > >
>

Reply via email to