Hi Alex

> - The resolver hides dropped entities. [...]

Thanks for flagging this.

I tend to think that the Resolver should not hide anything because it is a
critical piece in ensuing catalog consistency with respect to the Entity
Cache.

If the Resolver is a convenient place to enforce soft-deleted states,
perhaps we could do that during the "get" calls, while still allowing the
soft-deleted entities to be resolved normally.

The request signer could then use a different getter method returning all
resolved entities.

> - Which policy makes sense for deletes during the hold?

I think we should block file removal for soft-deleted entities where we
can. It seems possible with request signing.

Cheers,
Dmitri.

On Fri, Nov 13, 2026 at 11:55 AM Alex Dutra via dev <[email protected]>
wrote:

> Hi Prithvi, hi all,
>
> Two points from the remote signing work that relate to this proposal.
>
> 1. Spark's DROP TABLE ... PURGE is a drop without purge.
>
> Spark doesn't ask the catalog to purge:
>
> - It loads the table and calls the catalog's drop with purge=false.
> - It then deletes the table's files itself, through the table's FileIO.
>
> With Spark, the catalog never sees a purge. On a catalog with
> soft-delete enabled, a Spark DROP TABLE ... PURGE would:
>
> - put the table on hold, with its name reserved, even though the user
> asked for a purge;
> - delete its files right away from the client.
>
> The hold would then keep a table whose files are gone, which matters
> once recovery is added. With vended credentials, Polaris can't see or
> prevent those deletes. This might be worth documenting.
>
> 2. Soft-delete would help remote signing.
>
> I'm reviving S3 remote signing. Polaris authorizes each sign request
> against the table, so it does see those deletes. Today, after a drop,
> the table and its grants are gone. My current approach records the
> principal's DROP_TABLE_WITH_PURGE decision in the remote signing token
> when the table is loaded. After the drop, it then allows only reads
> and deletes within the table's locations.
>
> In [5544], a soft-deleted table keeps its row, id and grants. So for
> those tables Polaris could:
>
> - authorize the deletes against the table itself when they happen,
> instead of relying on a decision recorded at load time;
> - choose a policy for the hold: either allow the deletes, since the
> client asked for a purge, or refuse them, so the table stays
> recoverable. Only remote signing makes the second option possible.
>
> Two questions:
>
> - Which policy makes sense for deletes during the hold?
> - The resolver hides dropped entities. Would an option to resolve a
> soft-deleted table, for this purpose only, be acceptable?
>
> Neither point should block Phase 1. I just want to make sure the two
> features fit together.
>
> Thanks,
> Alex
>
> [5544]: https://github.com/apache/polaris/pull/5544
>
>
> On Thu, Nov 12, 2026 at 12:44 AM Dmitri Bourlatchkov via dev
> <[email protected]> wrote:
> >
> > Hi Prithvi and all,
> >
> > I did another review round in GH.
> >
> > Re-posting here the key comment: I'm very hesitant about
> > expiring soft-deleted entities during REST requests against other
> entities.
> > I think the resulting behaviours are too "magical" from the end user's
> > perspective.
> >
> > I believe the expiry use case belongs more naturally in the Admin Tool
> > given the current codebase. Eventually it could also be handled by the
> > delegation service [3990].
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> > [3990] https://github.com/apache/polaris/pull/3990
> >
> > On Mon, Oct 26, 2026 at 4:34 PM Prithvi S <[email protected]>
> > wrote:
> >
> > > Hi all,
> > >
> > > Quick follow-up on the cleanup gating for Phase 1:
> > > holding the identifier until the entity cleanup and its
> > > child tasks finish, then releasing it.
> > >
> > > While implementing this, I found that the coupling is
> > > in the metastore manager. dropEntityIfExists removes
> > > the entity (freeing the name) and schedules the cleanup
> > > task in the same transaction, by design.
> > >
> > > To keep the name reserved until cleanup completes, I'd
> > > need to separate these operations: keep the dropped
> > > entity, schedule cleanup, and remove the entity only
> > > after the cleanup task and its child tasks have finished.
> > >
> > > We don't currently have a way to detect when a task
> > > tree has completed. Tasks are fire-and-forget and
> > > self-remove on success, so I'd also need to add a small
> > > completion check.
> > >
> > > This touches PolarisMetaStoreManager, which has both
> > > JDBC (atomic and transactional) and NoSQL
> > > implementations. I wanted to check the direction before
> > > coding, in line with our interface-change guidance.
> > >
> > > Does keeping the entity reserved and gating its removal
> > > on cleanup completion sound right, or would you prefer
> > > a different approach, such as a dedicated purge task
> > > that handles the final entity removal?
> > >
> > > Thanks,
> > > Prithvi
> > >
> > > >
> > >
>

Reply via email to