Hi Prithvi, hi all, Two points from the remote signing work that relate to this proposal.
1. Spark's DROP TABLE ... PURGE is a drop without purge. Spark doesn't ask the catalog to purge: - It loads the table and calls the catalog's drop with purge=false. - It then deletes the table's files itself, through the table's FileIO. With Spark, the catalog never sees a purge. On a catalog with soft-delete enabled, a Spark DROP TABLE ... PURGE would: - put the table on hold, with its name reserved, even though the user asked for a purge; - delete its files right away from the client. The hold would then keep a table whose files are gone, which matters once recovery is added. With vended credentials, Polaris can't see or prevent those deletes. This might be worth documenting. 2. Soft-delete would help remote signing. I'm reviving S3 remote signing. Polaris authorizes each sign request against the table, so it does see those deletes. Today, after a drop, the table and its grants are gone. My current approach records the principal's DROP_TABLE_WITH_PURGE decision in the remote signing token when the table is loaded. After the drop, it then allows only reads and deletes within the table's locations. In [5544], a soft-deleted table keeps its row, id and grants. So for those tables Polaris could: - authorize the deletes against the table itself when they happen, instead of relying on a decision recorded at load time; - choose a policy for the hold: either allow the deletes, since the client asked for a purge, or refuse them, so the table stays recoverable. Only remote signing makes the second option possible. Two questions: - Which policy makes sense for deletes during the hold? - The resolver hides dropped entities. Would an option to resolve a soft-deleted table, for this purpose only, be acceptable? Neither point should block Phase 1. I just want to make sure the two features fit together. Thanks, Alex [5544]: https://github.com/apache/polaris/pull/5544 On Thu, Nov 12, 2026 at 12:44 AM Dmitri Bourlatchkov via dev <[email protected]> wrote: > > Hi Prithvi and all, > > I did another review round in GH. > > Re-posting here the key comment: I'm very hesitant about > expiring soft-deleted entities during REST requests against other entities. > I think the resulting behaviours are too "magical" from the end user's > perspective. > > I believe the expiry use case belongs more naturally in the Admin Tool > given the current codebase. Eventually it could also be handled by the > delegation service [3990]. > > WDYT? > > Thanks, > Dmitri. > [3990] https://github.com/apache/polaris/pull/3990 > > On Mon, Oct 26, 2026 at 4:34 PM Prithvi S <[email protected]> > wrote: > > > Hi all, > > > > Quick follow-up on the cleanup gating for Phase 1: > > holding the identifier until the entity cleanup and its > > child tasks finish, then releasing it. > > > > While implementing this, I found that the coupling is > > in the metastore manager. dropEntityIfExists removes > > the entity (freeing the name) and schedules the cleanup > > task in the same transaction, by design. > > > > To keep the name reserved until cleanup completes, I'd > > need to separate these operations: keep the dropped > > entity, schedule cleanup, and remove the entity only > > after the cleanup task and its child tasks have finished. > > > > We don't currently have a way to detect when a task > > tree has completed. Tasks are fire-and-forget and > > self-remove on success, so I'd also need to add a small > > completion check. > > > > This touches PolarisMetaStoreManager, which has both > > JDBC (atomic and transactional) and NoSQL > > implementations. I wanted to check the direction before > > coding, in line with our interface-change guidance. > > > > Does keeping the entity reserved and gating its removal > > on cleanup completion sound right, or would you prefer > > a different approach, such as a dedicated purge task > > that handles the final entity removal? > > > > Thanks, > > Prithvi > > > > > > >
