Hi Alex, Thanks for the detailed analysis and benchmarks!
I think authorizing each request makes sense as the default approach. A few thoughts: 1. For per-request authz and decision caching, I think we all agree that SigV4 signing itself happens on the Polaris server and is relatively cheap. So the authZ process may be a factor here, but we are not sure how much it slows things down. JDBC looks fine; I'm a bit surprised by the NoSQL results. Is there any particular reason for the -60% in NoSQL? My overall concern is that a cached allow decision could still be used to issue new S3 signatures, even after permissions are revoked. If we introduce this cache, I think we should clarify how permission changes invalidate cached decisions(which is tricky for OPA and ranger since the permission changes don't happen in Polaris). 2. For signing tokens for staged creation, I understand the table may not be registered yet, so checking permissions against an existing table may not work. However, I'm concerned about tokens remaining valid for hours without a way to revoke them. Could we use the token to carry the staged creation context, while still checking whether the caller has the required creation permissions on each signing request? We should clarify the token's scope, lifetime, and behavior when permissions change. 3. I raised batch signing in our earlier discussion as well. Iceberg has a proposal for a batch signing endpoint ([PR #18254](https://github.com/apache/iceberg/pull/18254)). Reducing network round trips may have a bigger impact than optimizing signing itself(decision cache and signing tokens). 4. Do we need a separate permission for remote signing? Remote signing is a mechanism for accessing storage, and we already have TABLE_READ_DATA and TABLE_WRITE_DATA. Adding another permission requires admins to grant it separately. What additional security boundary does TABLE_REMOTE_SIGN provide beyond the existing data access permissions? Overall, thanks a lot for picking this up. Looking forward to your new design. Yufei On Fri, Nov 6, 2026 at 9:28 AM Alex Dutra via dev <[email protected]> wrote: > > Hi again, > > As part of the remote signing work, here are the privileges and the > authorizable operations I'm proposing to introduce. Feedback welcome, > in particular on naming. > > New privilege: TABLE_REMOTE_SIGN > > - A table-level privilege, grantable like other table privileges (on a > table, or on a namespace or catalog for all tables below). > - It must be combined with TABLE_READ_DATA (to sign reads) and/or > TABLE_WRITE_DATA (to sign writes): on its own, it grants nothing. > - It is not implied by TABLE_FULL_METADATA, TABLE_READ_DATA or > TABLE_WRITE_DATA, so that remote signing is always granted > explicitly. It is implied though by CATALOG_MANAGE_CONTENT. > - With the Ranger authorizer, the corresponding access type is > table-remote-sign. > > New authorizable operations: > > - REMOTE_SIGN (requires TABLE_REMOTE_SIGN): checked when a > table is loaded, created or registered with remote signing delegation, in > addition to the existing *_WITH_READ/WRITE_DELEGATION operations. > For a table that does not exist yet, it is checked on the parent > namespace. > > - REMOTE_SIGN_READ and REMOTE_SIGN_WRITE (requiring > TABLE_REMOTE_SIGN plus TABLE_READ_DATA or TABLE_WRITE_DATA, > respectively): only needed if we choose to authorize each sign request > individually rather than through a signing token (see my previous message). > This allows us to have a single operation per sign request, which means a > single query to external authorizers such as OPA. > > Thanks, > Alex > > On Fri, Nov 6, 2026 at 6:02 PM Alex Dutra <[email protected]> wrote: > > > > Hi Tommy, hi all, > > > > Your assistance in testing and evaluating this feature is very welcome! > > > > Here are a few updates on my work: > > > > Since the old design doc [1] is mostly out of date, I'm currently > > exploring two design choices for authorizing sign requests that we > > briefly discussed, but never settled on: > > > > 1. A signing token, issued when the table is loaded and validated by > > the sign endpoint; this only works with Iceberg 1.12+. This is similar > > to how Project Nessie does remote signing. In this mode, the > > authorization decision is expected to be fast, because it is > > pre-computed in the token. > > > > 2. Per-request authorization, with or without a short-lived decision > > cache; this works for any Iceberg client. In this mode, authorization > > is done the normal way, which may incur in calls to the database > > and/or to the external authorizer, if any. The decision cache is a > > small enhancement where an authorization decision is cached for 30 > > seconds by default. > > > > To compare them, I created a new Gatling benchmark for remote signing > > [2], and I'm collecting preliminary results by running it locally > > against my working branch. A few early numbers (single machine, > > Polaris capped at 4 CPUs, 1000 sign requests per table load, so take > > them as indicative only), in max sign requests/s per node: > > > > - JDBC: token 12.7k, per-request without cache 10.3k (-19%), > > per-request with cache 12.4k > > > > - NoSQL: token 11.8k, per-request without cache 4.8k (-60%), > > per-request with cache 12.0k > > > > Using per-request authz without a decision cache is clearly the worst > > scenario, as expected: the gap is mostly due to entity resolution > > costs, and also cache deserialization for the NoSQL backend. I also > > tested external principals (credential-mode=external) and OPA: the > > numbers are similar or slightly worse due to repeated calls to OPA. > > > > However, per-request authz with a decision cache is roughly on par > > with signing tokens. That's an interesting observation and begs the > > question: which method should Polaris implement? Both? > > > > Each approach has pros and cons: > > > > - Signing tokens: > > - Pros: works out of the box with staged table creations: we capture > > the authz decision at load time > > - Cons: tokens are valid for X hours, and cannot be revoked or > > refreshed (at least not for now - maybe later) > > > > - Per-request authz with decision cache: > > - Pros: cache is short-lived (a few seconds) compared to token lifetime > > - Cons: does not work well with staged table creations (the sign > > request arrives before the table commit) > > > > I am inclined to use a mix of both: per-request authz by default, > > except for staged table creations where we'd use signing tokens. > > > > What are your thoughts? > > > > Thanks, > > Alex > > > > [1]: > > https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?usp=sharing > > [2]: https://github.com/apache/polaris-tools/pull/304 > > > > On Mon, Nov 2, 2026 at 9:27 PM Tommy Bollhofer <[email protected]> > > wrote: > > > > > > Hi Alex, > > > > > > Thank you for the update, and for linking the original thread. > > > > > > Great to hear that you're planning to revisit the previous PR. We'd be > > > happy to test an experimental build against our on-premises FlashBlade > > > S3-compatible storage and share our findings, including compatibility > > > across the various engines and versions we're currently using. > > > > > > Our client stack includes Trino with Iceberg 1.12, Spark with the Iceberg > > > 1.10 runtime, PyIceberg, and DuckDB. Given the relatively short duration > > > of our workloads, we should be able to begin testing without requiring > > > credential refresh support. > > > > > > Please let us know when there's a build available to evaluate. We'd be > > > glad to help validate the implementation and provide feedback. > > > > > > Thanks again for your work on this! > > > > > > Best, > > > Tommy > > > > > > > > > ________________________________________ > > > From: Alex Dutra <[email protected]> > > > Sent: Tuesday, October 27, 2026 13:19 > > > To: [email protected] <[email protected]> > > > Cc: Tommy Bollhofer <[email protected]> > > > Subject: Re: [DISCUSS] Resume remote signing work > > > > > > > > > Hi Tommy, > > > > > > Thanks for chiming in! Your reply got disconnected from the thread > > > though, so here is the link to my first message just in case: > > > > > > https://lists.apache.org/thread.html/vk5zj7qtrztl36c9c1qt6d09lryjltdk > > > > > > The details you provide are very valuable: since FlashBlade doesn't > > > support AssumeRole, remote signing becomes indeed a good alternative. And > > > since you can live for now without auth-credentials refresh, it seems to > > > me we have enough evidence that we should move forward and provide a > > > first, experimental implementation of remote signing. > > > > > > I will start preparing a revisited PR, built on top of the old one. > > > > > > Thanks, > > > Alex > > > > > > > > > > > > > > > On Thu, Oct 22, 2026 at 7:06 PM Tommy Bollhofer via dev > > > <[email protected]> wrote: > > > Hi all, > > > > > > +1 to resuming the remote signing work now, rather than waiting for the > > > spec on refreshing auth data. Alex asked on the issue (#32) for our use > > > case, so here it is. > > > > > > We run Polaris against an on-prem S3-compatible store (a FlashBlade). > > > Its STS support is limited to AssumeRoleWithWebIdentity, so Polaris > > > cannot vend credentials through its current AssumeRole flow. As a > > > result we turn vending off and distribute static storage keys to every > > > service in every environment: Polaris, Trino, Spark (table maintenance), > > > and PyIceberg and DuckDB clients. That is a rotation and blast-radius > > > burden that grows with each environment we add. Remote signing would > > > remove the static keys from the clients, which is why we would like to > > > see it move. > > > > > > On the auth-refresh gap: our workloads are mostly short queries and > > > pipeline runs, and our longest jobs are Spark maintenance operations > > > that take minutes, so as far as we can tell we could start without > > > refresh support and pick it up incrementally, as Alex suggested. > > > > > > Client versions are mixed on our side (Trino on Iceberg 1.12, Spark on > > > the Iceberg 1.10 runtime, PyIceberg and DuckDB with their own Iceberg > > > implementations), so we expect client-side support to vary. We would be > > > happy to test early builds against FlashBlade and report which engines > > > and versions work. > > > > > > (On #32 we also described a smaller option, AssumeRoleWithWebIdentity > > > as an alternative STS flow. Remote signing is the fuller fix and the > > > one we would rather see.) > > > > > > Best, > > > Tommy > > > > > > On 2026-10-02 15:00 UTC Alex Dutra wrote: > > > > Hi all, > > > > > > > > Polaris will soon be updated to Iceberg 1.12. > > > > > > > > This release includes the new remote-signing-config field within > > > > LoadTableResult, which addresses nearly all remaining requirements for > > > > Polaris to complete its remote signing implementation, abandoned a > > > > while ago due to impending spec changes. > > > > > > > > However, one detail remains: when we added the remote-signing-config > > > > structure, refreshing the contained authentication or authorization > > > > data was intentionally left out of scope. I plan to start a separate > > > > discussion and spec change proposal with the Iceberg community > > > > regarding this shortly. > > > > > > > > Despite this, the core remote signing functionality is again viable in > > > > Polaris today, with Iceberg 1.12. The only missing piece being that > > > > auth data cannot be refreshed yet, though this capability could be > > > > added incrementally in a future update. > > > > > > > > With that in mind, I would like to ask the Polaris community: > > > > > > > > Should we resume the remote signing work where we left [1]? Or would > > > > it be preferable to wait until the specification for refreshing auth > > > > data is finalized? > > > > > > > > For additional background, you can refer to the original DISCUSS > > > > thread [2], but keep in mind that much of it is now outdated. > > > > > > > > Thanks, > > > > Alex > > > > > > > > [1]: https://github.com/apache/polaris/pull/3995 > > > > [2]: > > > > https://lists.apache.org/thread.html/xpf60ko823c7mlg5bhnddq3btvdsq5s1 > > > > > > > > > > Tommy Bollhofer > > > Senior Data Engineer > > > Voloridge.com > > > [email protected] > > > 561.231.5770 > > > > > > This email and any attachments are intended only for the intended > > > recipient(s) and may contain information that is privileged and/or > > > confidential. If you are not an intended recipient, or have received this > > > email in error, please notify the sender immediately and permanently > > > delete this email, its attachments, and any copies. Any disclosure, > > > copying, distribution, or reliance on the contents by anyone other than > > > an intended recipient is unauthorized. This email message (including any > > > attachments) is not to be construed as investment advice or the > > > solicitation of an offer to buy or sell interests in any securities. Any > > > information provided in this message must be considered in conjunction > > > with the appropriate disclosure documents. PAST PERFORMANCE IS NOT > > > INDICATIVE OF FUTURE RESULTS.
