Hi Alex, It's great to see progress on remote signing.
I think the dual approach you proposed is reasonable. Re: staged creates: the long-lived signing token is similar to a vended credential. I believe it has the same problem that if the staging operation takes a very long time, the token (or credential) can expire [1]. However, with signing tokens the Polaris administrator has more control over assigning the appropriate credential validity window, so I think it is a useful feature to have. It might be worth supporting different validity window configurations and different authorizable operations for normal table access and staged create requests. WDYT? [1] https://lists.apache.org/thread.html/by053ptrxrxqx9fm57q64bl3bv41vy46 Cheers, Dmitri. On Fri, Nov 6, 2026 at 12:04 PM Alex Dutra via dev <[email protected]> wrote: > Hi Tommy, hi all, > > Your assistance in testing and evaluating this feature is very welcome! > > Here are a few updates on my work: > > Since the old design doc [1] is mostly out of date, I'm currently > exploring two design choices for authorizing sign requests that we > briefly discussed, but never settled on: > > 1. A signing token, issued when the table is loaded and validated by > the sign endpoint; this only works with Iceberg 1.12+. This is similar > to how Project Nessie does remote signing. In this mode, the > authorization decision is expected to be fast, because it is > pre-computed in the token. > > 2. Per-request authorization, with or without a short-lived decision > cache; this works for any Iceberg client. In this mode, authorization > is done the normal way, which may incur in calls to the database > and/or to the external authorizer, if any. The decision cache is a > small enhancement where an authorization decision is cached for 30 > seconds by default. > > To compare them, I created a new Gatling benchmark for remote signing > [2], and I'm collecting preliminary results by running it locally > against my working branch. A few early numbers (single machine, > Polaris capped at 4 CPUs, 1000 sign requests per table load, so take > them as indicative only), in max sign requests/s per node: > > - JDBC: token 12.7k, per-request without cache 10.3k (-19%), > per-request with cache 12.4k > > - NoSQL: token 11.8k, per-request without cache 4.8k (-60%), > per-request with cache 12.0k > > Using per-request authz without a decision cache is clearly the worst > scenario, as expected: the gap is mostly due to entity resolution > costs, and also cache deserialization for the NoSQL backend. I also > tested external principals (credential-mode=external) and OPA: the > numbers are similar or slightly worse due to repeated calls to OPA. > > However, per-request authz with a decision cache is roughly on par > with signing tokens. That's an interesting observation and begs the > question: which method should Polaris implement? Both? > > Each approach has pros and cons: > > - Signing tokens: > - Pros: works out of the box with staged table creations: we capture > the authz decision at load time > - Cons: tokens are valid for X hours, and cannot be revoked or > refreshed (at least not for now - maybe later) > > - Per-request authz with decision cache: > - Pros: cache is short-lived (a few seconds) compared to token lifetime > - Cons: does not work well with staged table creations (the sign > request arrives before the table commit) > > I am inclined to use a mix of both: per-request authz by default, > except for staged table creations where we'd use signing tokens. > > What are your thoughts? > > Thanks, > Alex > > [1]: > https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?usp=sharing > [2]: https://github.com/apache/polaris-tools/pull/304 > > On Mon, Nov 2, 2026 at 9:27 PM Tommy Bollhofer <[email protected]> > wrote: > > > > Hi Alex, > > > > Thank you for the update, and for linking the original thread. > > > > Great to hear that you're planning to revisit the previous PR. We'd be > happy to test an experimental build against our on-premises FlashBlade > S3-compatible storage and share our findings, including compatibility > across the various engines and versions we're currently using. > > > > Our client stack includes Trino with Iceberg 1.12, Spark with the > Iceberg 1.10 runtime, PyIceberg, and DuckDB. Given the relatively short > duration of our workloads, we should be able to begin testing without > requiring credential refresh support. > > > > Please let us know when there's a build available to evaluate. We'd be > glad to help validate the implementation and provide feedback. > > > > Thanks again for your work on this! > > > > Best, > > Tommy > > > > > > ________________________________________ > > From: Alex Dutra <[email protected]> > > Sent: Tuesday, October 27, 2026 13:19 > > To: [email protected] <[email protected]> > > Cc: Tommy Bollhofer <[email protected]> > > Subject: Re: [DISCUSS] Resume remote signing work > > > > > > Hi Tommy, > > > > Thanks for chiming in! Your reply got disconnected from the thread > though, so here is the link to my first message just in case: > > > > https://lists.apache.org/thread.html/vk5zj7qtrztl36c9c1qt6d09lryjltdk > > > > The details you provide are very valuable: since FlashBlade doesn't > support AssumeRole, remote signing becomes indeed a good alternative. And > since you can live for now without auth-credentials refresh, it seems to me > we have enough evidence that we should move forward and provide a first, > experimental implementation of remote signing. > > > > I will start preparing a revisited PR, built on top of the old one. > > > > Thanks, > > Alex > > > > > > > > > > On Thu, Oct 22, 2026 at 7:06 PM Tommy Bollhofer via dev < > [email protected]> wrote: > > Hi all, > > > > +1 to resuming the remote signing work now, rather than waiting for the > > spec on refreshing auth data. Alex asked on the issue (#32) for our use > > case, so here it is. > > > > We run Polaris against an on-prem S3-compatible store (a FlashBlade). > > Its STS support is limited to AssumeRoleWithWebIdentity, so Polaris > > cannot vend credentials through its current AssumeRole flow. As a > > result we turn vending off and distribute static storage keys to every > > service in every environment: Polaris, Trino, Spark (table maintenance), > > and PyIceberg and DuckDB clients. That is a rotation and blast-radius > > burden that grows with each environment we add. Remote signing would > > remove the static keys from the clients, which is why we would like to > > see it move. > > > > On the auth-refresh gap: our workloads are mostly short queries and > > pipeline runs, and our longest jobs are Spark maintenance operations > > that take minutes, so as far as we can tell we could start without > > refresh support and pick it up incrementally, as Alex suggested. > > > > Client versions are mixed on our side (Trino on Iceberg 1.12, Spark on > > the Iceberg 1.10 runtime, PyIceberg and DuckDB with their own Iceberg > > implementations), so we expect client-side support to vary. We would be > > happy to test early builds against FlashBlade and report which engines > > and versions work. > > > > (On #32 we also described a smaller option, AssumeRoleWithWebIdentity > > as an alternative STS flow. Remote signing is the fuller fix and the > > one we would rather see.) > > > > Best, > > Tommy > > > > On 2026-10-02 15:00 UTC Alex Dutra wrote: > > > Hi all, > > > > > > Polaris will soon be updated to Iceberg 1.12. > > > > > > This release includes the new remote-signing-config field within > > > LoadTableResult, which addresses nearly all remaining requirements for > > > Polaris to complete its remote signing implementation, abandoned a > > > while ago due to impending spec changes. > > > > > > However, one detail remains: when we added the remote-signing-config > > > structure, refreshing the contained authentication or authorization > > > data was intentionally left out of scope. I plan to start a separate > > > discussion and spec change proposal with the Iceberg community > > > regarding this shortly. > > > > > > Despite this, the core remote signing functionality is again viable in > > > Polaris today, with Iceberg 1.12. The only missing piece being that > > > auth data cannot be refreshed yet, though this capability could be > > > added incrementally in a future update. > > > > > > With that in mind, I would like to ask the Polaris community: > > > > > > Should we resume the remote signing work where we left [1]? Or would > > > it be preferable to wait until the specification for refreshing auth > > > data is finalized? > > > > > > For additional background, you can refer to the original DISCUSS > > > thread [2], but keep in mind that much of it is now outdated. > > > > > > Thanks, > > > Alex > > > > > > [1]: https://github.com/apache/polaris/pull/3995 > > > [2]: > https://lists.apache.org/thread.html/xpf60ko823c7mlg5bhnddq3btvdsq5s1 > > > > > > > Tommy Bollhofer > > Senior Data Engineer > > Voloridge.com > > [email protected] > > 561.231.5770 <(561)%20231-5770> > > > > This email and any attachments are intended only for the intended > recipient(s) and may contain information that is privileged and/or > confidential. If you are not an intended recipient, or have received this > email in error, please notify the sender immediately and permanently delete > this email, its attachments, and any copies. Any disclosure, copying, > distribution, or reliance on the contents by anyone other than an intended > recipient is unauthorized. This email message (including any attachments) > is not to be construed as investment advice or the solicitation of an offer > to buy or sell interests in any securities. Any information provided in > this message must be considered in conjunction with the appropriate > disclosure documents. PAST PERFORMANCE IS NOT INDICATIVE OF FUTURE RESULTS. >
