Hi Yufei, Yes I think the cache is reasonable as long as it becomes part of the security constraints and is clearly documented.
However, I would not take that -19% as granted: my benchmarks were on my local laptop only, where network calls to Postgres are cheap. I still need to reproduce the benchmark on k8s on some cloud provider, which I will shortly. But I expect those -19% to become worse, in which case, the cache may turn out to be an essential part of the setup. Thanks, Alex On Tue, Nov 10, 2026 at 7:24 PM Yufei Gu <[email protected]> wrote: > > On second thought, I believe invalidating the decision cache is hard even > with native RBAC, potentially making the system really complicated. Given > that OPA/Ranger cannot even support it, we'd prefer the simpler solution(TTL). > > It looks like other systems provide the same decision cache with certain > constraints. However, A cache is acceptable only when its maximum revocation > delay is an explicit part of the security contract. AWS Verified > Permissions[1] caches authorization decisions for 120 seconds by default, and > says policy changes may not take effect until the cache expires. OpenFGA has > caching off by default and offers a higher consistency mode that bypasses it. > > I think we should make the decision cache configurable and off by default. > When it is on, we must explicitly state the maximum revocation delay as part > of the security contract. > > Tommy, would you mind sharing which persistence you are using? I think -19% > for JDBC isn't too bad as the initial version, given that both > communities(Iceberg, Polaris) are actively working on batch signing. > > 1. > https://docs.aws.amazon.com/verifiedpermissions/latest/userguide/policy-stores-api-userpool-considerations-troubleshooting.html > > Yufei > > > > On Mon, Nov 9, 2026 at 11:36 AM Alex Dutra via dev <[email protected]> > wrote: >> >> Hi Tommy, >> >> Thanks for offering to test! >> >> Before answering your questions, I forgot to say that I am also >> planning to introduce a "compatibility mode" for older clients: this >> mode is automatically activated when there is no signing token in the >> request. Then Polaris would proceed as follows: >> >> - Sign requests without a token, on an existing table: accepted. >> - Sign requests without a token, on a missing table: rejected. >> >> In this mode, older clients can read and write existing tables, but >> get a clear error on CTAS (staged-creates). I think that strikes a >> good balance between support for older clients, while also taking >> security seriously by rejecting staged-creates. >> >> With that in mind: >> >> - Spark: with the Iceberg 1.12 runtime, all operations will be >> supported. With the 1.10 runtime, we would be in compat mode, and >> reads and writes on existing tables should work too, but not CTAS. >> >> - PyIceberg: should work for existing tables too, with the same >> limitation, but only with its fsspec FileIO >> (py-io-impl=pyiceberg.io.fsspec.FsspecFileIO); the default PyArrow >> FileIO does not support remote signing. I haven't tested it yet, so >> your feedback would be welcome. >> >> - Trino: no. Trino accesses S3 through its own file system, which does >> not implement remote signing, whatever the Iceberg library version. >> >> - DuckDB: no; as far as I know, its Iceberg extension does not support >> remote signing. Could you confirm please? >> >> I'll let you know when a build is available. >> >> Thanks, >> Alex >> >> >> >> >> >> >> On Mon, Nov 9, 2026 at 4:13 PM Tommy Bollhofer <[email protected]> >> wrote: >> > >> > Hi Alex, Yufei and Dmitri, >> > >> > Thanks for the write-up and the numbers, Alex. Some input from a user of a >> > FlashBlade (S3-compatible, no AssumeRole) deployment, where every client >> > today holds static storage keys: >> > >> > - Per-request authorization with the short decision cache as the default, >> > and signing tokens only where staged creates need them, sounds right to >> > us. Our workloads are short, and a revocation taking effect within the >> > cache TTL (30 seconds) is fine for us. We would not want a no-cache mode >> > to be the default, given the numbers. >> > - On TABLE_REMOTE_SIGN: either way works for us, because our data-writer >> > roles already hold CATALOG_MANAGE_CONTENT (which would imply it). We have >> > no objection to dropping it and relying on TABLE_READ_DATA / >> > TABLE_WRITE_DATA plus the catalog-level REMOTE_SIGNING_ENABLED flag, which >> > keeps the privilege model simpler. >> > - A 24-hour default for the signing token, configurable, is fine for us. >> > >> > One question: Do you have a view on which engines and clients will be able >> > to use remote signing in the first release (Trino, Spark, PyIceberg, >> > DuckDB)? Our stack is Trino with Iceberg 1.12, Spark on the Iceberg 1.10 >> > runtime (we can move it up for a test), PyIceberg and DuckDB. We would >> > test whichever of them support it as soon as a build is available, and >> > report back what works on FlashBlade. >> > >> > Best, >> > Tommy >> > >> > >> > >> > >> > ________________________________________ >> > From: Alex Dutra via dev <[email protected]> >> > Sent: Monday, November 09, 2026 09:16 >> > To: [email protected] <[email protected]> >> > Cc: Dmitri Bourlatchkov <[email protected]>; Alex Dutra <[email protected]> >> > Subject: Re: [DISCUSS] Resume remote signing work >> > >> > >> > Hi Yufei, Dmitri, >> > >> > >> > >> > Thanks for the feedback! >> > >> > >> > >> > @Yufei: >> > >> > >> > >> > The NoSQL overhead is due to cache deserialization. The NoSQL cache >> > >> > was designed to be GC friendly and holds serialized entities; but in a >> > >> > path that is mostly CPU bound like the sign endpoint, deserialization >> > >> > quickly becomes the bottleneck, even with zero database calls. We are >> > >> > aware of that, and already have a few ideas to improve its efficiency. >> > >> > >> > >> > Cached decisions: we could invalidate the cache on permission changes, >> > >> > but that would only work for built-in RBAC indeed. For OPA or Ranger, >> > >> > revocations would take effect within the cache TTL. That's why I think >> > >> > the cache TTL should be low: 30 seconds by default. I can benchmark >> > >> > shorter TTLs. That's also why the cache should only be applied to the >> > >> > sign endpoint, not to others. >> > >> > >> > >> > “Signing tokens”: the benchmarks showed that per-request authz remains >> > >> > doable performance-wise (no token needed), but for staged creates, the >> > >> > authorizer needs some extra info that only a token can carry. With >> > >> > this context in mind, I would like to repurpose signing tokens as >> > >> > follows: every table load, create, register and staged create would >> > >> > return a token carrying only the table context (operation, catalog, >> > >> > table; locations as well for staged-creates) and the principal hash, >> > >> > but not its privileges. Each sign request would still be authorized >> > >> > individually: on the table for existing tables, or on the namespace >> > >> > for staged-creates. Permission changes therefore take effect >> > >> > immediately (modulo the decision cache TTL). The token allows >> > >> > staged-creates to operate optimally, and ties each sign request to its >> > >> > initial operation, which improves auditing. Also, since it carries >> > >> > nothing secret anymore, it could be signed rather than encrypted, >> > >> > which is cheaper to verify. >> > >> > >> > >> > Batch sign: agreed, batch signing would help both models; but note >> > >> > that once it lands in the spec, it still needs to get support at least >> > >> > in the Java signer client, so it's not something we can count on >> > >> > today. >> > >> > >> > >> > Permissions: my rationale is that Polaris signs requests with its own >> > >> > credentials, and I didn’t want this to be possible without an explicit >> > >> > permission. But the catalog-level REMOTE_SIGNING_ENABLED flag already >> > >> > lets admins opt in, so I'm OK to drop TABLE_REMOTE_SIGN and rely on >> > >> > TABLE_READ_DATA / TABLE_WRITE_DATA, if everybody is on board. >> > >> > >> > >> > @Dmitri: >> > >> > >> > >> > The new token I'm proposing doesn’t need to expire for security >> > >> > reasons (it's not a security token), only for housekeeping sake (allow >> > >> > key rotation, for instance). Therefore, I think one validity window is >> > >> > enough, and one that is relatively generous is fine: I'd go with 24 >> > >> > hours by default, configurable. >> > >> > >> > >> > Thanks, >> > >> > Alex >> > >> > >> > >> > On Sat, Nov 7, 2026 at 2:18 AM Dmitri Bourlatchkov via dev >> > >> > <[email protected]> wrote: >> > >> > > >> > >> > > Hi Alex, >> > >> > > >> > >> > > It's great to see progress on remote signing. >> > >> > > >> > >> > > I think the dual approach you proposed is reasonable. >> > >> > > >> > >> > > Re: staged creates: the long-lived signing token is similar to a vended >> > >> > > credential. I believe it has the same problem that if the staging >> > > operation >> > >> > > takes a very long time, the token (or credential) can expire [1]. >> > >> > > >> > >> > > However, with signing tokens the Polaris administrator has more control >> > >> > > over assigning the appropriate credential validity window, so I think it >> > > is >> > >> > > a useful feature to have. >> > >> > > >> > >> > > It might be worth supporting different validity window configurations and >> > >> > > different authorizable operations for normal table access and staged >> > > create >> > >> > > requests. WDYT? >> > >> > > >> > >> > > [1] https://lists.apache.org/thread.html/by053ptrxrxqx9fm57q64bl3bv41vy46 >> > >> > > >> > >> > > Cheers, >> > >> > > Dmitri. >> > >> > > >> > >> > > On Fri, Nov 6, 2026 at 12:04 PM Alex Dutra via dev >> > > <[email protected]> >> > >> > > wrote: >> > >> > > >> > >> > > > Hi Tommy, hi all, >> > >> > > > >> > >> > > > Your assistance in testing and evaluating this feature is very welcome! >> > >> > > > >> > >> > > > Here are a few updates on my work: >> > >> > > > >> > >> > > > Since the old design doc [1] is mostly out of date, I'm currently >> > >> > > > exploring two design choices for authorizing sign requests that we >> > >> > > > briefly discussed, but never settled on: >> > >> > > > >> > >> > > > 1. A signing token, issued when the table is loaded and validated by >> > >> > > > the sign endpoint; this only works with Iceberg 1.12+. This is similar >> > >> > > > to how Project Nessie does remote signing. In this mode, the >> > >> > > > authorization decision is expected to be fast, because it is >> > >> > > > pre-computed in the token. >> > >> > > > >> > >> > > > 2. Per-request authorization, with or without a short-lived decision >> > >> > > > cache; this works for any Iceberg client. In this mode, authorization >> > >> > > > is done the normal way, which may incur in calls to the database >> > >> > > > and/or to the external authorizer, if any. The decision cache is a >> > >> > > > small enhancement where an authorization decision is cached for 30 >> > >> > > > seconds by default. >> > >> > > > >> > >> > > > To compare them, I created a new Gatling benchmark for remote signing >> > >> > > > [2], and I'm collecting preliminary results by running it locally >> > >> > > > against my working branch. A few early numbers (single machine, >> > >> > > > Polaris capped at 4 CPUs, 1000 sign requests per table load, so take >> > >> > > > them as indicative only), in max sign requests/s per node: >> > >> > > > >> > >> > > > - JDBC: token 12.7k, per-request without cache 10.3k (-19%), >> > >> > > > per-request with cache 12.4k >> > >> > > > >> > >> > > > - NoSQL: token 11.8k, per-request without cache 4.8k (-60%), >> > >> > > > per-request with cache 12.0k >> > >> > > > >> > >> > > > Using per-request authz without a decision cache is clearly the worst >> > >> > > > scenario, as expected: the gap is mostly due to entity resolution >> > >> > > > costs, and also cache deserialization for the NoSQL backend. I also >> > >> > > > tested external principals (credential-mode=external) and OPA: the >> > >> > > > numbers are similar or slightly worse due to repeated calls to OPA. >> > >> > > > >> > >> > > > However, per-request authz with a decision cache is roughly on par >> > >> > > > with signing tokens. That's an interesting observation and begs the >> > >> > > > question: which method should Polaris implement? Both? >> > >> > > > >> > >> > > > Each approach has pros and cons: >> > >> > > > >> > >> > > > - Signing tokens: >> > >> > > > - Pros: works out of the box with staged table creations: we capture >> > >> > > > the authz decision at load time >> > >> > > > - Cons: tokens are valid for X hours, and cannot be revoked or >> > >> > > > refreshed (at least not for now - maybe later) >> > >> > > > >> > >> > > > - Per-request authz with decision cache: >> > >> > > > - Pros: cache is short-lived (a few seconds) compared to token >> > > > lifetime >> > >> > > > - Cons: does not work well with staged table creations (the sign >> > >> > > > request arrives before the table commit) >> > >> > > > >> > >> > > > I am inclined to use a mix of both: per-request authz by default, >> > >> > > > except for staged table creations where we'd use signing tokens. >> > >> > > > >> > >> > > > What are your thoughts? >> > >> > > > >> > >> > > > Thanks, >> > >> > > > Alex >> > >> > > > >> > >> > > > [1]: >> > >> > > > https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?usp=sharing >> > >> > > > [2]: https://github.com/apache/polaris-tools/pull/304 >> > >> > > > >> > >> > > > On Mon, Nov 2, 2026 at 9:27 PM Tommy Bollhofer >> > > > <[email protected]> >> > >> > > > wrote: >> > >> > > > > >> > >> > > > > Hi Alex, >> > >> > > > > >> > >> > > > > Thank you for the update, and for linking the original thread. >> > >> > > > > >> > >> > > > > Great to hear that you're planning to revisit the previous PR. We'd >> > > > > be >> > >> > > > happy to test an experimental build against our on-premises FlashBlade >> > >> > > > S3-compatible storage and share our findings, including compatibility >> > >> > > > across the various engines and versions we're currently using. >> > >> > > > > >> > >> > > > > Our client stack includes Trino with Iceberg 1.12, Spark with the >> > >> > > > Iceberg 1.10 runtime, PyIceberg, and DuckDB. Given the relatively short >> > >> > > > duration of our workloads, we should be able to begin testing without >> > >> > > > requiring credential refresh support. >> > >> > > > > >> > >> > > > > Please let us know when there's a build available to evaluate. We'd >> > > > > be >> > >> > > > glad to help validate the implementation and provide feedback. >> > >> > > > > >> > >> > > > > Thanks again for your work on this! >> > >> > > > > >> > >> > > > > Best, >> > >> > > > > Tommy >> > >> > > > > >> > >> > > > > >> > >> > > > > ________________________________________ >> > >> > > > > From: Alex Dutra <[email protected]> >> > >> > > > > Sent: Tuesday, October 27, 2026 13:19 >> > >> > > > > To: [email protected] <[email protected]> >> > >> > > > > Cc: Tommy Bollhofer <[email protected]> >> > >> > > > > Subject: Re: [DISCUSS] Resume remote signing work >> > >> > > > > >> > >> > > > > >> > >> > > > > Hi Tommy, >> > >> > > > > >> > >> > > > > Thanks for chiming in! Your reply got disconnected from the thread >> > >> > > > though, so here is the link to my first message just in case: >> > >> > > > > >> > >> > > > > https://lists.apache.org/thread.html/vk5zj7qtrztl36c9c1qt6d09lryjltdk >> > >> > > > > >> > >> > > > > The details you provide are very valuable: since FlashBlade doesn't >> > >> > > > support AssumeRole, remote signing becomes indeed a good alternative. >> > > > And >> > >> > > > since you can live for now without auth-credentials refresh, it seems >> > > > to me >> > >> > > > we have enough evidence that we should move forward and provide a >> > > > first, >> > >> > > > experimental implementation of remote signing. >> > >> > > > > >> > >> > > > > I will start preparing a revisited PR, built on top of the old one. >> > >> > > > > >> > >> > > > > Thanks, >> > >> > > > > Alex >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > On Thu, Oct 22, 2026 at 7:06 PM Tommy Bollhofer via dev < >> > >> > > > [email protected]> wrote: >> > >> > > > > Hi all, >> > >> > > > > >> > >> > > > > +1 to resuming the remote signing work now, rather than waiting for >> > > > > the >> > >> > > > > spec on refreshing auth data. Alex asked on the issue (#32) for our >> > > > > use >> > >> > > > > case, so here it is. >> > >> > > > > >> > >> > > > > We run Polaris against an on-prem S3-compatible store (a FlashBlade). >> > >> > > > > Its STS support is limited to AssumeRoleWithWebIdentity, so Polaris >> > >> > > > > cannot vend credentials through its current AssumeRole flow. As a >> > >> > > > > result we turn vending off and distribute static storage keys to >> > > > > every >> > >> > > > > service in every environment: Polaris, Trino, Spark (table >> > > > > maintenance), >> > >> > > > > and PyIceberg and DuckDB clients. That is a rotation and blast-radius >> > >> > > > > burden that grows with each environment we add. Remote signing would >> > >> > > > > remove the static keys from the clients, which is why we would like >> > > > > to >> > >> > > > > see it move. >> > >> > > > > >> > >> > > > > On the auth-refresh gap: our workloads are mostly short queries and >> > >> > > > > pipeline runs, and our longest jobs are Spark maintenance operations >> > >> > > > > that take minutes, so as far as we can tell we could start without >> > >> > > > > refresh support and pick it up incrementally, as Alex suggested. >> > >> > > > > >> > >> > > > > Client versions are mixed on our side (Trino on Iceberg 1.12, Spark >> > > > > on >> > >> > > > > the Iceberg 1.10 runtime, PyIceberg and DuckDB with their own Iceberg >> > >> > > > > implementations), so we expect client-side support to vary. We would >> > > > > be >> > >> > > > > happy to test early builds against FlashBlade and report which >> > > > > engines >> > >> > > > > and versions work. >> > >> > > > > >> > >> > > > > (On #32 we also described a smaller option, AssumeRoleWithWebIdentity >> > >> > > > > as an alternative STS flow. Remote signing is the fuller fix and the >> > >> > > > > one we would rather see.) >> > >> > > > > >> > >> > > > > Best, >> > >> > > > > Tommy >> > >> > > > > >> > >> > > > > On 2026-10-02 15:00 UTC Alex Dutra wrote: >> > >> > > > > > Hi all, >> > >> > > > > > >> > >> > > > > > Polaris will soon be updated to Iceberg 1.12. >> > >> > > > > > >> > >> > > > > > This release includes the new remote-signing-config field within >> > >> > > > > > LoadTableResult, which addresses nearly all remaining requirements >> > > > > > for >> > >> > > > > > Polaris to complete its remote signing implementation, abandoned a >> > >> > > > > > while ago due to impending spec changes. >> > >> > > > > > >> > >> > > > > > However, one detail remains: when we added the >> > > > > > remote-signing-config >> > >> > > > > > structure, refreshing the contained authentication or authorization >> > >> > > > > > data was intentionally left out of scope. I plan to start a >> > > > > > separate >> > >> > > > > > discussion and spec change proposal with the Iceberg community >> > >> > > > > > regarding this shortly. >> > >> > > > > > >> > >> > > > > > Despite this, the core remote signing functionality is again >> > > > > > viable in >> > >> > > > > > Polaris today, with Iceberg 1.12. The only missing piece being that >> > >> > > > > > auth data cannot be refreshed yet, though this capability could be >> > >> > > > > > added incrementally in a future update. >> > >> > > > > > >> > >> > > > > > With that in mind, I would like to ask the Polaris community: >> > >> > > > > > >> > >> > > > > > Should we resume the remote signing work where we left [1]? Or >> > > > > > would >> > >> > > > > > it be preferable to wait until the specification for refreshing >> > > > > > auth >> > >> > > > > > data is finalized? >> > >> > > > > > >> > >> > > > > > For additional background, you can refer to the original DISCUSS >> > >> > > > > > thread [2], but keep in mind that much of it is now outdated. >> > >> > > > > > >> > >> > > > > > Thanks, >> > >> > > > > > Alex >> > >> > > > > > >> > >> > > > > > [1]: https://github.com/apache/polaris/pull/3995 >> > >> > > > > > [2]: >> > >> > > > https://lists.apache.org/thread.html/xpf60ko823c7mlg5bhnddq3btvdsq5s1 >> > >> > > > > > >> > >> > > > > >> > >> > > > > Tommy Bollhofer >> > >> > > > > Senior Data Engineer >> > >> > > > > Voloridge.com >> > >> > > > > [email protected] >> > >> > > > > 561.231.5770 <(561)%20231-5770> >> > >> > > > > >> > >> > > > > This email and any attachments are intended only for the intended >> > >> > > > recipient(s) and may contain information that is privileged and/or >> > >> > > > confidential. If you are not an intended recipient, or have received >> > > > this >> > >> > > > email in error, please notify the sender immediately and permanently >> > > > delete >> > >> > > > this email, its attachments, and any copies. Any disclosure, copying, >> > >> > > > distribution, or reliance on the contents by anyone other than an >> > > > intended >> > >> > > > recipient is unauthorized. This email message (including any >> > > > attachments) >> > >> > > > is not to be construed as investment advice or the solicitation of an >> > > > offer >> > >> > > > to buy or sell interests in any securities. Any information provided in >> > >> > > > this message must be considered in conjunction with the appropriate >> > >> > > > disclosure documents. PAST PERFORMANCE IS NOT INDICATIVE OF FUTURE >> > > > RESULTS. >> > >> > > > >> >
