Hi all, As promised, here are the results of the remote signing benchmarks I ran on AWS.
TLDR: with per-request authorization, one Polaris node signs enough requests for a large Spark deployment. Queries run about as fast as with vended credentials, except on tables made of many small files, where they are ~1/3 slower. Lakekeeper, measured the same way, showed similar performance. The detailed results are here: https://docs.google.com/document/d/1itPK_H7HWT1Ar-KrYFDvB0OvZUJZRbPGIqjdgNVW-bY/edit?usp=sharing All in all, the results seem to indicate that the approach is sound and could sustain even heavy loads. Of course, the feature would be labeled "beta" initially and off by default, but I am confident that we can proceed to the first PR now. Let me know if you have any questions. Thanks, Alex On Tue, Nov 17, 2026 at 11:28 AM Alex Dutra <[email protected]> wrote: > > Hi all, > > @Tommy: > > Thank you for confirming that the decision cache makes sense for your > use case, and for your input on the drop-with-purge vs soft-delete > discussion. > > I am finalizing some benchmarks on AWS, I will share the numbers > shortly. After that, I think we can proceed to opening the first PR! > > @Dmitri: > > > However, can those checks slow down the signing process? > > Not really. The overlap check only runs on the staged-create and > dropped-table paths. Sign requests for existing tables don't run such > checks. > > > extending the validity of the signing token beyond the lifetime of the > > catalog entity does not look like an authorization decision to me. [...] > > However, using the authorizer for this with a different operation name > > (e.g. DELETE_FILES_AFTER_DROP) works too. > > I agree that the deletes themselves can be done by any principal with > write access. What the recorded decision prevents is a principal with > read-only access deleting files once the table is gone. So some > decision about the principal is needed. > > I also agree that DROP_TABLE_WITH_PURGE isn't the right operation for > it: the server performs no purge here. A dedicated operation such as > your DELETE_FILES_AFTER_DROP is clearer. > > > This feels more like a catalog-level flag to me with the value being the > > maximum time period allowed for use on deleted entities. > > The token's lifespan is already bounded (24 hours by default). This > setting is global currently, and applied for both staged-creates and > purged tables; I'd rather not add a separate, per-realm or per-catalog > setting for now. Do you feel strongly about the catalog-level > override? > > > I think leveraging the soft-delete feature to support Spark use cases is > > more robust and easier for users to manage. > > Agreed: a per-request overlap check can't exclude a table being > created at the same location concurrently, while a soft-deleted entity > keeps its hold on the locations. When soft delete is enabled, Polaris > should authorize these requests against the retained entity. Since > soft delete is opt-in, though, I'd keep the token-based path as the > fallback for catalogs that don't enable it. > > @Yufei: > > > engine side purging seems related to staging creation: signing needs to > > work when the table either no longer exists or has not been committed yet. > > Exactly, and the token handles both the same way: it carries what the > missing entity can't provide. > > > Could you elaborate on how signing tokens work with staged table creation > > out of the box? > > Here is the sequence with an Iceberg 1.12 client: > > 1. The client creates the table with stage-create. Polaris authorizes > the staged create on the namespace, validates the table's locations, > and returns the usual load-table response. That response includes a > remote signing token. The token records the table identifier, its > locations, that the operation is a staged create, the principal, and > an expiry. > > 2. The client calls the sign endpoint and echoes the token in the > request. The endpoint first tries to authorize against the table. > Since there's no table, it falls back to the token: the token must be > valid and mark a staged create. The request is then authorized against > the parent namespace, on every request, with the caller's current > privileges: the token only supplies the locations, not the decision. > > 3. Once the client commits the staged table, subsequent requests are > authorized against the table itself, rather than from the token. > > Note: clients older than Iceberg 1.12 don't send the token, so staged > creates with remote signing are refused for them. That will be > documented as a limitation. > > > One possible workaround for purge would be for the engine to enumerate the > > files and obtain signatures for the delete requests before dropping the > > table from the catalog > > I see 2 obstacles: the number of files to keep in memory can be huge, > and S3 refuses a request if the x-amz-date header is more than 15 > minutes old. The engine would have to enumerate the files, read the > manifests, sign the deletes, then execute them within 15 minutes. > Also, this needs the Iceberg client changes you mention, whereas the > token approach works with Spark as it is today. > > So, concretely, I'd change my current prototype as follows: > > - rename the recorded decision to a dedicated operation > (DELETE_FILES_AFTER_DROP); > - document that the token lifespan bounds both the staged-create and > post-drop windows; > - document that soft deletes, if enabled, should be preferred over > token-based authz. > > Does that sound reasonable to everybody? AWS numbers coming soon. > > Thanks, > Alex > > > On Tue, Nov 17, 2026 at 3:34 AM Yufei Gu <[email protected]> wrote: > > > > Tommy, thanks for the context. That's helpful. > > > > Alex, engine side purging seems related to staging creation: signing needs > > to work when the table either no longer exists or has not been committed > > yet. One possible workaround for purge would be for the engine to enumerate > > the files and obtain signatures for the delete requests before dropping the > > table from the catalog, then execute those deletes after the drop succeeds. > > In that model, I think a bounded cleanup window is reasonable. Credential > > vending already has a similar limitation: existing credentials can outlive > > the table, but refreshing them through the table endpoint stops working > > after the drop. The TTL may not be enough for a large table, but this > > limitation applies to both. We need to modify the IRC clients to make this > > work, though. > > > > Could you elaborate on how signing tokens work with staged table creation > > out of the box? I understand your updated proposal carries the staged > > locations in the token and checks permissions on the parent namespace. What > > I’m missing is how the signing endpoint is accessed when a table does not > > exist yet. During staged creation, you can not pre-create all files, > > engines must hit the sign endpoint constantly. > > > > Yufei > > > > > > On Mon, Nov 16, 2026 at 11:49 AM Tommy Bollhofer <[email protected]> > > wrote: > >> > >> Hi Alex, Yufei and Dmitri, > >> > >> Thanks for the detailed answers, Alex. Replies to the open points: > >> > >> - DuckDB: from what I can see it does not support remote signing today. > >> The Iceberg extension's ACCESS_DELEGATION_MODE option only accepts > >> vended_credentials (the default) and none [1]. Support is tracked in > >> duckdb-iceberg #670 (open since January; a maintainer said it is wanted > >> but lower priority) [2], and a community pull request, #1318, adds a > >> remote_signing mode through an Iceberg-owned filesystem; it is still open, > >> and the maintainers asked whether it can be done in httpfs instead [3]. > >> > >> - Persistence (Yufei): we use the relational JDBC persistence on > >> PostgreSQL. If it helps your Kubernetes benchmark, that is the setup I can > >> test against. > >> > >> - PyIceberg: we run PyIceberg 0.11.1 against our catalog, with the s3fs > >> extra installed, so the fsspec FileIO is available. Once a build exists I > >> will test it with py-io-impl=pyiceberg.io.fsspec.FsspecFileIO on existing > >> tables and report back what works with an S3-compatible store. > >> > >> - Spark: our compaction runs on Spark with the Iceberg 1.10 runtime, so it > >> would be in compat mode (existing tables only). Good to know that works; I > >> can try the 1.12 runtime for a test if that is useful. > >> > >> - Decision cache: from a user's side, a configurable TTL that is off by > >> default, with the maximum revocation delay documented as part of the > >> contract, is fine for us. > >> > >> - Drop with purge: we run with table purge disabled and delete files > >> outside the catalog, so we have no stake in the post-drop signing question > >> and no preference between the token-recorded decision and the soft-delete > >> approach. > >> > >> Best, > >> Tommy > >> > >> [1] https://duckdb.org/docs/stable/core_extensions/iceberg/iceberg_options > >> (ACCESS_DELEGATION_MODE: "Allowed values are vended_credentials and none") > >> [2] https://github.com/duckdb/duckdb-iceberg/issues/670 > >> [3] https://github.com/duckdb/duckdb-iceberg/pull/1318 > >> > >> > >> ________________________________________ > >> From: Dmitri Bourlatchkov <[email protected]> > >> Sent: Friday, November 13, 2026 18:19 > >> To: [email protected] <[email protected]> > >> Cc: Yufei Gu <[email protected]>; Tommy Bollhofer > >> <[email protected]>; Alex Dutra <[email protected]> > >> Subject: Re: [DISCUSS] Resume remote signing work > >> > >> > >> Hi Alex, > >> > >> Re: the DROP_TABLE_WITH_PURGE proposal: > >> > >> It looks like it should work with the location overlap checks that you > >> mentioned. However, can those checks slow down the signing process? > >> > >> Assuming the performance is acceptable, I think it is an interesting > >> approach and is worth exploring. > >> > >> DROP_TABLE_WITH_PURGE protects server-side actions invoked via dedicated > >> API methods / parameters. > >> > >> This use case is different in that all deletes are performed on the client > >> side. Any client with normal write access can perform these deletes before > >> dropping the table from the catalog. So extending the validity of the > >> signing token beyond the lifetime of the catalog entity does not look like > >> an authorization decision to me. Since we validate that locations do not > >> overlap, signed requests will stay within the scope of the original entity > >> anyway. > >> > >> This feels more like a catalog-level flag to me with the value being the > >> maximum time period allowed for use on deleted entities. However, using > >> the authorizer for this with different operation name (e.g. > >> DELETE_FILES_AFTER_DROP) works too. > >> > >> Re: soft delete: > >> > >> I think leveraging the soft-delete feature to support Spark use cases is > >> more robust and easier for users to manage. > >> > >> The soft-deleted entity still maintains a hold on table locations. I think > >> this offers a stronger protection against overlaps than one-off checks in > >> each signing request (check/create races are still possible). > >> > >> Undropping the table could check if (some key) files are still present. > >> > >> And yes, we can indeed offer the option of denying engines all access to > >> soft-deleted table files. > >> > >> Cheers, > >> Dmitri. > >> > >> On Fri, Nov 13, 2026 at 5:15 PM Alex Dutra via dev > >> <[email protected]> wrote: > >> Hi all, > >> > >> One more remote signing question, raised while testing with Spark. > >> > >> Spark implements DROP TABLE ... PURGE client-side: it loads the table, > >> drops it without purging it, then deletes its files itself, with the > >> FileIO of the loaded table [1]. With vended credentials this works, as > >> the credentials outlive the drop. With per-request authorization, each > >> sign request is authorized against a table that no longer exists, so > >> the files are left behind. > >> > >> Proposal: when a table is loaded, created or registered, Polaris also > >> asks the authorizer whether the caller is allowed to > >> DROP_TABLE_WITH_PURGE, and records the outcome in the remote > >> signing token, along with the table's ID and locations. Once the table is > >> dropped, sign requests carrying a valid token are allowed if the recorded > >> decision allows it, the request only reads or deletes objects at the > >> recorded locations, > >> the table was dropped rather than renamed, and no other table or > >> namespace now uses those locations. Requests without a token are > >> rejected. > >> > >> Requests for tables that exist are still authorized per request, with > >> or without a token. The decision is only used once the table's grants > >> are gone with it, and the authorizer (including OPA and Ranger) makes > >> it, at load time. Spark loads the table right before purging it, so > >> the decision is recent. > >> > >> The soft-delete proposal [2] would let Polaris authorize such requests > >> against the retained table instead, when enabled. Note that Spark's > >> purge is a drop without purge followed by client-side deletes, so with > >> soft delete on, Spark would delete the files of a table that can still > >> be undropped, unless the catalog refuses it. Remote signing would make > >> refusing possible. > >> > >> Thoughts? > >> > >> Thanks, > >> Alex > >> > >> [1] > >> https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java > >> [2] https://lists.apache.org/thread/1djyz0x3k7m71d71bqhw11c3syyshtdr > >> > >> > >> On Wed, Nov 11, 2026 at 9:21 AM Alex Dutra <[email protected]> wrote: > >> > > >> > Hi Yufei, > >> > > >> > Yes I think the cache is reasonable as long as it becomes part of the > >> > security constraints and is clearly documented. > >> > > >> > However, I would not take that -19% as granted: my benchmarks were on > >> > my local laptop only, where network calls to Postgres are cheap. I > >> > still need to reproduce the benchmark on k8s on some cloud provider, > >> > which I will shortly. But I expect those -19% to become worse, in > >> > which case, the cache may turn out to be an essential part of the > >> > setup. > >> > > >> > Thanks, > >> > Alex > >> > > >> > On Tue, Nov 10, 2026 at 7:24 PM Yufei Gu <[email protected]> wrote: > >> > > > >> > > On second thought, I believe invalidating the decision cache is hard > >> > > even with native RBAC, potentially making the system really > >> > > complicated. Given that OPA/Ranger cannot even support it, we'd prefer > >> > > the simpler solution(TTL). > >> > > > >> > > It looks like other systems provide the same decision cache with > >> > > certain constraints. However, A cache is acceptable only when its > >> > > maximum revocation delay is an explicit part of the security contract. > >> > > AWS Verified Permissions[1] caches authorization decisions for 120 > >> > > seconds by default, and says policy changes may not take effect until > >> > > the cache expires. OpenFGA has caching off by default and offers a > >> > > higher consistency mode that bypasses it. > >> > > > >> > > I think we should make the decision cache configurable and off by > >> > > default. When it is on, we must explicitly state the maximum > >> > > revocation delay as part of the security contract. > >> > > > >> > > Tommy, would you mind sharing which persistence you are using? I think > >> > > -19% for JDBC isn't too bad as the initial version, given that both > >> > > communities(Iceberg, Polaris) are actively working on batch signing. > >> > > > >> > > 1. > >> > > https://docs.aws.amazon.com/verifiedpermissions/latest/userguide/policy-stores-api-userpool-considerations-troubleshooting.html > >> > > > >> > > Yufei > >> > > > >> > > > >> > > > >> > > On Mon, Nov 9, 2026 at 11:36 AM Alex Dutra via dev > >> > > <[email protected]> wrote: > >> > >> > >> > >> Hi Tommy, > >> > >> > >> > >> Thanks for offering to test! > >> > >> > >> > >> Before answering your questions, I forgot to say that I am also > >> > >> planning to introduce a "compatibility mode" for older clients: this > >> > >> mode is automatically activated when there is no signing token in the > >> > >> request. Then Polaris would proceed as follows: > >> > >> > >> > >> - Sign requests without a token, on an existing table: accepted. > >> > >> - Sign requests without a token, on a missing table: rejected. > >> > >> > >> > >> In this mode, older clients can read and write existing tables, but > >> > >> get a clear error on CTAS (staged-creates). I think that strikes a > >> > >> good balance between support for older clients, while also taking > >> > >> security seriously by rejecting staged-creates. > >> > >> > >> > >> With that in mind: > >> > >> > >> > >> - Spark: with the Iceberg 1.12 runtime, all operations will be > >> > >> supported. With the 1.10 runtime, we would be in compat mode, and > >> > >> reads and writes on existing tables should work too, but not CTAS. > >> > >> > >> > >> - PyIceberg: should work for existing tables too, with the same > >> > >> limitation, but only with its fsspec FileIO > >> > >> (py-io-impl=pyiceberg.io.fsspec.FsspecFileIO); the default PyArrow > >> > >> FileIO does not support remote signing. I haven't tested it yet, so > >> > >> your feedback would be welcome. > >> > >> > >> > >> - Trino: no. Trino accesses S3 through its own file system, which does > >> > >> not implement remote signing, whatever the Iceberg library version. > >> > >> > >> > >> - DuckDB: no; as far as I know, its Iceberg extension does not support > >> > >> remote signing. Could you confirm please? > >> > >> > >> > >> I'll let you know when a build is available. > >> > >> > >> > >> Thanks, > >> > >> Alex > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> On Mon, Nov 9, 2026 at 4:13 PM Tommy Bollhofer > >> > >> <[email protected]> wrote: > >> > >> > > >> > >> > Hi Alex, Yufei and Dmitri, > >> > >> > > >> > >> > Thanks for the write-up and the numbers, Alex. Some input from a > >> > >> > user of a FlashBlade (S3-compatible, no AssumeRole) deployment, > >> > >> > where every client today holds static storage keys: > >> > >> > > >> > >> > - Per-request authorization with the short decision cache as the > >> > >> > default, and signing tokens only where staged creates need them, > >> > >> > sounds right to us. Our workloads are short, and a revocation > >> > >> > taking effect within the cache TTL (30 seconds) is fine for us. We > >> > >> > would not want a no-cache mode to be the default, given the numbers. > >> > >> > - On TABLE_REMOTE_SIGN: either way works for us, because our > >> > >> > data-writer roles already hold CATALOG_MANAGE_CONTENT (which would > >> > >> > imply it). We have no objection to dropping it and relying on > >> > >> > TABLE_READ_DATA / TABLE_WRITE_DATA plus the catalog-level > >> > >> > REMOTE_SIGNING_ENABLED flag, which keeps the privilege model > >> > >> > simpler. > >> > >> > - A 24-hour default for the signing token, configurable, is fine > >> > >> > for us. > >> > >> > > >> > >> > One question: Do you have a view on which engines and clients will > >> > >> > be able to use remote signing in the first release (Trino, Spark, > >> > >> > PyIceberg, DuckDB)? Our stack is Trino with Iceberg 1.12, Spark on > >> > >> > the Iceberg 1.10 runtime (we can move it up for a test), PyIceberg > >> > >> > and DuckDB. We would test whichever of them support it as soon as a > >> > >> > build is available, and report back what works on FlashBlade. > >> > >> > > >> > >> > Best, > >> > >> > Tommy > >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > ________________________________________ > >> > >> > From: Alex Dutra via dev <[email protected]> > >> > >> > Sent: Monday, November 09, 2026 09:16 > >> > >> > To: [email protected] <[email protected]> > >> > >> > Cc: Dmitri Bourlatchkov <[email protected]>; Alex Dutra > >> > >> > <[email protected]> > >> > >> > Subject: Re: [DISCUSS] Resume remote signing work > >> > >> > > >> > >> > > >> > >> > Hi Yufei, Dmitri, > >> > >> > > >> > >> > > >> > >> > > >> > >> > Thanks for the feedback! > >> > >> > > >> > >> > > >> > >> > > >> > >> > @Yufei: > >> > >> > > >> > >> > > >> > >> > > >> > >> > The NoSQL overhead is due to cache deserialization. The NoSQL cache > >> > >> > > >> > >> > was designed to be GC friendly and holds serialized entities; but > >> > >> > in a > >> > >> > > >> > >> > path that is mostly CPU bound like the sign endpoint, > >> > >> > deserialization > >> > >> > > >> > >> > quickly becomes the bottleneck, even with zero database calls. We > >> > >> > are > >> > >> > > >> > >> > aware of that, and already have a few ideas to improve its > >> > >> > efficiency. > >> > >> > > >> > >> > > >> > >> > > >> > >> > Cached decisions: we could invalidate the cache on permission > >> > >> > changes, > >> > >> > > >> > >> > but that would only work for built-in RBAC indeed. For OPA or > >> > >> > Ranger, > >> > >> > > >> > >> > revocations would take effect within the cache TTL. That's why I > >> > >> > think > >> > >> > > >> > >> > the cache TTL should be low: 30 seconds by default. I can benchmark > >> > >> > > >> > >> > shorter TTLs. That's also why the cache should only be applied to > >> > >> > the > >> > >> > > >> > >> > sign endpoint, not to others. > >> > >> > > >> > >> > > >> > >> > > >> > >> > “Signing tokens”: the benchmarks showed that per-request authz > >> > >> > remains > >> > >> > > >> > >> > doable performance-wise (no token needed), but for staged creates, > >> > >> > the > >> > >> > > >> > >> > authorizer needs some extra info that only a token can carry. With > >> > >> > > >> > >> > this context in mind, I would like to repurpose signing tokens as > >> > >> > > >> > >> > follows: every table load, create, register and staged create would > >> > >> > > >> > >> > return a token carrying only the table context (operation, catalog, > >> > >> > > >> > >> > table; locations as well for staged-creates) and the principal hash, > >> > >> > > >> > >> > but not its privileges. Each sign request would still be authorized > >> > >> > > >> > >> > individually: on the table for existing tables, or on the namespace > >> > >> > > >> > >> > for staged-creates. Permission changes therefore take effect > >> > >> > > >> > >> > immediately (modulo the decision cache TTL). The token allows > >> > >> > > >> > >> > staged-creates to operate optimally, and ties each sign request to > >> > >> > its > >> > >> > > >> > >> > initial operation, which improves auditing. Also, since it carries > >> > >> > > >> > >> > nothing secret anymore, it could be signed rather than encrypted, > >> > >> > > >> > >> > which is cheaper to verify. > >> > >> > > >> > >> > > >> > >> > > >> > >> > Batch sign: agreed, batch signing would help both models; but note > >> > >> > > >> > >> > that once it lands in the spec, it still needs to get support at > >> > >> > least > >> > >> > > >> > >> > in the Java signer client, so it's not something we can count on > >> > >> > > >> > >> > today. > >> > >> > > >> > >> > > >> > >> > > >> > >> > Permissions: my rationale is that Polaris signs requests with its > >> > >> > own > >> > >> > > >> > >> > credentials, and I didn’t want this to be possible without an > >> > >> > explicit > >> > >> > > >> > >> > permission. But the catalog-level REMOTE_SIGNING_ENABLED flag > >> > >> > already > >> > >> > > >> > >> > lets admins opt in, so I'm OK to drop TABLE_REMOTE_SIGN and rely on > >> > >> > > >> > >> > TABLE_READ_DATA / TABLE_WRITE_DATA, if everybody is on board. > >> > >> > > >> > >> > > >> > >> > > >> > >> > @Dmitri: > >> > >> > > >> > >> > > >> > >> > > >> > >> > The new token I'm proposing doesn’t need to expire for security > >> > >> > > >> > >> > reasons (it's not a security token), only for housekeeping sake > >> > >> > (allow > >> > >> > > >> > >> > key rotation, for instance). Therefore, I think one validity window > >> > >> > is > >> > >> > > >> > >> > enough, and one that is relatively generous is fine: I'd go with 24 > >> > >> > > >> > >> > hours by default, configurable. > >> > >> > > >> > >> > > >> > >> > > >> > >> > Thanks, > >> > >> > > >> > >> > Alex > >> > >> > > >> > >> > > >> > >> > > >> > >> > On Sat, Nov 7, 2026 at 2:18 AM Dmitri Bourlatchkov via dev > >> > >> > > >> > >> > <[email protected]> wrote: > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > Hi Alex, > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > It's great to see progress on remote signing. > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > I think the dual approach you proposed is reasonable. > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > Re: staged creates: the long-lived signing token is similar to a > >> > >> > > vended > >> > >> > > >> > >> > > credential. I believe it has the same problem that if the staging > >> > >> > > operation > >> > >> > > >> > >> > > takes a very long time, the token (or credential) can expire [1]. > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > However, with signing tokens the Polaris administrator has more > >> > >> > > control > >> > >> > > >> > >> > > over assigning the appropriate credential validity window, so I > >> > >> > > think it is > >> > >> > > >> > >> > > a useful feature to have. > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > It might be worth supporting different validity window > >> > >> > > configurations and > >> > >> > > >> > >> > > different authorizable operations for normal table access and > >> > >> > > staged create > >> > >> > > >> > >> > > requests. WDYT? > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > [1] > >> > >> > > https://lists.apache.org/thread.html/by053ptrxrxqx9fm57q64bl3bv41vy46 > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > Cheers, > >> > >> > > >> > >> > > Dmitri. > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > On Fri, Nov 6, 2026 at 12:04 PM Alex Dutra via dev > >> > >> > > <[email protected]> > >> > >> > > >> > >> > > wrote: > >> > >> > > >> > >> > > > >> > >> > > >> > >> > > > Hi Tommy, hi all, > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Your assistance in testing and evaluating this feature is very > >> > >> > > > welcome! > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Here are a few updates on my work: > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Since the old design doc [1] is mostly out of date, I'm > >> > >> > > > currently > >> > >> > > >> > >> > > > exploring two design choices for authorizing sign requests that > >> > >> > > > we > >> > >> > > >> > >> > > > briefly discussed, but never settled on: > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > 1. A signing token, issued when the table is loaded and > >> > >> > > > validated by > >> > >> > > >> > >> > > > the sign endpoint; this only works with Iceberg 1.12+. This is > >> > >> > > > similar > >> > >> > > >> > >> > > > to how Project Nessie does remote signing. In this mode, the > >> > >> > > >> > >> > > > authorization decision is expected to be fast, because it is > >> > >> > > >> > >> > > > pre-computed in the token. > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > 2. Per-request authorization, with or without a short-lived > >> > >> > > > decision > >> > >> > > >> > >> > > > cache; this works for any Iceberg client. In this mode, > >> > >> > > > authorization > >> > >> > > >> > >> > > > is done the normal way, which may incur in calls to the database > >> > >> > > >> > >> > > > and/or to the external authorizer, if any. The decision cache > >> > >> > > > is a > >> > >> > > >> > >> > > > small enhancement where an authorization decision is cached for > >> > >> > > > 30 > >> > >> > > >> > >> > > > seconds by default. > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > To compare them, I created a new Gatling benchmark for remote > >> > >> > > > signing > >> > >> > > >> > >> > > > [2], and I'm collecting preliminary results by running it > >> > >> > > > locally > >> > >> > > >> > >> > > > against my working branch. A few early numbers (single machine, > >> > >> > > >> > >> > > > Polaris capped at 4 CPUs, 1000 sign requests per table load, so > >> > >> > > > take > >> > >> > > >> > >> > > > them as indicative only), in max sign requests/s per node: > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > - JDBC: token 12.7k, per-request without cache 10.3k (-19%), > >> > >> > > >> > >> > > > per-request with cache 12.4k > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > - NoSQL: token 11.8k, per-request without cache 4.8k (-60%), > >> > >> > > >> > >> > > > per-request with cache 12.0k > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Using per-request authz without a decision cache is clearly the > >> > >> > > > worst > >> > >> > > >> > >> > > > scenario, as expected: the gap is mostly due to entity > >> > >> > > > resolution > >> > >> > > >> > >> > > > costs, and also cache deserialization for the NoSQL backend. I > >> > >> > > > also > >> > >> > > >> > >> > > > tested external principals (credential-mode=external) and OPA: > >> > >> > > > the > >> > >> > > >> > >> > > > numbers are similar or slightly worse due to repeated calls to > >> > >> > > > OPA. > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > However, per-request authz with a decision cache is roughly on > >> > >> > > > par > >> > >> > > >> > >> > > > with signing tokens. That's an interesting observation and begs > >> > >> > > > the > >> > >> > > >> > >> > > > question: which method should Polaris implement? Both? > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Each approach has pros and cons: > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > - Signing tokens: > >> > >> > > >> > >> > > > - Pros: works out of the box with staged table creations: we > >> > >> > > > capture > >> > >> > > >> > >> > > > the authz decision at load time > >> > >> > > >> > >> > > > - Cons: tokens are valid for X hours, and cannot be revoked or > >> > >> > > >> > >> > > > refreshed (at least not for now - maybe later) > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > - Per-request authz with decision cache: > >> > >> > > >> > >> > > > - Pros: cache is short-lived (a few seconds) compared to > >> > >> > > > token lifetime > >> > >> > > >> > >> > > > - Cons: does not work well with staged table creations (the > >> > >> > > > sign > >> > >> > > >> > >> > > > request arrives before the table commit) > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > I am inclined to use a mix of both: per-request authz by > >> > >> > > > default, > >> > >> > > >> > >> > > > except for staged table creations where we'd use signing tokens. > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > What are your thoughts? > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > Thanks, > >> > >> > > >> > >> > > > Alex > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > [1]: > >> > >> > > >> > >> > > > https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?usp=sharing > >> > >> > > >> > >> > > > [2]: https://github.com/apache/polaris-tools/pull/304 > >> > >> > > >> > >> > > > > >> > >> > > >> > >> > > > On Mon, Nov 2, 2026 at 9:27 PM Tommy Bollhofer > >> > >> > > > <[email protected]> > >> > >> > > >> > >> > > > wrote: > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Hi Alex, > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Thank you for the update, and for linking the original thread. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Great to hear that you're planning to revisit the previous > >> > >> > > > > PR. We'd be > >> > >> > > >> > >> > > > happy to test an experimental build against our on-premises > >> > >> > > > FlashBlade > >> > >> > > >> > >> > > > S3-compatible storage and share our findings, including > >> > >> > > > compatibility > >> > >> > > >> > >> > > > across the various engines and versions we're currently using. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Our client stack includes Trino with Iceberg 1.12, Spark with > >> > >> > > > > the > >> > >> > > >> > >> > > > Iceberg 1.10 runtime, PyIceberg, and DuckDB. Given the > >> > >> > > > relatively short > >> > >> > > >> > >> > > > duration of our workloads, we should be able to begin testing > >> > >> > > > without > >> > >> > > >> > >> > > > requiring credential refresh support. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Please let us know when there's a build available to > >> > >> > > > > evaluate. We'd be > >> > >> > > >> > >> > > > glad to help validate the implementation and provide feedback. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Thanks again for your work on this! > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Best, > >> > >> > > >> > >> > > > > Tommy > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > ________________________________________ > >> > >> > > >> > >> > > > > From: Alex Dutra <[email protected]> > >> > >> > > >> > >> > > > > Sent: Tuesday, October 27, 2026 13:19 > >> > >> > > >> > >> > > > > To: [email protected] <[email protected]> > >> > >> > > >> > >> > > > > Cc: Tommy Bollhofer <[email protected]> > >> > >> > > >> > >> > > > > Subject: Re: [DISCUSS] Resume remote signing work > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Hi Tommy, > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Thanks for chiming in! Your reply got disconnected from the > >> > >> > > > > thread > >> > >> > > >> > >> > > > though, so here is the link to my first message just in case: > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > https://lists.apache.org/thread.html/vk5zj7qtrztl36c9c1qt6d09lryjltdk > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > The details you provide are very valuable: since FlashBlade > >> > >> > > > > doesn't > >> > >> > > >> > >> > > > support AssumeRole, remote signing becomes indeed a good > >> > >> > > > alternative. And > >> > >> > > >> > >> > > > since you can live for now without auth-credentials refresh, it > >> > >> > > > seems to me > >> > >> > > >> > >> > > > we have enough evidence that we should move forward and provide > >> > >> > > > a first, > >> > >> > > >> > >> > > > experimental implementation of remote signing. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > I will start preparing a revisited PR, built on top of the > >> > >> > > > > old one. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Thanks, > >> > >> > > >> > >> > > > > Alex > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > On Thu, Oct 22, 2026 at 7:06 PM Tommy Bollhofer via dev < > >> > >> > > >> > >> > > > [email protected]> wrote: > >> > >> > > >> > >> > > > > Hi all, > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > +1 to resuming the remote signing work now, rather than > >> > >> > > > > waiting for the > >> > >> > > >> > >> > > > > spec on refreshing auth data. Alex asked on the issue (#32) > >> > >> > > > > for our use > >> > >> > > >> > >> > > > > case, so here it is. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > We run Polaris against an on-prem S3-compatible store (a > >> > >> > > > > FlashBlade). > >> > >> > > >> > >> > > > > Its STS support is limited to AssumeRoleWithWebIdentity, so > >> > >> > > > > Polaris > >> > >> > > >> > >> > > > > cannot vend credentials through its current AssumeRole flow. > >> > >> > > > > As a > >> > >> > > >> > >> > > > > result we turn vending off and distribute static storage keys > >> > >> > > > > to every > >> > >> > > >> > >> > > > > service in every environment: Polaris, Trino, Spark (table > >> > >> > > > > maintenance), > >> > >> > > >> > >> > > > > and PyIceberg and DuckDB clients. That is a rotation and > >> > >> > > > > blast-radius > >> > >> > > >> > >> > > > > burden that grows with each environment we add. Remote > >> > >> > > > > signing would > >> > >> > > >> > >> > > > > remove the static keys from the clients, which is why we > >> > >> > > > > would like to > >> > >> > > >> > >> > > > > see it move. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > On the auth-refresh gap: our workloads are mostly short > >> > >> > > > > queries and > >> > >> > > >> > >> > > > > pipeline runs, and our longest jobs are Spark maintenance > >> > >> > > > > operations > >> > >> > > >> > >> > > > > that take minutes, so as far as we can tell we could start > >> > >> > > > > without > >> > >> > > >> > >> > > > > refresh support and pick it up incrementally, as Alex > >> > >> > > > > suggested. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Client versions are mixed on our side (Trino on Iceberg 1.12, > >> > >> > > > > Spark on > >> > >> > > >> > >> > > > > the Iceberg 1.10 runtime, PyIceberg and DuckDB with their own > >> > >> > > > > Iceberg > >> > >> > > >> > >> > > > > implementations), so we expect client-side support to vary. > >> > >> > > > > We would be > >> > >> > > >> > >> > > > > happy to test early builds against FlashBlade and report > >> > >> > > > > which engines > >> > >> > > >> > >> > > > > and versions work. > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > (On #32 we also described a smaller option, > >> > >> > > > > AssumeRoleWithWebIdentity > >> > >> > > >> > >> > > > > as an alternative STS flow. Remote signing is the fuller fix > >> > >> > > > > and the > >> > >> > > >> > >> > > > > one we would rather see.) > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Best, > >> > >> > > >> > >> > > > > Tommy > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > On 2026-10-02 15:00 UTC Alex Dutra wrote: > >> > >> > > >> > >> > > > > > Hi all, > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > Polaris will soon be updated to Iceberg 1.12. > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > This release includes the new remote-signing-config field > >> > >> > > > > > within > >> > >> > > >> > >> > > > > > LoadTableResult, which addresses nearly all remaining > >> > >> > > > > > requirements for > >> > >> > > >> > >> > > > > > Polaris to complete its remote signing implementation, > >> > >> > > > > > abandoned a > >> > >> > > >> > >> > > > > > while ago due to impending spec changes. > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > However, one detail remains: when we added the > >> > >> > > > > > remote-signing-config > >> > >> > > >> > >> > > > > > structure, refreshing the contained authentication or > >> > >> > > > > > authorization > >> > >> > > >> > >> > > > > > data was intentionally left out of scope. I plan to start a > >> > >> > > > > > separate > >> > >> > > >> > >> > > > > > discussion and spec change proposal with the Iceberg > >> > >> > > > > > community > >> > >> > > >> > >> > > > > > regarding this shortly. > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > Despite this, the core remote signing functionality is > >> > >> > > > > > again viable in > >> > >> > > >> > >> > > > > > Polaris today, with Iceberg 1.12. The only missing piece > >> > >> > > > > > being that > >> > >> > > >> > >> > > > > > auth data cannot be refreshed yet, though this capability > >> > >> > > > > > could be > >> > >> > > >> > >> > > > > > added incrementally in a future update. > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > With that in mind, I would like to ask the Polaris > >> > >> > > > > > community: > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > Should we resume the remote signing work where we left [1]? > >> > >> > > > > > Or would > >> > >> > > >> > >> > > > > > it be preferable to wait until the specification for > >> > >> > > > > > refreshing auth > >> > >> > > >> > >> > > > > > data is finalized? > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > For additional background, you can refer to the original > >> > >> > > > > > DISCUSS > >> > >> > > >> > >> > > > > > thread [2], but keep in mind that much of it is now > >> > >> > > > > > outdated. > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > Thanks, > >> > >> > > >> > >> > > > > > Alex > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > [1]: https://github.com/apache/polaris/pull/3995 > >> > >> > > >> > >> > > > > > [2]: > >> > >> > > >> > >> > > > https://lists.apache.org/thread.html/xpf60ko823c7mlg5bhnddq3btvdsq5s1 > >> > >> > > >> > >> > > > > > > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > Tommy Bollhofer > >> > >> > > >> > >> > > > > Senior Data Engineer > >> > >> > > >> > >> > > > > Voloridge.com > >> > >> > > >> > >> > > > > [email protected] > >> > >> > > >> > >> > > > > 561.231.5770 <(561)%20231-5770> > >> > >> > > >> > >> > > > > > >> > >> > > >> > >> > > > > This email and any attachments are intended only for the > >> > >> > > > > intended > >> > >> > > >> > >> > > > recipient(s) and may contain information that is privileged > >> > >> > > > and/or > >> > >> > > >> > >> > > > confidential. If you are not an intended recipient, or have > >> > >> > > > received this > >> > >> > > >> > >> > > > email in error, please notify the sender immediately and > >> > >> > > > permanently delete > >> > >> > > >> > >> > > > this email, its attachments, and any copies. Any disclosure, > >> > >> > > > copying, > >> > >> > > >> > >> > > > distribution, or reliance on the contents by anyone other than > >> > >> > > > an intended > >> > >> > > >> > >> > > > recipient is unauthorized. This email message (including any > >> > >> > > > attachments) > >> > >> > > >> > >> > > > is not to be construed as investment advice or the solicitation > >> > >> > > > of an offer > >> > >> > > >> > >> > > > to buy or sell interests in any securities. Any information > >> > >> > > > provided in > >> > >> > > >> > >> > > > this message must be considered in conjunction with the > >> > >> > > > appropriate > >> > >> > > >> > >> > > > disclosure documents. PAST PERFORMANCE IS NOT INDICATIVE OF > >> > >> > > > FUTURE RESULTS. > >> > >> > > >> > >> > > > > >> > >> >
