Hi Alex,

> What the recorded decision prevents is a principal with
> read-only access deleting files once the table is gone. So some
> decision about the principal is needed.

Do not get me wrong. I fully support recording the user's allows operations
in the token.

What I meant in my earlier email is that the time frame when the token is
valid beyond the entity's lifetime can be decided without the Authorizer in
the loop. For example, a sample config setting might work. WDYT?

So a read-only user can never use the token to sign delete requests.

However, a user with basic drop privileges could still get a longer term
token without being granted a server-side "PURGE" permission, or a
dedicated permission for DELETE_FILES_AFTER_DROP.

That said, it may be a moon point. My impression is that the discussion on
the soft-delete thread [1] to moving towards performing authZ checks on the
entity for every signing request.

Cheers,
Dmitri.

[1] https://lists.apache.org/thread/xf4kmddfkv5o5f1qmzxtmz23z4vtdn0x

On Tue, Nov 17, 2026 at 5:30 AM Alex Dutra via dev <[email protected]>
wrote:

> Hi all,
>
> @Tommy:
>
> Thank you for confirming that the decision cache makes sense for your
> use case, and for your input on the drop-with-purge vs soft-delete
> discussion.
>
> I am finalizing some benchmarks on AWS, I will share the numbers
> shortly. After that, I think we can proceed to opening the first PR!
>
> @Dmitri:
>
> > However, can those checks slow down the signing process?
>
> Not really. The overlap check only runs on the staged-create and
> dropped-table paths. Sign requests for existing tables don't run such
> checks.
>
> > extending the validity of the signing token beyond the lifetime of the
> catalog entity does not look like an authorization decision to me.  [...]
> However, using the authorizer for this with a different operation name
> (e.g. DELETE_FILES_AFTER_DROP) works too.
>
> I agree that the deletes themselves can be done by any principal with
> write access. What the recorded decision prevents is a principal with
> read-only access deleting files once the table is gone. So some
> decision about the principal is needed.
>
> I also agree that DROP_TABLE_WITH_PURGE isn't the right operation for
> it: the server performs no purge here. A dedicated operation such as
> your DELETE_FILES_AFTER_DROP is clearer.
>
> > This feels more like a catalog-level flag to me with the value being the
> maximum time period allowed for use on deleted entities.
>
> The token's lifespan is already bounded (24 hours by default). This
> setting is global currently, and applied for both staged-creates and
> purged tables; I'd rather not add a separate, per-realm or per-catalog
> setting for now. Do you feel strongly about the catalog-level
> override?
>
> > I think leveraging the soft-delete feature to support Spark use cases is
> more robust and easier for users to manage.
>
> Agreed: a per-request overlap check can't exclude a table being
> created at the same location concurrently, while a soft-deleted entity
> keeps its hold on the locations. When soft delete is enabled, Polaris
> should authorize these requests against the retained entity. Since
> soft delete is opt-in, though, I'd keep the token-based path as the
> fallback for catalogs that don't enable it.
>
> @Yufei:
>
> > engine side purging seems related to staging creation: signing needs to
> work when the table either no longer exists or has not been committed yet.
>
> Exactly, and the token handles both the same way: it carries what the
> missing entity can't provide.
>
> > Could you elaborate on how signing tokens work with staged table
> creation out of the box?
>
> Here is the sequence with an Iceberg 1.12 client:
>
> 1. The client creates the table with stage-create. Polaris authorizes
> the staged create on the namespace, validates the table's locations,
> and returns the usual load-table response. That response includes a
> remote signing token. The token records the table identifier, its
> locations, that the operation is a staged create, the principal, and
> an expiry.
>
> 2. The client calls the sign endpoint and echoes the token in the
> request. The endpoint first tries to authorize against the table.
> Since there's no table, it falls back to the token: the token must be
> valid and mark a staged create. The request is then authorized against
> the parent namespace, on every request, with the caller's current
> privileges: the token only supplies the locations, not the decision.
>
> 3. Once the client commits the staged table, subsequent requests are
> authorized against the table itself, rather than from the token.
>
> Note: clients older than Iceberg 1.12 don't send the token, so staged
> creates with remote signing are refused for them. That will be
> documented as a limitation.
>
> > One possible workaround for purge would be for the engine to enumerate
> the files and obtain signatures for the delete requests before dropping the
> table from the catalog
>
> I see 2 obstacles: the number of files to keep in memory can be huge,
> and S3 refuses a request if the x-amz-date header is more than 15
> minutes old. The engine would have to enumerate the files, read the
> manifests, sign the deletes, then execute them within 15 minutes.
> Also, this needs the Iceberg client changes you mention, whereas the
> token approach works with Spark as it is today.
>
> So, concretely, I'd change my current prototype as follows:
>
> - rename the recorded decision to a dedicated operation
> (DELETE_FILES_AFTER_DROP);
> - document that the token lifespan bounds both the staged-create and
> post-drop windows;
> - document that soft deletes, if enabled, should be preferred over
> token-based authz.
>
> Does that sound reasonable to everybody? AWS numbers coming soon.
>
> Thanks,
> Alex
>
>
> On Tue, Nov 17, 2026 at 3:34 AM Yufei Gu <[email protected]> wrote:
> >
> > Tommy, thanks for the context. That's helpful.
> >
> > Alex, engine side purging seems related to staging creation: signing
> needs to work when the table either no longer exists or has not been
> committed yet. One possible workaround for purge would be for the engine to
> enumerate the files and obtain signatures for the delete requests before
> dropping the table from the catalog, then execute those deletes after the
> drop succeeds. In that model, I think a bounded cleanup window is
> reasonable. Credential vending already has a similar limitation: existing
> credentials can outlive the table, but refreshing them through the table
> endpoint stops working after the drop. The TTL may not be enough for a
> large table, but this limitation applies to both. We need to modify the IRC
> clients to make this work, though.
> >
> > Could you elaborate on how signing tokens work with staged table
> creation out of the box? I understand your updated proposal carries the
> staged locations in the token and checks permissions on the parent
> namespace. What I’m missing is how the signing endpoint is accessed when a
> table does not exist yet. During staged creation, you can not pre-create
> all files, engines must hit the sign endpoint constantly.
> >
> > Yufei
> >
> >
> > On Mon, Nov 16, 2026 at 11:49 AM Tommy Bollhofer <
> [email protected]> wrote:
> >>
> >> Hi Alex, Yufei and Dmitri,
> >>
> >> Thanks for the detailed answers, Alex. Replies to the open points:
> >>
> >> - DuckDB: from what I can see it does not support remote signing today.
> The Iceberg extension's ACCESS_DELEGATION_MODE option only accepts
> vended_credentials (the default) and none [1]. Support is tracked in
> duckdb-iceberg #670 (open since January; a maintainer said it is wanted but
> lower priority) [2], and a community pull request, #1318, adds a
> remote_signing mode through an Iceberg-owned filesystem; it is still open,
> and the maintainers asked whether it can be done in httpfs instead [3].
> >>
> >> - Persistence (Yufei): we use the relational JDBC persistence on
> PostgreSQL. If it helps your Kubernetes benchmark, that is the setup I can
> test against.
> >>
> >> - PyIceberg: we run PyIceberg 0.11.1 against our catalog, with the s3fs
> extra installed, so the fsspec FileIO is available. Once a build exists I
> will test it with py-io-impl=pyiceberg.io.fsspec.FsspecFileIO on existing
> tables and report back what works with an S3-compatible store.
> >>
> >> - Spark: our compaction runs on Spark with the Iceberg 1.10 runtime, so
> it would be in compat mode (existing tables only). Good to know that works;
> I can try the 1.12 runtime for a test if that is useful.
> >>
> >> - Decision cache: from a user's side, a configurable TTL that is off by
> default, with the maximum revocation delay documented as part of the
> contract, is fine for us.
> >>
> >> - Drop with purge: we run with table purge disabled and delete files
> outside the catalog, so we have no stake in the post-drop signing question
> and no preference between the token-recorded decision and the soft-delete
> approach.
> >>
> >> Best,
> >> Tommy
> >>
> >> [1]
> https://duckdb.org/docs/stable/core_extensions/iceberg/iceberg_options
> (ACCESS_DELEGATION_MODE: "Allowed values are vended_credentials and none")
> >> [2] https://github.com/duckdb/duckdb-iceberg/issues/670
> >> [3] https://github.com/duckdb/duckdb-iceberg/pull/1318
> >>
> >>
> >> ________________________________________
> >> From: Dmitri Bourlatchkov <[email protected]>
> >> Sent: Friday, November 13, 2026 18:19
> >> To: [email protected] <[email protected]>
> >> Cc: Yufei Gu <[email protected]>; Tommy Bollhofer <
> [email protected]>; Alex Dutra <[email protected]>
> >> Subject: Re: [DISCUSS] Resume remote signing work
> >>
> >>
> >> Hi Alex,
> >>
> >> Re: the DROP_TABLE_WITH_PURGE proposal:
> >>
> >> It looks like it should work with the location overlap checks that you
> mentioned. However, can those checks slow down the signing process?
> >>
> >> Assuming the performance is acceptable, I think it is an interesting
> approach and is worth exploring.
> >>
> >> DROP_TABLE_WITH_PURGE protects server-side actions invoked via
> dedicated API methods / parameters.
> >>
> >> This use case is different in that all deletes are performed on the
> client side. Any client with normal write access can perform these deletes
> before dropping the table from the catalog. So extending the validity of
> the signing token beyond the lifetime of the catalog entity does not look
> like an authorization decision to me. Since we validate that locations do
> not overlap, signed requests will stay within the scope of the original
> entity anyway.
> >>
> >> This feels more like a catalog-level flag to me with the value being
> the maximum time period allowed for use on deleted entities. However, using
> the authorizer for this with different operation name (e.g.
> DELETE_FILES_AFTER_DROP) works too.
> >>
> >> Re: soft delete:
> >>
> >> I think leveraging the soft-delete feature to support Spark use cases
> is more robust and easier for users to manage.
> >>
> >> The soft-deleted entity still maintains a hold on table locations. I
> think this offers a stronger protection against overlaps than one-off
> checks in each signing request (check/create races are still possible).
> >>
> >> Undropping the table could check if (some key) files are still present.
> >>
> >> And yes, we can indeed offer the option of denying engines all access
> to soft-deleted table files.
> >>
> >> Cheers,
> >> Dmitri.
> >>
> >> On Fri, Nov 13, 2026 at 5:15 PM Alex Dutra via dev <
> [email protected]> wrote:
> >> Hi all,
> >>
> >> One more remote signing question, raised while testing with Spark.
> >>
> >> Spark implements DROP TABLE ... PURGE client-side: it loads the table,
> >> drops it without purging it, then deletes its files itself, with the
> >> FileIO of the loaded table [1]. With vended credentials this works, as
> >> the credentials outlive the drop. With per-request authorization, each
> >> sign request is authorized against a table that no longer exists, so
> >> the files are left behind.
> >>
> >> Proposal: when a table is loaded, created or registered, Polaris also
> >> asks the authorizer whether the caller is allowed to
> >> DROP_TABLE_WITH_PURGE, and records the outcome in the remote
> >> signing token, along with the table's ID and locations. Once the table
> is
> >> dropped, sign requests carrying a valid token are allowed if the
> recorded
> >> decision allows it, the request only reads or deletes objects at the
> >> recorded locations,
> >> the table was dropped rather than renamed, and no other table or
> >> namespace now uses those locations. Requests without a token are
> >> rejected.
> >>
> >> Requests for tables that exist are still authorized per request, with
> >> or without a token. The decision is only used once the table's grants
> >> are gone with it, and the authorizer (including OPA and Ranger) makes
> >> it, at load time. Spark loads the table right before purging it, so
> >> the decision is recent.
> >>
> >> The soft-delete proposal [2] would let Polaris authorize such requests
> >> against the retained table instead, when enabled. Note that Spark's
> >> purge is a drop without purge followed by client-side deletes, so with
> >> soft delete on, Spark would delete the files of a table that can still
> >> be undropped, unless the catalog refuses it. Remote signing would make
> >> refusing possible.
> >>
> >> Thoughts?
> >>
> >> Thanks,
> >> Alex
> >>
> >> [1]
> https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
> >> [2] https://lists.apache.org/thread/1djyz0x3k7m71d71bqhw11c3syyshtdr
> >>
> >>
> >> On Wed, Nov 11, 2026 at 9:21 AM Alex Dutra <[email protected]> wrote:
> >> >
> >> > Hi Yufei,
> >> >
> >> > Yes I think the cache is reasonable as long as it becomes part of the
> >> > security constraints and is clearly documented.
> >> >
> >> > However, I would not take that -19% as granted: my benchmarks were on
> >> > my local laptop only, where network calls to Postgres are cheap. I
> >> > still need to reproduce the benchmark on k8s on some cloud provider,
> >> > which I will shortly. But I expect those -19% to become worse, in
> >> > which case, the cache may turn out to be an essential part of the
> >> > setup.
> >> >
> >> > Thanks,
> >> > Alex
> >> >
> >> > On Tue, Nov 10, 2026 at 7:24 PM Yufei Gu <[email protected]>
> wrote:
> >> > >
> >> > > On second thought, I believe invalidating the decision cache is
> hard even with native RBAC, potentially making the system really
> complicated. Given that OPA/Ranger cannot even support it, we'd prefer the
> simpler solution(TTL).
> >> > >
> >> > > It looks like other systems provide the same decision cache with
> certain constraints. However, A cache is acceptable only when its maximum
> revocation delay is an explicit part of the security contract. AWS Verified
> Permissions[1] caches authorization decisions for 120 seconds by default,
> and says policy changes may not take effect until the cache expires.
> OpenFGA has caching off by default and offers a higher consistency mode
> that bypasses it.
> >> > >
> >> > > I think we should make the decision cache configurable and off by
> default. When it is on, we must explicitly state the maximum revocation
> delay as part of the security contract.
> >> > >
> >> > > Tommy, would you mind sharing which persistence you are using? I
> think -19% for JDBC isn't too bad as the initial version, given that both
> communities(Iceberg, Polaris) are actively working on batch signing.
> >> > >
> >> > > 1.
> https://docs.aws.amazon.com/verifiedpermissions/latest/userguide/policy-stores-api-userpool-considerations-troubleshooting.html
> >> > >
> >> > > Yufei
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Nov 9, 2026 at 11:36 AM Alex Dutra via dev <
> [email protected]> wrote:
> >> > >>
> >> > >> Hi Tommy,
> >> > >>
> >> > >> Thanks for offering to test!
> >> > >>
> >> > >> Before answering your questions, I forgot to say that I am also
> >> > >> planning to introduce a "compatibility mode" for older clients:
> this
> >> > >> mode is automatically activated when there is no signing token in
> the
> >> > >> request. Then Polaris would proceed as follows:
> >> > >>
> >> > >> - Sign requests without a token, on an existing table: accepted.
> >> > >> - Sign requests without a token, on a missing table: rejected.
> >> > >>
> >> > >> In this mode, older clients can read and write existing tables, but
> >> > >> get a clear error on CTAS (staged-creates). I think that strikes a
> >> > >> good balance between support for older clients, while also taking
> >> > >> security seriously by rejecting staged-creates.
> >> > >>
> >> > >> With that in mind:
> >> > >>
> >> > >> - Spark: with the Iceberg 1.12 runtime, all operations will be
> >> > >> supported. With the 1.10 runtime, we would be in compat mode, and
> >> > >> reads and writes on existing tables should work too, but not CTAS.
> >> > >>
> >> > >> - PyIceberg: should work for existing tables too, with the same
> >> > >> limitation, but only with its fsspec FileIO
> >> > >> (py-io-impl=pyiceberg.io.fsspec.FsspecFileIO); the default PyArrow
> >> > >> FileIO does not support remote signing. I haven't tested it yet, so
> >> > >> your feedback would be welcome.
> >> > >>
> >> > >> - Trino: no. Trino accesses S3 through its own file system, which
> does
> >> > >> not implement remote signing, whatever the Iceberg library version.
> >> > >>
> >> > >> - DuckDB: no; as far as I know, its Iceberg extension does not
> support
> >> > >> remote signing. Could you confirm please?
> >> > >>
> >> > >> I'll let you know when a build is available.
> >> > >>
> >> > >> Thanks,
> >> > >> Alex
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >> On Mon, Nov 9, 2026 at 4:13 PM Tommy Bollhofer <
> [email protected]> wrote:
> >> > >> >
> >> > >> > Hi Alex, Yufei and Dmitri,
> >> > >> >
> >> > >> > Thanks for the write-up and the numbers, Alex. Some input from a
> user of a FlashBlade (S3-compatible, no AssumeRole) deployment, where every
> client today holds static storage keys:
> >> > >> >
> >> > >> > - Per-request authorization with the short decision cache as the
> default, and signing tokens only where staged creates need them, sounds
> right to us. Our workloads are short, and a revocation taking effect within
> the cache TTL (30 seconds) is fine for us. We would not want a no-cache
> mode to be the default, given the numbers.
> >> > >> > - On TABLE_REMOTE_SIGN: either way works for us, because our
> data-writer roles already hold CATALOG_MANAGE_CONTENT (which would imply
> it). We have no objection to dropping it and relying on TABLE_READ_DATA /
> TABLE_WRITE_DATA plus the catalog-level REMOTE_SIGNING_ENABLED flag, which
> keeps the privilege model simpler.
> >> > >> > - A 24-hour default for the signing token, configurable, is fine
> for us.
> >> > >> >
> >> > >> > One question: Do you have a view on which engines and clients
> will be able to use remote signing in the first release (Trino, Spark,
> PyIceberg, DuckDB)? Our stack is Trino with Iceberg 1.12, Spark on the
> Iceberg 1.10 runtime (we can move it up for a test), PyIceberg and DuckDB.
> We would test whichever of them support it as soon as a build is available,
> and report back what works on FlashBlade.
> >> > >> >
> >> > >> > Best,
> >> > >> > Tommy
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > ________________________________________
> >> > >> > From: Alex Dutra via dev <[email protected]>
> >> > >> > Sent: Monday, November 09, 2026 09:16
> >> > >> > To: [email protected] <[email protected]>
> >> > >> > Cc: Dmitri Bourlatchkov <[email protected]>; Alex Dutra <
> [email protected]>
> >> > >> > Subject: Re: [DISCUSS] Resume remote signing work
> >> > >> >
> >> > >> >
> >> > >> > Hi Yufei, Dmitri,
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > Thanks for the feedback!
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > @Yufei:
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > The NoSQL overhead is due to cache deserialization. The NoSQL
> cache
> >> > >> >
> >> > >> > was designed to be GC friendly and holds serialized entities;
> but in a
> >> > >> >
> >> > >> > path that is mostly CPU bound like the sign endpoint,
> deserialization
> >> > >> >
> >> > >> > quickly becomes the bottleneck, even with zero database calls.
> We are
> >> > >> >
> >> > >> > aware of that, and already have a few ideas to improve its
> efficiency.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > Cached decisions: we could invalidate the cache on permission
> changes,
> >> > >> >
> >> > >> > but that would only work for built-in RBAC indeed. For OPA or
> Ranger,
> >> > >> >
> >> > >> > revocations would take effect within the cache TTL. That's why I
> think
> >> > >> >
> >> > >> > the cache TTL should be low: 30 seconds by default. I can
> benchmark
> >> > >> >
> >> > >> > shorter TTLs. That's also why the cache should only be applied
> to the
> >> > >> >
> >> > >> > sign endpoint, not to others.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > “Signing tokens”: the benchmarks showed that per-request authz
> remains
> >> > >> >
> >> > >> > doable performance-wise (no token needed), but for staged
> creates, the
> >> > >> >
> >> > >> > authorizer needs some extra info that only a token can carry.
> With
> >> > >> >
> >> > >> > this context in mind, I would like to repurpose signing tokens as
> >> > >> >
> >> > >> > follows: every table load, create, register and staged create
> would
> >> > >> >
> >> > >> > return a token carrying only the table context (operation,
> catalog,
> >> > >> >
> >> > >> > table; locations as well for staged-creates) and the principal
> hash,
> >> > >> >
> >> > >> > but not its privileges. Each sign request would still be
> authorized
> >> > >> >
> >> > >> > individually: on the table for existing tables, or on the
> namespace
> >> > >> >
> >> > >> > for staged-creates. Permission changes therefore take effect
> >> > >> >
> >> > >> > immediately (modulo the decision cache TTL). The token allows
> >> > >> >
> >> > >> > staged-creates to operate optimally, and ties each sign request
> to its
> >> > >> >
> >> > >> > initial operation, which improves auditing. Also, since it
> carries
> >> > >> >
> >> > >> > nothing secret anymore, it could be signed rather than encrypted,
> >> > >> >
> >> > >> > which is cheaper to verify.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > Batch sign: agreed, batch signing would help both models; but
> note
> >> > >> >
> >> > >> > that once it lands in the spec, it still needs to get support at
> least
> >> > >> >
> >> > >> > in the Java signer client, so it's not something we can count on
> >> > >> >
> >> > >> > today.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > Permissions: my rationale is that Polaris signs requests with
> its own
> >> > >> >
> >> > >> > credentials, and I didn’t want this to be possible without an
> explicit
> >> > >> >
> >> > >> > permission. But the catalog-level REMOTE_SIGNING_ENABLED flag
> already
> >> > >> >
> >> > >> > lets admins opt in, so I'm OK to drop TABLE_REMOTE_SIGN and rely
> on
> >> > >> >
> >> > >> > TABLE_READ_DATA / TABLE_WRITE_DATA, if everybody is on board.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > @Dmitri:
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > The new token I'm proposing doesn’t need to expire for security
> >> > >> >
> >> > >> > reasons (it's not a security token), only for housekeeping sake
> (allow
> >> > >> >
> >> > >> > key rotation, for instance). Therefore, I think one validity
> window is
> >> > >> >
> >> > >> > enough, and one that is relatively generous is fine: I'd go with
> 24
> >> > >> >
> >> > >> > hours by default, configurable.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > Thanks,
> >> > >> >
> >> > >> > Alex
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Sat, Nov 7, 2026 at 2:18 AM Dmitri Bourlatchkov via dev
> >> > >> >
> >> > >> > <[email protected]> wrote:
> >> > >> >
> >> > >> > >
> >> > >> >
> >> > >> > > Hi Alex,
> >> > >> >
> >> > >> > >
> >> > >> >
> >> > >> > > It's great to see progress on remote signing.
> >> > >> >
> >> > >> > >
> >> > >> >
> >> > >> > > I think the dual approach you proposed is reasonable.
> >> > >> >
> >> > >> > >
> >> > >> >
> >> > >> > > Re: staged creates: the long-lived signing token is similar to
> a vended
> >> > >> >
> >> > >> > > credential. I believe it has the same problem that if the
> staging operation
> >> > >> >
> >> > >> > > takes a very long time, the token (or credential) can expire
> [1].
> >> > >> >
> >> > >> > >
> >> > >> >
> >> > >> > > However, with signing tokens the Polaris administrator has
> more control
> >> > >> >
> >> > >> > > over assigning the appropriate credential validity window, so
> I think it is
> >> > >> >
> >> > >> > > a useful feature to have.
> >> > >> >
> >> > >> > >
> >> > >> >
> >> > >> > > It might be worth supporting different validity window
> configurations and
> >> > >> >
> >> > >> > > different authorizable operations for normal table access and
> staged create
> >> > >> >
> >> > >> > > requests. WDYT?
> >> > >> >
> >> > >> > >
> >> > >> >
> >> > >> > > [1]
> https://lists.apache.org/thread.html/by053ptrxrxqx9fm57q64bl3bv41vy46
> >> > >> >
> >> > >> > >
> >> > >> >
> >> > >> > > Cheers,
> >> > >> >
> >> > >> > > Dmitri.
> >> > >> >
> >> > >> > >
> >> > >> >
> >> > >> > > On Fri, Nov 6, 2026 at 12:04 PM Alex Dutra via dev <
> [email protected]>
> >> > >> >
> >> > >> > > wrote:
> >> > >> >
> >> > >> > >
> >> > >> >
> >> > >> > > > Hi Tommy, hi all,
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > Your assistance in testing and evaluating this feature is
> very welcome!
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > Here are a few updates on my work:
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > Since the old design doc [1] is mostly out of date, I'm
> currently
> >> > >> >
> >> > >> > > > exploring two design choices for authorizing sign requests
> that we
> >> > >> >
> >> > >> > > > briefly discussed, but never settled on:
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > 1. A signing token, issued when the table is loaded and
> validated by
> >> > >> >
> >> > >> > > > the sign endpoint; this only works with Iceberg 1.12+. This
> is similar
> >> > >> >
> >> > >> > > > to how Project Nessie does remote signing. In this mode, the
> >> > >> >
> >> > >> > > > authorization decision is expected to be fast, because it is
> >> > >> >
> >> > >> > > > pre-computed in the token.
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > 2. Per-request authorization, with or without a short-lived
> decision
> >> > >> >
> >> > >> > > > cache; this works for any Iceberg client. In this mode,
> authorization
> >> > >> >
> >> > >> > > > is done the normal way, which may incur in calls to the
> database
> >> > >> >
> >> > >> > > > and/or to the external authorizer, if any. The decision
> cache is a
> >> > >> >
> >> > >> > > > small enhancement where an authorization decision is cached
> for 30
> >> > >> >
> >> > >> > > > seconds by default.
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > To compare them, I created a new Gatling benchmark for
> remote signing
> >> > >> >
> >> > >> > > > [2], and I'm collecting preliminary results by running it
> locally
> >> > >> >
> >> > >> > > > against my working branch. A few early numbers (single
> machine,
> >> > >> >
> >> > >> > > > Polaris capped at 4 CPUs, 1000 sign requests per table load,
> so take
> >> > >> >
> >> > >> > > > them as indicative only), in max sign requests/s per node:
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > - JDBC: token 12.7k, per-request without cache 10.3k (-19%),
> >> > >> >
> >> > >> > > > per-request with cache 12.4k
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > - NoSQL: token 11.8k, per-request without cache 4.8k (-60%),
> >> > >> >
> >> > >> > > > per-request with cache 12.0k
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > Using per-request authz without a decision cache is clearly
> the worst
> >> > >> >
> >> > >> > > > scenario, as expected: the gap is mostly due to entity
> resolution
> >> > >> >
> >> > >> > > > costs, and also cache deserialization for the NoSQL backend.
> I also
> >> > >> >
> >> > >> > > > tested external principals (credential-mode=external) and
> OPA: the
> >> > >> >
> >> > >> > > > numbers are similar or slightly worse due to repeated calls
> to OPA.
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > However, per-request authz with a decision cache is roughly
> on par
> >> > >> >
> >> > >> > > > with signing tokens. That's an interesting observation and
> begs the
> >> > >> >
> >> > >> > > > question: which method should Polaris implement? Both?
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > Each approach has pros and cons:
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > - Signing tokens:
> >> > >> >
> >> > >> > > >   - Pros: works out of the box with staged table creations:
> we capture
> >> > >> >
> >> > >> > > > the authz decision at load time
> >> > >> >
> >> > >> > > >   - Cons: tokens are valid for X hours, and cannot be
> revoked or
> >> > >> >
> >> > >> > > > refreshed (at least not for now - maybe later)
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > - Per-request authz with decision cache:
> >> > >> >
> >> > >> > > >   - Pros: cache is short-lived (a few seconds) compared to
> token lifetime
> >> > >> >
> >> > >> > > >   - Cons: does not work well with staged table creations
> (the sign
> >> > >> >
> >> > >> > > > request arrives before the table commit)
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > I am inclined to use a mix of both: per-request authz by
> default,
> >> > >> >
> >> > >> > > > except for staged table creations where we'd use signing
> tokens.
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > What are your thoughts?
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > Thanks,
> >> > >> >
> >> > >> > > > Alex
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > [1]:
> >> > >> >
> >> > >> > > >
> https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?usp=sharing
> >> > >> >
> >> > >> > > > [2]: https://github.com/apache/polaris-tools/pull/304
> >> > >> >
> >> > >> > > >
> >> > >> >
> >> > >> > > > On Mon, Nov 2, 2026 at 9:27 PM Tommy Bollhofer <
> [email protected]>
> >> > >> >
> >> > >> > > > wrote:
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Hi Alex,
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Thank you for the update, and for linking the original
> thread.
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Great to hear that you're planning to revisit the previous
> PR. We'd be
> >> > >> >
> >> > >> > > > happy to test an experimental build against our on-premises
> FlashBlade
> >> > >> >
> >> > >> > > > S3-compatible storage and share our findings, including
> compatibility
> >> > >> >
> >> > >> > > > across the various engines and versions we're currently
> using.
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Our client stack includes Trino with Iceberg 1.12, Spark
> with the
> >> > >> >
> >> > >> > > > Iceberg 1.10 runtime, PyIceberg, and DuckDB. Given the
> relatively short
> >> > >> >
> >> > >> > > > duration of our workloads, we should be able to begin
> testing without
> >> > >> >
> >> > >> > > > requiring credential refresh support.
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Please let us know when there's a build available to
> evaluate. We'd be
> >> > >> >
> >> > >> > > > glad to help validate the implementation and provide
> feedback.
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Thanks again for your work on this!
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Best,
> >> > >> >
> >> > >> > > > > Tommy
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > ________________________________________
> >> > >> >
> >> > >> > > > > From: Alex Dutra <[email protected]>
> >> > >> >
> >> > >> > > > > Sent: Tuesday, October 27, 2026 13:19
> >> > >> >
> >> > >> > > > > To: [email protected] <[email protected]>
> >> > >> >
> >> > >> > > > > Cc: Tommy Bollhofer <[email protected]>
> >> > >> >
> >> > >> > > > > Subject: Re: [DISCUSS] Resume remote signing work
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Hi Tommy,
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Thanks for chiming in! Your reply got disconnected from
> the thread
> >> > >> >
> >> > >> > > > though, so here is the link to my first message just in case:
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > >
> https://lists.apache.org/thread.html/vk5zj7qtrztl36c9c1qt6d09lryjltdk
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > The details you provide are very valuable: since
> FlashBlade doesn't
> >> > >> >
> >> > >> > > > support AssumeRole, remote signing becomes indeed a good
> alternative. And
> >> > >> >
> >> > >> > > > since you can live for now without auth-credentials refresh,
> it seems to me
> >> > >> >
> >> > >> > > > we have enough evidence that we should move forward and
> provide a first,
> >> > >> >
> >> > >> > > > experimental implementation of remote signing.
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > I will start preparing a revisited PR, built on top of the
> old one.
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Thanks,
> >> > >> >
> >> > >> > > > > Alex
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > On Thu, Oct 22, 2026 at 7:06 PM Tommy Bollhofer via dev <
> >> > >> >
> >> > >> > > > [email protected]> wrote:
> >> > >> >
> >> > >> > > > > Hi all,
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > +1 to resuming the remote signing work now, rather than
> waiting for the
> >> > >> >
> >> > >> > > > > spec on refreshing auth data. Alex asked on the issue
> (#32) for our use
> >> > >> >
> >> > >> > > > > case, so here it is.
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > We run Polaris against an on-prem S3-compatible store (a
> FlashBlade).
> >> > >> >
> >> > >> > > > > Its STS support is limited to AssumeRoleWithWebIdentity,
> so Polaris
> >> > >> >
> >> > >> > > > > cannot vend credentials through its current AssumeRole
> flow. As a
> >> > >> >
> >> > >> > > > > result we turn vending off and distribute static storage
> keys to every
> >> > >> >
> >> > >> > > > > service in every environment: Polaris, Trino, Spark (table
> maintenance),
> >> > >> >
> >> > >> > > > > and PyIceberg and DuckDB clients. That is a rotation and
> blast-radius
> >> > >> >
> >> > >> > > > > burden that grows with each environment we add. Remote
> signing would
> >> > >> >
> >> > >> > > > > remove the static keys from the clients, which is why we
> would like to
> >> > >> >
> >> > >> > > > > see it move.
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > On the auth-refresh gap: our workloads are mostly short
> queries and
> >> > >> >
> >> > >> > > > > pipeline runs, and our longest jobs are Spark maintenance
> operations
> >> > >> >
> >> > >> > > > > that take minutes, so as far as we can tell we could start
> without
> >> > >> >
> >> > >> > > > > refresh support and pick it up incrementally, as Alex
> suggested.
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Client versions are mixed on our side (Trino on Iceberg
> 1.12, Spark on
> >> > >> >
> >> > >> > > > > the Iceberg 1.10 runtime, PyIceberg and DuckDB with their
> own Iceberg
> >> > >> >
> >> > >> > > > > implementations), so we expect client-side support to
> vary. We would be
> >> > >> >
> >> > >> > > > > happy to test early builds against FlashBlade and report
> which engines
> >> > >> >
> >> > >> > > > > and versions work.
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > (On #32 we also described a smaller option,
> AssumeRoleWithWebIdentity
> >> > >> >
> >> > >> > > > > as an alternative STS flow. Remote signing is the fuller
> fix and the
> >> > >> >
> >> > >> > > > > one we would rather see.)
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Best,
> >> > >> >
> >> > >> > > > > Tommy
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > On 2026-10-02 15:00 UTC Alex Dutra wrote:
> >> > >> >
> >> > >> > > > > > Hi all,
> >> > >> >
> >> > >> > > > > >
> >> > >> >
> >> > >> > > > > > Polaris will soon be updated to Iceberg 1.12.
> >> > >> >
> >> > >> > > > > >
> >> > >> >
> >> > >> > > > > > This release includes the new remote-signing-config
> field within
> >> > >> >
> >> > >> > > > > > LoadTableResult, which addresses nearly all remaining
> requirements for
> >> > >> >
> >> > >> > > > > > Polaris to complete its remote signing implementation,
> abandoned a
> >> > >> >
> >> > >> > > > > > while ago due to impending spec changes.
> >> > >> >
> >> > >> > > > > >
> >> > >> >
> >> > >> > > > > > However, one detail remains: when we added the
> remote-signing-config
> >> > >> >
> >> > >> > > > > > structure, refreshing the contained authentication or
> authorization
> >> > >> >
> >> > >> > > > > > data was intentionally left out of scope. I plan to
> start a separate
> >> > >> >
> >> > >> > > > > > discussion and spec change proposal with the Iceberg
> community
> >> > >> >
> >> > >> > > > > > regarding this shortly.
> >> > >> >
> >> > >> > > > > >
> >> > >> >
> >> > >> > > > > > Despite this, the core remote signing functionality is
> again viable in
> >> > >> >
> >> > >> > > > > > Polaris today, with Iceberg 1.12. The only missing piece
> being that
> >> > >> >
> >> > >> > > > > > auth data cannot be refreshed yet, though this
> capability could be
> >> > >> >
> >> > >> > > > > > added incrementally in a future update.
> >> > >> >
> >> > >> > > > > >
> >> > >> >
> >> > >> > > > > > With that in mind, I would like to ask the Polaris
> community:
> >> > >> >
> >> > >> > > > > >
> >> > >> >
> >> > >> > > > > > Should we resume the remote signing work where we left
> [1]? Or would
> >> > >> >
> >> > >> > > > > > it be preferable to wait until the specification for
> refreshing auth
> >> > >> >
> >> > >> > > > > > data is finalized?
> >> > >> >
> >> > >> > > > > >
> >> > >> >
> >> > >> > > > > > For additional background, you can refer to the original
> DISCUSS
> >> > >> >
> >> > >> > > > > > thread [2], but keep in mind that much of it is now
> outdated.
> >> > >> >
> >> > >> > > > > >
> >> > >> >
> >> > >> > > > > > Thanks,
> >> > >> >
> >> > >> > > > > > Alex
> >> > >> >
> >> > >> > > > > >
> >> > >> >
> >> > >> > > > > > [1]: https://github.com/apache/polaris/pull/3995
> >> > >> >
> >> > >> > > > > > [2]:
> >> > >> >
> >> > >> > > >
> https://lists.apache.org/thread.html/xpf60ko823c7mlg5bhnddq3btvdsq5s1
> >> > >> >
> >> > >> > > > > >
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > Tommy Bollhofer
> >> > >> >
> >> > >> > > > > Senior Data Engineer
> >> > >> >
> >> > >> > > > > Voloridge.com
> >> > >> >
> >> > >> > > > > [email protected]
> >> > >> >
> >> > >> > > > > 561.231.5770 <(561)%20231-5770> <(561)%20231-5770>
> >> > >> >
> >> > >> > > > >
> >> > >> >
> >> > >> > > > > This email and any attachments are intended only for the
> intended
> >> > >> >
> >> > >> > > > recipient(s) and may contain information that is privileged
> and/or
> >> > >> >
> >> > >> > > > confidential. If you are not an intended recipient, or have
> received this
> >> > >> >
> >> > >> > > > email in error, please notify the sender immediately and
> permanently delete
> >> > >> >
> >> > >> > > > this email, its attachments, and any copies. Any disclosure,
> copying,
> >> > >> >
> >> > >> > > > distribution, or reliance on the contents by anyone other
> than an intended
> >> > >> >
> >> > >> > > > recipient is unauthorized. This email message (including any
> attachments)
> >> > >> >
> >> > >> > > > is not to be construed as investment advice or the
> solicitation of an offer
> >> > >> >
> >> > >> > > > to buy or sell interests in any securities. Any information
> provided in
> >> > >> >
> >> > >> > > > this message must be considered in conjunction with the
> appropriate
> >> > >> >
> >> > >> > > > disclosure documents. PAST PERFORMANCE IS NOT INDICATIVE OF
> FUTURE RESULTS.
> >> > >> >
> >> > >> > > >
> >> > >> >
>

Reply via email to