Hey Dan, The two scenarios you mentioned make sense to me and PR [1] has been focused on implementing the first scenario. Please take a look when you have the chance. Also I would like to second Prashant's point of scenario 1 not requiring a spec change, let us know your thoughts on this.
For scenario 2 of using normal planning mode, I agree that we should reuse the general `/sign` endpoint with an additional spec change for adding a new option to access delegation like `X-Iceberg-Access-Delegation: pre-signed-urls`. Additionally, as Prashant mentions above, considering the per-file nature of the current `/sign` endpoint, I believe this would be a good opportunity to discuss the option of a bulk-sign endpoint. WDYT? I can also provide a POC for this spec change. Thank you again and please take a look at [1]. Best, William [1] https://github.com/apache/iceberg/pull/17457 On 2026/08/25 03:03:01 Prashant Singh wrote: > Hey Dan, > The 2 scenarios you mentioned make sense to me specially from the pov that > we need an HTTPInputFile / HTTPInputStream and fileIO should use the > relevant input file/stream based on detecting if the url is presigned, we > have trying to achieve the same via [1], where these two ly in the *core* > and is > available to all the fileIO and are generic enough to be repurposed. We > kept your feedback of integrating metrics especially in mind while > implementing this [2], please do have a look when you get some time ! > I feel for scenario 1, you mentioned, we would not require spec change, > please let us know wdyt ? > > For scenario 2 i.e client reaching out to server to get pre-signed url on > per file basis (assuming this because that what we have for remote > signing), based on the feedbacks we got during this discussion, i wonder if > it's the right time to think about the bulk sign api (which could return > pre-signed / remote signed response based on access delegation requested) > Since one of the concerns of the current remote signing was that it was > requested was per file and it can cause server overwhelm for very huge > tables, i know we have been parallely working on an effort to return back > server issue signer configs in signer requests to potentially optimize > authZ per request > but a bulk signing api would be really helpful to mitigate those concerns. > > Looking forward to your feedback ! > > [1] https://github.com/apache/iceberg/pull/17457 > [2] https://github.com/apache/iceberg/pull/17457#discussion_r3696073044 > > Thanks, > Prashant Singh > > On Mon, Aug 17, 2026 at 1:35 PM Daniel Weeks <[email protected]> wrote: > > > Hey William, > > > > I had a chance to sync with Russell on this topic and I agree that Azure > > has some unique differences that make the current remote signing > > difficult. > > > > In terms of how to best support the pre-signing models, I believe that we > > can actually accommodate that implementation under the current remote > > signing APIs. > > > > Back when we designed the remote signing APIs, we considered the case > > where the signer wanted to return a different path than what was originally > > requested (primarily for cases like routing/redirection). However, that > > also allows for the catalog to return a presigned URI instead of the > > originally requested path. The endpoint itself is general (just `/sign`) > > and adding a new option to access delegation would be a good signal for > > what a client is requesting. > > > > In terms of use cases, there are two different scenarios: > > 1. Where a scan/plan request returns presigned URLs: in this case the > > FileIO should detect the presigned url and execute it directly instead of > > using the SDK client. > > 2. The normal planning mode, where the FileIO should contact the signing > > service instead of going directly through the SDK to get a presigned URL. > > > > I believe both of these scenarios can be achieved by introducing a common > > BaseInputFile/BaseInputStream that shares the identification or redirection > > logic for presigning. > > > > Does this make sense? I'll also follow up on the PR, but I think there's > > very little we need to do in order to accommodate both paths. > > > > -Dan > > > > > > On Fri, Aug 14, 2026 at 1:43 PM William Hyun <[email protected]> wrote: > > > >> Here is a POC for following along with this claim about Azure not > >> supporting S3-style remote signing natively. > >> - https://github.com/williamhyun/Azure-SAS-Header-POC > >> > >> On 2026/07/21 20:01:16 William Hyun wrote: > >> > Hi All, > >> > > >> > Following up on the ongoing discussion thread regarding the file-level > >> access delegation modes in the Iceberg REST spec, I wanted to share some > >> findings from a cross-cloud perspective of how the major Cloud Service > >> Providers handle these protocols. > >> > > >> > When evaluating remote signing as an access mode, two major operational > >> hurdles emerge: strict 15-minute Time-To-Live constraints and a lack of > >> native support across all CSPs, particularly Azure. > >> > > >> > 1. Non-Configurable TTLs > >> > AWS: In the AWS SigV4 protocol, the calculated signature with a > >> timestamp inherently expires 15 minutes after. [1] > >> > GCS: GCS enforces an identical constraint for remote header signing. A > >> signature passed via request headers is valid only from 15 minutes prior to > >> the active datetime until exactly 15 minutes after. [2] > >> > > >> > 2. Lack of Native Remote Signing Support in Azure > >> > While AWS (SigV4) and GCS (V4) natively accept short-lived, granular > >> signatures via the Authorization header, Microsoft Azure Blob Storage and > >> ADLS Gen2 fundamentally reject this paradigm. > >> > > >> > In Azure Storage, a Shared Access Signature (SAS) is strictly a token > >> designed to be appended to the resource URI as query parameters. [3] > >> > To execute true remote header signing for Azure Storage, a catalog > >> would be forced to use the legacy SharedKey authorization scheme, > >> formatting the header as Authorization: SharedKey > >> <AccountName>:<Signature>. Constructing this signature mathematically > >> requires the catalog to hold the storage account's master access key. [4] > >> [5] > >> > Because of this, Microsoft explicitly recommends disallowing SharedKey > >> authorization entirely, advising modern architectures to rely instead on > >> User Delegation SAS tokens (via the URI) or Microsoft Entra ID. [5] > >> > > >> > On the other hand, pre-signed URLs are natively and uniformly supported > >> across all three major CSPs. While the underlying cryptographic signatures > >> and generation methods differ (AWS SigV4, GCS V4, Azure SAS), the resulting > >> access artifact is universally a standard, fetchable HTTPS URL. > >> Furthermore, pre-signed URLs allow for highly customizable expiration > >> times. The maximum TTL is upper bound by the validity of the signing > >> credential itself (e.g. temporary STS credentials which are default 1-hour > >> sessions but can be configured up to 12 hours). This provides significantly > >> more operational flexibility than the rigid 15-minute window enforced on > >> header-based signatures. > >> > > >> > Given the findings above, I would still like to discuss the possibility > >> of pre-signed URLs. > >> > During our last sync, a valid concern was raised: our current SDKs and > >> FileIO implementations don't support processing raw, pre-signed HTTPS URLs. > >> Thus requiring a new, separate read path to be built. To help us evaluate > >> exactly what this effort entails and to discuss the trade-offs, I’ve put > >> together a POC for what this pre-signed URL read path might look like: > >> > https://github.com/williamhyun/iceberg/pull/206 > >> > > >> > Looking forward to hearing your thoughts! > >> > > >> > Best, > >> > William > >> > > >> > [1] > >> https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-api-request-authentication.html#sqs-api-basic-authentication-process > >> > [2] > >> https://docs.cloud.google.com/storage/docs/authentication/signatures#active_datetime > >> > [3] > >> https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview#how-a-shared-access-signature-works > >> > [4] > >> https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#specifying-the-authorization-header > >> > [5] > >> https://learn.microsoft.com/en-us/azure/storage/common/authorize-data-access?tabs=blobs#authorization-for-data-operations > >> > > >> > > >> > On 2026/07/08 01:16:35 William Hyun wrote: > >> > > Hi all, > >> > > > >> > > Following up on the earlier discussion around file-level access > >> delegation > >> > > in the REST Catalog spec, I put together a small POC PR against my > >> personal > >> > > Iceberg fork to explore the remote signing + caching direction we > >> discussed: > >> > > > >> > > https://github.com/williamhyun/iceberg/pull/198 > >> > > > >> > > The goal is to make the tradeoffs more concrete and evaluate whether > >> this > >> > > could be a practical approach for finer-grained delegated access. > >> > > Feedback is very welcome. > >> > > > >> > > Best, > >> > > William > >> > > > >> > > > >> > > > >> > > On Thu, Jul 2, 2026 at 5:29 PM William Hyun <[email protected]> > >> wrote: > >> > > > >> > > > Hi all, > >> > > > > >> > > > Thank you again for the discussion during the 7/1 catalog community > >> sync. > >> > > > I wanted to summarize two alternative directions that came up to > >> > > > pre-signed URLs: > >> > > > 1. scoped credential vending > >> > > > 2. remote signing with caching and bulk-signing > >> > > > > >> > > > I think scoped credential vending is worth considering in cases > >> where the > >> > > > authorization boundary maps cleanly to a small number of stable > >> storage > >> > > > prefixes. > >> > > > Scoped credential vending seems less practical when access needs to > >> be > >> > > > granted for many specific files rather than a small number of > >> prefixes. > >> > > > > >> > > > In AWS, session policies are passed as parameters during > >> AssumeRole-style > >> > > > issuance when the temporary session is created. [1] > >> > > > This means that scoping down an already-issued credential cannot be > >> done > >> > > > locally by simply altering the existing token. > >> > > > Instead, it requires minting another STS session whose policy > >> encodes the > >> > > > new scope. > >> > > > This turns authorization into an STS issuance problem with > >> limitations on > >> > > > STS request frequency [2] and policy size. [3] > >> > > > > >> > > > By contrast, pre-signed URLs and remote signing can authorize exact > >> object > >> > > > requests using already-held credentials, without creating a new STS > >> session > >> > > > for each file set. > >> > > > Based on that discussion, I am exploring a POC around bulk remote > >> > > > signing + caching to evaluate whether it could be a practical > >> alternative > >> > > > to pre-signed URLs for these finer-grained access patterns. > >> > > > > >> > > > Best, > >> > > > William > >> > > > > >> > > > [1] > >> https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html > >> > > > [2] > >> > > > > >> https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html#reference_iam-quotas-sts-requests > >> > > > [3] > >> > > > > >> https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html > >> > > > > >> > > > On Tue, Jun 30, 2026 at 7:43 PM William Hyun <[email protected]> > >> wrote: > >> > > > > >> > > >> Hi everyone, > >> > > >> > >> > > >> I'd like to open a discussion on an extension to the REST Catalog > >> spec to > >> > > >> enable file-level access delegation during scan planning. > >> > > >> Today, delegated access is table-scoped, which can force > >> > > >> over-provisioning or table fragmentation when consumers should > >> only see a > >> > > >> subset of partitions or records. > >> > > >> This proposal adds a path using pre-signed URLs to make > >> partition-scoped > >> > > >> sharing practical in the REST Catalog model. > >> > > >> > >> > > >> Here is my proposal: https://s.apache.org/n16st > >> > > >> > >> > > >> I look forward to hearing your thoughts. > >> > > >> > >> > > >> Relevant discussion: > >> > > >> - https://lists.apache.org/thread/ko9kp0gvzhx85n7cvoxqnpw4vwnhmdg6 > >> > > >> > >> > > >> Best, > >> > > >> William > >> > > >> > >> > > >> > >> > > > >> > > >> > > >
