Here is a POC for following along with this claim about Azure not supporting S3-style remote signing natively. - https://github.com/williamhyun/Azure-SAS-Header-POC
On 2026/07/21 20:01:16 William Hyun wrote: > Hi All, > > Following up on the ongoing discussion thread regarding the file-level access > delegation modes in the Iceberg REST spec, I wanted to share some findings > from a cross-cloud perspective of how the major Cloud Service Providers > handle these protocols. > > When evaluating remote signing as an access mode, two major operational > hurdles emerge: strict 15-minute Time-To-Live constraints and a lack of > native support across all CSPs, particularly Azure. > > 1. Non-Configurable TTLs > AWS: In the AWS SigV4 protocol, the calculated signature with a timestamp > inherently expires 15 minutes after. [1] > GCS: GCS enforces an identical constraint for remote header signing. A > signature passed via request headers is valid only from 15 minutes prior to > the active datetime until exactly 15 minutes after. [2] > > 2. Lack of Native Remote Signing Support in Azure > While AWS (SigV4) and GCS (V4) natively accept short-lived, granular > signatures via the Authorization header, Microsoft Azure Blob Storage and > ADLS Gen2 fundamentally reject this paradigm. > > In Azure Storage, a Shared Access Signature (SAS) is strictly a token > designed to be appended to the resource URI as query parameters. [3] > To execute true remote header signing for Azure Storage, a catalog would be > forced to use the legacy SharedKey authorization scheme, formatting the > header as Authorization: SharedKey <AccountName>:<Signature>. Constructing > this signature mathematically requires the catalog to hold the storage > account's master access key. [4] [5] > Because of this, Microsoft explicitly recommends disallowing SharedKey > authorization entirely, advising modern architectures to rely instead on User > Delegation SAS tokens (via the URI) or Microsoft Entra ID. [5] > > On the other hand, pre-signed URLs are natively and uniformly supported > across all three major CSPs. While the underlying cryptographic signatures > and generation methods differ (AWS SigV4, GCS V4, Azure SAS), the resulting > access artifact is universally a standard, fetchable HTTPS URL. Furthermore, > pre-signed URLs allow for highly customizable expiration times. The maximum > TTL is upper bound by the validity of the signing credential itself (e.g. > temporary STS credentials which are default 1-hour sessions but can be > configured up to 12 hours). This provides significantly more operational > flexibility than the rigid 15-minute window enforced on header-based > signatures. > > Given the findings above, I would still like to discuss the possibility of > pre-signed URLs. > During our last sync, a valid concern was raised: our current SDKs and FileIO > implementations don't support processing raw, pre-signed HTTPS URLs. Thus > requiring a new, separate read path to be built. To help us evaluate exactly > what this effort entails and to discuss the trade-offs, I’ve put together a > POC for what this pre-signed URL read path might look like: > https://github.com/williamhyun/iceberg/pull/206 > > Looking forward to hearing your thoughts! > > Best, > William > > [1] > https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-api-request-authentication.html#sqs-api-basic-authentication-process > [2] > https://docs.cloud.google.com/storage/docs/authentication/signatures#active_datetime > [3] > https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview#how-a-shared-access-signature-works > [4] > https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#specifying-the-authorization-header > [5] > https://learn.microsoft.com/en-us/azure/storage/common/authorize-data-access?tabs=blobs#authorization-for-data-operations > > > On 2026/07/08 01:16:35 William Hyun wrote: > > Hi all, > > > > Following up on the earlier discussion around file-level access delegation > > in the REST Catalog spec, I put together a small POC PR against my personal > > Iceberg fork to explore the remote signing + caching direction we discussed: > > > > https://github.com/williamhyun/iceberg/pull/198 > > > > The goal is to make the tradeoffs more concrete and evaluate whether this > > could be a practical approach for finer-grained delegated access. > > Feedback is very welcome. > > > > Best, > > William > > > > > > > > On Thu, Jul 2, 2026 at 5:29 PM William Hyun <[email protected]> wrote: > > > > > Hi all, > > > > > > Thank you again for the discussion during the 7/1 catalog community sync. > > > I wanted to summarize two alternative directions that came up to > > > pre-signed URLs: > > > 1. scoped credential vending > > > 2. remote signing with caching and bulk-signing > > > > > > I think scoped credential vending is worth considering in cases where the > > > authorization boundary maps cleanly to a small number of stable storage > > > prefixes. > > > Scoped credential vending seems less practical when access needs to be > > > granted for many specific files rather than a small number of prefixes. > > > > > > In AWS, session policies are passed as parameters during AssumeRole-style > > > issuance when the temporary session is created. [1] > > > This means that scoping down an already-issued credential cannot be done > > > locally by simply altering the existing token. > > > Instead, it requires minting another STS session whose policy encodes the > > > new scope. > > > This turns authorization into an STS issuance problem with limitations on > > > STS request frequency [2] and policy size. [3] > > > > > > By contrast, pre-signed URLs and remote signing can authorize exact object > > > requests using already-held credentials, without creating a new STS > > > session > > > for each file set. > > > Based on that discussion, I am exploring a POC around bulk remote > > > signing + caching to evaluate whether it could be a practical alternative > > > to pre-signed URLs for these finer-grained access patterns. > > > > > > Best, > > > William > > > > > > [1] https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html > > > [2] > > > https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html#reference_iam-quotas-sts-requests > > > [3] > > > https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html > > > > > > On Tue, Jun 30, 2026 at 7:43 PM William Hyun <[email protected]> wrote: > > > > > >> Hi everyone, > > >> > > >> I'd like to open a discussion on an extension to the REST Catalog spec to > > >> enable file-level access delegation during scan planning. > > >> Today, delegated access is table-scoped, which can force > > >> over-provisioning or table fragmentation when consumers should only see a > > >> subset of partitions or records. > > >> This proposal adds a path using pre-signed URLs to make partition-scoped > > >> sharing practical in the REST Catalog model. > > >> > > >> Here is my proposal: https://s.apache.org/n16st > > >> > > >> I look forward to hearing your thoughts. > > >> > > >> Relevant discussion: > > >> - https://lists.apache.org/thread/ko9kp0gvzhx85n7cvoxqnpw4vwnhmdg6 > > >> > > >> Best, > > >> William > > >> > > >> > > >
