> No R2-specific client implementation, signer, or plugin was needed for basic > reads and writes.
Awesome! The overall direction LGTM. Details can be discussed in PRs. Thanks Austen! Yufei On Thu, Sep 10, 2026 at 7:00 AM Austen Tomek <[email protected]> wrote: > > Just to confirm my understanding: "local signing" happens on the Polaris > server, which generates scoped, temporary credentials without calling > Cloudflare's credential API. > Yes that’s correct. Polaris generates the temporary credentials without > calling Cloudflare’s credential API. The client uses them to sign its S3 > requests, and R2 checks the permissions, prefix scope, and expiry. > Does this work out of the box with existing Iceberg REST Catalog clients that > support credential vending and S3FileIO, or does it require a special client > impl. or changes? > No R2-specific client implementation, signer, or plugin was needed for basic > reads and writes. Here’s what I’ve tested against a local patched Polaris > backed by real R2 buckets: > > > * > PyIceberg 0.11.1 and 0.12.0, with PyArrow 25.0.1: create, append, and read on > 0.11.1; append and read on 0.12.0. > * > DuckDB 1.5.5, with the Iceberg and httpfs extensions: create, insert, and > read. > * > Iceberg Java 1.11.0, with AWS SDK 2.44.7: reads through the unmodified > RESTCatalog and S3FileIO. Separate tests through our Java client covered > writes and multipart uploads, including the SDK’s CRC32 trailers. > * > Spark, Trino, and Flink: not tested yet. > > The clients consumed the normal Iceberg REST credential response, no > separately configured R2 storage credentials were necessary. > Option B: R2 as S3-compatible storage … > If this is the case, I'd lean toward considering R2 an S3-compatible storage, > which may not need a separate storage config type > I agree that the current approach does seem to be straddling two concepts as > Sushant pointed out. If we go with Option A, R2FileIO would basically be a > wrapper around S3FileIO given the R2 data plane is the S3 API and nothing R2 > specific happens on the client. I’m happy to explore R2 under the existing S3 > storage configuration. I originally chose a separate type because credential > issuance differs from AWS STS, but that difference seems clearly server side. > > My next step would be to prototype Option B on my fork: put local signing > behind the S3 configuration, work through how to represent the account ID and > jurisdiction, and rerun the client test matrix.. I’ll check that the > configuration changes preserve existing S3 and STS behavior. > > Agreed on keeping the PRs small. Using the existing S3 type should remove > R2’s need for the StorageTypeFileIO change in #4486. I expect we can separate > the configuration/API changes from the credential-vending implementation, but > I want to confirm that with the prototype before actually proposing the split. > > Does that sound OK to everyone? > > Thanks, Austen > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > From: Sushant Raikar via dev <[email protected]> > Date: Wednesday, September 9, 2026 at 19:16 > To: [email protected] <[email protected]> > Cc: Sushant Raikar <[email protected]> > Subject: Re: [DISCUSS] Cloudflare R2 support with scoped credential vending > > [You don't often get email from [email protected]. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] > > Thanks for kicking this off, Austen. R2 interest is clearly picking up, so > it's great to see this worked out in the open. > > One thing I noticed is that the design sits between two approaches, and I > wonder whether committing to one direction is cleaner than blending them. > > *Option A:* R2 as a fully first-class storage type. Own config, own > credential vending (the local JWT signing), and its own R2FileIO rather > than reusing S3FileIO. My thinking is that a dedicated FileIO keeps the > boundaries clean and sidesteps the "one FileIO maps to one storage type" > question in #4486, since R2 wouldn't share S3FileIO at all. The cost is > real though: a new FileIO probably means contributing R2FileIO upstream to > Iceberg, not just changes in Polaris. Curious whether that upstream path > seems reasonable to folks who've done it before. > > *Option B:* R2 as S3-compatible storage. If we go here, I think the main > design question is defining the lowest common denominator of credential > vending that the majority of s3.* contract can actually express, not just > for R2 but across S3-compatible backends generally (MinIO, Ceph RGW, > Blackblaze, Wasabi and so on). R2 would then fit inside that shared vending > path and accept its limits, rather than each backend bending s3.* in its > own direction. Would it make sense to scope that LCD first, before wiring > up R2 specifically? > > What I'd personally want to avoid is the middle ground: a distinct type > that still leans on S3FileIO and S3 assumptions underneath. That seems like > it carries the maintenance cost of a new type without the clean separation > that justifies it, though I might be underestimating how much the two can > safely share. > > On process, +1 to Yufei on splitting into small PRs. And I think his > framing is right: the local-signing path does look a lot like STS-style > vending from the client's side, so much of this may come down to whether > the config differences are big enough to warrant a separate type. > > Thanks again for putting this together. Happy to pitch in however's useful > as it moves forward. > > > On Wed, Sep 9, 2026 at 3:57 PM Yufei Gu <[email protected]> wrote: > > > Hi Austen, > > > > Just to confirm my understanding: “local signing” happens on the Polaris > > server, which generates scoped, temporary credentials without calling > > Cloudflare’s credential API. The client then uses those credentials to sign > > its own S3 requests, and R2 enforces the scope and expiry. This sounds like > > STS-style credential vending, where Polaris performs the issuance without > > talking to an object storage server. If this is the case, I'd lean toward > > considering R2 an S3-compatible storage, which may not need a separate > > storage config type. However, we will still need to check all configuration > > details to ensure no incompatible changes are introduced. > > > > Does this work out of the box with existing Iceberg REST Catalog clients > > that support credential vending and S3FileIO, or does it require a special > > client impl. or changes? You mentioned validation with your internal Python > > client. Have you also tested unmodified OSS clients(e.g., Spark with > > Iceberg plugins, Trino)? > > > > > Would you prefer the shared FileIO validation change to land separately, > > and how should we sequence the management API and R2 integration changes? > > > > Yes, landing the changes separately would be preferable. Small and > > well-scoped PRs usually merge faster. > > > > Yufei > > > > > > On Wed, Sep 9, 2026 at 11:18 AM Austen Tomek < > > [email protected]> wrote: > > > > > Hi all, > > > > > > I’m Austen Tomek from Chicago Trading Company. We use Polaris for our > > > Iceberg catalogs and we're evaluating Cloudflare R2 as an object store > > (the > > > no egress fees make it a very compelling product). I’d like to discuss > > > contributing R2 support and credential vending to Polaris with the goal > > of > > > getting feedback before opening a ready-for-review upstream PR. This is > > my > > > first contribution, so trying to do this right. > > > > > > R2 exposes S3-compatible APIs but it does not provide AWS STS. Cloudflare > > > instead supports these short-lived, scoped credentials generated by > > locally > > > signed JWT with the server holding the parent API token [1]. This allows > > us > > > to retain Polaris’s authorization and credential-vending model while > > > storing our table data in R2. > > > > > > I have a prototype with the following design: > > > > > > > > > * > > > An opt-in R2 storage type, with catalog configuration identifying the > > > Cloudflare account and optional jurisdiction. Parent credentials remain > > in > > > server configuration and can be selected through storageName. > > > * > > > Local credential generation using Cloudflare’s documented format. > > > Credentials expire, are restricted to one bucket, and carry prefix and > > > access scopes derived from Polaris’s location grants. The integration > > uses > > > Polaris’s existing credential cache. > > > * > > > Credentials returned through the existing Iceberg REST vending flow as > > > s3.* properties, including the endpoint, session token, and expiry. > > > Compatible clients use their existing S3 FileIO. > > > * > > > Conservative handling of scopes the implementation cannot represent: > > > cross-bucket and mixed read/write grants are rejected rather than > > > broadened. R2 is excluded from the default supported-storage-types list. > > > > > > The main design questions I feel this runs into: > > > > > > * > > > Should this be a distinct storage type or a credential-vending mechanism > > > selected through the existing S3 configuration? > > > * > > > I chose a separate type because the identity, credential generation, and > > > scoping model differ from AWS STS, as this seemed more appropriate > > > > > > There is a draft preview PR against my fork [2], including > > implementation, > > > management API changes, CLI support, tests, and documentation. It also > > > discloses the AI assistance used during implementation. I leaned heavily > > on > > > Fable 5.1 for this implementation so I want to be upfront about it. > > > > > > We are running the implementation in a non-production environment right > > > now as a POC. Validation includes live R2 reads/writes, multipart > > > operations, negative scope-boundary tests, our internal Python client > > > regression matrix, and credential rotation/recovery exercises. The > > upstream > > > CI workflow has passed on the fork. > > > > > > The change also encounters the existing assumption that a FileIO > > > implementation identifies exactly one storage type: both S3 and R2 use > > > S3FileIO. This overlaps with #4486 [3]. > > > > > > I’d particularly appreciate feedback on: > > > > > > > > > 1. > > > Should R2 have its own storage configuration/type, or should non-STS > > > vending be modeled within the existing S3 integration? > > > 2. > > > Does the local-signing approach fit Polaris’s credential-vending model? > > > Any additional constraints or validations necessary? > > > 3. > > > Would you prefer the shared FileIO validation change to land separately, > > > and how should we sequence the management API and R2 integration changes? > > > > > > Happy to adapt the design and split the contribution into more focused > > > PRs. Wanted to start the discussion > > > > > > Thanks, > > > Austen > > > > > > [1] https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > > > [2] https://github.com/deepdishgary/polaris/pull/1 > > > [3] https://github.com/apache/polaris/issues/4486 > > > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > > > intended for the exclusive use of the individual or entity to whom it is > > > addressed and may contain information that is proprietary, confidential > > > and/or exempt from disclosure under applicable law. If you are not the > > > intended recipient, you are hereby notified that any viewing, copying, > > > disclosure or distribution of this information may be subject to legal > > > restriction or sanction. Please notify the sender, by electronic mail or > > > telephone, of any unintended recipients and delete the original message > > > without making any copies. > > > > > > > > > > > > > This electronic mail message and any attached files contain information > intended for the exclusive use of the individual or entity to whom it is > addressed and may contain information that is proprietary, confidential > and/or exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any viewing, copying, > disclosure or distribution of this information may be subject to legal > restriction or sanction. Please notify the sender, by electronic mail or > telephone, of any unintended recipients and delete the original message > without making any copies.
