Hi Austen, Thanks for starting this contribution!
After briefly looking through your PR [1] I think adding R2StorageConfigInfo makes sense. Even though "aws" in AwsStorageConfigurationInfo is longer implies AWS services, and MinIO, Rust, Ozone are supported just as well, R2 appears to be significantly different from STS-based S3 systems that a separate config is probably the most natural way to support it in the Polaris codebase. The alternative, I guess, is to add R2 accountId and jurisdiction as optional properties to AwsStorageConfigurationInfo (or overload existing properties), but that looks like mixing unrelated concepts together. Adding a generic property bag to AwsStorageConfigurationInfo is not convenient because PolarisStorageIntegrationProviderImpl will probably have to know how to interpret it in order to redirect credential vending requests to R2-specific code. I wonder whether R2StorageConfigurationInfo and AwsStorageConfigurationInfo (and corresponding "storage integration" classes) could share a common base class for common properties (e.g. endpoint). Even though "endpoint" might not be relevant to R2 in the cloud, the fact that S3FileIO uses it, and R2 is accessed via S3FileIO means we should probably allow users to configure the full set of FileIO properties just in case. I think it is fine for different PolarisStorageIntegration implementations to produce the same kind of StorageAccessConfig (S3 in this case) and thus cause S3FileIO to be used by clients (including Polaris' own storage access paths). Changes in StorageTypeFileIO in [1] look a bit concerning to me. I wonder if this code could be refactored to avoid dependencies on storage config types completely.... but TBH, I did not look too deeply into this today. Anticipating future PRs for this in the Polaris repo, I'd like to ask to separate Python code changes from java changes since these areas usually attract different reviewers :) >From my POV, management API and matching java changes can be in the same PR. I'd say docs should come later (to keep PRs small). All in all, thank you again and I hope this contribution lands in Polaris :) [1] https://github.com/deepdishgary/polaris/pull/1 Cheers, Dmitri. On Thu, Sep 10, 2026 at 1:52 PM Jean-Baptiste Onofré <[email protected]> wrote: > Hi Austen, > > Nice discussion! Thanks for that, and welcome aboard :) > > I think it makes sense to have R2 built under the existing S3 storage > configuration. It matches how Polaris is already structured: > StorageType is keyed on the URI scheme (s3://, s3a://), and > AwsStorageConfigurationInfo already carries endpoint, stsEndpoint, > pathStyleAccess and, importantly, an stsUnavailable flag. > So, what you need is largely there already: correct me if I'm wrong, > R2 is essentially "S3 compatible store where STS is unavailable, plus > a way to actually vend credentials in that case". Going this route > also keeps this work "isolated" from the FileIO-per-storage-type > discussion. > > The interesting design question is a pluggable credential vending path > for S3 compatible stores when STS is not available, with Cloudflare's > local signing as the first concrete implementation, rather than R2 > specific code. It would be good to sketch how MinIO would slot into > the same config fields before the vending path hardens around R2. > > I believe we have to check if Cloudflare account ID and jurisdiction > really need dedicated config fields, or can we use the endpoint value > (with documentation)? Keeping the configuration surface minimal will > help the design generalize. > > I would love to discuss two things: > 1. Polaris shipping a Cloudflare API token and locally minting scoped > credentials makes it effectively an STS for R2, which increases the > blast radius of a server compromise. Your "reject rather than boarden" > handling of unrepresentable scopes is the right call. > 2. The PyIceberg/DuckDB/Java results are promising. Just curious about > Spark or Trino. > > We love small, well-scoped PRs. I would suggest to start with the S3 > config and the management API as first shoot, and we can follow with > the credential vending implementation. > > Regards > JB > > On Wed, Sep 9, 2026 at 8:18 PM Austen Tomek > <[email protected]> wrote: > > > > Hi all, > > > > I’m Austen Tomek from Chicago Trading Company. We use Polaris for our > Iceberg catalogs and we're evaluating Cloudflare R2 as an object store (the > no egress fees make it a very compelling product). I’d like to discuss > contributing R2 support and credential vending to Polaris with the goal of > getting feedback before opening a ready-for-review upstream PR. This is my > first contribution, so trying to do this right. > > > > R2 exposes S3-compatible APIs but it does not provide AWS STS. > Cloudflare instead supports these short-lived, scoped credentials generated > by locally signed JWT with the server holding the parent API token [1]. > This allows us to retain Polaris’s authorization and credential-vending > model while storing our table data in R2. > > > > I have a prototype with the following design: > > > > > > * > > An opt-in R2 storage type, with catalog configuration identifying the > Cloudflare account and optional jurisdiction. Parent credentials remain in > server configuration and can be selected through storageName. > > * > > Local credential generation using Cloudflare’s documented format. > Credentials expire, are restricted to one bucket, and carry prefix and > access scopes derived from Polaris’s location grants. The integration uses > Polaris’s existing credential cache. > > * > > Credentials returned through the existing Iceberg REST vending flow as > s3.* properties, including the endpoint, session token, and expiry. > Compatible clients use their existing S3 FileIO. > > * > > Conservative handling of scopes the implementation cannot represent: > cross-bucket and mixed read/write grants are rejected rather than > broadened. R2 is excluded from the default supported-storage-types list. > > > > The main design questions I feel this runs into: > > > > * > > Should this be a distinct storage type or a credential-vending mechanism > selected through the existing S3 configuration? > > * > > I chose a separate type because the identity, credential generation, and > scoping model differ from AWS STS, as this seemed more appropriate > > > > There is a draft preview PR against my fork [2], including > implementation, management API changes, CLI support, tests, and > documentation. It also discloses the AI assistance used during > implementation. I leaned heavily on Fable 5.1 for this implementation so I > want to be upfront about it. > > > > We are running the implementation in a non-production environment right > now as a POC. Validation includes live R2 reads/writes, multipart > operations, negative scope-boundary tests, our internal Python client > regression matrix, and credential rotation/recovery exercises. The upstream > CI workflow has passed on the fork. > > > > The change also encounters the existing assumption that a FileIO > implementation identifies exactly one storage type: both S3 and R2 use > S3FileIO. This overlaps with #4486 [3]. > > > > I’d particularly appreciate feedback on: > > > > > > 1. > > Should R2 have its own storage configuration/type, or should non-STS > vending be modeled within the existing S3 integration? > > 2. > > Does the local-signing approach fit Polaris’s credential-vending model? > Any additional constraints or validations necessary? > > 3. > > Would you prefer the shared FileIO validation change to land separately, > and how should we sequence the management API and R2 integration changes? > > > > Happy to adapt the design and split the contribution into more focused > PRs. Wanted to start the discussion > > > > Thanks, > > Austen > > > > [1] https://developers.cloudflare.com/r2/api/s3/temporary-credentials/ > > [2] https://github.com/deepdishgary/polaris/pull/1 > > [3] https://github.com/apache/polaris/issues/4486 > > > > Get Outlook for Mac<https://aka.ms/GetOutlookForMac> > > > > > > > > > > > > > > This electronic mail message and any attached files contain information > intended for the exclusive use of the individual or entity to whom it is > addressed and may contain information that is proprietary, confidential > and/or exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any viewing, copying, > disclosure or distribution of this information may be subject to legal > restriction or sanction. Please notify the sender, by electronic mail or > telephone, of any unintended recipients and delete the original message > without making any copies. >
