Hi all,
I’m Austen Tomek from Chicago Trading Company. We use Polaris for our Iceberg
catalogs and we're evaluating Cloudflare R2 as an object store (the no egress
fees make it a very compelling product). I’d like to discuss contributing R2
support and credential vending to Polaris with the goal of getting feedback
before opening a ready-for-review upstream PR. This is my first contribution,
so trying to do this right.
R2 exposes S3-compatible APIs but it does not provide AWS STS. Cloudflare
instead supports these short-lived, scoped credentials generated by locally
signed JWT with the server holding the parent API token [1]. This allows us to
retain Polaris’s authorization and credential-vending model while storing our
table data in R2.
I have a prototype with the following design:
*
An opt-in R2 storage type, with catalog configuration identifying the
Cloudflare account and optional jurisdiction. Parent credentials remain in
server configuration and can be selected through storageName.
*
Local credential generation using Cloudflare’s documented format. Credentials
expire, are restricted to one bucket, and carry prefix and access scopes
derived from Polaris’s location grants. The integration uses Polaris’s existing
credential cache.
*
Credentials returned through the existing Iceberg REST vending flow as s3.*
properties, including the endpoint, session token, and expiry. Compatible
clients use their existing S3 FileIO.
*
Conservative handling of scopes the implementation cannot represent:
cross-bucket and mixed read/write grants are rejected rather than broadened. R2
is excluded from the default supported-storage-types list.
The main design questions I feel this runs into:
*
Should this be a distinct storage type or a credential-vending mechanism
selected through the existing S3 configuration?
*
I chose a separate type because the identity, credential generation, and
scoping model differ from AWS STS, as this seemed more appropriate
There is a draft preview PR against my fork [2], including implementation,
management API changes, CLI support, tests, and documentation. It also
discloses the AI assistance used during implementation. I leaned heavily on
Fable 5.1 for this implementation so I want to be upfront about it.
We are running the implementation in a non-production environment right now as
a POC. Validation includes live R2 reads/writes, multipart operations, negative
scope-boundary tests, our internal Python client regression matrix, and
credential rotation/recovery exercises. The upstream CI workflow has passed on
the fork.
The change also encounters the existing assumption that a FileIO implementation
identifies exactly one storage type: both S3 and R2 use S3FileIO. This overlaps
with #4486 [3].
I’d particularly appreciate feedback on:
1.
Should R2 have its own storage configuration/type, or should non-STS vending be
modeled within the existing S3 integration?
2.
Does the local-signing approach fit Polaris’s credential-vending model? Any
additional constraints or validations necessary?
3.
Would you prefer the shared FileIO validation change to land separately, and
how should we sequence the management API and R2 integration changes?
Happy to adapt the design and split the contribution into more focused PRs.
Wanted to start the discussion
Thanks,
Austen
[1] https://developers.cloudflare.com/r2/api/s3/temporary-credentials/
[2] https://github.com/deepdishgary/polaris/pull/1
[3] https://github.com/apache/polaris/issues/4486
Get Outlook for Mac<https://aka.ms/GetOutlookForMac>
This electronic mail message and any attached files contain information
intended for the exclusive use of the individual or entity to whom it is
addressed and may contain information that is proprietary, confidential and/or
exempt from disclosure under applicable law. If you are not the intended
recipient, you are hereby notified that any viewing, copying, disclosure or
distribution of this information may be subject to legal restriction or
sanction. Please notify the sender, by electronic mail or telephone, of any
unintended recipients and delete the original message without making any copies.
