Hi Dennis, While I still support the idea of splitting OpenAPI yaml files and URI bases between the Management and Sharing APIs, I do not think it is tightly coupled with my proposal for the authentication-neural Sharing API spec.
To give an example, currently [5446] CreateListingRequest includes a "consumerName". This immediately binds the Listing definition to an ExternalConsumer and by extension to ConsumerCredential. ConsumerCredential is naturally part of the authentication process during Share access. I believe ConsumerCredential (and related entities) is worth isolating from the Shares API to increase interoperability. Implementations that choose to rely on ConsumerCredential for Share access can still do so. However, ConsumerCredential will not be a requirement for all possible Share API implementations. This can work with any OpenAPI yaml and URI paths layout. I'd like to resolve this aspect before we go back to URI paths and YAML file locations. Actually, I do not feel too strongly about path prefixes. I do believe separate prefixes make implementations easier in many aspects (including network setup), but interleaving those paths with the Management API is not a blocker, from my POV, only an inconvenience. Cheers, Dmitri. [5446] https://github.com/apache/polaris/pull/5446 On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <[email protected]> wrote: > Yeah, that's a good topic to discuss; I guess it touches on our core > (unresolved?) conventions on how we organize API url-prefixes and yaml > files. Reposting my response to the github comment as well: > > Right, I think we could certainly split the API routes and yaml-definitions > even further if we want to ensure a well-defined auth-detail-free core, but > for the standalone-Polaris use case we'd still need to define it > *somewhere* > under a Sharing-specific surface, right? > > @dimas-b <https://github.com/dimas-b> Were you thinking of just splitting > the yaml file but keeping the share-management root, or adding a third > sharing prefix like > > /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials... > ? > > Or just yaml-file split would be keeping > /api/shares-management/v1/external-consumers/{name}/credentials... but > putting it in a separate yaml file to make it easy for service-runners to > choose not to support that whole yaml file's concrete APIs. > > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi All, > > > > Re: the shating API PR [5446], I'm repeating my GH comment here for > > visibility. > > > > I'd like the API to be neutral from the authentication perspective. > > > > This is how we handle the IRC spec in Polaris. The API declares that > > clients need to authenticate, but specific details about principals, > > clients and secrets are delegated to the server. > > > > I think this is a sound approach for the sharing API too. Specific > details > > related to native Polaris principals (ConsumerCredential, etc.) can > > probably remain in the Management API. > > > > WDYT? > > > > Thanks, > > Dmitri. > > > > [5446] https://github.com/apache/polaris/pull/5446 > > > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi > > > > > > As I said during the community meeting, let's focus on the API PR > > > first. We will build from there. > > > > > > Regards > > > JB > > > > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <[email protected]> wrote: > > > > > > > > +1, thanks for the discussion everyone! > > > > > > > > Yes, I meant to flip it to "ready to review" earlier; flipped it now. > > I'm > > > > still working through the existing set of comments and may need to > come > > > > back to it in the next few days before getting proper replies, but > feel > > > > free to add additional comments. > > > > > > > > Recapping the community sync discussion re: Dmitri's last message, > I'm > > > > fully aligned on building the sharing on top of existing authz > > > mechanisms, > > > > instead of being a new mechanism, and indeed external IdP is intended > > to > > > be > > > > a first-class supported use case. > > > > > > > > The latest distinction is that the ExternalConsumer isn't > fundamentally > > > > identified by any internal PolarisPrincipal, but rather that it > serves > > > as a > > > > wrapper that references some identity, whether that identity is > defined > > > in > > > > an external IdP without any latent in-Polaris representation or if > > it's a > > > > hidden PolarisPrincipal (for the Polaris-standalone RBAC use case). > > > > > > > > The sharing endpoint via ExternalConsumer is the only layer of > > > indirection > > > > at the API entry-point, but then once the set of activePrincipalRoles > > is > > > > resolved, the underlying authz engine doesn't behave any differently > > from > > > > normal Polaris authz. > > > > > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu <[email protected]> > wrote: > > > > > > > > > Thanks everyone for the discussion! > > > > > > > > > > As a next step discussed in today's community sync, we will start > > > reviewing > > > > > the spec change. > > > > > > > > > > Hi Dennis, is this PR ready for review? > > > > > https://github.com/apache/polaris/pull/5446 > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Thanks for this discussion Dennis! > > > > > > > > > > > > Wanted to weigh in with my thoughts and questions. > > > > > > > > > > > > > > > > > > > > Do you prefer /api/shares-management/v1/ > > > > > > > instead of adding to /api/management/v1? > > > > > > > > > > > > > > > > > > While I can see the argument for a share specific management > > endpoint > > > > > there > > > > > > doesn't seem to be a clear obvious choice from what I can see - I > > > lean > > > > > > towards /api/management as being the central place for all the > > > > > management - > > > > > > share or not. As long as we're careful not to leak share types or > > non > > > > > share > > > > > > types into the wrong api (which could be helped by having > separate > > > yamls) > > > > > > then it feels reasonable to have all under the management api. > > > > > > > > > > > > Also went through the doc and PR and made several comments as > well. > > > > > > > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > Prithvi mentioned ExternalConsumer and principals. I'd like to > > > > > > > reinforce the need to further discuss "consumers" and > principals > > > and > > > > > how > > > > > > > they relate to IdP and Polaris Authentication. > > > > > > > > > > > > > > From my POV, share access control should not be a new mechanism > > but > > > > > > should > > > > > > > build on top of the existing extensible authentication / > > > authorization > > > > > > > framework. Supporting external IdP for shares would be a > valuable > > > > > feature > > > > > > > IMHO. > > > > > > > > > > > > > > Side note: I'm posting comments on PR 5446 too. > > > > > > > > > > > > > > Cheers, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > Thank you for reviving this, and for the concrete spec in PR > > > 5446.. > > > > > the > > > > > > > > journeys doc was especially helpful for seeing the end-to-end > > > flow. > > > > > > > > > > > > > > > > I agree this is worth doing, and that v1 is more than a > > tutorial > > > on > > > > > > > today's > > > > > > > > RBAC. A partner principal on /api/catalog is already > possible; > > > what > > > > > > seems > > > > > > > > valuable is a first-class share (enumerated members, a > > restricted > > > > > > > consumer, > > > > > > > > a listing as the binding and audit unit) plus a segregated, > > > read-only > > > > > > > > Iceberg REST surface with credential vending. Stock Iceberg > > REST > > > as > > > > > the > > > > > > > > consumer protocol, and keeping Flight and Polaris-to-Polaris > > > > > federation > > > > > > > out > > > > > > > > of v1, all sound right to me. > > > > > > > > > > > > > > > > I don't have a strong view yet on the URL layout (management > vs > > > > > shares > > > > > > > > prefix, nested vs top-level, prefix keyed by share vs > listing) > > > and am > > > > > > > happy > > > > > > > > to follow whatever the community prefers there. > > > > > > > > > > > > > > > > A few things I wanted to check on the spec itself, several of > > > these > > > > > > echo > > > > > > > > points already raised on the list: > > > > > > > > > > > > > > > > 1. Membership as source of truth. If a hidden catalog role is > > > > > > > independently > > > > > > > > grantable, a share could be widened without going through the > > > share > > > > > > APIs. > > > > > > > > Would it make sense to say the data plane authorizes against > > > > > > membership + > > > > > > > > listing, and that any backing roles are not user-visible / > not > > > > > > > > independently grantable? > > > > > > > > 2. ExternalConsumer and principals. The Aug 7 recap already > > > noted we > > > > > > > should > > > > > > > > not assume a PrincipalEntity behind every ExternalConsumer. > The > > > > > > journeys > > > > > > > > still describe granting the consumer a share principal role, > I > > > assume > > > > > > > that > > > > > > > > is an implementation sketch for v1 client_id/secret, not the > > > public > > > > > > > model? > > > > > > > > 3. Listing expiration. JB's original note was "share these > > tables > > > > > with > > > > > > > this > > > > > > > > consumer until this date." Credential expiry is a bit > > different. > > > > > Would > > > > > > a > > > > > > > > listing-level expiration (and maybe ACTIVE/SUSPENDED) be in > > > scope for > > > > > > v1? > > > > > > > > 4. Time-travel. As raised in the community sync, sharing via > > > Iceberg > > > > > > REST > > > > > > > > means historical snapshots remain readable. Totally fine for > v1 > > > to > > > > > > > support > > > > > > > > only full history, would an explicit field on the share help > > so a > > > > > later > > > > > > > > CURRENT_ONLY option is not a breaking surprise? > > > > > > > > 5. Consumer data plane. The current file is control-plane > only, > > > which > > > > > > is > > > > > > > a > > > > > > > > good first step. It might help to list the Iceberg routes > that > > > are > > > > > > > mounted, > > > > > > > > that writes 404, that non-members 404, and that /config does > > not > > > > > return > > > > > > > > provider catalog properties, so implementations don't > diverge. > > > > > > > > > > > > > > > > Two smaller notes, if useful: in the J1 journey the consumer > > > still > > > > > > posts > > > > > > > to > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are not supposed to > > > touch > > > > > the > > > > > > > > internal catalog API, should the default token endpoint live > on > > > the > > > > > > > shares > > > > > > > > surface (or an external IdP)? And on Dmitri's point about > share > > > > > > operators > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege on the bound > > > catalog > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make that work > on > > > any > > > > > URL > > > > > > > > prefix. > > > > > > > > > > > > > > > > Happy to review the next revision, and thanks again to both > of > > > you > > > > > for > > > > > > > > driving this. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Prithvi S > > > > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo <[email protected]> > > > wrote: > > > > > > > > > > > > > > > > > As discussed live, looks like we both might have some > drafts > > > we can > > > > > > > > compare > > > > > > > > > and choose/merge from :) > > > > > > > > > > > > > > > > > > Here's my current draft spec PR; it opts for my preferred > > > syntax > > > > > > > choices > > > > > > > > > for now (/api/management/ for share-management, > /api/shares/ > > > for > > > > > > > consumer > > > > > > > > > data-plane): https://github.com/apache/polaris/pull/5446 > > > > > > > > > > > > > > > > > > Since we were talking about preferring mdfiles for design > > > iteration > > > > > > > > here's > > > > > > > > > a companion design-addendum which just enumerates what the > > > > > end-to-end > > > > > > > > user > > > > > > > > > journeys look like syntactically for each combination of > > design > > > > > > > choices: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md > > > > > > > > > > > > > > > > > > Please let me know if anyone prefers a Google Doc version > of > > > the > > > > > > > > > design-addendum for easier commenting, or if the > > > design-addendum > > > > > > itself > > > > > > > > > should also be modeled as a PR on some free-floating branch > > > just to > > > > > > > > capture > > > > > > > > > comments. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste Onofré < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > I am resuming work on this proposal. > > > > > > > > > > > > > > > > > > > > Regarding our recent discussion, I will move forward with > > > > > opening a > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to help > illustrate > > > the > > > > > > > > > > design. > > > > > > > > > > > > > > > > > > > > I will update this thread once the PR is ready. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri Bourlatchkov < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi Dennis, > > > > > > > > > > > > > > > > > > > > > > Thanks for the recap! > > > > > > > > > > > > > > > > > > > > > > I was thinking that share management is probably > distinct > > > from > > > > > > > > catalog > > > > > > > > > > > management. A user who already has a realm+catalog may > > > have an > > > > > > > option > > > > > > > > > to > > > > > > > > > > > manage shares, but not the catalog itself. Would it > make > > > sense > > > > > to > > > > > > > > > > separate > > > > > > > > > > > the share management API from the general Polaris > > > management > > > > > > API? I > > > > > > > > > mean > > > > > > > > > > > using a different prefix for URIs. > > > > > > > > > > > > > > > > > > > > > > For example: /api/shares/v1/ > > > > > > > > > > > > > > > > > > > > > > Obviously clients should not make any assumptions about > > the > > > > > base > > > > > > > path > > > > > > > > > > > (/api/shares), which could be different in different > > > > > deployments > > > > > > > > (e.g. > > > > > > > > > > > going through a gateway). Clients must be able to take > > the > > > base > > > > > > URI > > > > > > > > as > > > > > > > > > a > > > > > > > > > > > config option. Path segments after /v1/ will be > specified > > > by > > > > > > > Polaris > > > > > > > > > > (Open > > > > > > > > > > > API yaml). > > > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Just to recap some of the discussion from today's > > > community > > > > > > sync, > > > > > > > > > > Dmitri, > > > > > > > > > > > > Robert, and Alex helped raise a few considerations to > > > roll > > > > > into > > > > > > > the > > > > > > > > > > next > > > > > > > > > > > > design/prototyping phase: > > > > > > > > > > > > > > > > > > > > > > > > 1. Given the way sharing is packaged as a separate > > > thing, it > > > > > > may > > > > > > > > not > > > > > > > > > be > > > > > > > > > > > > obvious to users that historical snapshots are > equally > > > > > > available > > > > > > > to > > > > > > > > > > > > consumers as the latest snapshot (by virtue of > sharing > > > via > > > > > the > > > > > > > > > > standard IRC > > > > > > > > > > > > protocol) - we should make sure to document that fact > > > well, > > > > > > > > including > > > > > > > > > > the > > > > > > > > > > > > pitfall that deleting rows by updating a table > doesn't > > > > > > > necessarily > > > > > > > > > mean > > > > > > > > > > > > those rows become inaccessible to consumers > > > > > > > > > > > > 2. Make sure not to hard-code assumptions about the > > > existence > > > > > > of > > > > > > > an > > > > > > > > > > > > internal PrincipalEntity backing an ExternalConsumer > - > > > even > > > > > if > > > > > > > the > > > > > > > > > > first > > > > > > > > > > > > MVP implementation uses a nested/hidden Principal of > > some > > > > > type > > > > > > > that > > > > > > > > > > holds a > > > > > > > > > > > > Polaris-managed client_id/client_secret, the > 3rd-party > > > IdP > > > > > use > > > > > > > > cases > > > > > > > > > > > > require "implicit" Principals resolved via the oauth > > > token. > > > > > In > > > > > > > such > > > > > > > > > > cases > > > > > > > > > > > > the ExternalConsumer contains the metadata > coordinating > > > the > > > > > > > > three-way > > > > > > > > > > > > handshake between the data-provider, data-consumer, > and > > > IdP, > > > > > > but > > > > > > > > > there > > > > > > > > > > is > > > > > > > > > > > > no PrincipalEntity underlying it. > > > > > > > > > > > > 3. Longer-term, we may want to tackle "on-behalf-of" > > > > > semantics > > > > > > - > > > > > > > > > > initially, > > > > > > > > > > > > the way an ExternalConsumer works is that the > > > finer-grained > > > > > > > > consumer > > > > > > > > > > users > > > > > > > > > > > > share a single ExternalConsumer identity, just like > > how a > > > > > > > FEDERATED > > > > > > > > > > catalog > > > > > > > > > > > > shares a single service principal in its > > > ConnectionConfig. > > > > > But > > > > > > in > > > > > > > > > some > > > > > > > > > > > > cases the consumer may want some end-to-end > attribution > > > of > > > > > > > > > fine-grained > > > > > > > > > > > > users, like "on-behalf-of: bob @ consumercompany.com > " > > > when > > > > > the > > > > > > > > > > > > ExternalConsumer implicit principal is something like > > > > > > > > > > > > "externalconsumer$consumercompany" > > > > > > > > > > > > 4. We can iterate more rapidly on a draft spec PR and > > > also > > > > > > expand > > > > > > > > on > > > > > > > > > > the > > > > > > > > > > > > detailed technical design > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Dennis > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo < > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Thanks JB for reviving this thread, and belated > > thanks > > > to > > > > > all > > > > > > > > > > reviewers > > > > > > > > > > > > > who have left feedback in the doc or this thread so > > > far! > > > > > And > > > > > > > > > > apologies > > > > > > > > > > > > for > > > > > > > > > > > > > the silence on this, it's still been on my radar > but > > I > > > got > > > > > > > > > > distracted by > > > > > > > > > > > > > some other work for awhile. > > > > > > > > > > > > > > > > > > > > > > > > > > Echoing JB's summary, the proposal is indeed that > > we're > > > > > > mostly > > > > > > > > just > > > > > > > > > > > > > reusing existing Authz/RBAC machinery, and the > > > translation > > > > > > > layer > > > > > > > > to > > > > > > > > > > the > > > > > > > > > > > > new > > > > > > > > > > > > > sharing concepts sits *on top of* the existing > authz > > > engine > > > > > > as > > > > > > > > much > > > > > > > > > > as > > > > > > > > > > > > > possible. > > > > > > > > > > > > > > > > > > > > > > > > > > So even if we're adding layers of indirection on > > > > > > > > > > ExternalConsumer/Listing > > > > > > > > > > > > > -> Principal/PrincipalRole/CatalogRole+grants, the > > > parts > > > > > that > > > > > > > end > > > > > > > > > up > > > > > > > > > > > > doing > > > > > > > > > > > > > the underlying authz just reuse existing Authorizer > > > logic. > > > > > > > > > > > > > > > > > > > > > > > > > > Based on some other offline discussion with > Dmitri, I > > > > > > realized > > > > > > > > why > > > > > > > > > > the > > > > > > > > > > > > > discussion around "requirements" may have been > > > confusing: > > > > > > > > > > > > > > > > > > > > > > > > > > 1. My attempts to shorten the doc inadvertently cut > > out > > > > > some > > > > > > of > > > > > > > > the > > > > > > > > > > more > > > > > > > > > > > > > "abstract requirements" formulations > > > > > > > > > > > > > 2. The project in itself is somewhat more of a > "form > > > *as* > > > > > > > > function" > > > > > > > > > > > > > feature than *strictly* functional in nature -- for > > > > > example, > > > > > > > many > > > > > > > > > > > > correctly > > > > > > > > > > > > > pointed out that "sharing" via the existing > > > RBAC/Principals > > > > > > > > syntax > > > > > > > > > is > > > > > > > > > > > > > already *possible* just by letting a cross-org > > > "consumer" > > > > > be > > > > > > a > > > > > > > > > > > > first-class > > > > > > > > > > > > > principal and access your own company's main > Polaris > > > > > catalog > > > > > > > > > > endpoints. > > > > > > > > > > > > > > > > > > > > > > > > > > For (1) I added this to the Requirements section in > > > the doc > > > > > > to > > > > > > > > > better > > > > > > > > > > > > > clarify the abstract form of the data model > element: > > > > > > > > > > > > > > > > > > > > > > > > > > *A. Sharing Container*: Container concept for > holding > > > > > things > > > > > > > you > > > > > > > > > > want to > > > > > > > > > > > > > share - “ShareEntity” > > > > > > > > > > > > > *B. Consumer Identity*: Special flavor of > > > principal-like > > > > > > entity > > > > > > > > > > > > > representing a consumer relationship - > > > “ExternalConsumer” > > > > > > > > > > > > > *C. Map Containers to Consumers*: A way to specify > > > letting > > > > > a > > > > > > > > given > > > > > > > > > > > > > consumer access a given sharing container - > “Listing” > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated API > access-point > > > for > > > > > > > > > consumers - > > > > > > > > > > > > > “EndpointConfig” > > > > > > > > > > > > > *E. Objects in Containers*: A way to specify which > > > > > in-catalog > > > > > > > > > objects > > > > > > > > > > > > > belong in the big sharing container - > > “ShareMembership” > > > > > > > > > > > > > > > > > > > > > > > > > > This still bakes in a bit of implementation detail > > > since it > > > > > > > begs > > > > > > > > > the > > > > > > > > > > > > > question why not just let core RBAC/grants handle > (C) > > > and > > > > > > (E). > > > > > > > To > > > > > > > > > > that > > > > > > > > > > > > end, > > > > > > > > > > > > > and addressing problem (2) of where the "form *is* > > > > > > function", I > > > > > > > > > > added a > > > > > > > > > > > > > "Concept Mapping" tab: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0 > > > > > > > > > > > > > > > > > > > > > > > > > > I think the details of more advanced features can > > > still be > > > > > > > hashed > > > > > > > > > out > > > > > > > > > > > > > incrementally without blocking work on an MVP (and > > > > > certainly > > > > > > > the > > > > > > > > > MVP > > > > > > > > > > > > scope > > > > > > > > > > > > > could be trimmed so we can launch-and-iterate), so > as > > > long > > > > > as > > > > > > > > > there's > > > > > > > > > > > > > community alignment on the general direction I > > propose > > > we > > > > > > could > > > > > > > > > move > > > > > > > > > > > > > forward on basic building blocks. Or if building > > blocks > > > > > > > > themselves > > > > > > > > > > are > > > > > > > > > > > > > controversial we could still proceed with some > > > prototyping > > > > > > > across > > > > > > > > > > > > different > > > > > > > > > > > > > options. > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > Dennis > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste > > Onofré < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi everyone, > > > > > > > > > > > > >> > > > > > > > > > > > > >> I would like to resume the discussion about Data > > > Sharing > > > > > in > > > > > > > > > Polaris > > > > > > > > > > > > >> and propose focusing on three major items to help > us > > > move > > > > > > > > forward. > > > > > > > > > > > > >> > > > > > > > > > > > > >> 1. What is data sharing? > > > > > > > > > > > > >> In a nutshell, it is an open protocol proposal > > > > > allowing > > > > > > > > > > > > >> organizations to share live data without copying > or > > > > > > > replication. > > > > > > > > > The > > > > > > > > > > > > >> main goal is zero-copy sharing achieved through > > shared > > > > > > > metadata > > > > > > > > > and > > > > > > > > > > > > >> authorization. > > > > > > > > > > > > >> > > > > > > > > > > > > >> On Polaris, this would mean: > > > > > > > > > > > > >> a. The consumer is authenticated as usual. > > > > > > > > > > > > >> b. The Polaris server provides the consumer with > > > direct > > > > > > access > > > > > > > > to > > > > > > > > > > the > > > > > > > > > > > > >> data files (e.g., Parquet, as specified in the > > > > > > metadata.json). > > > > > > > > > > > > >> c. The consumer reads the data files directly > using > > > their > > > > > > > > > preferred > > > > > > > > > > > > >> tool (e.g., a query engine). > > > > > > > > > > > > >> > > > > > > > > > > > > >> While Polaris already has most of the necessary > > > underlying > > > > > > > > > > > > >> capabilities, we need clear branding/packaging > > around > > > this > > > > > > use > > > > > > > > > case. > > > > > > > > > > > > >> Specifically, this means: > > > > > > > > > > > > >> a. A share is a dedicated catalog in the Polaris > > > server, > > > > > or > > > > > > a > > > > > > > > > > catalog > > > > > > > > > > > > >> role scoped to specific namespaces. > > > > > > > > > > > > >> b. A recipient is represented by a principal and a > > > > > principal > > > > > > > > role. > > > > > > > > > > > > >> c. A consumer profile consists of the catalog URI > > and > > > the > > > > > > > OAuth2 > > > > > > > > > > > > >> client_id/client_secret. > > > > > > > > > > > > >> d. We use credential vending (temporary STS > > > credentials) > > > > > for > > > > > > > > > > consumer > > > > > > > > > > > > >> access to the data files. > > > > > > > > > > > > >> e. We use the Polaris IRC layer to expose the > > > metadata and > > > > > > the > > > > > > > > > file > > > > > > > > > > > > >> locations/access. > > > > > > > > > > > > >> f. We can also support Polaris-to-Polaris sharing > > via > > > > > > > > Federation. > > > > > > > > > > > > >> > > > > > > > > > > > > >> Concretely: > > > > > > > > > > > > >> > > > > > > > > > > > > >> - On the provider side, we use the standard > > Polaris > > > RBAC > > > > > > > model > > > > > > > > > > > > >> (assigning grants to catalog roles, and assigning > > > those > > > > > > > catalog > > > > > > > > > > roles > > > > > > > > > > > > >> to principal roles): > > > > > > > > > > > > >> - Create a dedicated principal with its own > > > > > > > > > > > > client_id/client_secret. > > > > > > > > > > > > >> - Create a catalog role with > TABLE_READ_DATA, > > > > > > > TABLE_LIST, > > > > > > > > > and > > > > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces to be > shared > > > (no > > > > > > write > > > > > > > > > > > > >> grants). > > > > > > > > > > > > >> - Create a principal role per destination, > > > attach > > > > > the > > > > > > > > > catalog > > > > > > > > > > > > >> role, and assign the principal. > > > > > > > > > > > > >> - The destination engine uses loadTable with > > the > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation: vended-credentials" > > > header. > > > > > > > > Polaris > > > > > > > > > > then > > > > > > > > > > > > >> emits temporary, read-only storage credentials, > > > allowing > > > > > the > > > > > > > > > engine > > > > > > > > > > to > > > > > > > > > > > > >> read the data files directly from the owner's > bucket > > > > > without > > > > > > > > > > accessing > > > > > > > > > > > > >> the rest of the bucket. This achieves zero-copy > > > sharing. > > > > > > > > > > > > >> - On the recipient/consumer side, no special > > > > > configuration > > > > > > > is > > > > > > > > > > > > >> needed; any IRC client can use it. > > > > > > > > > > > > >> > > > > > > > > > > > > >> 2. What is data sharing not? > > > > > > > > > > > > >> I previously considered adding an Arrow Flight > > > > > endpoint > > > > > > to > > > > > > > > > > > > >> Polaris, but I no longer think it is a good idea > for > > > the > > > > > > > initial > > > > > > > > > > > > >> phase. An Arrow Flight endpoint would serve the > data > > > > > > directly, > > > > > > > > > which > > > > > > > > > > > > >> would require Polaris to read the data and expose > > it. > > > I am > > > > > > > > against > > > > > > > > > > > > >> embedding a query engine in Polaris, as this tight > > > > > coupling > > > > > > > > feels > > > > > > > > > > like > > > > > > > > > > > > >> an anti-pattern. While we could use a delegation > > > service > > > > > to > > > > > > > > serve > > > > > > > > > > the > > > > > > > > > > > > >> Flight endpoint, I think it should be considered > > for a > > > > > later > > > > > > > > > phase, > > > > > > > > > > if > > > > > > > > > > > > >> at all, since a Flight endpoint might be better > > > suited as > > > > > a > > > > > > > > > consumer > > > > > > > > > > > > >> responsibility and could encourage data copying. > > > > > > > > > > > > >> > > > > > > > > > > > > >> The first phase should focus on packaging and > > > branding the > > > > > > > > > > > > >> capabilities we already have. > > > > > > > > > > > > >> > > > > > > > > > > > > >> 3. What could the Data Sharing "package" look > like > > in > > > > > > > Polaris? > > > > > > > > > > > > >> To make Data Sharing obvious and easy to use, > we > > > > > should > > > > > > > > > > introduce > > > > > > > > > > > > >> the missing concept of a "share"—a named > definition > > > that > > > > > > users > > > > > > > > can > > > > > > > > > > > > >> list, discover, and define. The goal is to > > > materialize the > > > > > > > > user's > > > > > > > > > > > > >> intent ("I want to share these tables with this > > > consumer > > > > > > until > > > > > > > > > this > > > > > > > > > > > > >> date") into standard Polaris entities (principal, > > > > > principal > > > > > > > > role, > > > > > > > > > > > > >> catalog role, and grants). > > > > > > > > > > > > >> > > > > > > > > > > > > >> Therefore, I propose adding a /shares endpoint to > > the > > > > > > Polaris > > > > > > > > > > > > >> Management API to package and materialize these > > share > > > > > > > > definitions > > > > > > > > > > into > > > > > > > > > > > > >> Polaris entities. This would provide a usable, > > > > > first-version > > > > > > > > > package > > > > > > > > > > > > >> for Polaris Data Sharing. > > > > > > > > > > > > >> > > > > > > > > > > > > >> I hope this addresses Dmitri's questions and > > provides > > > a > > > > > more > > > > > > > > > > concrete > > > > > > > > > > > > >> scenario for Polaris Data Sharing. > > > > > > > > > > > > >> > > > > > > > > > > > > >> Thoughts? > > > > > > > > > > > > >> > > > > > > > > > > > > >> Regards, > > > > > > > > > > > > >> JB > > > > > > > > > > > > >> > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis Huo < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Hi All, > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > One big emerging enterprise use case coming up > as > > > more > > > > > > > people > > > > > > > > > > > > >> consolidate > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is something > commonly > > > known > > > > > > as > > > > > > > > > "Data > > > > > > > > > > > > >> Sharing", > > > > > > > > > > > > >> > an more specifically over the course of adoption > > of > > > open > > > > > > > table > > > > > > > > > > formats > > > > > > > > > > > > >> > "Open Data Sharing". > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Examples of existing managed service providers' > > Data > > > > > > Sharing > > > > > > > > > > features: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > https://www.databricks.com/product/delta-sharing > > > > > > > > > > > > >> > > > > > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > The basic idea is that when you share data > between > > > > > > different > > > > > > > > > > > > companies, > > > > > > > > > > > > >> you > > > > > > > > > > > > >> > need a first-class governance/management layer > and > > > extra > > > > > > > > > > > > >> bells-and-whistles > > > > > > > > > > > > >> > that are distinct from just the basic > capabilities > > > of > > > > > RBAC > > > > > > > or > > > > > > > > > > > > >> generalized > > > > > > > > > > > > >> > access-control (i.e. if you're sharing across > > > > > > > > > partially-untrusted > > > > > > > > > > org > > > > > > > > > > > > >> > boundaries, you don't just let the consumer > > > organization > > > > > > log > > > > > > > > > into > > > > > > > > > > your > > > > > > > > > > > > >> > datalake like one of your own employees). > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > JB and I put together this high-level proposal > for > > > > > > > supporting > > > > > > > > > Open > > > > > > > > > > > > >> Sharing > > > > > > > > > > > > >> > in Polaris: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Tentatively, it means adding ~5 logical data > model > > > > > > > constructs, > > > > > > > > > > some of > > > > > > > > > > > > >> > which may be a first-class PolarisEntity type, > > > others > > > > > > > subtypes > > > > > > > > > of > > > > > > > > > > > > >> existing > > > > > > > > > > > > >> > entities, and others just a nested construct: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > - ShareEntity (would behave similarly to a > > > Catalog) > > > > > > > > > > > > >> > - ExternalConsumer (mostly inherits from > > > Principal) > > > > > > > > > > > > >> > - Listing (Similar to a "role grant" but has > > > > > different > > > > > > > > > > metadata) > > > > > > > > > > > > >> > - EndpointConfig (nested config under > Listing) > > > > > > > > > > > > >> > - ShareMembership (Similar to a "securable > > > grant" but > > > > > > > > > different > > > > > > > > > > > > >> metadata) > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Feedback/comments welcome! I'll also bring it up > > for > > > > > live > > > > > > > > > > discussion > > > > > > > > > > > > if > > > > > > > > > > > > >> > there's time in the community sync. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Cheers, > > > > > > > > > > > > >> > Dennis > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
