Right, sorry, to clarify, my stated bullets in the last email were my hypothetical portrayal of what your proposed core set of APIs would be, not what my current proposal says (so that you can confirm or deny whether those three bullets convey your ideal state).
It sounds like (2) and (3) match your intent, and (1) is exactly where the current spec PR differs, since it contains consumers and listings. For the particular use cases that went into the scoping for the Polaris Shares proposal, the ExternalConsumer is fairly fundamental to extracting as a first-class object. But you're right that there could be a trimmed scoping that allows implementors to fully abstract away the ExternalConsumer/Listing semantics instead. In the same way, there's even a fully-trimmed formulation where we don't implement *any* management APIs at all: Where "Sharing" is just data providers providing dedicated lightweight Polaris servers, or adding a very simple config-file-based mapping for what data to make accessible to dedicated "sharing" Principals or PrincipalRoles. We could reconcile this by also moving the ExternalConsumer/Listings APIs to the same "optional" yaml file I moved the other ConsumerCredential-related endpoints into and keeping the "core Shares API" limited to what you wanted. Broadly, the two pillars of the proposal are: 1. *What* to share 2. *With whom* to share The core Shares API would only cover the first, in line with your proposal: CRUD on Shares and the securables contained in them (listShares, createShare, getShare, updateShare, deleteShare, listShareMembers, updateShareMembers). The optional file would cover the second: ExternalConsumers, Listings, the endpoint configuration, and credentials for the polaris-standalone case. Polaris service-runners could then decide if they just want sharing consumers to live in the exact same identity domain as their other native Polaris readers, or segregate them by enabling the "With whom" half of the API. One thing to discuss for the "core-only" case - today the management API doesn't syntactically support granting a share to a PrincipalRole, so we'd need to extend grant types or expose some of the internals of a share's backing catalog role. I guess we can iteratively refine any necessary API changes as we discover these in the core-only implementation. Note that the "with whom" half is necessary for the full "segregation" reqiurements stated in the original proposal doc, such as preventing assignment to other roles or accessing the "normal" catalog endpoints, which is why it can't be *fully* subsumed by only the native Polaris management API. It also doesn't conflict with external IdPs: Compare to Databricks' RECIPIENT for example, which can wrap a Databricks identity, a provider-issued token, or the recipients own IdP through an OIDC federatoin policy, and shares are granted uniformly to it in all three cases: https://docs.databricks.com/aws/en/delta-sharing/create-recipient-oidc-fed Even in the full API model my PR proposes, we could let the Listing be something the authorizer looks at alongside other authz conext, so the pluggable authorizers/OPA/Ranger/etc can integrate as the authoritative layer. On Thu, Oct 22, 2026 at 2:00 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Dennis, > > > - Shares API defines only shares and the objects placed in shares, with > no > > Consumer or Listing > > POST /shares/{shareName}/listings taking a CreateListingRequest with a > `consumerName` is still part of `spec/polaris-shares-api.yaml`. > > I'm kind of confused about the scope of "shares" API vs. other APIs :) > Where are the boundaries? > > Would could be the minimal isoalated API sufficient for managing Shares? > > > - Controlling who can read shares is only ordinary Polaris authz, e.g. > > assignPrincipalRoletoPrincipal or > addGrantOnSecurableToPrincipalRole(share, > > principalRole) > > Sounds good. > > > - The partner organization consuming the share is just a principal, which > > could either be Polaris-managed or from an external IdP > > This also sounds good to me. > > I guess the only concern / confusion area is around Listing / Consumer > (first point above). > > Cheers, > Dmitri. > > > On Thu, Oct 22, 2026 at 2:07 PM Dennis Huo <[email protected]> wrote: > > > Thanks Dmitri, that does help highlight the the concern. Just to make > sure > > I understand what you have in mind: > > > > - Shares API defines only shares and the objects placed in shares, with > no > > Consumer or Listing > > - Controlling who can read shares is only ordinary Polaris authz, e.g. > > assignPrincipalRoletoPrincipal or > addGrantOnSecurableToPrincipalRole(share, > > principalRole) > > - The partner organization consuming the share is just a principal, which > > could either be Polaris-managed or from an external IdP > > > > Is that correct? > > > > On Wed, Oct 21, 2026 at 4:34 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi Dennis, > > > > > > Let me try to restate my concern from another angle. Apologies that I > was > > > not very clear in my previous messages. > > > > > > Currently the create listing doc says: "Create a listing, which grants > an > > > external consumer access to this share". > > > > > > So, this binds authorization decisions about a Share (Listing) to > > > ExternalConsumer. > > > > > > In IRC use cases a PolarisPrincipal is established by authenticating > the > > > client request (either via internal JWT or via external IdP), > > Subsequently, > > > the PolarisPrincipal is passed to PolarisAuthorizer for making > > > authorization decisions. The latter part can be external too (e.g. OPA > / > > > Ranger). > > > > > > (Side note: the "external" work in ExternalConsumer is actually quite > > > confusing with respect to prior usage of that word in external IdP use > > > cases.) > > > > > > In my view, the ExternalConsumer and related concepts in current Shares > > API > > > do not align with the external IdP and Authorizer workflows. > > > > > > IIRC, actual access to shared tables is going to use the IRC API, > right? > > > That access path in existing Polaris use cases maps well to external > IdP > > > and external authorizers. > > > > > > Why does the Shares API need to define a "Consumer" as a separate > entity > > > and narrow authorization decisions to its relationship with Listings? > > > > > > I believe authentication and authorization options should remain as > wide > > as > > > possible because these matters are usually addressed very differently > in > > > each deployment. > > > > > > To be clear: I do not mind defining Share "Consumers" in the internal > > > Polaris RBAC domain. It can fit well there, I'm only suggesting keeping > > the > > > Shares API itself neutral to Auth N/Z concerns. > > > > > > Cheers, > > > Dmitri. > > > > > > On Wed, Oct 21, 2026 at 7:02 PM Dennis Huo <[email protected]> wrote: > > > > > > > Ah thanks for clarifying - to be clear, the ConsumerCredential is > *not* > > > > strictly necessary for the Listing in terms of spec expressibility, > > only > > > > the ExternalConsumer is. > > > > > > > > An ExternalConsumer doesn't need to be backed by a > ConsumerCredential; > > > the > > > > ExternalConsumer is an abstract Sharing-specific pointer to whatever > > does > > > > the indirection in the sharing implementation. > > > > > > > > Responded to your github comment - I'll split anything that touches > > > > "ConsumerCredential" into a separate yaml file so it's easier to see > > the > > > > core auth-agnostic sharing spec in one file, with the more > > > > implementation-specific decorations around ConsumerCredential in an > > > > auxiliary file. > > > > > > > > I suppose another option is to even split it into separate PRs (in > > > addition > > > > to separate yaml files?). Then we can easily see one commit that's > just > > > the > > > > pure abstract auth-agnostic API, that in theory private Polaris > > > > implementations could already take and implement on external-IdP > > > > architectures to support the same agnostic spec. > > > > > > > > On Wed, Oct 21, 2026 at 1:54 PM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi Dennis, > > > > > > > > > > While I still support the idea of splitting OpenAPI yaml files and > > URI > > > > > bases between the Management and Sharing APIs, I do not think it is > > > > tightly > > > > > coupled with my proposal for the authentication-neural Sharing API > > > spec. > > > > > > > > > > To give an example, currently [5446] CreateListingRequest includes > a > > > > > "consumerName". This immediately binds the Listing definition to > > > > > an ExternalConsumer and by extension to ConsumerCredential. > > > > > > > > > > ConsumerCredential is naturally part of the authentication process > > > during > > > > > Share access. I believe ConsumerCredential (and related entities) > is > > > > worth > > > > > isolating from the Shares API to increase interoperability. > > > > > > > > > > Implementations that choose to rely on ConsumerCredential for Share > > > > access > > > > > can still do so. However, ConsumerCredential will not be a > > requirement > > > > for > > > > > all possible Share API implementations. > > > > > > > > > > This can work with any OpenAPI yaml and URI paths layout. > > > > > > > > > > I'd like to resolve this aspect before we go back to URI paths and > > YAML > > > > > file locations. Actually, I do not feel too strongly about path > > > > prefixes. I > > > > > do believe separate prefixes make implementations easier in many > > > aspects > > > > > (including network setup), but interleaving those paths with the > > > > Management > > > > > API is not a blocker, from my POV, only an inconvenience. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > [5446] https://github.com/apache/polaris/pull/5446 > > > > > > > > > > On Wed, Oct 21, 2026 at 3:19 PM Dennis Huo <[email protected]> > wrote: > > > > > > > > > > > Yeah, that's a good topic to discuss; I guess it touches on our > > core > > > > > > (unresolved?) conventions on how we organize API url-prefixes and > > > yaml > > > > > > files. Reposting my response to the github comment as well: > > > > > > > > > > > > Right, I think we could certainly split the API routes and > > > > > yaml-definitions > > > > > > even further if we want to ensure a well-defined auth-detail-free > > > core, > > > > > but > > > > > > for the standalone-Polaris use case we'd still need to define it > > > > > > *somewhere* > > > > > > under a Sharing-specific surface, right? > > > > > > > > > > > > @dimas-b <https://github.com/dimas-b> Were you thinking of just > > > > > splitting > > > > > > the yaml file but keeping the share-management root, or adding a > > > third > > > > > > sharing prefix like > > > > > > > > > > > > > > > > > > > > > > > > > > > /api/shares-management-consumer-credentials/v1/external-consumers/{name}/credentials... > > > > > > ? > > > > > > > > > > > > Or just yaml-file split would be keeping > > > > > > > /api/shares-management/v1/external-consumers/{name}/credentials... > > > but > > > > > > putting it in a separate yaml file to make it easy for > > > service-runners > > > > to > > > > > > choose not to support that whole yaml file's concrete APIs. > > > > > > > > > > > > On Tue, Oct 20, 2026 at 3:23 PM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > Re: the shating API PR [5446], I'm repeating my GH comment here > > for > > > > > > > visibility. > > > > > > > > > > > > > > I'd like the API to be neutral from the authentication > > perspective. > > > > > > > > > > > > > > This is how we handle the IRC spec in Polaris. The API declares > > > that > > > > > > > clients need to authenticate, but specific details about > > > principals, > > > > > > > clients and secrets are delegated to the server. > > > > > > > > > > > > > > I think this is a sound approach for the sharing API too. > > Specific > > > > > > details > > > > > > > related to native Polaris principals (ConsumerCredential, etc.) > > can > > > > > > > probably remain in the Management API. > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > Thanks, > > > > > > > Dmitri. > > > > > > > > > > > > > > [5446] https://github.com/apache/polaris/pull/5446 > > > > > > > > > > > > > > On Fri, Oct 2, 2026 at 12:50 AM Jean-Baptiste Onofré < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > > > As I said during the community meeting, let's focus on the > API > > PR > > > > > > > > first. We will build from there. > > > > > > > > > > > > > > > > Regards > > > > > > > > JB > > > > > > > > > > > > > > > > On Fri, Oct 2, 2026 at 3:18 AM Dennis Huo <[email protected]> > > > wrote: > > > > > > > > > > > > > > > > > > +1, thanks for the discussion everyone! > > > > > > > > > > > > > > > > > > Yes, I meant to flip it to "ready to review" earlier; > flipped > > > it > > > > > now. > > > > > > > I'm > > > > > > > > > still working through the existing set of comments and may > > need > > > > to > > > > > > come > > > > > > > > > back to it in the next few days before getting proper > > replies, > > > > but > > > > > > feel > > > > > > > > > free to add additional comments. > > > > > > > > > > > > > > > > > > Recapping the community sync discussion re: Dmitri's last > > > > message, > > > > > > I'm > > > > > > > > > fully aligned on building the sharing on top of existing > > authz > > > > > > > > mechanisms, > > > > > > > > > instead of being a new mechanism, and indeed external IdP > is > > > > > intended > > > > > > > to > > > > > > > > be > > > > > > > > > a first-class supported use case. > > > > > > > > > > > > > > > > > > The latest distinction is that the ExternalConsumer isn't > > > > > > fundamentally > > > > > > > > > identified by any internal PolarisPrincipal, but rather > that > > it > > > > > > serves > > > > > > > > as a > > > > > > > > > wrapper that references some identity, whether that > identity > > is > > > > > > defined > > > > > > > > in > > > > > > > > > an external IdP without any latent in-Polaris > representation > > or > > > > if > > > > > > > it's a > > > > > > > > > hidden PolarisPrincipal (for the Polaris-standalone RBAC > use > > > > case). > > > > > > > > > > > > > > > > > > The sharing endpoint via ExternalConsumer is the only layer > > of > > > > > > > > indirection > > > > > > > > > at the API entry-point, but then once the set of > > > > > activePrincipalRoles > > > > > > > is > > > > > > > > > resolved, the underlying authz engine doesn't behave any > > > > > differently > > > > > > > from > > > > > > > > > normal Polaris authz. > > > > > > > > > > > > > > > > > > On Thu, Oct 1, 2026 at 5:58 PM Yufei Gu < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks everyone for the discussion! > > > > > > > > > > > > > > > > > > > > As a next step discussed in today's community sync, we > will > > > > start > > > > > > > > reviewing > > > > > > > > > > the spec change. > > > > > > > > > > > > > > > > > > > > Hi Dennis, is this PR ready for review? > > > > > > > > > > https://github.com/apache/polaris/pull/5446 > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Sep 26, 2026 at 12:23 PM Travis Bowen < > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks for this discussion Dennis! > > > > > > > > > > > > > > > > > > > > > > Wanted to weigh in with my thoughts and questions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you prefer /api/shares-management/v1/ > > > > > > > > > > > > instead of adding to /api/management/v1? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > While I can see the argument for a share specific > > > management > > > > > > > endpoint > > > > > > > > > > there > > > > > > > > > > > doesn't seem to be a clear obvious choice from what I > can > > > see > > > > > - I > > > > > > > > lean > > > > > > > > > > > towards /api/management as being the central place for > > all > > > > the > > > > > > > > > > management - > > > > > > > > > > > share or not. As long as we're careful not to leak > share > > > > types > > > > > or > > > > > > > non > > > > > > > > > > share > > > > > > > > > > > types into the wrong api (which could be helped by > having > > > > > > separate > > > > > > > > yamls) > > > > > > > > > > > then it feels reasonable to have all under the > management > > > > api. > > > > > > > > > > > > > > > > > > > > > > Also went through the doc and PR and made several > > comments > > > as > > > > > > well. > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 18, 2026 at 9:54 AM Dmitri Bourlatchkov < > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > Prithvi mentioned ExternalConsumer and principals. > I'd > > > like > > > > > to > > > > > > > > > > > > reinforce the need to further discuss "consumers" and > > > > > > principals > > > > > > > > and > > > > > > > > > > how > > > > > > > > > > > > they relate to IdP and Polaris Authentication. > > > > > > > > > > > > > > > > > > > > > > > > From my POV, share access control should not be a new > > > > > mechanism > > > > > > > but > > > > > > > > > > > should > > > > > > > > > > > > build on top of the existing extensible > authentication > > / > > > > > > > > authorization > > > > > > > > > > > > framework. Supporting external IdP for shares would > be > > a > > > > > > valuable > > > > > > > > > > feature > > > > > > > > > > > > IMHO. > > > > > > > > > > > > > > > > > > > > > > > > Side note: I'm posting comments on PR 5446 too. > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 3:30 PM Prithvi S < > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for reviving this, and for the concrete > > spec > > > in > > > > > PR > > > > > > > > 5446.. > > > > > > > > > > the > > > > > > > > > > > > > journeys doc was especially helpful for seeing the > > > > > end-to-end > > > > > > > > flow. > > > > > > > > > > > > > > > > > > > > > > > > > > I agree this is worth doing, and that v1 is more > > than a > > > > > > > tutorial > > > > > > > > on > > > > > > > > > > > > today's > > > > > > > > > > > > > RBAC. A partner principal on /api/catalog is > already > > > > > > possible; > > > > > > > > what > > > > > > > > > > > seems > > > > > > > > > > > > > valuable is a first-class share (enumerated > members, > > a > > > > > > > restricted > > > > > > > > > > > > consumer, > > > > > > > > > > > > > a listing as the binding and audit unit) plus a > > > > segregated, > > > > > > > > read-only > > > > > > > > > > > > > Iceberg REST surface with credential vending. Stock > > > > Iceberg > > > > > > > REST > > > > > > > > as > > > > > > > > > > the > > > > > > > > > > > > > consumer protocol, and keeping Flight and > > > > > Polaris-to-Polaris > > > > > > > > > > federation > > > > > > > > > > > > out > > > > > > > > > > > > > of v1, all sound right to me. > > > > > > > > > > > > > > > > > > > > > > > > > > I don't have a strong view yet on the URL layout > > > > > (management > > > > > > vs > > > > > > > > > > shares > > > > > > > > > > > > > prefix, nested vs top-level, prefix keyed by share > vs > > > > > > listing) > > > > > > > > and am > > > > > > > > > > > > happy > > > > > > > > > > > > > to follow whatever the community prefers there. > > > > > > > > > > > > > > > > > > > > > > > > > > A few things I wanted to check on the spec itself, > > > > several > > > > > of > > > > > > > > these > > > > > > > > > > > echo > > > > > > > > > > > > > points already raised on the list: > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Membership as source of truth. If a hidden > catalog > > > > role > > > > > is > > > > > > > > > > > > independently > > > > > > > > > > > > > grantable, a share could be widened without going > > > through > > > > > the > > > > > > > > share > > > > > > > > > > > APIs. > > > > > > > > > > > > > Would it make sense to say the data plane > authorizes > > > > > against > > > > > > > > > > > membership + > > > > > > > > > > > > > listing, and that any backing roles are not > > > user-visible > > > > / > > > > > > not > > > > > > > > > > > > > independently grantable? > > > > > > > > > > > > > 2. ExternalConsumer and principals. The Aug 7 recap > > > > already > > > > > > > > noted we > > > > > > > > > > > > should > > > > > > > > > > > > > not assume a PrincipalEntity behind every > > > > ExternalConsumer. > > > > > > The > > > > > > > > > > > journeys > > > > > > > > > > > > > still describe granting the consumer a share > > principal > > > > > role, > > > > > > I > > > > > > > > assume > > > > > > > > > > > > that > > > > > > > > > > > > > is an implementation sketch for v1 > client_id/secret, > > > not > > > > > the > > > > > > > > public > > > > > > > > > > > > model? > > > > > > > > > > > > > 3. Listing expiration. JB's original note was > "share > > > > these > > > > > > > tables > > > > > > > > > > with > > > > > > > > > > > > this > > > > > > > > > > > > > consumer until this date." Credential expiry is a > bit > > > > > > > different. > > > > > > > > > > Would > > > > > > > > > > > a > > > > > > > > > > > > > listing-level expiration (and maybe > ACTIVE/SUSPENDED) > > > be > > > > in > > > > > > > > scope for > > > > > > > > > > > v1? > > > > > > > > > > > > > 4. Time-travel. As raised in the community sync, > > > sharing > > > > > via > > > > > > > > Iceberg > > > > > > > > > > > REST > > > > > > > > > > > > > means historical snapshots remain readable. Totally > > > fine > > > > > for > > > > > > v1 > > > > > > > > to > > > > > > > > > > > > support > > > > > > > > > > > > > only full history, would an explicit field on the > > share > > > > > help > > > > > > > so a > > > > > > > > > > later > > > > > > > > > > > > > CURRENT_ONLY option is not a breaking surprise? > > > > > > > > > > > > > 5. Consumer data plane. The current file is > > > control-plane > > > > > > only, > > > > > > > > which > > > > > > > > > > > is > > > > > > > > > > > > a > > > > > > > > > > > > > good first step. It might help to list the Iceberg > > > routes > > > > > > that > > > > > > > > are > > > > > > > > > > > > mounted, > > > > > > > > > > > > > that writes 404, that non-members 404, and that > > /config > > > > > does > > > > > > > not > > > > > > > > > > return > > > > > > > > > > > > > provider catalog properties, so implementations > don't > > > > > > diverge. > > > > > > > > > > > > > > > > > > > > > > > > > > Two smaller notes, if useful: in the J1 journey the > > > > > consumer > > > > > > > > still > > > > > > > > > > > posts > > > > > > > > > > > > to > > > > > > > > > > > > > /api/catalog/v1/oauth/tokens, if partners are not > > > > supposed > > > > > to > > > > > > > > touch > > > > > > > > > > the > > > > > > > > > > > > > internal catalog API, should the default token > > endpoint > > > > > live > > > > > > on > > > > > > > > the > > > > > > > > > > > > shares > > > > > > > > > > > > > surface (or an external IdP)? And on Dmitri's point > > > about > > > > > > share > > > > > > > > > > > operators > > > > > > > > > > > > > vs catalog operators, a SHARE_MANAGE privilege on > the > > > > bound > > > > > > > > catalog > > > > > > > > > > > > > (independent of CATALOG_MANAGE_ACCESS) would make > > that > > > > work > > > > > > on > > > > > > > > any > > > > > > > > > > URL > > > > > > > > > > > > > prefix. > > > > > > > > > > > > > > > > > > > > > > > > > > Happy to review the next revision, and thanks again > > to > > > > both > > > > > > of > > > > > > > > you > > > > > > > > > > for > > > > > > > > > > > > > driving this. > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > Prithvi S > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 4, 2026 at 4:38 PM Dennis Huo < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > As discussed live, looks like we both might have > > some > > > > > > drafts > > > > > > > > we can > > > > > > > > > > > > > compare > > > > > > > > > > > > > > and choose/merge from :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here's my current draft spec PR; it opts for my > > > > preferred > > > > > > > > syntax > > > > > > > > > > > > choices > > > > > > > > > > > > > > for now (/api/management/ for share-management, > > > > > > /api/shares/ > > > > > > > > for > > > > > > > > > > > > consumer > > > > > > > > > > > > > > data-plane): > > > > https://github.com/apache/polaris/pull/5446 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Since we were talking about preferring mdfiles > for > > > > design > > > > > > > > iteration > > > > > > > > > > > > > here's > > > > > > > > > > > > > > a companion design-addendum which just enumerates > > > what > > > > > the > > > > > > > > > > end-to-end > > > > > > > > > > > > > user > > > > > > > > > > > > > > journeys look like syntactically for each > > combination > > > > of > > > > > > > design > > > > > > > > > > > > choices: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/dennishuo/dhuo-public-provenance/blob/main/polaris-sharing/design/shares-journeys.md > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please let me know if anyone prefers a Google Doc > > > > version > > > > > > of > > > > > > > > the > > > > > > > > > > > > > > design-addendum for easier commenting, or if the > > > > > > > > design-addendum > > > > > > > > > > > itself > > > > > > > > > > > > > > should also be modeled as a PR on some > > free-floating > > > > > branch > > > > > > > > just to > > > > > > > > > > > > > capture > > > > > > > > > > > > > > comments. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 3, 2026 at 7:22 AM Jean-Baptiste > > Onofré < > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am resuming work on this proposal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding our recent discussion, I will move > > > forward > > > > > with > > > > > > > > > > opening a > > > > > > > > > > > > > > > draft PR for the /api/shares/v1 endpoint to > help > > > > > > illustrate > > > > > > > > the > > > > > > > > > > > > > > > design. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I will update this thread once the PR is ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 5:04 PM Dmitri > > Bourlatchkov > > > < > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Dennis, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the recap! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I was thinking that share management is > > probably > > > > > > distinct > > > > > > > > from > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > management. A user who already has a > > > realm+catalog > > > > > may > > > > > > > > have an > > > > > > > > > > > > option > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > manage shares, but not the catalog itself. > > Would > > > it > > > > > > make > > > > > > > > sense > > > > > > > > > > to > > > > > > > > > > > > > > > separate > > > > > > > > > > > > > > > > the share management API from the general > > Polaris > > > > > > > > management > > > > > > > > > > > API? I > > > > > > > > > > > > > > mean > > > > > > > > > > > > > > > > using a different prefix for URIs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For example: /api/shares/v1/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Obviously clients should not make any > > assumptions > > > > > about > > > > > > > the > > > > > > > > > > base > > > > > > > > > > > > path > > > > > > > > > > > > > > > > (/api/shares), which could be different in > > > > different > > > > > > > > > > deployments > > > > > > > > > > > > > (e.g. > > > > > > > > > > > > > > > > going through a gateway). Clients must be > able > > to > > > > > take > > > > > > > the > > > > > > > > base > > > > > > > > > > > URI > > > > > > > > > > > > > as > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > config option. Path segments after /v1/ will > be > > > > > > specified > > > > > > > > by > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > (Open > > > > > > > > > > > > > > > > API yaml). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 7, 2026 at 3:52 AM Dennis Huo < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just to recap some of the discussion from > > > today's > > > > > > > > community > > > > > > > > > > > sync, > > > > > > > > > > > > > > > Dmitri, > > > > > > > > > > > > > > > > > Robert, and Alex helped raise a few > > > > considerations > > > > > to > > > > > > > > roll > > > > > > > > > > into > > > > > > > > > > > > the > > > > > > > > > > > > > > > next > > > > > > > > > > > > > > > > > design/prototyping phase: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Given the way sharing is packaged as a > > > > separate > > > > > > > > thing, it > > > > > > > > > > > may > > > > > > > > > > > > > not > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > obvious to users that historical snapshots > > are > > > > > > equally > > > > > > > > > > > available > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > consumers as the latest snapshot (by virtue > > of > > > > > > sharing > > > > > > > > via > > > > > > > > > > the > > > > > > > > > > > > > > > standard IRC > > > > > > > > > > > > > > > > > protocol) - we should make sure to document > > > that > > > > > fact > > > > > > > > well, > > > > > > > > > > > > > including > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > pitfall that deleting rows by updating a > > table > > > > > > doesn't > > > > > > > > > > > > necessarily > > > > > > > > > > > > > > mean > > > > > > > > > > > > > > > > > those rows become inaccessible to consumers > > > > > > > > > > > > > > > > > 2. Make sure not to hard-code assumptions > > about > > > > the > > > > > > > > existence > > > > > > > > > > > of > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > internal PrincipalEntity backing an > > > > > ExternalConsumer > > > > > > - > > > > > > > > even > > > > > > > > > > if > > > > > > > > > > > > the > > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > > > MVP implementation uses a nested/hidden > > > Principal > > > > > of > > > > > > > some > > > > > > > > > > type > > > > > > > > > > > > that > > > > > > > > > > > > > > > holds a > > > > > > > > > > > > > > > > > Polaris-managed client_id/client_secret, > the > > > > > > 3rd-party > > > > > > > > IdP > > > > > > > > > > use > > > > > > > > > > > > > cases > > > > > > > > > > > > > > > > > require "implicit" Principals resolved via > > the > > > > > oauth > > > > > > > > token. > > > > > > > > > > In > > > > > > > > > > > > such > > > > > > > > > > > > > > > cases > > > > > > > > > > > > > > > > > the ExternalConsumer contains the metadata > > > > > > coordinating > > > > > > > > the > > > > > > > > > > > > > three-way > > > > > > > > > > > > > > > > > handshake between the data-provider, > > > > data-consumer, > > > > > > and > > > > > > > > IdP, > > > > > > > > > > > but > > > > > > > > > > > > > > there > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > no PrincipalEntity underlying it. > > > > > > > > > > > > > > > > > 3. Longer-term, we may want to tackle > > > > > "on-behalf-of" > > > > > > > > > > semantics > > > > > > > > > > > - > > > > > > > > > > > > > > > initially, > > > > > > > > > > > > > > > > > the way an ExternalConsumer works is that > the > > > > > > > > finer-grained > > > > > > > > > > > > > consumer > > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > > > share a single ExternalConsumer identity, > > just > > > > like > > > > > > > how a > > > > > > > > > > > > FEDERATED > > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > shares a single service principal in its > > > > > > > > ConnectionConfig. > > > > > > > > > > But > > > > > > > > > > > in > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > cases the consumer may want some end-to-end > > > > > > attribution > > > > > > > > of > > > > > > > > > > > > > > fine-grained > > > > > > > > > > > > > > > > > users, like "on-behalf-of: bob @ > > > > > consumercompany.com > > > > > > " > > > > > > > > when > > > > > > > > > > the > > > > > > > > > > > > > > > > > ExternalConsumer implicit principal is > > > something > > > > > like > > > > > > > > > > > > > > > > > "externalconsumer$consumercompany" > > > > > > > > > > > > > > > > > 4. We can iterate more rapidly on a draft > > spec > > > PR > > > > > and > > > > > > > > also > > > > > > > > > > > expand > > > > > > > > > > > > > on > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > detailed technical design > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > Dennis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2026 at 6:12 PM Dennis Huo > < > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks JB for reviving this thread, and > > > belated > > > > > > > thanks > > > > > > > > to > > > > > > > > > > all > > > > > > > > > > > > > > > reviewers > > > > > > > > > > > > > > > > > > who have left feedback in the doc or this > > > > thread > > > > > so > > > > > > > > far! > > > > > > > > > > And > > > > > > > > > > > > > > > apologies > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > the silence on this, it's still been on > my > > > > radar > > > > > > but > > > > > > > I > > > > > > > > got > > > > > > > > > > > > > > > distracted by > > > > > > > > > > > > > > > > > > some other work for awhile. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Echoing JB's summary, the proposal is > > indeed > > > > that > > > > > > > we're > > > > > > > > > > > mostly > > > > > > > > > > > > > just > > > > > > > > > > > > > > > > > > reusing existing Authz/RBAC machinery, > and > > > the > > > > > > > > translation > > > > > > > > > > > > layer > > > > > > > > > > > > > to > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > sharing concepts sits *on top of* the > > > existing > > > > > > authz > > > > > > > > engine > > > > > > > > > > > as > > > > > > > > > > > > > much > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So even if we're adding layers of > > indirection > > > > on > > > > > > > > > > > > > > > ExternalConsumer/Listing > > > > > > > > > > > > > > > > > > -> > > > Principal/PrincipalRole/CatalogRole+grants, > > > > > the > > > > > > > > parts > > > > > > > > > > that > > > > > > > > > > > > end > > > > > > > > > > > > > > up > > > > > > > > > > > > > > > > > doing > > > > > > > > > > > > > > > > > > the underlying authz just reuse existing > > > > > Authorizer > > > > > > > > logic. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Based on some other offline discussion > with > > > > > > Dmitri, I > > > > > > > > > > > realized > > > > > > > > > > > > > why > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > discussion around "requirements" may have > > > been > > > > > > > > confusing: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. My attempts to shorten the doc > > > inadvertently > > > > > cut > > > > > > > out > > > > > > > > > > some > > > > > > > > > > > of > > > > > > > > > > > > > the > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > "abstract requirements" formulations > > > > > > > > > > > > > > > > > > 2. The project in itself is somewhat more > > of > > > a > > > > > > "form > > > > > > > > *as* > > > > > > > > > > > > > function" > > > > > > > > > > > > > > > > > > feature than *strictly* functional in > > nature > > > -- > > > > > for > > > > > > > > > > example, > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > correctly > > > > > > > > > > > > > > > > > > pointed out that "sharing" via the > existing > > > > > > > > RBAC/Principals > > > > > > > > > > > > > syntax > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > already *possible* just by letting a > > > cross-org > > > > > > > > "consumer" > > > > > > > > > > be > > > > > > > > > > > a > > > > > > > > > > > > > > > > > first-class > > > > > > > > > > > > > > > > > > principal and access your own company's > > main > > > > > > Polaris > > > > > > > > > > catalog > > > > > > > > > > > > > > > endpoints. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For (1) I added this to the Requirements > > > > section > > > > > in > > > > > > > > the doc > > > > > > > > > > > to > > > > > > > > > > > > > > better > > > > > > > > > > > > > > > > > > clarify the abstract form of the data > model > > > > > > element: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A. Sharing Container*: Container concept > > for > > > > > > holding > > > > > > > > > > things > > > > > > > > > > > > you > > > > > > > > > > > > > > > want to > > > > > > > > > > > > > > > > > > share - “ShareEntity” > > > > > > > > > > > > > > > > > > *B. Consumer Identity*: Special flavor of > > > > > > > > principal-like > > > > > > > > > > > entity > > > > > > > > > > > > > > > > > > representing a consumer relationship - > > > > > > > > “ExternalConsumer” > > > > > > > > > > > > > > > > > > *C. Map Containers to Consumers*: A way > to > > > > > specify > > > > > > > > letting > > > > > > > > > > a > > > > > > > > > > > > > given > > > > > > > > > > > > > > > > > > consumer access a given sharing > container - > > > > > > “Listing” > > > > > > > > > > > > > > > > > > *D. Consumer Endpoint*: A segregated API > > > > > > access-point > > > > > > > > for > > > > > > > > > > > > > > consumers - > > > > > > > > > > > > > > > > > > “EndpointConfig” > > > > > > > > > > > > > > > > > > *E. Objects in Containers*: A way to > > specify > > > > > which > > > > > > > > > > in-catalog > > > > > > > > > > > > > > objects > > > > > > > > > > > > > > > > > > belong in the big sharing container - > > > > > > > “ShareMembership” > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This still bakes in a bit of > implementation > > > > > detail > > > > > > > > since it > > > > > > > > > > > > begs > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > question why not just let core > RBAC/grants > > > > handle > > > > > > (C) > > > > > > > > and > > > > > > > > > > > (E). > > > > > > > > > > > > To > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > end, > > > > > > > > > > > > > > > > > > and addressing problem (2) of where the > > "form > > > > > *is* > > > > > > > > > > > function", I > > > > > > > > > > > > > > > added a > > > > > > > > > > > > > > > > > > "Concept Mapping" tab: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think the details of more advanced > > features > > > > can > > > > > > > > still be > > > > > > > > > > > > hashed > > > > > > > > > > > > > > out > > > > > > > > > > > > > > > > > > incrementally without blocking work on an > > MVP > > > > > (and > > > > > > > > > > certainly > > > > > > > > > > > > the > > > > > > > > > > > > > > MVP > > > > > > > > > > > > > > > > > scope > > > > > > > > > > > > > > > > > > could be trimmed so we can > > > launch-and-iterate), > > > > > so > > > > > > as > > > > > > > > long > > > > > > > > > > as > > > > > > > > > > > > > > there's > > > > > > > > > > > > > > > > > > community alignment on the general > > direction > > > I > > > > > > > propose > > > > > > > > we > > > > > > > > > > > could > > > > > > > > > > > > > > move > > > > > > > > > > > > > > > > > > forward on basic building blocks. Or if > > > > building > > > > > > > blocks > > > > > > > > > > > > > themselves > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > controversial we could still proceed with > > > some > > > > > > > > prototyping > > > > > > > > > > > > across > > > > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > > options. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > Dennis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 21, 2026 at 10:44 PM > > > Jean-Baptiste > > > > > > > Onofré < > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi everyone, > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> I would like to resume the discussion > > about > > > > Data > > > > > > > > Sharing > > > > > > > > > > in > > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > > >> and propose focusing on three major > items > > to > > > > > help > > > > > > us > > > > > > > > move > > > > > > > > > > > > > forward. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> 1. What is data sharing? > > > > > > > > > > > > > > > > > >> In a nutshell, it is an open > protocol > > > > > proposal > > > > > > > > > > allowing > > > > > > > > > > > > > > > > > >> organizations to share live data without > > > > copying > > > > > > or > > > > > > > > > > > > replication. > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > >> main goal is zero-copy sharing achieved > > > > through > > > > > > > shared > > > > > > > > > > > > metadata > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > >> authorization. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> On Polaris, this would mean: > > > > > > > > > > > > > > > > > >> a. The consumer is authenticated as > usual. > > > > > > > > > > > > > > > > > >> b. The Polaris server provides the > > consumer > > > > with > > > > > > > > direct > > > > > > > > > > > access > > > > > > > > > > > > > to > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >> data files (e.g., Parquet, as specified > in > > > the > > > > > > > > > > > metadata.json). > > > > > > > > > > > > > > > > > >> c. The consumer reads the data files > > > directly > > > > > > using > > > > > > > > their > > > > > > > > > > > > > > preferred > > > > > > > > > > > > > > > > > >> tool (e.g., a query engine). > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> While Polaris already has most of the > > > > necessary > > > > > > > > underlying > > > > > > > > > > > > > > > > > >> capabilities, we need clear > > > branding/packaging > > > > > > > around > > > > > > > > this > > > > > > > > > > > use > > > > > > > > > > > > > > case. > > > > > > > > > > > > > > > > > >> Specifically, this means: > > > > > > > > > > > > > > > > > >> a. A share is a dedicated catalog in the > > > > Polaris > > > > > > > > server, > > > > > > > > > > or > > > > > > > > > > > a > > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > >> role scoped to specific namespaces. > > > > > > > > > > > > > > > > > >> b. A recipient is represented by a > > principal > > > > > and a > > > > > > > > > > principal > > > > > > > > > > > > > role. > > > > > > > > > > > > > > > > > >> c. A consumer profile consists of the > > > catalog > > > > > URI > > > > > > > and > > > > > > > > the > > > > > > > > > > > > OAuth2 > > > > > > > > > > > > > > > > > >> client_id/client_secret. > > > > > > > > > > > > > > > > > >> d. We use credential vending (temporary > > STS > > > > > > > > credentials) > > > > > > > > > > for > > > > > > > > > > > > > > > consumer > > > > > > > > > > > > > > > > > >> access to the data files. > > > > > > > > > > > > > > > > > >> e. We use the Polaris IRC layer to > expose > > > the > > > > > > > > metadata and > > > > > > > > > > > the > > > > > > > > > > > > > > file > > > > > > > > > > > > > > > > > >> locations/access. > > > > > > > > > > > > > > > > > >> f. We can also support > Polaris-to-Polaris > > > > > sharing > > > > > > > via > > > > > > > > > > > > > Federation. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> Concretely: > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> - On the provider side, we use the > > > standard > > > > > > > Polaris > > > > > > > > RBAC > > > > > > > > > > > > model > > > > > > > > > > > > > > > > > >> (assigning grants to catalog roles, and > > > > > assigning > > > > > > > > those > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > roles > > > > > > > > > > > > > > > > > >> to principal roles): > > > > > > > > > > > > > > > > > >> - Create a dedicated principal > with > > > its > > > > > own > > > > > > > > > > > > > > > > > client_id/client_secret. > > > > > > > > > > > > > > > > > >> - Create a catalog role with > > > > > > TABLE_READ_DATA, > > > > > > > > > > > > TABLE_LIST, > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > >> NAMESPACE_LIST grants on the namespaces > to > > > be > > > > > > shared > > > > > > > > (no > > > > > > > > > > > write > > > > > > > > > > > > > > > > > >> grants). > > > > > > > > > > > > > > > > > >> - Create a principal role per > > > > destination, > > > > > > > > attach > > > > > > > > > > the > > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > >> role, and assign the principal. > > > > > > > > > > > > > > > > > >> - The destination engine uses > > > loadTable > > > > > with > > > > > > > the > > > > > > > > > > > > > > > > > >> "X-Iceberg-Access-Delegation: > > > > > vended-credentials" > > > > > > > > header. > > > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > then > > > > > > > > > > > > > > > > > >> emits temporary, read-only storage > > > > credentials, > > > > > > > > allowing > > > > > > > > > > the > > > > > > > > > > > > > > engine > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > >> read the data files directly from the > > > owner's > > > > > > bucket > > > > > > > > > > without > > > > > > > > > > > > > > > accessing > > > > > > > > > > > > > > > > > >> the rest of the bucket. This achieves > > > > zero-copy > > > > > > > > sharing. > > > > > > > > > > > > > > > > > >> - On the recipient/consumer side, no > > > special > > > > > > > > > > configuration > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > >> needed; any IRC client can use it. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> 2. What is data sharing not? > > > > > > > > > > > > > > > > > >> I previously considered adding an > > Arrow > > > > > Flight > > > > > > > > > > endpoint > > > > > > > > > > > to > > > > > > > > > > > > > > > > > >> Polaris, but I no longer think it is a > > good > > > > idea > > > > > > for > > > > > > > > the > > > > > > > > > > > > initial > > > > > > > > > > > > > > > > > >> phase. An Arrow Flight endpoint would > > serve > > > > the > > > > > > data > > > > > > > > > > > directly, > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > >> would require Polaris to read the data > and > > > > > expose > > > > > > > it. > > > > > > > > I am > > > > > > > > > > > > > against > > > > > > > > > > > > > > > > > >> embedding a query engine in Polaris, as > > this > > > > > tight > > > > > > > > > > coupling > > > > > > > > > > > > > feels > > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > > >> an anti-pattern. While we could use a > > > > delegation > > > > > > > > service > > > > > > > > > > to > > > > > > > > > > > > > serve > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >> Flight endpoint, I think it should be > > > > considered > > > > > > > for a > > > > > > > > > > later > > > > > > > > > > > > > > phase, > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > >> at all, since a Flight endpoint might be > > > > better > > > > > > > > suited as > > > > > > > > > > a > > > > > > > > > > > > > > consumer > > > > > > > > > > > > > > > > > >> responsibility and could encourage data > > > > copying. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> The first phase should focus on > packaging > > > and > > > > > > > > branding the > > > > > > > > > > > > > > > > > >> capabilities we already have. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> 3. What could the Data Sharing > "package" > > > look > > > > > > like > > > > > > > in > > > > > > > > > > > > Polaris? > > > > > > > > > > > > > > > > > >> To make Data Sharing obvious and > easy > > to > > > > > use, > > > > > > we > > > > > > > > > > should > > > > > > > > > > > > > > > introduce > > > > > > > > > > > > > > > > > >> the missing concept of a "share"—a named > > > > > > definition > > > > > > > > that > > > > > > > > > > > users > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > >> list, discover, and define. The goal is > to > > > > > > > > materialize the > > > > > > > > > > > > > user's > > > > > > > > > > > > > > > > > >> intent ("I want to share these tables > with > > > > this > > > > > > > > consumer > > > > > > > > > > > until > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > >> date") into standard Polaris entities > > > > > (principal, > > > > > > > > > > principal > > > > > > > > > > > > > role, > > > > > > > > > > > > > > > > > >> catalog role, and grants). > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> Therefore, I propose adding a /shares > > > endpoint > > > > > to > > > > > > > the > > > > > > > > > > > Polaris > > > > > > > > > > > > > > > > > >> Management API to package and > materialize > > > > these > > > > > > > share > > > > > > > > > > > > > definitions > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > >> Polaris entities. This would provide a > > > usable, > > > > > > > > > > first-version > > > > > > > > > > > > > > package > > > > > > > > > > > > > > > > > >> for Polaris Data Sharing. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> I hope this addresses Dmitri's questions > > and > > > > > > > provides > > > > > > > > a > > > > > > > > > > more > > > > > > > > > > > > > > > concrete > > > > > > > > > > > > > > > > > >> scenario for Polaris Data Sharing. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> Thoughts? > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> Regards, > > > > > > > > > > > > > > > > > >> JB > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> On Thu, May 28, 2026 at 7:46 AM Dennis > > Huo < > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > Hi All, > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > One big emerging enterprise use case > > > coming > > > > up > > > > > > as > > > > > > > > more > > > > > > > > > > > > people > > > > > > > > > > > > > > > > > >> consolidate > > > > > > > > > > > > > > > > > >> > Data Lakehouses and Catalogs is > > something > > > > > > commonly > > > > > > > > known > > > > > > > > > > > as > > > > > > > > > > > > > > "Data > > > > > > > > > > > > > > > > > >> Sharing", > > > > > > > > > > > > > > > > > >> > an more specifically over the course > of > > > > > adoption > > > > > > > of > > > > > > > > open > > > > > > > > > > > > table > > > > > > > > > > > > > > > formats > > > > > > > > > > > > > > > > > >> > "Open Data Sharing". > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > Examples of existing managed service > > > > > providers' > > > > > > > Data > > > > > > > > > > > Sharing > > > > > > > > > > > > > > > features: > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > https://www.databricks.com/product/delta-sharing > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > https://docs.snowflake.com/en/user-guide/data-sharing-intro > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > The basic idea is that when you share > > data > > > > > > between > > > > > > > > > > > different > > > > > > > > > > > > > > > > > companies, > > > > > > > > > > > > > > > > > >> you > > > > > > > > > > > > > > > > > >> > need a first-class > governance/management > > > > layer > > > > > > and > > > > > > > > extra > > > > > > > > > > > > > > > > > >> bells-and-whistles > > > > > > > > > > > > > > > > > >> > that are distinct from just the basic > > > > > > capabilities > > > > > > > > of > > > > > > > > > > RBAC > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > >> generalized > > > > > > > > > > > > > > > > > >> > access-control (i.e. if you're sharing > > > > across > > > > > > > > > > > > > > partially-untrusted > > > > > > > > > > > > > > > org > > > > > > > > > > > > > > > > > >> > boundaries, you don't just let the > > > consumer > > > > > > > > organization > > > > > > > > > > > log > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > your > > > > > > > > > > > > > > > > > >> > datalake like one of your own > > employees). > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > JB and I put together this high-level > > > > proposal > > > > > > for > > > > > > > > > > > > supporting > > > > > > > > > > > > > > Open > > > > > > > > > > > > > > > > > >> Sharing > > > > > > > > > > > > > > > > > >> > in Polaris: > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > Tentatively, it means adding ~5 > logical > > > data > > > > > > model > > > > > > > > > > > > constructs, > > > > > > > > > > > > > > > some of > > > > > > > > > > > > > > > > > >> > which may be a first-class > PolarisEntity > > > > type, > > > > > > > > others > > > > > > > > > > > > subtypes > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > >> existing > > > > > > > > > > > > > > > > > >> > entities, and others just a nested > > > > construct: > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > - ShareEntity (would behave > similarly > > > to > > > > a > > > > > > > > Catalog) > > > > > > > > > > > > > > > > > >> > - ExternalConsumer (mostly inherits > > > from > > > > > > > > Principal) > > > > > > > > > > > > > > > > > >> > - Listing (Similar to a "role > grant" > > > but > > > > > has > > > > > > > > > > different > > > > > > > > > > > > > > > metadata) > > > > > > > > > > > > > > > > > >> > - EndpointConfig (nested config > under > > > > > > Listing) > > > > > > > > > > > > > > > > > >> > - ShareMembership (Similar to a > > > > "securable > > > > > > > > grant" but > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > >> metadata) > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > Feedback/comments welcome! I'll also > > bring > > > > it > > > > > up > > > > > > > for > > > > > > > > > > live > > > > > > > > > > > > > > > discussion > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > >> > there's time in the community sync. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > Cheers, > > > > > > > > > > > > > > > > > >> > Dennis > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
