Thanks JB for reviving this thread, and belated thanks to all reviewers who have left feedback in the doc or this thread so far! And apologies for the silence on this, it's still been on my radar but I got distracted by some other work for awhile.
Echoing JB's summary, the proposal is indeed that we're mostly just reusing existing Authz/RBAC machinery, and the translation layer to the new sharing concepts sits *on top of* the existing authz engine as much as possible. So even if we're adding layers of indirection on ExternalConsumer/Listing -> Principal/PrincipalRole/CatalogRole+grants, the parts that end up doing the underlying authz just reuse existing Authorizer logic. Based on some other offline discussion with Dmitri, I realized why the discussion around "requirements" may have been confusing: 1. My attempts to shorten the doc inadvertently cut out some of the more "abstract requirements" formulations 2. The project in itself is somewhat more of a "form *as* function" feature than *strictly* functional in nature -- for example, many correctly pointed out that "sharing" via the existing RBAC/Principals syntax is already *possible* just by letting a cross-org "consumer" be a first-class principal and access your own company's main Polaris catalog endpoints. For (1) I added this to the Requirements section in the doc to better clarify the abstract form of the data model element: *A. Sharing Container*: Container concept for holding things you want to share - “ShareEntity” *B. Consumer Identity*: Special flavor of principal-like entity representing a consumer relationship - “ExternalConsumer” *C. Map Containers to Consumers*: A way to specify letting a given consumer access a given sharing container - “Listing” *D. Consumer Endpoint*: A segregated API access-point for consumers - “EndpointConfig” *E. Objects in Containers*: A way to specify which in-catalog objects belong in the big sharing container - “ShareMembership” This still bakes in a bit of implementation detail since it begs the question why not just let core RBAC/grants handle (C) and (E). To that end, and addressing problem (2) of where the "form *is* function", I added a "Concept Mapping" tab: https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?tab=t.rms5bxy0jntc#heading=h.wuh830gzomt0 I think the details of more advanced features can still be hashed out incrementally without blocking work on an MVP (and certainly the MVP scope could be trimmed so we can launch-and-iterate), so as long as there's community alignment on the general direction I propose we could move forward on basic building blocks. Or if building blocks themselves are controversial we could still proceed with some prototyping across different options. Cheers, Dennis On Tue, Jul 21, 2026 at 10:44 PM Jean-Baptiste Onofré <[email protected]> wrote: > Hi everyone, > > I would like to resume the discussion about Data Sharing in Polaris > and propose focusing on three major items to help us move forward. > > 1. What is data sharing? > In a nutshell, it is an open protocol proposal allowing > organizations to share live data without copying or replication. The > main goal is zero-copy sharing achieved through shared metadata and > authorization. > > On Polaris, this would mean: > a. The consumer is authenticated as usual. > b. The Polaris server provides the consumer with direct access to the > data files (e.g., Parquet, as specified in the metadata.json). > c. The consumer reads the data files directly using their preferred > tool (e.g., a query engine). > > While Polaris already has most of the necessary underlying > capabilities, we need clear branding/packaging around this use case. > Specifically, this means: > a. A share is a dedicated catalog in the Polaris server, or a catalog > role scoped to specific namespaces. > b. A recipient is represented by a principal and a principal role. > c. A consumer profile consists of the catalog URI and the OAuth2 > client_id/client_secret. > d. We use credential vending (temporary STS credentials) for consumer > access to the data files. > e. We use the Polaris IRC layer to expose the metadata and the file > locations/access. > f. We can also support Polaris-to-Polaris sharing via Federation. > > Concretely: > > - On the provider side, we use the standard Polaris RBAC model > (assigning grants to catalog roles, and assigning those catalog roles > to principal roles): > - Create a dedicated principal with its own client_id/client_secret. > - Create a catalog role with TABLE_READ_DATA, TABLE_LIST, and > NAMESPACE_LIST grants on the namespaces to be shared (no write > grants). > - Create a principal role per destination, attach the catalog > role, and assign the principal. > - The destination engine uses loadTable with the > "X-Iceberg-Access-Delegation: vended-credentials" header. Polaris then > emits temporary, read-only storage credentials, allowing the engine to > read the data files directly from the owner's bucket without accessing > the rest of the bucket. This achieves zero-copy sharing. > - On the recipient/consumer side, no special configuration is > needed; any IRC client can use it. > > 2. What is data sharing not? > I previously considered adding an Arrow Flight endpoint to > Polaris, but I no longer think it is a good idea for the initial > phase. An Arrow Flight endpoint would serve the data directly, which > would require Polaris to read the data and expose it. I am against > embedding a query engine in Polaris, as this tight coupling feels like > an anti-pattern. While we could use a delegation service to serve the > Flight endpoint, I think it should be considered for a later phase, if > at all, since a Flight endpoint might be better suited as a consumer > responsibility and could encourage data copying. > > The first phase should focus on packaging and branding the > capabilities we already have. > > 3. What could the Data Sharing "package" look like in Polaris? > To make Data Sharing obvious and easy to use, we should introduce > the missing concept of a "share"—a named definition that users can > list, discover, and define. The goal is to materialize the user's > intent ("I want to share these tables with this consumer until this > date") into standard Polaris entities (principal, principal role, > catalog role, and grants). > > Therefore, I propose adding a /shares endpoint to the Polaris > Management API to package and materialize these share definitions into > Polaris entities. This would provide a usable, first-version package > for Polaris Data Sharing. > > I hope this addresses Dmitri's questions and provides a more concrete > scenario for Polaris Data Sharing. > > Thoughts? > > Regards, > JB > > On Thu, May 28, 2026 at 7:46 AM Dennis Huo <[email protected]> wrote: > > > > Hi All, > > > > One big emerging enterprise use case coming up as more people consolidate > > Data Lakehouses and Catalogs is something commonly known as "Data > Sharing", > > an more specifically over the course of adoption of open table formats > > "Open Data Sharing". > > > > Examples of existing managed service providers' Data Sharing features: > > > > https://www.databricks.com/product/delta-sharing > > https://docs.snowflake.com/en/user-guide/data-sharing-intro > > https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html > > https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction > > > https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview > > > > The basic idea is that when you share data between different companies, > you > > need a first-class governance/management layer and extra > bells-and-whistles > > that are distinct from just the basic capabilities of RBAC or generalized > > access-control (i.e. if you're sharing across partially-untrusted org > > boundaries, you don't just let the consumer organization log into your > > datalake like one of your own employees). > > > > JB and I put together this high-level proposal for supporting Open > Sharing > > in Polaris: > > > > > https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing > > > > Tentatively, it means adding ~5 logical data model constructs, some of > > which may be a first-class PolarisEntity type, others subtypes of > existing > > entities, and others just a nested construct: > > > > - ShareEntity (would behave similarly to a Catalog) > > - ExternalConsumer (mostly inherits from Principal) > > - Listing (Similar to a "role grant" but has different metadata) > > - EndpointConfig (nested config under Listing) > > - ShareMembership (Similar to a "securable grant" but different > metadata) > > > > Feedback/comments welcome! I'll also bring it up for live discussion if > > there's time in the community sync. > > > > Cheers, > > Dennis >
