Hi everyone,
I would like to resume the discussion about Data Sharing in Polaris
and propose focusing on three major items to help us move forward.
1. What is data sharing?
In a nutshell, it is an open protocol proposal allowing
organizations to share live data without copying or replication. The
main goal is zero-copy sharing achieved through shared metadata and
authorization.
On Polaris, this would mean:
a. The consumer is authenticated as usual.
b. The Polaris server provides the consumer with direct access to the
data files (e.g., Parquet, as specified in the metadata.json).
c. The consumer reads the data files directly using their preferred
tool (e.g., a query engine).
While Polaris already has most of the necessary underlying
capabilities, we need clear branding/packaging around this use case.
Specifically, this means:
a. A share is a dedicated catalog in the Polaris server, or a catalog
role scoped to specific namespaces.
b. A recipient is represented by a principal and a principal role.
c. A consumer profile consists of the catalog URI and the OAuth2
client_id/client_secret.
d. We use credential vending (temporary STS credentials) for consumer
access to the data files.
e. We use the Polaris IRC layer to expose the metadata and the file
locations/access.
f. We can also support Polaris-to-Polaris sharing via Federation.
Concretely:
- On the provider side, we use the standard Polaris RBAC model
(assigning grants to catalog roles, and assigning those catalog roles
to principal roles):
- Create a dedicated principal with its own client_id/client_secret.
- Create a catalog role with TABLE_READ_DATA, TABLE_LIST, and
NAMESPACE_LIST grants on the namespaces to be shared (no write
grants).
- Create a principal role per destination, attach the catalog
role, and assign the principal.
- The destination engine uses loadTable with the
"X-Iceberg-Access-Delegation: vended-credentials" header. Polaris then
emits temporary, read-only storage credentials, allowing the engine to
read the data files directly from the owner's bucket without accessing
the rest of the bucket. This achieves zero-copy sharing.
- On the recipient/consumer side, no special configuration is
needed; any IRC client can use it.
2. What is data sharing not?
I previously considered adding an Arrow Flight endpoint to
Polaris, but I no longer think it is a good idea for the initial
phase. An Arrow Flight endpoint would serve the data directly, which
would require Polaris to read the data and expose it. I am against
embedding a query engine in Polaris, as this tight coupling feels like
an anti-pattern. While we could use a delegation service to serve the
Flight endpoint, I think it should be considered for a later phase, if
at all, since a Flight endpoint might be better suited as a consumer
responsibility and could encourage data copying.
The first phase should focus on packaging and branding the
capabilities we already have.
3. What could the Data Sharing "package" look like in Polaris?
To make Data Sharing obvious and easy to use, we should introduce
the missing concept of a "share"—a named definition that users can
list, discover, and define. The goal is to materialize the user's
intent ("I want to share these tables with this consumer until this
date") into standard Polaris entities (principal, principal role,
catalog role, and grants).
Therefore, I propose adding a /shares endpoint to the Polaris
Management API to package and materialize these share definitions into
Polaris entities. This would provide a usable, first-version package
for Polaris Data Sharing.
I hope this addresses Dmitri's questions and provides a more concrete
scenario for Polaris Data Sharing.
Thoughts?
Regards,
JB
On Thu, May 28, 2026 at 7:46 AM Dennis Huo <[email protected]> wrote:
>
> Hi All,
>
> One big emerging enterprise use case coming up as more people consolidate
> Data Lakehouses and Catalogs is something commonly known as "Data Sharing",
> an more specifically over the course of adoption of open table formats
> "Open Data Sharing".
>
> Examples of existing managed service providers' Data Sharing features:
>
> https://www.databricks.com/product/delta-sharing
> https://docs.snowflake.com/en/user-guide/data-sharing-intro
> https://docs.aws.amazon.com/redshift/latest/dg/datashare-overview.html
> https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction
> https://learn.microsoft.com/en-us/fabric/governance/external-data-sharing-overview
>
> The basic idea is that when you share data between different companies, you
> need a first-class governance/management layer and extra bells-and-whistles
> that are distinct from just the basic capabilities of RBAC or generalized
> access-control (i.e. if you're sharing across partially-untrusted org
> boundaries, you don't just let the consumer organization log into your
> datalake like one of your own employees).
>
> JB and I put together this high-level proposal for supporting Open Sharing
> in Polaris:
>
> https://docs.google.com/document/d/1Y0yQi5iWbmuTHPkFiIs7WjIiC3EXJTl1PzZ-wtoRnZ0/edit?usp=sharing
>
> Tentatively, it means adding ~5 logical data model constructs, some of
> which may be a first-class PolarisEntity type, others subtypes of existing
> entities, and others just a nested construct:
>
> - ShareEntity (would behave similarly to a Catalog)
> - ExternalConsumer (mostly inherits from Principal)
> - Listing (Similar to a "role grant" but has different metadata)
> - EndpointConfig (nested config under Listing)
> - ShareMembership (Similar to a "securable grant" but different metadata)
>
> Feedback/comments welcome! I'll also bring it up for live discussion if
> there's time in the community sync.
>
> Cheers,
> Dennis
