Hi all, Thanks for the thorough discussion. I'm on board with labels as a read-only projection over Tags, and with landing Tags first.
On adding origin to the Tag contract before merge, I think a few assumptions in the thread are implicit, and they lead to different designs. I'd like to make them explicit and walk through this from first principles (duplicated from my replies here: https://github.com/apache/polaris/pull/5366#issuecomment-6529593730): 0. What is the federation model for tags? The thread seems to assume foreign tags are read-only mirrors, synced one-way from the source (Andrei: "A Polaris user cannot edit them (they change only when Polaris re-syncs from the source)"). Is that agreed, or could federated tags ever be writable, or written through to the source catalog? This changes what origin means: in a write-through model it drives where writes go, not just provenance on reads. 1. Given that model, does any client need origin at all? Under read-only one-way sync, read-only enforcement, re-sync, authorization of imported tags, and strict collision handling all look like things Polaris can do server-side. JB noted provenance "stays available through the Tag API for anyone who needs it". Could we name who that is, e.g. a governance/masking engine, a UI, or a sync tool, and what it decides based on origin? If there is no such consumer, origin can stay server-internal. 2. Even if one does, a generic native-vs-external flag isn't enough. The thread consistently says foreign tags are "attributed to the source catalog", and Yufei's point about "who is asserting it" needs a concrete source, especially with Sung's "possibly others" as additional sources. So origin would have to identify the specific source, e.g. {type, source} where source references a Polaris entity (federated catalog/connection id) rather than a mutable name. But if foreign tags only exist inside their own federated catalog, isn't the containing catalog already that answer? 3. Even if origin names the source, is same-name native + foreign coexistence on one table a real requirement? Andrei mentioned allowing it, but with strict collisions that state would fail the load anyway. Namespacing foreign definitions per source would avoid changing name uniqueness and assignment identity. Unless we settle these before merge, I'd propose #5366 reserve only a read-only, server-set origin {type, source}, with type always NATIVE and source unset in v1, and defer identity/uniqueness changes to the federation design. Thanks, EJ On Thu, Oct 22, 2026 at 11:29 AM Andrei Tserakhau via dev < [email protected]> wrote: > Hi Dmitri, > > Agreed, that makes sense. Tag assignments are keyed by the target's entity > ID (plus field ID for columns), so without mirrored table entities there's > nothing for imported tags to attach to. Gating federated tag import on > ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS looks like a reasonable > approach > > When the flag is off, Polaris wouldn't import federated labels as tags. > I'll capture this in the design next to the per-catalog suppress option. > > Thanks, > Andrei > > On Thu, Oct 22, 2026 at 7:33 PM Dmitri Bourlatchkov < > [email protected]> wrote: > > > Hi Andrei, > > > > Thanks for the clarification about label import. I must have missed that > > we're going to create mirror Polaris entities for them. > > > > I think this approach will work fine. However, I think it still requires > > the ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS flag to be set. > > Otherwise, Polaris will not create mirror entities for tables themselves. > > > > We may want to gate federated tag imports on that flag too. WDYT? > > > > Thanks, > > Dmitri. > > > > On Thu, 22 Oct 2026 at 02:56, Andrei Tserakhau < > > [email protected]> wrote: > > > >> Hi Dmitri, > >> > >> Thanks, that's a good point. I agree there's a skew if local tags are > >> filtered per user while federated labels are forwarded based on the > service > >> identity. > >> > >> I'd see identity propagation as nice to have rather than required. It > >> would remove the skew, but labels don't need to depend on it. > >> > >> The consumer role is close to the mirror approach used for tables. > >> Foreign labels are imported as Polaris tags, attributed to the source > and > >> read-only. Once they're Polaris entities, the Polaris authorizer decides > >> who sees them. What we can't reproduce is the source's own per-user > rules, > >> because those don't travel over the wire. So in practice: > >> > >> - the federation service identity defines the maximum set Polaris can > see; > >> - Polaris permissions on the imported tags narrow it per user; > >> - the per-catalog option to suppress federated labels covers deployments > >> where the service identity is too broad. > >> > >> That's the same trade-off federated tables have today, and I'll call it > >> out explicitly as a deployment consideration in the design. If identity > >> propagation lands in federation later, labels would just benefit from > it. > >> > >> Thanks, > >> Andrei > >> > >> On Thu, Oct 22, 2026 at 1:02 AM Dmitri Bourlatchkov <[email protected]> > >> wrote: > >> > >>> Hi All, > >>> > >>> On permissions and federated catalogs: If we assume that tag/label > >>> visibility is a product of authorization rules, then it will be > critical to > >>> propagate the client's identity to Federated catalogs, I think. > >>> > >>> Note that this is not how Federation currently works. > >>> > >>> If we filter local tags by local Authorizer decisions, but > >>> forward Federated labels based on the service identity, it will create > an > >>> authorization skew. > >>> > >>> A similar skew exists with vended credentials for federated tables. > >>> However, for tables the recommendation currently is to create mirror > >>> entities in Polaris [5528][5548], which subsequently allows > Polaris-side > >>> Authorization deciusions. (Note that the referenced Federation RBAC doc > >>> changes have not been released yet.) > >>> > >>> However, this does not seem to be possible with labels (due to limited > >>> provenance information, as discussed previously). > >>> > >>> Cheers, > >>> Dmitri. > >>> > >>> [5528] https://github.com/apache/polaris/issues/5528 > >>> > >>> [5548] https://github.com/apache/polaris/pull/5548 > >>> > >>> On Mon, Oct 12, 2026 at 6:49 AM Andrei Tserakhau via dev < > >>> [email protected]> wrote: > >>> > >>>> Hi JB, > >>>> > >>>> Thanks, good to keep the Glasgow conversation going. Glad we're > aligned > >>>> on > >>>> opt B. > >>>> > >>>> On your questions: > >>>> > >>>> 1/ Permissions / leakage: I'd reuse the existing tag visibility rules > >>>> rather than add a label-specific privilege. On `loadTable` / > `loadView`, > >>>> labels would only include tags the caller is already allowed to see > >>>> through > >>>> the Tag API. Table access alone would not expose anything extra. > >>>> > >>>> 2/ Performance: Agreed, `loadTable` is too hot to resolve inheritance > on > >>>> every call. I'd cache the effective tags per table, invalidate on tag > >>>> changes, then apply the caller-specific visibility filter on top. That > >>>> should keep the added cost small. > >>>> > >>>> 3/ Default vs gated: I'd gate it initially with something like > >>>> `polaris.label.enabled` per catalog, default off. We can move toward > >>>> on-by-default later once the behavior and cost are proven. > >>>> > >>>> Best, > >>>> Andrei > >>>> > >>>> On Mon, Oct 12, 2026 at 9:43 AM Jean-Baptiste Onofré <[email protected] > > > >>>> wrote: > >>>> > >>>> > Hi Andrei > >>>> > > >>>> > Thanks for starting this thread after our discussion during > Lakehouse > >>>> > Day/CommunityOverCode in Glasgow. > >>>> > > >>>> > I'm +1 to option B (labels as a read-only projection over tags). > Tags > >>>> > already have definitions, allowed values, inheritance and reverse > >>>> lookup, > >>>> > and both models are keyed by Iceberg field-id. A separate labels > >>>> entity > >>>> > would give us two source of truth to reconcile, with no benefit > imho. > >>>> > > >>>> > The tag work isn't merged yet, so it's cheap (good timing :)) to > >>>> reserve > >>>> > what the consumer role needs. Postponing this later would be much > more > >>>> > painful. I would keep the actual federation impl out of the first > tag > >>>> PR. > >>>> > > >>>> > About collisions, "native wins" is reasonable. Polaris is the > >>>> asserting > >>>> > catalog at the point of access, so I'm fine with that rule. > >>>> > > >>>> > I have a few questions: > >>>> > - We have to be careful that tag-definition visibility (permission) > >>>> > doesn't leak through labels. Do we plan to see labels on > >>>> loadTable/loadView > >>>> > relying on the table read privileges? Or do we want specific > >>>> privileges for > >>>> > label/tag? > >>>> > - Resolving tags with inheritence on every loadTable can be > >>>> significant > >>>> > for performence. What are your thoughts about that? Opt caching? > >>>> > - Do we want to have label emitted by default, or do you want to > >>>> guate by > >>>> > catalog configuration (polaris.label.enabled=on)? > >>>> > > >>>> > Thanks! > >>>> > Regards > >>>> > JB > >>>> > > >>>> > On Sun, Oct 11, 2026 at 11:26 AM Andrei Tserakhau via dev < > >>>> > [email protected]> wrote: > >>>> > > >>>> >> Hi all, > >>>> >> > >>>> >> Now that the IRC "labels" field has merged in Iceberg Spec ([0]) > and > >>>> the > >>>> >> Polaris Tag work is close ([1]), I'd like to open a thread on how > >>>> Polaris > >>>> >> should support labels. > >>>> >> > >>>> >> There are two ways Polaris could do labels: > >>>> >> > >>>> >> A) Labels as a separate entity - their own definitions/store, > >>>> parallel to > >>>> >> Tags. > >>>> >> B) Labels as a projection over Tags - read-only IRC view / > >>>> projection of > >>>> >> the existing tag model. > >>>> >> > >>>> >> I propose B. Tags already are the structured model: definitions, > >>>> >> assignments, allowed values, inheritance, reverse lookup. IRC > labels > >>>> are > >>>> >> designed as the flat read surface of such models (object-level > >>>> key/values > >>>> >> plus per-field labels by field-id). A separate labels entity would > >>>> >> duplicate the tag model and create two sources of truth to > >>>> reconcile; a > >>>> >> projection keeps one model with two surfaces, a structured > >>>> authoring API > >>>> >> and an interoperable flat read field. Column identity already lines > >>>> up: > >>>> >> Tags use the Iceberg field-id, and IRC labels fields are keyed by > >>>> >> field-id. > >>>> >> > >>>> >> Given B, a Polaris catalog plays two roles: > >>>> >> > >>>> >> 1. Producer. Polaris resolves a target's effective tags > (inheritance > >>>> >> applied, closest-wins) and projects them into IRC labels on > >>>> >> loadTable/loadView. The Tag API stays the authoring/management > >>>> surface and > >>>> >> labels is the read projection for IRC compliant engines. > >>>> >> > >>>> >> 2. Consumer. When Polaris federates a foreign IRC catalog, it reads > >>>> that > >>>> >> catalog's labels and stores them as tags owned by the source > catalog: > >>>> >> read-only in Polaris and clearly marked as foreign. A Polaris user > >>>> cannot > >>>> >> edit them (they change only when Polaris re-syncs from the source) > >>>> and > >>>> >> they > >>>> >> stay separate from Polaris's own natively-authored tags. That is > how > >>>> a > >>>> >> foreign catalog's metadata becomes usable in Polaris without > >>>> pretending it > >>>> >> is local. > >>>> >> > >>>> >> Mostly I'd like the community read on whether projection-over-tags > >>>> feels > >>>> >> right, and whether the producer/consumer split is worth keeping in > >>>> mind in > >>>> >> the Tag design so the contract leaves room for it. Would like to > >>>> hear if > >>>> >> this framing is wrong, or where a separate entity would actually > >>>> serve > >>>> >> better. > >>>> >> > >>>> >> Thanks, > >>>> >> Andrei > >>>> >> > >>>> >> [0] IRC Labels spec PR - > >>>> https://github.com/apache/iceberg/pull/15750 > >>>> >> [1] Tags discussion thread - > >>>> >> https://lists.apache.org/thread/28h8bvg3pwwmpob2p4gcrpfs1x5cgkmn > >>>> >> > >>>> > > >>>> > >>> > >>> > >>> -- > >>> Dmitri Bourlatchkov > >>> Senior Staff Software Engineer, Dremio > >>> Dremio.com > >>> < > https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature> > / > >>> Follow Us on LinkedIn <https://www.linkedin.com/company/dremio> / Get > >>> Started <https://www.dremio.com/get-started/> > >>> > >>> > >>> The Agentic Lakehouse > >>> The only lakehouse built for agents, managed by agents > >>> > >>> > >>> >
