Thanks Ej, Dmitrii, +1 to Dmitrii's idea about having Tags api as Beta, merge and continue on main.
That removes my concern about locking things in, so I don't need anything extra in v1. To cover other questions: - Model: read-only, one-way sync. No write-through. - Who needs origin: owners can add native Polaris tags to a federated table, so one federated catalog can hold both native and imported tags. The containing catalog alone doesn't tell them apart. A UI or audit tool that separates the two is the consumer. - Same-name coexistence: fine to drop. Namespacing foreign definitions per source works. - Allowed values: good point. Imported definitions can't have a fixed allowed-values list, since values arrive from the source and can change on each sync. They'd need a source-managed value mode. That belongs in the federation design, we can design it's later once 5366 merged. Thanks, Andrei On Fri, Oct 30, 2026 at 7:21 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi EJ, Andrei, > > I'd like to clarify two points: > > A) From what I understand, tag definitions live inside a catalog. A > federated catalog has a Polaris Catalog entity in its own right. So, why do > we need to track tags' origin? > > Are we planning to allow Polaris users to alter tag data imported from the > federated catalog?... perhaps add new Polaris-specific tags to a Federated > catalog? > > B) From my POV the bigger question about creating tags from federated > labels is the enum value set. With native tags we know ahead of time all > possible values. However, with Federated catalogs we do not know what > values may exist until we sync. Moreover, each sync may bring new, > previously unknown values. How can this play with Polaris Tag enums > (CreateTagRequest.allowed-values in [5366])? > > WDYT? > > That said, from a practical perspective, IIRC the general practice in > Polaris is to flag new APIs as "beta" to allow follow-up changes to be made > without worrying about strict backward compatibility. > > Did we miss to label the Tags API as beta/experimental in [5366]? > > I'd propose accepting the current API shape as "beta", merging and > continuing this discussion with respect to the code on `main`. If we have > to alter the Tags API, I think it will be perfectly fine. We'll drop the > "beta" label after the feature stabilizes. Can this work? > > Thanks, > Dmitri. > > [5366] https://github.com/apache/polaris/pull/5366 > > > > > On Fri, Oct 30, 2026 at 1:51 PM EJ Wang <[email protected]> > wrote: > > > Hi all, > > > > Thanks for the thorough discussion. I'm on board with labels as a > read-only > > projection over Tags, and with landing Tags first. > > > > On adding origin to the Tag contract before merge, I think a few > > assumptions in the thread are implicit, and they lead to different > designs. > > I'd like to make them explicit and walk through this from first > principles > > (duplicated from my replies here: > > https://github.com/apache/polaris/pull/5366#issuecomment-6529593730): > > > > 0. What is the federation model for tags? The thread seems to assume > > foreign tags are read-only mirrors, synced one-way from the source > (Andrei: > > "A Polaris user cannot edit them (they change only when Polaris re-syncs > > from the source)"). Is that agreed, or could federated tags ever be > > writable, or written through to the source catalog? This changes what > > origin means: in a write-through model it drives where writes go, not > just > > provenance on reads. > > > > 1. Given that model, does any client need origin at all? Under read-only > > one-way sync, read-only enforcement, re-sync, authorization of imported > > tags, and strict collision handling all look like things Polaris can do > > server-side. JB noted provenance "stays available through the Tag API for > > anyone who needs it". Could we name who that is, e.g. a > governance/masking > > engine, a UI, or a sync tool, and what it decides based on origin? If > there > > is no such consumer, origin can stay server-internal. > > > > 2. Even if one does, a generic native-vs-external flag isn't enough. The > > thread consistently says foreign tags are "attributed to the source > > catalog", and Yufei's point about "who is asserting it" needs a concrete > > source, especially with Sung's "possibly others" as additional sources. > So > > origin would have to identify the specific source, e.g. {type, source} > > where source references a Polaris entity (federated catalog/connection > id) > > rather than a mutable name. But if foreign tags only exist inside their > own > > federated catalog, isn't the containing catalog already that answer? > > > > 3. Even if origin names the source, is same-name native + foreign > > coexistence on one table a real requirement? Andrei mentioned allowing > it, > > but with strict collisions that state would fail the load anyway. > > Namespacing foreign definitions per source would avoid changing name > > uniqueness and assignment identity. > > > > Unless we settle these before merge, I'd propose #5366 reserve only a > > read-only, server-set origin {type, source}, with type always NATIVE and > > source unset in v1, and defer identity/uniqueness changes to the > federation > > design. > > > > Thanks, > > EJ > > > > On Thu, Oct 22, 2026 at 11:29 AM Andrei Tserakhau via dev < > > [email protected]> wrote: > > > > > Hi Dmitri, > > > > > > Agreed, that makes sense. Tag assignments are keyed by the target's > > entity > > > ID (plus field ID for columns), so without mirrored table entities > > there's > > > nothing for imported tags to attach to. Gating federated tag import on > > > ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS looks like a reasonable > > > approach > > > > > > When the flag is off, Polaris wouldn't import federated labels as tags. > > > I'll capture this in the design next to the per-catalog suppress > option. > > > > > > Thanks, > > > Andrei > > > > > > On Thu, Oct 22, 2026 at 7:33 PM Dmitri Bourlatchkov < > > > [email protected]> wrote: > > > > > > > Hi Andrei, > > > > > > > > Thanks for the clarification about label import. I must have missed > > that > > > > we're going to create mirror Polaris entities for them. > > > > > > > > I think this approach will work fine. However, I think it still > > requires > > > > the ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS flag to be set. > > > > Otherwise, Polaris will not create mirror entities for tables > > themselves. > > > > > > > > We may want to gate federated tag imports on that flag too. WDYT? > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > On Thu, 22 Oct 2026 at 02:56, Andrei Tserakhau < > > > > [email protected]> wrote: > > > > > > > >> Hi Dmitri, > > > >> > > > >> Thanks, that's a good point. I agree there's a skew if local tags > are > > > >> filtered per user while federated labels are forwarded based on the > > > service > > > >> identity. > > > >> > > > >> I'd see identity propagation as nice to have rather than required. > It > > > >> would remove the skew, but labels don't need to depend on it. > > > >> > > > >> The consumer role is close to the mirror approach used for tables. > > > >> Foreign labels are imported as Polaris tags, attributed to the > source > > > and > > > >> read-only. Once they're Polaris entities, the Polaris authorizer > > decides > > > >> who sees them. What we can't reproduce is the source's own per-user > > > rules, > > > >> because those don't travel over the wire. So in practice: > > > >> > > > >> - the federation service identity defines the maximum set Polaris > can > > > see; > > > >> - Polaris permissions on the imported tags narrow it per user; > > > >> - the per-catalog option to suppress federated labels covers > > deployments > > > >> where the service identity is too broad. > > > >> > > > >> That's the same trade-off federated tables have today, and I'll call > > it > > > >> out explicitly as a deployment consideration in the design. If > > identity > > > >> propagation lands in federation later, labels would just benefit > from > > > it. > > > >> > > > >> Thanks, > > > >> Andrei > > > >> > > > >> On Thu, Oct 22, 2026 at 1:02 AM Dmitri Bourlatchkov < > [email protected] > > > > > > >> wrote: > > > >> > > > >>> Hi All, > > > >>> > > > >>> On permissions and federated catalogs: If we assume that tag/label > > > >>> visibility is a product of authorization rules, then it will be > > > critical to > > > >>> propagate the client's identity to Federated catalogs, I think. > > > >>> > > > >>> Note that this is not how Federation currently works. > > > >>> > > > >>> If we filter local tags by local Authorizer decisions, but > > > >>> forward Federated labels based on the service identity, it will > > create > > > an > > > >>> authorization skew. > > > >>> > > > >>> A similar skew exists with vended credentials for federated tables. > > > >>> However, for tables the recommendation currently is to create > mirror > > > >>> entities in Polaris [5528][5548], which subsequently allows > > > Polaris-side > > > >>> Authorization deciusions. (Note that the referenced Federation RBAC > > doc > > > >>> changes have not been released yet.) > > > >>> > > > >>> However, this does not seem to be possible with labels (due to > > limited > > > >>> provenance information, as discussed previously). > > > >>> > > > >>> Cheers, > > > >>> Dmitri. > > > >>> > > > >>> [5528] https://github.com/apache/polaris/issues/5528 > > > >>> > > > >>> [5548] https://github.com/apache/polaris/pull/5548 > > > >>> > > > >>> On Mon, Oct 12, 2026 at 6:49 AM Andrei Tserakhau via dev < > > > >>> [email protected]> wrote: > > > >>> > > > >>>> Hi JB, > > > >>>> > > > >>>> Thanks, good to keep the Glasgow conversation going. Glad we're > > > aligned > > > >>>> on > > > >>>> opt B. > > > >>>> > > > >>>> On your questions: > > > >>>> > > > >>>> 1/ Permissions / leakage: I'd reuse the existing tag visibility > > rules > > > >>>> rather than add a label-specific privilege. On `loadTable` / > > > `loadView`, > > > >>>> labels would only include tags the caller is already allowed to > see > > > >>>> through > > > >>>> the Tag API. Table access alone would not expose anything extra. > > > >>>> > > > >>>> 2/ Performance: Agreed, `loadTable` is too hot to resolve > > inheritance > > > on > > > >>>> every call. I'd cache the effective tags per table, invalidate on > > tag > > > >>>> changes, then apply the caller-specific visibility filter on top. > > That > > > >>>> should keep the added cost small. > > > >>>> > > > >>>> 3/ Default vs gated: I'd gate it initially with something like > > > >>>> `polaris.label.enabled` per catalog, default off. We can move > toward > > > >>>> on-by-default later once the behavior and cost are proven. > > > >>>> > > > >>>> Best, > > > >>>> Andrei > > > >>>> > > > >>>> On Mon, Oct 12, 2026 at 9:43 AM Jean-Baptiste Onofré < > > [email protected] > > > > > > > >>>> wrote: > > > >>>> > > > >>>> > Hi Andrei > > > >>>> > > > > >>>> > Thanks for starting this thread after our discussion during > > > Lakehouse > > > >>>> > Day/CommunityOverCode in Glasgow. > > > >>>> > > > > >>>> > I'm +1 to option B (labels as a read-only projection over tags). > > > Tags > > > >>>> > already have definitions, allowed values, inheritance and > reverse > > > >>>> lookup, > > > >>>> > and both models are keyed by Iceberg field-id. A separate labels > > > >>>> entity > > > >>>> > would give us two source of truth to reconcile, with no benefit > > > imho. > > > >>>> > > > > >>>> > The tag work isn't merged yet, so it's cheap (good timing :)) to > > > >>>> reserve > > > >>>> > what the consumer role needs. Postponing this later would be > much > > > more > > > >>>> > painful. I would keep the actual federation impl out of the > first > > > tag > > > >>>> PR. > > > >>>> > > > > >>>> > About collisions, "native wins" is reasonable. Polaris is the > > > >>>> asserting > > > >>>> > catalog at the point of access, so I'm fine with that rule. > > > >>>> > > > > >>>> > I have a few questions: > > > >>>> > - We have to be careful that tag-definition visibility > > (permission) > > > >>>> > doesn't leak through labels. Do we plan to see labels on > > > >>>> loadTable/loadView > > > >>>> > relying on the table read privileges? Or do we want specific > > > >>>> privileges for > > > >>>> > label/tag? > > > >>>> > - Resolving tags with inheritence on every loadTable can be > > > >>>> significant > > > >>>> > for performence. What are your thoughts about that? Opt caching? > > > >>>> > - Do we want to have label emitted by default, or do you want to > > > >>>> guate by > > > >>>> > catalog configuration (polaris.label.enabled=on)? > > > >>>> > > > > >>>> > Thanks! > > > >>>> > Regards > > > >>>> > JB > > > >>>> > > > > >>>> > On Sun, Oct 11, 2026 at 11:26 AM Andrei Tserakhau via dev < > > > >>>> > [email protected]> wrote: > > > >>>> > > > > >>>> >> Hi all, > > > >>>> >> > > > >>>> >> Now that the IRC "labels" field has merged in Iceberg Spec > ([0]) > > > and > > > >>>> the > > > >>>> >> Polaris Tag work is close ([1]), I'd like to open a thread on > how > > > >>>> Polaris > > > >>>> >> should support labels. > > > >>>> >> > > > >>>> >> There are two ways Polaris could do labels: > > > >>>> >> > > > >>>> >> A) Labels as a separate entity - their own definitions/store, > > > >>>> parallel to > > > >>>> >> Tags. > > > >>>> >> B) Labels as a projection over Tags - read-only IRC view / > > > >>>> projection of > > > >>>> >> the existing tag model. > > > >>>> >> > > > >>>> >> I propose B. Tags already are the structured model: > definitions, > > > >>>> >> assignments, allowed values, inheritance, reverse lookup. IRC > > > labels > > > >>>> are > > > >>>> >> designed as the flat read surface of such models (object-level > > > >>>> key/values > > > >>>> >> plus per-field labels by field-id). A separate labels entity > > would > > > >>>> >> duplicate the tag model and create two sources of truth to > > > >>>> reconcile; a > > > >>>> >> projection keeps one model with two surfaces, a structured > > > >>>> authoring API > > > >>>> >> and an interoperable flat read field. Column identity already > > lines > > > >>>> up: > > > >>>> >> Tags use the Iceberg field-id, and IRC labels fields are keyed > by > > > >>>> >> field-id. > > > >>>> >> > > > >>>> >> Given B, a Polaris catalog plays two roles: > > > >>>> >> > > > >>>> >> 1. Producer. Polaris resolves a target's effective tags > > > (inheritance > > > >>>> >> applied, closest-wins) and projects them into IRC labels on > > > >>>> >> loadTable/loadView. The Tag API stays the authoring/management > > > >>>> surface and > > > >>>> >> labels is the read projection for IRC compliant engines. > > > >>>> >> > > > >>>> >> 2. Consumer. When Polaris federates a foreign IRC catalog, it > > reads > > > >>>> that > > > >>>> >> catalog's labels and stores them as tags owned by the source > > > catalog: > > > >>>> >> read-only in Polaris and clearly marked as foreign. A Polaris > > user > > > >>>> cannot > > > >>>> >> edit them (they change only when Polaris re-syncs from the > > source) > > > >>>> and > > > >>>> >> they > > > >>>> >> stay separate from Polaris's own natively-authored tags. That > is > > > how > > > >>>> a > > > >>>> >> foreign catalog's metadata becomes usable in Polaris without > > > >>>> pretending it > > > >>>> >> is local. > > > >>>> >> > > > >>>> >> Mostly I'd like the community read on whether > > projection-over-tags > > > >>>> feels > > > >>>> >> right, and whether the producer/consumer split is worth keeping > > in > > > >>>> mind in > > > >>>> >> the Tag design so the contract leaves room for it. Would like > to > > > >>>> hear if > > > >>>> >> this framing is wrong, or where a separate entity would > actually > > > >>>> serve > > > >>>> >> better. > > > >>>> >> > > > >>>> >> Thanks, > > > >>>> >> Andrei > > > >>>> >> > > > >>>> >> [0] IRC Labels spec PR - > > > >>>> https://github.com/apache/iceberg/pull/15750 > > > >>>> >> [1] Tags discussion thread - > > > >>>> >> > https://lists.apache.org/thread/28h8bvg3pwwmpob2p4gcrpfs1x5cgkmn > > > >>>> >> > > > >>>> > > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> Dmitri Bourlatchkov > > > >>> Senior Staff Software Engineer, Dremio > > > >>> Dremio.com > > > >>> < > > > > > > https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature > > > > > > / > > > >>> Follow Us on LinkedIn <https://www.linkedin.com/company/dremio> / > > Get > > > >>> Started <https://www.dremio.com/get-started/> > > > >>> > > > >>> > > > >>> The Agentic Lakehouse > > > >>> The only lakehouse built for agents, managed by agents > > > >>> > > > >>> > > > >>> > > > > > >
