Hi Yufei It was my call also in this thread: let's land the Tag work first and build Labels on top of that. Labels will be gated per catalog (off by default) anyway, so there's no rush on the implementation side.
However, I would like to make sure the Tag contract leaves room for what labels need before is merged. As said earlier in this thread, it's cheap to do now and much more painful once the API is released. Provenance is a good example. In the current spec, ObjectTag tells us where a tag is applied (apply-method and assigned-at), but not who asserts it: there's no origin and no owner. Andrei's consumer role (foreign, read-only tags attributed to the source catalog) needs this. It also covers your point about values whose meaning changes across catalogs: we keep them attributed to their source instead of merging them. I suggest adding an origin on tag assignments in #5366. @Andrei, can you comment on the PR with what the consumer role needs? About losing provenance in the label projection, I think it's acceptable, as long as we are explicit about it. From engine point of view, Polaris is the asserting catalog at the point of access (that 's IRC "spirit"). Provenance stays available through the Tag API for anyone who needs it. Strict collision by default and the per-catalog option to remove federated labels already cover the risky case. The design should clearly state what an engine can and can't infer from labels. @Andrei, can you please update the design with the points we agreed in this thread? Thanks! Regards JB On Wed, Oct 21, 2026 at 1:30 AM Yufei Gu <[email protected]> wrote: > Thanks for bringing this up, Andrei! Great discussion, everyone! Many > good questions were asked in the thread. > > Since tags are still in progress, I'd prefer to get that work in first > and then come back to labels. That should give us a better > understanding of how the two fit together, esp. with federation, where > values can conflict or their meaning can change across catalogs. > > > One limitation: the labels wire has no provenance > > Andrei makes a good point about labels lacking provenance. I wouldn't > be comfortable using an attribute for governance decisions without > knowing where it came from and who is asserting it. That's another > reason to get the tag model in place first. > > In general, projecting labels from tags makes sense to me. The tag > model should capture provenance and ownership, along with permissions > for reading and changing tags. We'd still need to work out what losing > that provenance in the label projection means for engines, given that > IRC labels are string-to-string key-value pairs. > > Yufei > > On Sun, Oct 18, 2026 at 3:16 PM Andrei Tserakhau via dev > <[email protected]> wrote: > > > > Hi JB, > > > > Thanks, agree on all three. > > > > 1/ Federated visibility. Fair point. The federation principal may see > more > > than an individual Polaris user should. I'll call that out in the design > > and add a per-catalog option to suppress federated labels when needed. > > 2/ Collisions. Agreed on strict by default. Since the engine has no > > provenance on the wire, failing is safer. Override can stay as an > explicit > > opt-in. > > 3/ Cache invalidation. Ancestor tag writes are the tricky case. Rather > than > > fan out invalidations to every table, I'd start with a catalog-level tag > > revision: any tag write bumps it, and cached entries are recomputed > lazily. > > Tag writes are rare enough that this should be fine initially. > > > > On schema evolution, field IDs mostly solve it. The IRC spec isn't > > opinionated here: field-id refers to the table's schema, not only the > > current one, so a catalog may either emit or omit labels for dropped > > columns. For Polaris I'd omit them, which matches the Tag design hiding > > assignments on removed field IDs. Renames keep their labels, and > re-adding > > the same name gets a new field ID so old labels don't carry over. Nested > > fields stay out of scope for v1, same as Tags. > > > > Thanks, > > Andrei > > > > On Wed, Oct 14, 2026 at 5:13 PM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi all, > > > > > > I re-iterate what I said earlier: my preference is clearly on building > > > labels as read-only projection over tags. > > > > > > I agree with the proposals on permissions and gating. Reusing the tag > > > visibility rules, with no label specific privilege, means table read > access > > > alone exposes nothing new/extra. Gating behind a per-catalog flag > (default > > > off for now) sounds good to me. > > > > > > I have three comments: > > > - About federated label visibility, I understand the argument that the > > > source catalog already filtered what it exposes to the federation > > > principal. In practice, that principal is often a single service > identity. > > > Every Polaris user who can read the federated table would then see > > > everything it can see, which could be more than they should. I would > love > > > the design to say this explicitly as a deployment consideration. Maybe > we > > > should suppress pass through of federated labels with a per-catalog > option. > > > - I share Sung's point about collision. With no "provenance," an engine > > > can't tell that a value was overridden. I think it would be better to > have > > > strict as the default, in addition of a warning. > > > - I'm happy to see that Andrei is considering caching. Caching > effective > > > tags per table and applying the caller-specific filter afterwards is > > > reasonable. The "challenge" is how the invalidation should work. A tag > set > > > on a namespace or catalog affects every table below it. So, we have to > > > careful design the cache eviction. > > > > > > I have one question for you guys: how should per-field labels behave > when > > > a schema evolves? For example, when a column is dropped. Field-id key > > > helps, but the expected behavior should be "clear". > > > > > > Regards > > > JB > > > > > > On Tue, Oct 13, 2026 at 6:23 PM Andrei Tserakhau via dev < > > > [email protected]> wrote: > > > > > >> Hi Sung, > > >> > > >> Thanks, both good points. > > >> > > >> On federated labels, I’d keep them visible to anyone who can read the > > >> federated table. The source catalog already decided what to expose to > the > > >> federation principal, and Polaris doesn’t really have enough context > to > > >> re-apply its own tag permissions on top. So the boundary is the > federation > > >> connection plus normal table access in Polaris. > > >> > > >> On collisions, I agree silent override is the bad case. I’d make this > > >> configurable per catalog: > > >> > > >> - > > >> > > >> override (default): native value wins, but Polaris records the > conflict > > >> and shows it clearly in UI/logs. > > >> - > > >> > > >> strict: collision is an error and loadTable fails. > > >> > > >> I’d also catch this when the native tag is created. If the same key > > >> already > > >> comes from the source, warn there instead of waiting until read time. > > >> > > >> One limitation: the labels wire has no provenance, so the engine can’t > > >> tell > > >> that a value was overridden. That signal has to stay on the Polaris > side, > > >> or strict mode has to reject the load. > > >> > > >> Thanks, > > >> Andrei > > >> > > >> On Tue, Oct 13, 2026 at 5:59 PM Sung Yun <[email protected]> wrote: > > >> > > >> > On permissions: I agree that reducing the number of privileges makes > > >> > sense. I'd also like us to think about how this applies to federated > > >> labels. > > >> > Here are the label sources we've talked about so far: > > >> > - Tags (internal to Polaris) > > >> > - Federated labels > > >> > - Possibly others > > >> > > > >> > If tag permissions decide which labels show up in > loadTable/loadView, > > >> what > > >> > happens to federated labels? Should they always be visible, or > should > > >> > Polaris have a permission that controls whether they're shown? > > >> > > > >> > On collisions: I'm concerned that a simple "native wins" approach > could > > >> > make the engine behave differently without anyone noticing. > > >> > > > >> > How much this matters depends on what the labels are used for. Take > data > > >> > governance: if the native label silently wins, the policy applied > to a > > >> > table could differ depending on whether it's accessed through the > source > > >> > catalog or the federated catalog. The engine would then pick up that > > >> > difference with no signal that anything changed. > > >> > To prevent this, I'd suggest we consider a louder failure mode, or > at > > >> > least logging when a collision happens. > > >> > > > >> > Cheers, > > >> > Sung > > >> > > > >> > On 2026/10/12 10:48:49 Andrei Tserakhau via dev wrote: > > >> > > Hi JB, > > >> > > > > >> > > Thanks, good to keep the Glasgow conversation going. Glad we're > > >> aligned > > >> > on > > >> > > opt B. > > >> > > > > >> > > On your questions: > > >> > > > > >> > > 1/ Permissions / leakage: I'd reuse the existing tag visibility > rules > > >> > > rather than add a label-specific privilege. On `loadTable` / > > >> `loadView`, > > >> > > labels would only include tags the caller is already allowed to > see > > >> > through > > >> > > the Tag API. Table access alone would not expose anything extra. > > >> > > > > >> > > 2/ Performance: Agreed, `loadTable` is too hot to resolve > inheritance > > >> on > > >> > > every call. I'd cache the effective tags per table, invalidate on > tag > > >> > > changes, then apply the caller-specific visibility filter on top. > That > > >> > > should keep the added cost small. > > >> > > > > >> > > 3/ Default vs gated: I'd gate it initially with something like > > >> > > `polaris.label.enabled` per catalog, default off. We can move > toward > > >> > > on-by-default later once the behavior and cost are proven. > > >> > > > > >> > > Best, > > >> > > Andrei > > >> > > > > >> > > On Mon, Oct 12, 2026 at 9:43 AM Jean-Baptiste Onofré < > [email protected] > > >> > > > >> > > wrote: > > >> > > > > >> > > > Hi Andrei > > >> > > > > > >> > > > Thanks for starting this thread after our discussion during > > >> Lakehouse > > >> > > > Day/CommunityOverCode in Glasgow. > > >> > > > > > >> > > > I'm +1 to option B (labels as a read-only projection over tags). > > >> Tags > > >> > > > already have definitions, allowed values, inheritance and > reverse > > >> > lookup, > > >> > > > and both models are keyed by Iceberg field-id. A separate labels > > >> entity > > >> > > > would give us two source of truth to reconcile, with no benefit > > >> imho. > > >> > > > > > >> > > > The tag work isn't merged yet, so it's cheap (good timing :)) to > > >> > reserve > > >> > > > what the consumer role needs. Postponing this later would be > much > > >> more > > >> > > > painful. I would keep the actual federation impl out of the > first > > >> tag > > >> > PR. > > >> > > > > > >> > > > About collisions, "native wins" is reasonable. Polaris is the > > >> asserting > > >> > > > catalog at the point of access, so I'm fine with that rule. > > >> > > > > > >> > > > I have a few questions: > > >> > > > - We have to be careful that tag-definition visibility > (permission) > > >> > > > doesn't leak through labels. Do we plan to see labels on > > >> > loadTable/loadView > > >> > > > relying on the table read privileges? Or do we want specific > > >> > privileges for > > >> > > > label/tag? > > >> > > > - Resolving tags with inheritence on every loadTable can be > > >> significant > > >> > > > for performence. What are your thoughts about that? Opt caching? > > >> > > > - Do we want to have label emitted by default, or do you want to > > >> guate > > >> > by > > >> > > > catalog configuration (polaris.label.enabled=on)? > > >> > > > > > >> > > > Thanks! > > >> > > > Regards > > >> > > > JB > > >> > > > > > >> > > > On Sun, Oct 11, 2026 at 11:26 AM Andrei Tserakhau via dev < > > >> > > > [email protected]> wrote: > > >> > > > > > >> > > >> Hi all, > > >> > > >> > > >> > > >> Now that the IRC "labels" field has merged in Iceberg Spec > ([0]) > > >> and > > >> > the > > >> > > >> Polaris Tag work is close ([1]), I'd like to open a thread on > how > > >> > Polaris > > >> > > >> should support labels. > > >> > > >> > > >> > > >> There are two ways Polaris could do labels: > > >> > > >> > > >> > > >> A) Labels as a separate entity - their own definitions/store, > > >> > parallel to > > >> > > >> Tags. > > >> > > >> B) Labels as a projection over Tags - read-only IRC view / > > >> projection > > >> > of > > >> > > >> the existing tag model. > > >> > > >> > > >> > > >> I propose B. Tags already are the structured model: > definitions, > > >> > > >> assignments, allowed values, inheritance, reverse lookup. IRC > > >> labels > > >> > are > > >> > > >> designed as the flat read surface of such models (object-level > > >> > key/values > > >> > > >> plus per-field labels by field-id). A separate labels entity > would > > >> > > >> duplicate the tag model and create two sources of truth to > > >> reconcile; > > >> > a > > >> > > >> projection keeps one model with two surfaces, a structured > > >> authoring > > >> > API > > >> > > >> and an interoperable flat read field. Column identity already > lines > > >> > up: > > >> > > >> Tags use the Iceberg field-id, and IRC labels fields are keyed > by > > >> > > >> field-id. > > >> > > >> > > >> > > >> Given B, a Polaris catalog plays two roles: > > >> > > >> > > >> > > >> 1. Producer. Polaris resolves a target's effective tags > > >> (inheritance > > >> > > >> applied, closest-wins) and projects them into IRC labels on > > >> > > >> loadTable/loadView. The Tag API stays the authoring/management > > >> > surface and > > >> > > >> labels is the read projection for IRC compliant engines. > > >> > > >> > > >> > > >> 2. Consumer. When Polaris federates a foreign IRC catalog, it > reads > > >> > that > > >> > > >> catalog's labels and stores them as tags owned by the source > > >> catalog: > > >> > > >> read-only in Polaris and clearly marked as foreign. A Polaris > user > > >> > cannot > > >> > > >> edit them (they change only when Polaris re-syncs from the > source) > > >> and > > >> > > >> they > > >> > > >> stay separate from Polaris's own natively-authored tags. That > is > > >> how a > > >> > > >> foreign catalog's metadata becomes usable in Polaris without > > >> > pretending it > > >> > > >> is local. > > >> > > >> > > >> > > >> Mostly I'd like the community read on whether > projection-over-tags > > >> > feels > > >> > > >> right, and whether the producer/consumer split is worth > keeping in > > >> > mind in > > >> > > >> the Tag design so the contract leaves room for it. Would like > to > > >> hear > > >> > if > > >> > > >> this framing is wrong, or where a separate entity would > actually > > >> serve > > >> > > >> better. > > >> > > >> > > >> > > >> Thanks, > > >> > > >> Andrei > > >> > > >> > > >> > > >> [0] IRC Labels spec PR - > > >> https://github.com/apache/iceberg/pull/15750 > > >> > > >> [1] Tags discussion thread - > > >> > > >> > https://lists.apache.org/thread/28h8bvg3pwwmpob2p4gcrpfs1x5cgkmn > > >> > > >> > > >> > > > > > >> > > > > >> > > > >> > > > >
