Thanks for bringing this up, Andrei! Great discussion, everyone! Many good questions were asked in the thread.
Since tags are still in progress, I'd prefer to get that work in first and then come back to labels. That should give us a better understanding of how the two fit together, esp. with federation, where values can conflict or their meaning can change across catalogs. > One limitation: the labels wire has no provenance Andrei makes a good point about labels lacking provenance. I wouldn't be comfortable using an attribute for governance decisions without knowing where it came from and who is asserting it. That's another reason to get the tag model in place first. In general, projecting labels from tags makes sense to me. The tag model should capture provenance and ownership, along with permissions for reading and changing tags. We'd still need to work out what losing that provenance in the label projection means for engines, given that IRC labels are string-to-string key-value pairs. Yufei On Sun, Oct 18, 2026 at 3:16 PM Andrei Tserakhau via dev <[email protected]> wrote: > > Hi JB, > > Thanks, agree on all three. > > 1/ Federated visibility. Fair point. The federation principal may see more > than an individual Polaris user should. I'll call that out in the design > and add a per-catalog option to suppress federated labels when needed. > 2/ Collisions. Agreed on strict by default. Since the engine has no > provenance on the wire, failing is safer. Override can stay as an explicit > opt-in. > 3/ Cache invalidation. Ancestor tag writes are the tricky case. Rather than > fan out invalidations to every table, I'd start with a catalog-level tag > revision: any tag write bumps it, and cached entries are recomputed lazily. > Tag writes are rare enough that this should be fine initially. > > On schema evolution, field IDs mostly solve it. The IRC spec isn't > opinionated here: field-id refers to the table's schema, not only the > current one, so a catalog may either emit or omit labels for dropped > columns. For Polaris I'd omit them, which matches the Tag design hiding > assignments on removed field IDs. Renames keep their labels, and re-adding > the same name gets a new field ID so old labels don't carry over. Nested > fields stay out of scope for v1, same as Tags. > > Thanks, > Andrei > > On Wed, Oct 14, 2026 at 5:13 PM Jean-Baptiste Onofré <[email protected]> > wrote: > > > Hi all, > > > > I re-iterate what I said earlier: my preference is clearly on building > > labels as read-only projection over tags. > > > > I agree with the proposals on permissions and gating. Reusing the tag > > visibility rules, with no label specific privilege, means table read access > > alone exposes nothing new/extra. Gating behind a per-catalog flag (default > > off for now) sounds good to me. > > > > I have three comments: > > - About federated label visibility, I understand the argument that the > > source catalog already filtered what it exposes to the federation > > principal. In practice, that principal is often a single service identity. > > Every Polaris user who can read the federated table would then see > > everything it can see, which could be more than they should. I would love > > the design to say this explicitly as a deployment consideration. Maybe we > > should suppress pass through of federated labels with a per-catalog option. > > - I share Sung's point about collision. With no "provenance," an engine > > can't tell that a value was overridden. I think it would be better to have > > strict as the default, in addition of a warning. > > - I'm happy to see that Andrei is considering caching. Caching effective > > tags per table and applying the caller-specific filter afterwards is > > reasonable. The "challenge" is how the invalidation should work. A tag set > > on a namespace or catalog affects every table below it. So, we have to > > careful design the cache eviction. > > > > I have one question for you guys: how should per-field labels behave when > > a schema evolves? For example, when a column is dropped. Field-id key > > helps, but the expected behavior should be "clear". > > > > Regards > > JB > > > > On Tue, Oct 13, 2026 at 6:23 PM Andrei Tserakhau via dev < > > [email protected]> wrote: > > > >> Hi Sung, > >> > >> Thanks, both good points. > >> > >> On federated labels, I’d keep them visible to anyone who can read the > >> federated table. The source catalog already decided what to expose to the > >> federation principal, and Polaris doesn’t really have enough context to > >> re-apply its own tag permissions on top. So the boundary is the federation > >> connection plus normal table access in Polaris. > >> > >> On collisions, I agree silent override is the bad case. I’d make this > >> configurable per catalog: > >> > >> - > >> > >> override (default): native value wins, but Polaris records the conflict > >> and shows it clearly in UI/logs. > >> - > >> > >> strict: collision is an error and loadTable fails. > >> > >> I’d also catch this when the native tag is created. If the same key > >> already > >> comes from the source, warn there instead of waiting until read time. > >> > >> One limitation: the labels wire has no provenance, so the engine can’t > >> tell > >> that a value was overridden. That signal has to stay on the Polaris side, > >> or strict mode has to reject the load. > >> > >> Thanks, > >> Andrei > >> > >> On Tue, Oct 13, 2026 at 5:59 PM Sung Yun <[email protected]> wrote: > >> > >> > On permissions: I agree that reducing the number of privileges makes > >> > sense. I'd also like us to think about how this applies to federated > >> labels. > >> > Here are the label sources we've talked about so far: > >> > - Tags (internal to Polaris) > >> > - Federated labels > >> > - Possibly others > >> > > >> > If tag permissions decide which labels show up in loadTable/loadView, > >> what > >> > happens to federated labels? Should they always be visible, or should > >> > Polaris have a permission that controls whether they're shown? > >> > > >> > On collisions: I'm concerned that a simple "native wins" approach could > >> > make the engine behave differently without anyone noticing. > >> > > >> > How much this matters depends on what the labels are used for. Take data > >> > governance: if the native label silently wins, the policy applied to a > >> > table could differ depending on whether it's accessed through the source > >> > catalog or the federated catalog. The engine would then pick up that > >> > difference with no signal that anything changed. > >> > To prevent this, I'd suggest we consider a louder failure mode, or at > >> > least logging when a collision happens. > >> > > >> > Cheers, > >> > Sung > >> > > >> > On 2026/10/12 10:48:49 Andrei Tserakhau via dev wrote: > >> > > Hi JB, > >> > > > >> > > Thanks, good to keep the Glasgow conversation going. Glad we're > >> aligned > >> > on > >> > > opt B. > >> > > > >> > > On your questions: > >> > > > >> > > 1/ Permissions / leakage: I'd reuse the existing tag visibility rules > >> > > rather than add a label-specific privilege. On `loadTable` / > >> `loadView`, > >> > > labels would only include tags the caller is already allowed to see > >> > through > >> > > the Tag API. Table access alone would not expose anything extra. > >> > > > >> > > 2/ Performance: Agreed, `loadTable` is too hot to resolve inheritance > >> on > >> > > every call. I'd cache the effective tags per table, invalidate on tag > >> > > changes, then apply the caller-specific visibility filter on top. That > >> > > should keep the added cost small. > >> > > > >> > > 3/ Default vs gated: I'd gate it initially with something like > >> > > `polaris.label.enabled` per catalog, default off. We can move toward > >> > > on-by-default later once the behavior and cost are proven. > >> > > > >> > > Best, > >> > > Andrei > >> > > > >> > > On Mon, Oct 12, 2026 at 9:43 AM Jean-Baptiste Onofré <[email protected] > >> > > >> > > wrote: > >> > > > >> > > > Hi Andrei > >> > > > > >> > > > Thanks for starting this thread after our discussion during > >> Lakehouse > >> > > > Day/CommunityOverCode in Glasgow. > >> > > > > >> > > > I'm +1 to option B (labels as a read-only projection over tags). > >> Tags > >> > > > already have definitions, allowed values, inheritance and reverse > >> > lookup, > >> > > > and both models are keyed by Iceberg field-id. A separate labels > >> entity > >> > > > would give us two source of truth to reconcile, with no benefit > >> imho. > >> > > > > >> > > > The tag work isn't merged yet, so it's cheap (good timing :)) to > >> > reserve > >> > > > what the consumer role needs. Postponing this later would be much > >> more > >> > > > painful. I would keep the actual federation impl out of the first > >> tag > >> > PR. > >> > > > > >> > > > About collisions, "native wins" is reasonable. Polaris is the > >> asserting > >> > > > catalog at the point of access, so I'm fine with that rule. > >> > > > > >> > > > I have a few questions: > >> > > > - We have to be careful that tag-definition visibility (permission) > >> > > > doesn't leak through labels. Do we plan to see labels on > >> > loadTable/loadView > >> > > > relying on the table read privileges? Or do we want specific > >> > privileges for > >> > > > label/tag? > >> > > > - Resolving tags with inheritence on every loadTable can be > >> significant > >> > > > for performence. What are your thoughts about that? Opt caching? > >> > > > - Do we want to have label emitted by default, or do you want to > >> guate > >> > by > >> > > > catalog configuration (polaris.label.enabled=on)? > >> > > > > >> > > > Thanks! > >> > > > Regards > >> > > > JB > >> > > > > >> > > > On Sun, Oct 11, 2026 at 11:26 AM Andrei Tserakhau via dev < > >> > > > [email protected]> wrote: > >> > > > > >> > > >> Hi all, > >> > > >> > >> > > >> Now that the IRC "labels" field has merged in Iceberg Spec ([0]) > >> and > >> > the > >> > > >> Polaris Tag work is close ([1]), I'd like to open a thread on how > >> > Polaris > >> > > >> should support labels. > >> > > >> > >> > > >> There are two ways Polaris could do labels: > >> > > >> > >> > > >> A) Labels as a separate entity - their own definitions/store, > >> > parallel to > >> > > >> Tags. > >> > > >> B) Labels as a projection over Tags - read-only IRC view / > >> projection > >> > of > >> > > >> the existing tag model. > >> > > >> > >> > > >> I propose B. Tags already are the structured model: definitions, > >> > > >> assignments, allowed values, inheritance, reverse lookup. IRC > >> labels > >> > are > >> > > >> designed as the flat read surface of such models (object-level > >> > key/values > >> > > >> plus per-field labels by field-id). A separate labels entity would > >> > > >> duplicate the tag model and create two sources of truth to > >> reconcile; > >> > a > >> > > >> projection keeps one model with two surfaces, a structured > >> authoring > >> > API > >> > > >> and an interoperable flat read field. Column identity already lines > >> > up: > >> > > >> Tags use the Iceberg field-id, and IRC labels fields are keyed by > >> > > >> field-id. > >> > > >> > >> > > >> Given B, a Polaris catalog plays two roles: > >> > > >> > >> > > >> 1. Producer. Polaris resolves a target's effective tags > >> (inheritance > >> > > >> applied, closest-wins) and projects them into IRC labels on > >> > > >> loadTable/loadView. The Tag API stays the authoring/management > >> > surface and > >> > > >> labels is the read projection for IRC compliant engines. > >> > > >> > >> > > >> 2. Consumer. When Polaris federates a foreign IRC catalog, it reads > >> > that > >> > > >> catalog's labels and stores them as tags owned by the source > >> catalog: > >> > > >> read-only in Polaris and clearly marked as foreign. A Polaris user > >> > cannot > >> > > >> edit them (they change only when Polaris re-syncs from the source) > >> and > >> > > >> they > >> > > >> stay separate from Polaris's own natively-authored tags. That is > >> how a > >> > > >> foreign catalog's metadata becomes usable in Polaris without > >> > pretending it > >> > > >> is local. > >> > > >> > >> > > >> Mostly I'd like the community read on whether projection-over-tags > >> > feels > >> > > >> right, and whether the producer/consumer split is worth keeping in > >> > mind in > >> > > >> the Tag design so the contract leaves room for it. Would like to > >> hear > >> > if > >> > > >> this framing is wrong, or where a separate entity would actually > >> serve > >> > > >> better. > >> > > >> > >> > > >> Thanks, > >> > > >> Andrei > >> > > >> > >> > > >> [0] IRC Labels spec PR - > >> https://github.com/apache/iceberg/pull/15750 > >> > > >> [1] Tags discussion thread - > >> > > >> https://lists.apache.org/thread/28h8bvg3pwwmpob2p4gcrpfs1x5cgkmn > >> > > >> > >> > > > > >> > > > >> > > >> > >
