Hi all, I re-iterate what I said earlier: my preference is clearly on building labels as read-only projection over tags.
I agree with the proposals on permissions and gating. Reusing the tag visibility rules, with no label specific privilege, means table read access alone exposes nothing new/extra. Gating behind a per-catalog flag (default off for now) sounds good to me. I have three comments: - About federated label visibility, I understand the argument that the source catalog already filtered what it exposes to the federation principal. In practice, that principal is often a single service identity. Every Polaris user who can read the federated table would then see everything it can see, which could be more than they should. I would love the design to say this explicitly as a deployment consideration. Maybe we should suppress pass through of federated labels with a per-catalog option. - I share Sung's point about collision. With no "provenance," an engine can't tell that a value was overridden. I think it would be better to have strict as the default, in addition of a warning. - I'm happy to see that Andrei is considering caching. Caching effective tags per table and applying the caller-specific filter afterwards is reasonable. The "challenge" is how the invalidation should work. A tag set on a namespace or catalog affects every table below it. So, we have to careful design the cache eviction. I have one question for you guys: how should per-field labels behave when a schema evolves? For example, when a column is dropped. Field-id key helps, but the expected behavior should be "clear". Regards JB On Tue, Oct 13, 2026 at 6:23 PM Andrei Tserakhau via dev < [email protected]> wrote: > Hi Sung, > > Thanks, both good points. > > On federated labels, I’d keep them visible to anyone who can read the > federated table. The source catalog already decided what to expose to the > federation principal, and Polaris doesn’t really have enough context to > re-apply its own tag permissions on top. So the boundary is the federation > connection plus normal table access in Polaris. > > On collisions, I agree silent override is the bad case. I’d make this > configurable per catalog: > > - > > override (default): native value wins, but Polaris records the conflict > and shows it clearly in UI/logs. > - > > strict: collision is an error and loadTable fails. > > I’d also catch this when the native tag is created. If the same key already > comes from the source, warn there instead of waiting until read time. > > One limitation: the labels wire has no provenance, so the engine can’t tell > that a value was overridden. That signal has to stay on the Polaris side, > or strict mode has to reject the load. > > Thanks, > Andrei > > On Tue, Oct 13, 2026 at 5:59 PM Sung Yun <[email protected]> wrote: > > > On permissions: I agree that reducing the number of privileges makes > > sense. I'd also like us to think about how this applies to federated > labels. > > Here are the label sources we've talked about so far: > > - Tags (internal to Polaris) > > - Federated labels > > - Possibly others > > > > If tag permissions decide which labels show up in loadTable/loadView, > what > > happens to federated labels? Should they always be visible, or should > > Polaris have a permission that controls whether they're shown? > > > > On collisions: I'm concerned that a simple "native wins" approach could > > make the engine behave differently without anyone noticing. > > > > How much this matters depends on what the labels are used for. Take data > > governance: if the native label silently wins, the policy applied to a > > table could differ depending on whether it's accessed through the source > > catalog or the federated catalog. The engine would then pick up that > > difference with no signal that anything changed. > > To prevent this, I'd suggest we consider a louder failure mode, or at > > least logging when a collision happens. > > > > Cheers, > > Sung > > > > On 2026/10/12 10:48:49 Andrei Tserakhau via dev wrote: > > > Hi JB, > > > > > > Thanks, good to keep the Glasgow conversation going. Glad we're aligned > > on > > > opt B. > > > > > > On your questions: > > > > > > 1/ Permissions / leakage: I'd reuse the existing tag visibility rules > > > rather than add a label-specific privilege. On `loadTable` / > `loadView`, > > > labels would only include tags the caller is already allowed to see > > through > > > the Tag API. Table access alone would not expose anything extra. > > > > > > 2/ Performance: Agreed, `loadTable` is too hot to resolve inheritance > on > > > every call. I'd cache the effective tags per table, invalidate on tag > > > changes, then apply the caller-specific visibility filter on top. That > > > should keep the added cost small. > > > > > > 3/ Default vs gated: I'd gate it initially with something like > > > `polaris.label.enabled` per catalog, default off. We can move toward > > > on-by-default later once the behavior and cost are proven. > > > > > > Best, > > > Andrei > > > > > > On Mon, Oct 12, 2026 at 9:43 AM Jean-Baptiste Onofré <[email protected]> > > > wrote: > > > > > > > Hi Andrei > > > > > > > > Thanks for starting this thread after our discussion during Lakehouse > > > > Day/CommunityOverCode in Glasgow. > > > > > > > > I'm +1 to option B (labels as a read-only projection over tags). Tags > > > > already have definitions, allowed values, inheritance and reverse > > lookup, > > > > and both models are keyed by Iceberg field-id. A separate labels > entity > > > > would give us two source of truth to reconcile, with no benefit imho. > > > > > > > > The tag work isn't merged yet, so it's cheap (good timing :)) to > > reserve > > > > what the consumer role needs. Postponing this later would be much > more > > > > painful. I would keep the actual federation impl out of the first tag > > PR. > > > > > > > > About collisions, "native wins" is reasonable. Polaris is the > asserting > > > > catalog at the point of access, so I'm fine with that rule. > > > > > > > > I have a few questions: > > > > - We have to be careful that tag-definition visibility (permission) > > > > doesn't leak through labels. Do we plan to see labels on > > loadTable/loadView > > > > relying on the table read privileges? Or do we want specific > > privileges for > > > > label/tag? > > > > - Resolving tags with inheritence on every loadTable can be > significant > > > > for performence. What are your thoughts about that? Opt caching? > > > > - Do we want to have label emitted by default, or do you want to > guate > > by > > > > catalog configuration (polaris.label.enabled=on)? > > > > > > > > Thanks! > > > > Regards > > > > JB > > > > > > > > On Sun, Oct 11, 2026 at 11:26 AM Andrei Tserakhau via dev < > > > > [email protected]> wrote: > > > > > > > >> Hi all, > > > >> > > > >> Now that the IRC "labels" field has merged in Iceberg Spec ([0]) and > > the > > > >> Polaris Tag work is close ([1]), I'd like to open a thread on how > > Polaris > > > >> should support labels. > > > >> > > > >> There are two ways Polaris could do labels: > > > >> > > > >> A) Labels as a separate entity - their own definitions/store, > > parallel to > > > >> Tags. > > > >> B) Labels as a projection over Tags - read-only IRC view / > projection > > of > > > >> the existing tag model. > > > >> > > > >> I propose B. Tags already are the structured model: definitions, > > > >> assignments, allowed values, inheritance, reverse lookup. IRC labels > > are > > > >> designed as the flat read surface of such models (object-level > > key/values > > > >> plus per-field labels by field-id). A separate labels entity would > > > >> duplicate the tag model and create two sources of truth to > reconcile; > > a > > > >> projection keeps one model with two surfaces, a structured > authoring > > API > > > >> and an interoperable flat read field. Column identity already lines > > up: > > > >> Tags use the Iceberg field-id, and IRC labels fields are keyed by > > > >> field-id. > > > >> > > > >> Given B, a Polaris catalog plays two roles: > > > >> > > > >> 1. Producer. Polaris resolves a target's effective tags (inheritance > > > >> applied, closest-wins) and projects them into IRC labels on > > > >> loadTable/loadView. The Tag API stays the authoring/management > > surface and > > > >> labels is the read projection for IRC compliant engines. > > > >> > > > >> 2. Consumer. When Polaris federates a foreign IRC catalog, it reads > > that > > > >> catalog's labels and stores them as tags owned by the source > catalog: > > > >> read-only in Polaris and clearly marked as foreign. A Polaris user > > cannot > > > >> edit them (they change only when Polaris re-syncs from the source) > and > > > >> they > > > >> stay separate from Polaris's own natively-authored tags. That is > how a > > > >> foreign catalog's metadata becomes usable in Polaris without > > pretending it > > > >> is local. > > > >> > > > >> Mostly I'd like the community read on whether projection-over-tags > > feels > > > >> right, and whether the producer/consumer split is worth keeping in > > mind in > > > >> the Tag design so the contract leaves room for it. Would like to > hear > > if > > > >> this framing is wrong, or where a separate entity would actually > serve > > > >> better. > > > >> > > > >> Thanks, > > > >> Andrei > > > >> > > > >> [0] IRC Labels spec PR - > https://github.com/apache/iceberg/pull/15750 > > > >> [1] Tags discussion thread - > > > >> https://lists.apache.org/thread/28h8bvg3pwwmpob2p4gcrpfs1x5cgkmn > > > >> > > > > > > > > > >
