roryqi commented on code in PR #12177: URL: https://github.com/apache/gravitino/pull/12177#discussion_r3718768354
########## design-docs/policy-on-tag.md: ########## @@ -0,0 +1,669 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of Policy-on-Tag in Gravitino + +--- + +## Background + +Gravitino currently has two independent governance concepts: + +| Concept | Current state | +|---------|---------------| +| Tag | A flat metalake-scoped metadata object used to classify or annotate metadata objects. Tags can be associated with catalogs, schemas, tables, filesets, topics, models, and columns. Tag listing follows the metadata object hierarchy, so a child object can receive tags from parent metadata objects. | +| Policy | A metalake-scoped metadata object with typed content, enabled state, and audit information. The current model allows policies to be associated directly with metadata objects. The system iceberg compaction policy is the first built-in policy type and is consumed by the table maintenance service. | + +The current object-side governance model is: + +```text +Tag -> Metadata Object +Policy -> Metadata Object +``` + +This direct object policy model is understandable for a small number of objects, but it creates +problems when governance needs to scale across many catalogs, schemas, tables, and columns: + +1. Users must manage both object tags and object policies on the same metadata object. +2. Policy assignment does not naturally follow classification. A table can be marked as + `maintenance_standard`, but the maintenance policy still has to be attached separately. +3. New objects can be missed unless administrators attach policies to every new object or rely on + ancestor-level direct policy assignment. +4. TMS needs maintenance policy selection now, while future ABAC needs tag-driven policy selection. + Direct object policies do not provide a shared selection layer for both scenarios. +5. Keeping two object-side governance paths makes the user model harder to explain and document. + +The proposed target model is: + +```text +Policy -> Tag -> Metadata Object +``` + +Policy remains a first-class object. Tags become the only object-side governance attachment point. +An object policy is a read-only policy result for a metadata object, derived from the tags the +object has or inherits from parent metadata objects. Tags themselves are not nested. + +--- + +## Goals + +1. **Single Object-Side Attachment Point**: Metadata objects receive governance behavior only + through tags, not through direct policy attachment. +2. **Reusable Policy Lifecycle**: Policies remain first-class objects with typed content, enabled + state, audit information, and metalake-scoped lifecycle operations. +3. **Immutable Policy Definitions**: Policy definitions are not modified in place. To change policy + behavior, administrators create a replacement policy and update policy-to-tag associations. +4. **Policy-to-Tag Association**: Administrators can associate policies with tags and inspect which + tags carry a policy. +5. **Flat Tags**: Tags remain flat metalake-scoped objects. The design does not add parent tags, + child tags, tag groups, or tag-to-tag inheritance. +6. **Object Policy Resolution**: Gravitino can compute object policies for a metadata object from + its effective tags, including inherited tags. +7. **Read-Only Object Policies**: Object policies are derived results. Users cannot create, alter, + enable, disable, delete, or associate policies directly on a metadata object. +8. **Explicit Visibility Privileges**: Tag and policy visibility are controlled by read-only + privileges that are separate from mutation privileges. +9. **Secure Policy Enforcement**: Row filter and column mask enforcement does not depend on whether + the end user can view policy details. +10. **TMS Integration**: TMS consumes the system iceberg compaction policy through object policy + lookup, not through direct object policy relations. +11. **ABAC Evolution Path**: The resolver boundary can later support tag-expression policies without + changing every policy consumer. +12. **Explicit Breaking Migration**: Existing direct object policy relations are migrated or retired + explicitly; runtime behavior does not read direct object policy relations. + +--- + +## Non-Goals + +1. **Direct Object Policy Compatibility**: Gravitino will not preserve direct policy attachment to + metadata objects in the target model. Object-side policy behavior comes only from tags. +2. **Object Policy Mutation**: Object policies are not mutable entities. The object policy API is a + lookup API, not a create, update, delete, enable, disable, or association API. +3. **Nested Tags**: This design does not introduce tag hierarchy, tag groups, parent tags, child + tags, tag-to-tag relations, or tag-to-tag inheritance. Policy selection is based on flat tags + assigned to metadata objects. +4. **Tag-Expression ABAC in Phase 1**: The first phase does not introduce an expression language. + Policy selection is a fixed relation from policy to tag. +5. **Full Explainability UI**: A dedicated UI is not required for the first milestone. APIs must + still return enough source information to trace object policy sources. +6. **Multi-Value Tag Append Semantics**: This design does not append multiple values to the same tag + key on one object. If assignment values are used, the same tag key has one effective value. +7. **Engine-Specific Enforcement Plugins**: This design defines policy selection inside Gravitino. + Trino, Spark, Iceberg, or OPA enforcement integrations are separate designs. + +--- + +## Solution Investigations + +### Policy Assignment Model + +| Approach | Pros | Cons | Decision | +|----------|------|------|----------| +| Direct object policy | Simple to understand for one object; current implementation already exists. | Does not scale well; duplicates object-side tag and policy management; does not align classification with action. | Rejected | +| Controls embedded in tags | Simplest user model; objects only receive tags. | Tags become heavy governance objects; policy lifecycle, reuse, audit, enable/disable, and versioning are weaker; does not keep tag and policy concepts clear. | Rejected | +| Policy-on-tag | Keeps policies reusable and auditable; makes tags the only object-side attachment point; works for TMS without an expression engine; keeps a clean path to ABAC. | Requires a new relation table, resolver, APIs, and breaking migration away from direct object policy. | **Chosen** | +| Nested tags | Can model classification hierarchy directly in tag objects. | Adds a second hierarchy beside metadata object hierarchy; complicates policy resolution, authorization, migration, and explainability. | Rejected | +| Tag-expression ABAC | Most expressive; supports complex conditions over tag names, tag values, principals, and scopes. | Requires expression language, matching engine, and more complex UX; too large for the next milestone. | Future | + +Policy-on-tag is the best next step because it is useful as a standalone model and keeps the +long-term ABAC path open. The consumer path can stay stable: + +```text +Consumer -> ObjectPolicyResolver -> object policies +``` + +Only the policy selection layer needs to evolve later from fixed `policy -> tag` relations to tag +expressions. + +### Row Filter and Column Mask Conflict Handling + +| Approach | Example | Trade-off | Decision | +|----------|---------|-----------|----------| +| Restrict conflicts at configuration time | Snowflake allows only one directly assigned row access policy on a table or view, and evaluates row access policies before masking policies. | Simple and predictable, but stricter for administrators. | Rejected for tag-driven policy selection because conflicts can still arise from multiple effective tags. | +| Combine row filters with OR semantics | BigQuery combines multiple row-level access policies with OR semantics. | Flexible, but can broaden access and is risky as a default for tag-driven governance. | Rejected | +| Priority-based resolution | Some systems can choose a winning policy by priority. | Flexible, but effective access becomes harder to reason about and easier to misconfigure. | Rejected | +| Fail closed on ambiguity | Databricks ABAC blocks access when multiple distinct row filters or column masks apply to the same target. | Safest default, but administrators must fix overlapping tags or policy associations. | **Chosen** | + +### Tag Value Semantics + +| Approach | Pros | Cons | Decision | +|----------|------|------|----------| +| Append values | Can express multiple values for the same tag key on one object. | Makes policy selection ambiguous and can trigger multiple policies for one logical classification dimension. | Rejected | +| Overwrite values | Keeps each tag key single-valued for one object; aligns with common tag and label systems. | Administrators must use separate tag keys for separate dimensions. | **Chosen** | + +--- + +## Proposal + +### Target Model + +The target model has four concepts: + +| Concept | Description | +|---------|-------------| +| Policy | A metalake-scoped governance rule with typed content, enabled state, audit information, and version history. | +| Tag | A flat metalake-scoped classification object associated with metadata objects. Tags do not have parent or child tags. | +| Policy-tag relation | A relation that binds one policy to one tag in the same metalake. | +| Object policy | A read-only policy result for a metadata object, derived from effective tags and policy-tag relations. | + +Object-side governance becomes: + +```text +Metadata Object -> Effective Tags -> Object Policies +``` + +Metadata objects do not store direct policy relations. Object policies are not persisted as separate +entities. + +### Effective Tag Semantics + +Policy-on-tag reuses the current metadata-object tag inheritance model. This is not tag nesting: +tags are flat, and only tag assignments flow through the metadata object hierarchy. Examples use +flat tag names without dot separators. + +1. Tags associated with an object are direct tags. +2. Tags associated with parent metadata objects are inherited tags. +3. If a child object has a direct assignment for a tag name, that direct tag becomes the effective + source for that tag and overrides inherited assignments with the same tag name. +4. Policy-on-tag uses tag presence for policy resolution. It does not need assignment values to + resolve the phase-1 policy set. +5. The effective tag set is the de-duplicated result of walking from the object to its ancestors. +6. Policies bound to effective tags become object policy candidates. + +### Policy Supported Object Types + +`PolicyContent.supportedObjectTypes()` should mean "the metadata object types on which this policy Review Comment: Updated in 4259c6c9d. is now deprecated in this design, and object policy resolution no longer filters by it. The resolver returns enabled policies from effective tags, while type-specific consumers decide which policy types they consume. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
