jerryshao commented on code in PR #13382:
URL: https://github.com/apache/gravitino/pull/13382#discussion_r4068263269
##########
docs/migration-guide.md:
##########
@@ -0,0 +1,102 @@
+---
+title: "Migration Guide"
+slug: "/migration-guide"
+keyword: "migration, upgrade, compatibility, Gravitino"
+license: "This software is licensed under the Apache License version 2."
+---
+
+This guide lists user-visible behavior changes by upgrade version and the
action needed for
+existing deployments. For database backup, schema scripts, and rollback
commands, see
+[Upgrade Gravitino](./how-to-upgrade.md).
+
+## Upgrading from Gravitino 1.3 to 2.0
+
+### Policies are selected through tags
+
+- Direct policy associations with metadata objects are no longer read or
written. The upgrade
+ does not convert existing direct associations. Before upgrading, export each
policy's directly
+ associated objects from `GET
/api/metalakes/{metalake}/policies/{policy}/objects`. Save the
+ policy name, object type, and full name, including direct associations on
ancestor objects.
+ Back up the metadata database before the schema upgrade.
+- To preserve the old scope, create one dedicated tag per policy, associate
the policy with that
+ tag using the `ALL_VALUES` selector, and assign the tag to every object that
had a direct
+ policy association. A tag on a catalog or schema reaches its descendants. If
a policy was
+ directly associated with both a parent and a child, assign the tag to both.
+- Object policy lookup at `GET
/api/metalakes/{metalake}/objects/{type}/{fullName}/policies`
+ remains available, but now returns enabled policies matching the object's
effective tags.
+ A policy matching through multiple tags appears once. With `details=true`,
`inherited` is
+ true only if the matching tag assignments are inherited. Disabled policies
remain associated
+ with tags but do not appear in object policy results.
Review Comment:
[Important] The enabled flag changed meaning in this release, and the guide
states the new behavior without flagging it as a change that needs action.
In 1.3 the flag was documentation-only: the text this PR deletes says so
explicitly (`docs/policies.md:126` on `main`: "Gravitino does not act on it, so
disabling a policy does not detach it or change what a consumer receives", and
the same claim in `manage-policies-in-gravitino.md`). After #13355,
`ObjectPolicyResolver.resolve` ends with `.filter(PolicyEntity::enabled)`
(`core/src/main/java/org/apache/gravitino/policy/ObjectPolicyResolver.java:128`),
so a disabled policy silently disappears from every object lookup.
So a deployment that disabled a policy while still relying on a consumer
acting on it (TMS is the concrete case) loses that behavior at upgrade,
independent of the tag migration, and nothing in the checklist tells the
operator to look. Suggest adding an action bullet to "Policies are selected
through tags" or step 1 of the checklist: inventory disabled policies before
upgrading and re-enable the ones consumers still need, because disabling is now
enforced rather than advisory.
Verified by: reading `ObjectPolicyResolver.java:126-128` on this head and
`git show origin/main:docs/policies.md` lines 120-130 for the previous
documented behavior.
##########
docs/policies.md:
##########
@@ -1,189 +1,105 @@
---
title: "Policies"
slug: "/policies"
-keyword: "policy, policies, governance, metadata object, Gravitino"
+keyword: "policy, policies, governance, tags, Gravitino"
license: "This software is licensed under the Apache License version 2."
---
## Introduction
-A policy is a named set of rules that you create once in a metalake and attach
to metadata objects.
-Attaching a policy to a catalog or schema applies it to everything beneath, so
a setting that varies
-by table can be expressed once at the level where it holds and overridden
where it does not.
+A policy is a named set of rules in a metalake. Associate it with a tag, then
assign that tag to
+metadata objects. When a client reads an object's policies, Gravitino finds
its effective tags and
+returns the enabled policies whose association selectors match. The policy
remains a separate
+object: changing its rules updates every object where it applies.
-Tags and policies are close cousins, and the difference is what they carry. A
tag classifies, and
-its content is its name. A policy prescribes, and its content is a set of
rules something acts on.
-
-Policies come in two kinds. A built-in policy has a type that Gravitino
defines and a consumer that
-acts on it. A custom policy carries rules of your own, which Gravitino stores,
inherits, and serves
-back to whatever system you build around it.
-
-Common uses:
-
-- Setting table maintenance behavior for a whole catalog rather than table by
table, and letting new
- tables pick it up without further work
-- Recording a rule once against metadata that lives in several catalogs, so
every engine reaching
- those objects through Gravitino sees the same rule
-- Feeding an external enforcement or scheduling system that reads policies
from Gravitino rather
- than keeping its own copy of what applies where
+Policies come in two kinds. Built-in policy types have rules and consumers
defined by Gravitino.
+Custom policies carry rules that your own system interprets. For example, the
table maintenance
+service consumes the built-in Iceberg compaction policy.
## Quick Start
-**1. Create the policy.** Policies are created from the policy list in the UI,
which creates custom
-policies. A policy needs a name, the object types it supports, and its rules.
Built-in policies are
-created over REST.
-
-**2. Attach it to an object.** Open the catalog, schema, table, fileset,
topic, model, view, or function you want to
-govern and add the policy from its policy control. Only policies that already
exist in the metalake
-are offered.
+1. [Create a policy](./manage-policies-in-gravitino.md#create-a-policy) and
+ [create a tag](./manage-tags-in-gravitino.md#create-a-tag) in the same
metalake.
+2. [Associate the policy with the
tag](./manage-policies-in-gravitino.md#associate-a-policy-with-a-tag).
+ Choose `ALL_VALUES` to match tag presence or `TAG_VALUE` to match one exact
assignment value.
+3. [Assign the tag](./manage-tags-in-gravitino.md#object-operations) to an
object or its ancestor.
+4. [List the object's
policies](./manage-policies-in-gravitino.md#list-policies-on-an-object) to
+ confirm the result.
-**3. See where the policy is attached.** Selecting a policy name in the policy
list shows the
-objects it is attached to directly.
+For example, associate `retention_30d` with `data_domain` using
+`TAG_VALUE("finance")`. A table with an effective `data_domain=finance`
assignment receives the
+policy. A table with only `data_domain=risk` does not.
## The Policy Model
-### Policy Types
+### Policy Types and Content
-| Type | Rules |
Consumed by |
+| Type | Rules |
Consumer |
|-----------------------------|--------------------------------------|---------------------------|
| `system_iceberg_compaction` | Compaction thresholds and scheduling | Table
maintenance service |
-| `custom` | A free-form map you define | A
system you provide |
-
-A built-in type has a name beginning with `system_` and a content shape
Gravitino defines. The
-compaction policy is documented in [Iceberg compaction
policy](./iceberg-compaction-policy.md), and
-the service that acts on it in
-[Table maintenance service](./table-maintenance-service/optimizer.md).
-
-A custom policy has type `custom`, and Gravitino makes no attempt to interpret
what is inside
-`customRules`. The rules are stored, inherited down the hierarchy, and
returned to any client that
-asks.
-
-The UI creates custom policies only. A built-in policy is created over REST
with its own content
-shape.
-
-### What Can Carry a Policy
-
-A metadata object is identified by a type and a name, with each level below
the catalog separated by
-a dot. Eight object types can carry a policy.
-
-| Object type | Name form |
-|-------------|-----------------------------------------------|
-| `CATALOG` | `{catalog_name}` |
-| `SCHEMA` | `{catalog_name}.{schema_name}` |
-| `TABLE` | `{catalog_name}.{schema_name}.{table_name}` |
-| `FILESET` | `{catalog_name}.{schema_name}.{fileset_name}` |
-| `TOPIC` | `{catalog_name}.{schema_name}.{topic_name}` |
-| `MODEL` | `{catalog_name}.{schema_name}.{model_name}` |
-| `VIEW` | `{catalog_name}.{schema_name}.{view_name}` |
-| `FUNCTION` | `{catalog_name}.{schema_name}.{function_name}`|
-
-Columns cannot carry a policy, which is narrower than
-[tags](./tags.md). A metalake cannot carry one either, so to reach every object
-in a catalog, attach the policy to the catalog.
-
-Each policy also declares its own `supportedObjectTypes`, which narrows the
list further for that
-policy.
-
-### Content
-
-Policy content has three parts: the `supportedObjectTypes` list, the rules,
and properties.
-
-`supportedObjectTypes` is fixed when the policy is created and cannot be
changed afterward, so a
-policy meant for tables only stays that way for its lifetime.
-
-The rules are what a consumer evaluates. For a custom policy they live under
`customRules` as a map
-you define, where the name is yours and the value is any JSON value.
-
-```json
-"customRules": {
- "retentionDays": 30,
- "maxTableSizeGb": 500,
- "requiresApproval": true
-}
-```
-
-Gravitino does not interpret those names or values. Whatever consumes the
policy decides what
-`retentionDays` means and what to do about it.
-
-A built-in policy has a rule set Gravitino defines, and the service that
consumes it documents how
-those rules are applied. The compaction policy carries `minDataFileMse`,
`minDeleteFileNumber`,
-`dataFileMseWeight`, `deleteFileNumberWeight`, `max-partition-num`, and a
trigger and score
-expression, plus any `job.options.` entries passed through to the job. Those
names and their
-meanings are covered in [Iceberg compaction
policy](./iceberg-compaction-policy.md).
-
-Properties describe the policy itself rather than the behavior it asks for.
Rules change as you
-adjust thresholds, and properties stay stable. The compaction policy uses
properties for its
-strategy type and job template name, which tell the table maintenance service
what to run, and those
-are set by Gravitino rather than by you. For a custom policy, properties are
yours, and suit facts
-such as which team owns the policy, which system consumes it, or which version
of a rule set it
-represents. Anything evaluated against an object belongs in rules instead.
-
-Properties sit on the policy rather than on an attachment, so every object
carrying the policy sees
-the same values.
-
-### The Enabled Flag
-
-The `enabled` flag marks a policy as active or inactive for readers. Gravitino
does not act on it,
-so disabling a policy does not detach it or change what a consumer receives.
Treat it as a signal to
-whoever reads the policy, useful for holding a policy through review without
deleting it.
-
-### Inheritance
-
-An object shows the policies attached to it plus the policies attached to each
of its ancestors, so
-a policy on a catalog applies to every schema, table, fileset, topic, model,
view, and function beneath it. For
-catalogs that support multi-level schemas, the intermediate schemas are
ancestors too.
-
-Each policy appears once, whether it reaches the object through one ancestor
or several. A policy
-attached directly to the object counts as direct even when an ancestor carries
it too.
-
-Direct and inherited attachments are distinguishable. In the UI an inherited
policy is marked with a
-lock icon. Over REST, a policy listing requested with `details=true` carries
an `inherited` field on
-each policy, which a plain listing of names does not.
+| `custom` | A free-form map that you define | A
system that you provide |
-A policy that reaches an object only by inheritance cannot be removed there.
Detach it from the
-ancestor that carries it, which affects every other object beneath that
ancestor as well.
+A custom policy's rules live in `customRules`. Gravitino stores them and
returns them to clients;
+it does not interpret their names or values. Built-in types have a defined
content shape. See
+[Iceberg compaction policy](./iceberg-compaction-policy.md) for the compaction
rules and
+[Table maintenance service](./table-maintenance-service/optimizer.md) for a
worked example.
-Inheritance is resolved when the object is read rather than stored on the
object, so attaching a
-policy to a catalog takes effect immediately for tables created afterward.
+Policy content also has `properties` and `supportedObjectTypes`. Properties
describe the policy
+itself, such as its owner or consumer. `supportedObjectTypes` is required when
creating a custom
+policy and cannot be changed later. Object policy lookup does not filter by
this field; each
+consumer decides whether a policy type applies to the object it is processing.
-## Working With Policies in the UI
+### Policy-to-Tag Associations
-### Managing the Policy Set
+Each association connects one policy to one tag and stores a selector. The
selector determines
+whether that association contributes the policy to an object's lookup result.
-The policy list holds every policy in the metalake and can be searched. A
policy can be renamed, its
-comment and rules edited, and its enabled flag switched from there. Policies
created over REST,
-including built-in ones, appear in the list alongside the rest.
+| Selector | When it matches
|
+|--------------|------------------------------------------------------------------------|
+| `ALL_VALUES` | The effective tag is present, including an assignment without
a value. |
+| `TAG_VALUE` | The effective tag has the specified exact assignment value.
|
Review Comment:
[Nit] "the specified exact assignment value" reads as if the assignment
holds a single value. An assignment can hold several
(`TagAssignment.ofValues(String...)`, and `migration-guide.md:53` in this PR
says as much), and the match is a containment check:
`Arrays.asList(assignment.values()).contains(expectedValue)`
(`core/src/main/java/org/apache/gravitino/policy/ObjectPolicyResolver.java:138`).
So `data_domain=[finance, risk]` matches both `TAG_VALUE("finance")` and
`TAG_VALUE("risk")`.
Suggest "one of the effective tag's assignment values equals the specified
value" here and in the same table at `manage-policies-in-gravitino.md:228`.
Verified by: reading `ObjectPolicyResolver.matches` and
`api/src/main/java/org/apache/gravitino/tag/TagAssignment.java`.
##########
docs/manage-policies-in-gravitino.md:
##########
@@ -215,35 +217,145 @@ client.deletePolicy("retention_30d");
</TabItem>
</Tabs>
+## Policy-to-Tag Associations
+
+Create a policy and a tag in the same metalake before associating them. Each
policy-to-tag
+association has a selector:
+
+| Selector | Match condition
|
+|--------------|------------------------------------------------------------------|
+| `ALL_VALUES` | The effective tag is present, with or without assignment
values. |
+| `TAG_VALUE` | The effective tag has the specified exact assignment value.
|
+
+The selector belongs to the association, not to the policy or the tag. An
existing association
+cannot be replaced by another add request. Remove it and add it again to
change its selector.
+
+### Associate a Policy with a Tag
+
+This example applies `retention_30d` when the effective `data_domain` tag has
the value `finance`.
+Create the tag first if it does not exist; see
+[Manage
tags](./manage-tags-in-gravitino.md#create-a-tag-with-a-value-constraint).
+
+<Tabs groupId='language' queryString>
+<TabItem value="shell" label="REST">
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" \
+ -d '{"selector": {"type": "TAG_VALUE", "value": "finance"}}' \
+
http://localhost:8090/api/metalakes/test/tags/data_domain/policies/retention_30d
+```
+
+Use `{"selector": {"type": "ALL_VALUES"}}` to match any assignment of
`data_domain`.
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+PolicyTagAssociation association = client.addPolicyForTag(
+ "data_domain", "retention_30d", TagValueSelector.of("finance"));
+
+// Match any assignment of data_domain instead:
+// client.addPolicyForTag("data_domain", "retention_30d");
+```
+
+</TabItem>
+</Tabs>
+
+The request fails with a conflict if the policy is already associated with the
tag.
+
+### List Associations
+
+List associations from either side. By default the response contains names.
Set `details=true`
+to get policy or tag details together with each association's selector. These
lists show direct
+associations even when their selectors do not match any object's current tag
values.
+
+<Tabs groupId='language' queryString>
+<TabItem value="shell" label="REST">
+
+```shell
+curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
+
"http://localhost:8090/api/metalakes/test/tags/data_domain/policies?details=true";
+
+curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
+
"http://localhost:8090/api/metalakes/test/policies/retention_30d/tags?details=true";
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+PolicyTagAssociation[] policies =
client.listPolicyAssociationsForTag("data_domain");
+PolicyTagAssociation[] tags =
client.listTagAssociationsForPolicy("retention_30d");
+```
+
+</TabItem>
+</Tabs>
+
+### Remove an Association
+
+Removing the association stops this policy from being selected through the
tag. It leaves the
+policy, tag, and tag assignments in place.
+
+<Tabs groupId='language' queryString>
+<TabItem value="shell" label="REST">
+
+```shell
+curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
+
http://localhost:8090/api/metalakes/test/tags/data_domain/policies/retention_30d
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+client.removePolicyFromTag("data_domain", "retention_30d");
+```
+
+</TabItem>
+</Tabs>
+
## Object Operations
Object policies are read-only results derived from effective tags. To change
the policies that apply
to an object, associate a policy with a tag and then assign or remove that tag
on the object or one
of its ancestors. See [Manage tags in
Gravitino](./manage-tags-in-gravitino.md) for tag assignment
operations.
+For the `TAG_VALUE("finance")` association above, assign `data_domain=finance`
to a table or one
+of its ancestors. A direct assignment of `data_domain` on the table overrides
an inherited
+assignment of the same tag. For example, assigning only `data_domain=risk` to
the table stops
+`retention_30d` from matching there, even if its catalog has
`data_domain=finance`.
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v2+json" \
+ -H "Content-Type: application/vnd.gravitino.v2+json" \
+ -d '{"tagsToAdd": [{"name": "data_domain", "value": "finance"}]}' \
+
http://localhost:8090/api/metalakes/test/objects/table/catalog1.schema1.customers/tags
+```
+
### List Policies on an Object
The response includes policies derived from effective tags assigned to the
object or its ancestors.
With `details=true`, the response returns full policy objects instead of
policy names.
Each policy includes an `inherited` field, which is `true` when it matches
only through a tag
-assigned to an ancestor of the object.
+assigned to an ancestor of the object. Disabled policies do not appear in this
result.
<Tabs groupId='language' queryString>
<TabItem value="shell" label="REST">
```shell
curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-
"http://localhost:8090/api/metalakes/test/objects/catalog/catalog1/policies?details=true";
+
"http://localhost:8090/api/metalakes/test/objects/table/catalog1.schema1.customers/policies?details=true";
```
</TabItem>
<TabItem value="java" label="Java">
```java
-Catalog catalog = client.loadCatalog("catalog1");
-String[] policyNames = catalog.supportsPolicies().listPolicies();
-Policy[] policies = catalog.supportsPolicies().listPolicyInfos();
+Table customers = ...
Review Comment:
[Nit] `Table customers = ...` replaces what used to be a runnable snippet
(`client.loadCatalog("catalog1")`), so the Java tab no longer shows how to get
to the object. `Table.supportsPolicies()` is a default method on the API
interface (`api/src/main/java/org/apache/gravitino/rel/Table.java:167`), so a
real line works, e.g. loading the catalog, `asTableCatalog()`, then
`loadTable(NameIdentifier.of("schema1", "customers"))`.
Verified by: reading
`api/src/main/java/org/apache/gravitino/rel/Table.java:159-167` and the REST
example directly above this tab.
##########
docs/policies.md:
##########
@@ -1,189 +1,105 @@
---
title: "Policies"
slug: "/policies"
-keyword: "policy, policies, governance, metadata object, Gravitino"
+keyword: "policy, policies, governance, tags, Gravitino"
license: "This software is licensed under the Apache License version 2."
---
## Introduction
-A policy is a named set of rules that you create once in a metalake and attach
to metadata objects.
-Attaching a policy to a catalog or schema applies it to everything beneath, so
a setting that varies
-by table can be expressed once at the level where it holds and overridden
where it does not.
+A policy is a named set of rules in a metalake. Associate it with a tag, then
assign that tag to
+metadata objects. When a client reads an object's policies, Gravitino finds
its effective tags and
+returns the enabled policies whose association selectors match. The policy
remains a separate
+object: changing its rules updates every object where it applies.
-Tags and policies are close cousins, and the difference is what they carry. A
tag classifies, and
-its content is its name. A policy prescribes, and its content is a set of
rules something acts on.
-
-Policies come in two kinds. A built-in policy has a type that Gravitino
defines and a consumer that
-acts on it. A custom policy carries rules of your own, which Gravitino stores,
inherits, and serves
-back to whatever system you build around it.
-
-Common uses:
-
-- Setting table maintenance behavior for a whole catalog rather than table by
table, and letting new
- tables pick it up without further work
-- Recording a rule once against metadata that lives in several catalogs, so
every engine reaching
- those objects through Gravitino sees the same rule
-- Feeding an external enforcement or scheduling system that reads policies
from Gravitino rather
- than keeping its own copy of what applies where
+Policies come in two kinds. Built-in policy types have rules and consumers
defined by Gravitino.
+Custom policies carry rules that your own system interprets. For example, the
table maintenance
+service consumes the built-in Iceberg compaction policy.
## Quick Start
-**1. Create the policy.** Policies are created from the policy list in the UI,
which creates custom
-policies. A policy needs a name, the object types it supports, and its rules.
Built-in policies are
-created over REST.
-
-**2. Attach it to an object.** Open the catalog, schema, table, fileset,
topic, model, view, or function you want to
-govern and add the policy from its policy control. Only policies that already
exist in the metalake
-are offered.
+1. [Create a policy](./manage-policies-in-gravitino.md#create-a-policy) and
+ [create a tag](./manage-tags-in-gravitino.md#create-a-tag) in the same
metalake.
+2. [Associate the policy with the
tag](./manage-policies-in-gravitino.md#associate-a-policy-with-a-tag).
+ Choose `ALL_VALUES` to match tag presence or `TAG_VALUE` to match one exact
assignment value.
+3. [Assign the tag](./manage-tags-in-gravitino.md#object-operations) to an
object or its ancestor.
+4. [List the object's
policies](./manage-policies-in-gravitino.md#list-policies-on-an-object) to
+ confirm the result.
-**3. See where the policy is attached.** Selecting a policy name in the policy
list shows the
-objects it is attached to directly.
+For example, associate `retention_30d` with `data_domain` using
+`TAG_VALUE("finance")`. A table with an effective `data_domain=finance`
assignment receives the
+policy. A table with only `data_domain=risk` does not.
## The Policy Model
-### Policy Types
+### Policy Types and Content
-| Type | Rules |
Consumed by |
+| Type | Rules |
Consumer |
|-----------------------------|--------------------------------------|---------------------------|
| `system_iceberg_compaction` | Compaction thresholds and scheduling | Table
maintenance service |
-| `custom` | A free-form map you define | A
system you provide |
-
-A built-in type has a name beginning with `system_` and a content shape
Gravitino defines. The
-compaction policy is documented in [Iceberg compaction
policy](./iceberg-compaction-policy.md), and
-the service that acts on it in
-[Table maintenance service](./table-maintenance-service/optimizer.md).
-
-A custom policy has type `custom`, and Gravitino makes no attempt to interpret
what is inside
-`customRules`. The rules are stored, inherited down the hierarchy, and
returned to any client that
-asks.
-
-The UI creates custom policies only. A built-in policy is created over REST
with its own content
-shape.
-
-### What Can Carry a Policy
-
-A metadata object is identified by a type and a name, with each level below
the catalog separated by
-a dot. Eight object types can carry a policy.
-
-| Object type | Name form |
-|-------------|-----------------------------------------------|
-| `CATALOG` | `{catalog_name}` |
-| `SCHEMA` | `{catalog_name}.{schema_name}` |
-| `TABLE` | `{catalog_name}.{schema_name}.{table_name}` |
-| `FILESET` | `{catalog_name}.{schema_name}.{fileset_name}` |
-| `TOPIC` | `{catalog_name}.{schema_name}.{topic_name}` |
-| `MODEL` | `{catalog_name}.{schema_name}.{model_name}` |
-| `VIEW` | `{catalog_name}.{schema_name}.{view_name}` |
-| `FUNCTION` | `{catalog_name}.{schema_name}.{function_name}`|
-
-Columns cannot carry a policy, which is narrower than
-[tags](./tags.md). A metalake cannot carry one either, so to reach every object
-in a catalog, attach the policy to the catalog.
-
-Each policy also declares its own `supportedObjectTypes`, which narrows the
list further for that
-policy.
-
-### Content
-
-Policy content has three parts: the `supportedObjectTypes` list, the rules,
and properties.
-
-`supportedObjectTypes` is fixed when the policy is created and cannot be
changed afterward, so a
-policy meant for tables only stays that way for its lifetime.
-
-The rules are what a consumer evaluates. For a custom policy they live under
`customRules` as a map
-you define, where the name is yours and the value is any JSON value.
-
-```json
-"customRules": {
- "retentionDays": 30,
- "maxTableSizeGb": 500,
- "requiresApproval": true
-}
-```
-
-Gravitino does not interpret those names or values. Whatever consumes the
policy decides what
-`retentionDays` means and what to do about it.
-
-A built-in policy has a rule set Gravitino defines, and the service that
consumes it documents how
-those rules are applied. The compaction policy carries `minDataFileMse`,
`minDeleteFileNumber`,
-`dataFileMseWeight`, `deleteFileNumberWeight`, `max-partition-num`, and a
trigger and score
-expression, plus any `job.options.` entries passed through to the job. Those
names and their
-meanings are covered in [Iceberg compaction
policy](./iceberg-compaction-policy.md).
-
-Properties describe the policy itself rather than the behavior it asks for.
Rules change as you
-adjust thresholds, and properties stay stable. The compaction policy uses
properties for its
-strategy type and job template name, which tell the table maintenance service
what to run, and those
-are set by Gravitino rather than by you. For a custom policy, properties are
yours, and suit facts
-such as which team owns the policy, which system consumes it, or which version
of a rule set it
-represents. Anything evaluated against an object belongs in rules instead.
-
-Properties sit on the policy rather than on an attachment, so every object
carrying the policy sees
-the same values.
-
-### The Enabled Flag
-
-The `enabled` flag marks a policy as active or inactive for readers. Gravitino
does not act on it,
-so disabling a policy does not detach it or change what a consumer receives.
Treat it as a signal to
-whoever reads the policy, useful for holding a policy through review without
deleting it.
-
-### Inheritance
-
-An object shows the policies attached to it plus the policies attached to each
of its ancestors, so
-a policy on a catalog applies to every schema, table, fileset, topic, model,
view, and function beneath it. For
-catalogs that support multi-level schemas, the intermediate schemas are
ancestors too.
-
-Each policy appears once, whether it reaches the object through one ancestor
or several. A policy
-attached directly to the object counts as direct even when an ancestor carries
it too.
-
-Direct and inherited attachments are distinguishable. In the UI an inherited
policy is marked with a
-lock icon. Over REST, a policy listing requested with `details=true` carries
an `inherited` field on
-each policy, which a plain listing of names does not.
+| `custom` | A free-form map that you define | A
system that you provide |
-A policy that reaches an object only by inheritance cannot be removed there.
Detach it from the
-ancestor that carries it, which affects every other object beneath that
ancestor as well.
+A custom policy's rules live in `customRules`. Gravitino stores them and
returns them to clients;
+it does not interpret their names or values. Built-in types have a defined
content shape. See
+[Iceberg compaction policy](./iceberg-compaction-policy.md) for the compaction
rules and
+[Table maintenance service](./table-maintenance-service/optimizer.md) for a
worked example.
-Inheritance is resolved when the object is read rather than stored on the
object, so attaching a
-policy to a catalog takes effect immediately for tables created afterward.
+Policy content also has `properties` and `supportedObjectTypes`. Properties
describe the policy
+itself, such as its owner or consumer. `supportedObjectTypes` is required when
creating a custom
+policy and cannot be changed later. Object policy lookup does not filter by
this field; each
+consumer decides whether a policy type applies to the object it is processing.
-## Working With Policies in the UI
+### Policy-to-Tag Associations
-### Managing the Policy Set
+Each association connects one policy to one tag and stores a selector. The
selector determines
+whether that association contributes the policy to an object's lookup result.
-The policy list holds every policy in the metalake and can be searched. A
policy can be renamed, its
-comment and rules edited, and its enabled flag switched from there. Policies
created over REST,
-including built-in ones, appear in the list alongside the rest.
+| Selector | When it matches
|
+|--------------|------------------------------------------------------------------------|
+| `ALL_VALUES` | The effective tag is present, including an assignment without
a value. |
+| `TAG_VALUE` | The effective tag has the specified exact assignment value.
|
-Deleting a policy removes it from every object it was attached to, with no
warning about how many
-objects that affects and no way to recover the attachments.
+A policy may be associated with multiple tags. Association listings show those
direct relations
+and their selectors, even if no object currently matches them. An object
policy lookup returns each
+matching policy once.
-### Attaching and Detaching
+A selector cannot be changed in place. Remove the policy-to-tag association
and add it again with
+the new selector. Removing an association leaves the policy, tag, and tag
assignments intact.
+Deleting a policy removes its associations.
-Policies attach from the object rather than from the policy, so open the
object and use the policy
-control there. Inherited policies carry no remove control. Detaching removes
the direct attachment
-only, so an object still shows a policy it inherits from an ancestor.
+### Effective Tags and Inheritance
-### Finding Where a Policy Is Used
+An object receives tags assigned directly to it and tags inherited from its
metadata object
+ancestors. When both the object and an ancestor assign the same tag name, the
object's direct
+assignment wins, including its assignment values. Tag names themselves are
flat; tags do not
Review Comment:
[Important] Rewriting this page dropped the object-type coverage, and under
the new model the answer changed rather than became irrelevant.
The old page had "What Can Carry a Policy" with the eight object types and
the sentence "Columns cannot carry a policy" (`docs/policies.md:80` on `main`).
Now policies come from effective tags, and tags *are* assignable to columns
(`docs/tags.md:60`). Nothing in the lookup path narrows that:
`EffectiveTagResolver.resolve` accepts whatever
`MetadataObjectUtil.checkMetadataObject` accepts, which includes `COLUMN`
(`core/src/main/java/org/apache/gravitino/utils/MetadataObjectUtil.java:262`),
the walk up is generic (`EffectiveTagResolver.java:70-71`),
`ObjectPolicyResolver` applies no object-type filter, and the REST layer parses
any type name
(`server/src/main/java/org/apache/gravitino/server/web/rest/MetadataObjectPolicyOperations.java:94`).
Two consequences a reader cannot get from this page: `GET
/objects/column/{catalog}.{schema}.{table}.{column}/policies` now returns
policies, and a migration tag assigned to a catalog (the recipe in
`migration-guide.md`) reaches columns too, which direct policy association
never did. Combined with `supportedObjectTypes` no longer filtering, a policy
created for `TABLE` only will show up in a column lookup.
Suggest restoring one short paragraph or table in this section stating which
object types object policy lookup covers, explicitly including columns.
Verified by: reading the three resolver/REST files named above on this head,
plus `docs/tags.md:54-64` for the tag-assignable types.
##########
docs/topics.md:
##########
@@ -57,13 +57,14 @@ Leaving either unset takes the broker's own default, from
`num.partition` and
### What Gravitino Stores and What It Does Not
-Gravitino stores the topic's place in the hierarchy and anything attached to
it, including tags,
-policies, and ownership. Message content, offsets, consumer groups, and lag
stay entirely in the
+Gravitino stores the topic's place in the hierarchy, its tag assignments,
derived policies,
Review Comment:
[Nit] "Gravitino stores ... derived policies" says the derived result is
persisted. It is computed per request: `ObjectPolicyResolver.resolve`
re-resolves effective tags and re-evaluates selectors on every lookup
(`core/src/main/java/org/apache/gravitino/policy/ObjectPolicyResolver.java:77-128`),
which is also what `policies.md` describes. What is stored is the tag
assignment and the policy-to-tag association.
Suggest "its tag assignments and ownership" here, with policies described as
derived at read time. `docs/catalogs-and-schemas.md:78` has the same phrasing
("anything associated with those objects such as tags, derived policies, and
ownership").
Verified by: reading the resolver on this head; no persisted object-policy
relation is written anywhere in that path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
