This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch branch-1.3
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/branch-1.3 by this push:
new 0be25c4961 [Cherry-pick to branch-1.3] [#14681] fix(tag): Normalize
metadata object names for tag operations (#14682) (#14739)
0be25c4961 is described below
commit 0be25c4961c904fa879bac6b85ef49be0ea8bc56
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Sat Nov 7 10:14:47 2026 +0800
[Cherry-pick to branch-1.3] [#14681] fix(tag): Normalize metadata object
names for tag operations (#14682) (#14739)
### What changes were proposed in this pull request?
Backport 3c63c74ff1d19d6592d617035b4b033687825341 (#14682) to
`branch-1.3`, resolving the cherry-pick conflicts against the 1.3 APIs.
TagManager normalizes metadata identifiers before listing, reading,
adding or removing tag relations. Schema, table and object names follow
the catalog's existing case rules. Column parents use their own
schema/table scopes, and an exact stored column name is preserved when
case folding changes it. MetadataIdConverter delegates its existing
normalization entry point to the same helper.
Existence checks receive the original request. Relation access and the
existing tree locks use the resulting canonical identifier. The 1.3
string-tag API, lock nesting, relation updates and error handling are
retained.
### Why are the changes needed?
An equivalent object name can pass the catalog's existence check but
fail the tag relation lookup when the original spelling is used as the
storage key. The same issue affects tag association and disassociation.
Backports the fix for #14681.
### Does this PR introduce _any_ user-facing change?
Tag operations use the existing catalog case rules across all nine
tag-supported metadata types in 1.3. Case-sensitive object names, exact
stored column names and tag names are preserved. A capability or
stored-column lookup failure propagates before changing tag relations.
Each supported non-catalog tag operation reads catalog capabilities
after its existence check. A catalog cache miss may reload the
connector. A column whose spelling changes also reads the stored table;
these extra reads can fail the operation before relation access. Missing
stored tables retain the folded-name fallback. This does not make
concurrent external metadata changes atomic.
This backport retains 1.3's existing authorization/cache and storage
support. It does not introduce main-only APIs: custom `normalizeName`,
physical table-name resolution, Semantic Models, tag-value assignments,
or the normalized authorization-cache entry points and per-catalog
failure memoization. No configuration, dependency or schema migration is
added.
### How was this patch tested?
Targeted tests exercise real H2 tag relation storage for all nine
supported target types, case-insensitive names, Glue-style column rules,
exact stored column names, original-request existence checks, add/remove
round trips, capability failures and stored-column read failures before
mutation. Shared-helper tests cover missing stored tables, preserved
error causes and avoiding unnecessary table reads. The branch's
authorization normalizer is tested for exact and folded column
identifiers.
All 133 targeted cases passed (67 core and 66 server-common, zero
failures/errors/skips). Core coverage includes TestTagManager,
TestCapabilityHelpers, TestPolicyManager, TestTagHookDispatcher,
TestTableNormalizeDispatcher, TestMetadataObjectUtil and
TestAuthorizationRequestContext. Server-common coverage includes
TestMetadataIdConverter, TestJcasbinAuthorizer,
TestJcasbinAuthorizationLookups, TestJcasbinAuthorizationCacheKeys,
TestJcasbinAuthorizerCacheHelpers and TestJcasbinLoadedRolesCache.
Restoring the old TagManager and MetadataIdConverter made 15 of their 30
cases fail; all 30 passed again after restoring the fix. Scoped Spotless
and Javadoc passed. Only explicit `--tests` filters were used; the full
core suite was not run. Column authorization assertions target the
existing `normalizeCaseSensitive` entry point: this backport does not
extend the 1.3 column entity/ID-loading API. Live connectors, Docker
database backends and HTTP/CLI integration are not exercised; no SQL
statement or schema is changed.
---------
Co-authored-by: Qi Yu <[email protected]>
Co-authored-by: yuqi <[email protected]>
---
.../gravitino/catalog/CapabilityHelpers.java | 55 +++
.../java/org/apache/gravitino/tag/TagManager.java | 33 +-
.../gravitino/catalog/TestCapabilityHelpers.java | 87 +++++
.../org/apache/gravitino/tag/TestTagManager.java | 402 ++++++++++++++++++++-
.../server/authorization/MetadataIdConverter.java | 5 +-
.../authorization/TestMetadataIdConverter.java | 57 +++
6 files changed, 627 insertions(+), 12 deletions(-)
diff --git
a/core/src/main/java/org/apache/gravitino/catalog/CapabilityHelpers.java
b/core/src/main/java/org/apache/gravitino/catalog/CapabilityHelpers.java
index e66dc69cdd..ede8ca1fa8 100644
--- a/core/src/main/java/org/apache/gravitino/catalog/CapabilityHelpers.java
+++ b/core/src/main/java/org/apache/gravitino/catalog/CapabilityHelpers.java
@@ -23,13 +23,20 @@ import static
org.apache.gravitino.rel.expressions.transforms.Transforms.NAME_OF
import static
org.apache.gravitino.utils.NameIdentifierUtil.getCatalogIdentifier;
import com.google.common.base.Preconditions;
+import com.google.common.collect.ObjectArrays;
+import java.io.IOException;
import java.util.Arrays;
+import java.util.function.Supplier;
+import org.apache.gravitino.Entity;
+import org.apache.gravitino.EntityStore;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
import org.apache.gravitino.connector.capability.Capability;
import org.apache.gravitino.connector.capability.CapabilityResult;
import org.apache.gravitino.exceptions.NoSuchCatalogException;
+import org.apache.gravitino.exceptions.NoSuchEntityException;
import org.apache.gravitino.file.FilesetChange;
+import org.apache.gravitino.meta.TableEntity;
import org.apache.gravitino.model.ModelChange;
import org.apache.gravitino.rel.Column;
import org.apache.gravitino.rel.TableChange;
@@ -64,6 +71,42 @@ public class CapabilityHelpers {
}
}
+ /**
+ * Normalizes an existing catalog object's identifier using the catalog's
case rules. Tag and
+ * authorization lookups share these rules so they address the same stored
object.
+ *
+ * <p>Column parents use the schema and table scopes independently. If
folding changes a column
+ * name that exactly matches a stored column, the requested spelling is
retained.
+ *
+ * @param ident the entity identifier of the object
+ * @param scope the capability scope of the object
+ * @param catalogManager the catalog manager used to load naming capabilities
+ * @param entityStore supplies the store containing table columns; called
only if column folding
+ * changes the requested name
+ * @return the normalized identifier
+ * @throws NoSuchCatalogException if the catalog does not exist
+ * @throws RuntimeException if capabilities or stored columns cannot be
loaded
+ */
+ public static NameIdentifier normalizeMetadataIdentifier(
+ NameIdentifier ident,
+ Capability.Scope scope,
+ CatalogManager catalogManager,
+ Supplier<EntityStore> entityStore) {
+ Capability capability = getCapability(ident, catalogManager);
+ if (scope == Capability.Scope.COLUMN) {
+ NameIdentifier table =
+ applyCaseSensitive(
+ NameIdentifier.of(ident.namespace().levels()),
Capability.Scope.TABLE, capability);
+ String column = applyCaseSensitiveOnName(scope, ident.name(),
capability);
+ if (!column.equals(ident.name()) && hasStoredColumn(table, ident.name(),
entityStore.get())) {
+ column = ident.name();
+ }
+ return NameIdentifier.of(
+ Namespace.of(ObjectArrays.concat(table.namespace().levels(),
table.name())), column);
+ }
+ return applyCaseSensitive(ident, scope, capability);
+ }
+
public static Column[] applyCapabilities(Column[] columns, Capability
capabilities) {
return Arrays.stream(columns)
.map(c -> applyCapabilities(c, capabilities))
@@ -585,4 +628,16 @@ public class CapabilityHelpers {
+ " Illegal name: "
+ name);
}
+
+ private static boolean hasStoredColumn(
+ NameIdentifier table, String column, EntityStore entityStore) {
+ try {
+ return entityStore.get(table, Entity.EntityType.TABLE,
TableEntity.class).columns().stream()
+ .anyMatch(storedColumn -> storedColumn.name().equals(column));
+ } catch (NoSuchEntityException e) {
+ return false;
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to load stored columns of table: " +
table, e);
+ }
+ }
}
diff --git a/core/src/main/java/org/apache/gravitino/tag/TagManager.java
b/core/src/main/java/org/apache/gravitino/tag/TagManager.java
index 971001ae6b..812b6e00e1 100644
--- a/core/src/main/java/org/apache/gravitino/tag/TagManager.java
+++ b/core/src/main/java/org/apache/gravitino/tag/TagManager.java
@@ -33,9 +33,12 @@ import java.util.Set;
import org.apache.gravitino.Entity;
import org.apache.gravitino.EntityAlreadyExistsException;
import org.apache.gravitino.EntityStore;
+import org.apache.gravitino.GravitinoEnv;
import org.apache.gravitino.MetadataObject;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.SupportsRelationOperations;
+import org.apache.gravitino.catalog.CapabilityHelpers;
+import org.apache.gravitino.connector.capability.Capability;
import org.apache.gravitino.exceptions.NoSuchEntityException;
import org.apache.gravitino.exceptions.NoSuchMetadataObjectException;
import org.apache.gravitino.exceptions.NoSuchTagException;
@@ -243,11 +246,9 @@ public class TagManager implements TagDispatcher {
public Tag[] listTagsInfoForMetadataObject(String metalake, MetadataObject
metadataObject)
throws NoSuchMetadataObjectException {
- NameIdentifier entityIdent = MetadataObjectUtil.toEntityIdent(metalake,
metadataObject);
+ NameIdentifier entityIdent = checkAndNormalizeMetadataObject(metalake,
metadataObject);
Entity.EntityType entityType =
MetadataObjectUtil.toEntityType(metadataObject);
- MetadataObjectUtil.checkMetadataObject(metalake, metadataObject);
-
return TreeLockUtils.doWithTreeLock(
entityIdent,
LockType.READ,
@@ -274,12 +275,10 @@ public class TagManager implements TagDispatcher {
public Tag getTagForMetadataObject(String metalake, MetadataObject
metadataObject, String name)
throws NoSuchMetadataObjectException {
- NameIdentifier entityIdent = MetadataObjectUtil.toEntityIdent(metalake,
metadataObject);
+ NameIdentifier entityIdent = checkAndNormalizeMetadataObject(metalake,
metadataObject);
Entity.EntityType entityType =
MetadataObjectUtil.toEntityType(metadataObject);
NameIdentifier tagIdent = NameIdentifierUtil.ofTag(metalake, name);
- MetadataObjectUtil.checkMetadataObject(metalake, metadataObject);
-
return TreeLockUtils.doWithTreeLock(
entityIdent,
LockType.READ,
@@ -316,11 +315,9 @@ public class TagManager implements TagDispatcher {
"Cannot associate tags for unsupported metadata object type %s",
metadataObject.type());
- NameIdentifier entityIdent = MetadataObjectUtil.toEntityIdent(metalake,
metadataObject);
+ NameIdentifier entityIdent = checkAndNormalizeMetadataObject(metalake,
metadataObject);
Entity.EntityType entityType =
MetadataObjectUtil.toEntityType(metadataObject);
- MetadataObjectUtil.checkMetadataObject(metalake, metadataObject);
-
// Remove all the tags that are both set to add and remove
Set<String> tagsToAddSet = tagsToAdd == null ? Sets.newHashSet() :
Sets.newHashSet(tagsToAdd);
Set<String> tagsToRemoveSet =
@@ -377,6 +374,24 @@ public class TagManager implements TagDispatcher {
}));
}
+ private NameIdentifier checkAndNormalizeMetadataObject(
+ String metalake, MetadataObject metadataObject) {
+ NameIdentifier ident = MetadataObjectUtil.toEntityIdent(metalake,
metadataObject);
+ // Keep provider existence checks on the original request. The canonical
identifier below is
+ // used for both relation access and tree locks, so equivalent names share
the same lock.
+ MetadataObjectUtil.checkMetadataObject(metalake, metadataObject);
+ if
(!SUPPORTED_METADATA_OBJECT_TYPES_FOR_TAGS.contains(metadataObject.type())
+ || metadataObject.type() == MetadataObject.Type.CATALOG) {
+ return ident;
+ }
+
+ return CapabilityHelpers.normalizeMetadataIdentifier(
+ ident,
+ Capability.Scope.valueOf(metadataObject.type().name()),
+ GravitinoEnv.getInstance().catalogManager(),
+ () -> entityStore);
+ }
+
private TagEntity updateTagEntity(TagEntity tagEntity, TagChange... changes)
{
Map<String, String> props =
tagEntity.properties() == null
diff --git
a/core/src/test/java/org/apache/gravitino/catalog/TestCapabilityHelpers.java
b/core/src/test/java/org/apache/gravitino/catalog/TestCapabilityHelpers.java
index 86207c2716..0aaffeba87 100644
--- a/core/src/test/java/org/apache/gravitino/catalog/TestCapabilityHelpers.java
+++ b/core/src/test/java/org/apache/gravitino/catalog/TestCapabilityHelpers.java
@@ -18,12 +18,19 @@
*/
package org.apache.gravitino.catalog;
+import java.io.IOException;
+import java.util.Collections;
+import org.apache.gravitino.Entity;
+import org.apache.gravitino.EntityStore;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
import org.apache.gravitino.connector.BaseCatalog;
import org.apache.gravitino.connector.capability.Capability;
import org.apache.gravitino.connector.capability.CapabilityResult;
import org.apache.gravitino.exceptions.NoSuchCatalogException;
+import org.apache.gravitino.exceptions.NoSuchEntityException;
+import org.apache.gravitino.meta.ColumnEntity;
+import org.apache.gravitino.meta.TableEntity;
import org.apache.gravitino.rel.TableChange;
import org.apache.gravitino.rel.types.Types;
import org.junit.jupiter.api.Assertions;
@@ -117,4 +124,84 @@ public class TestCapabilityHelpers {
Assertions.assertInstanceOf(IllegalStateException.class, e.getCause());
}
+
+ @Test
+ void testNormalizeMetadataIdentifierKeepsExactStoredColumnName() throws
IOException {
+ CatalogManager catalogManager = foldingCatalogManager();
+ EntityStore store = Mockito.mock(EntityStore.class);
+ NameIdentifier table = NameIdentifier.of("metalake", "catalog", "schema",
"orders");
+ TableEntity tableEntity = Mockito.mock(TableEntity.class);
+ ColumnEntity column = Mockito.mock(ColumnEntity.class);
+ Mockito.when(column.name()).thenReturn("MyCol");
+
Mockito.when(tableEntity.columns()).thenReturn(Collections.singletonList(column));
+ Mockito.when(store.get(table, Entity.EntityType.TABLE, TableEntity.class))
+ .thenReturn(tableEntity);
+
+ Assertions.assertEquals(
+ NameIdentifier.of("metalake", "catalog", "schema", "orders", "MyCol"),
+ CapabilityHelpers.normalizeMetadataIdentifier(
+ NameIdentifier.of("metalake", "catalog", "SCHEMA", "ORDERS",
"MyCol"),
+ Capability.Scope.COLUMN,
+ catalogManager,
+ () -> store));
+ Assertions.assertEquals(
+ NameIdentifier.of("metalake", "catalog", "schema", "orders",
"othercol"),
+ CapabilityHelpers.normalizeMetadataIdentifier(
+ NameIdentifier.of("metalake", "catalog", "SCHEMA", "ORDERS",
"OtherCol"),
+ Capability.Scope.COLUMN,
+ catalogManager,
+ () -> store));
+ Mockito.verify(store, Mockito.times(2)).get(table,
Entity.EntityType.TABLE, TableEntity.class);
+
+ // A name that folding does not change needs no stored-column read.
+ Assertions.assertEquals(
+ NameIdentifier.of("metalake", "catalog", "schema", "orders", "mycol"),
+ CapabilityHelpers.normalizeMetadataIdentifier(
+ NameIdentifier.of("metalake", "catalog", "SCHEMA", "ORDERS",
"mycol"),
+ Capability.Scope.COLUMN,
+ catalogManager,
+ () -> store));
+ Mockito.verify(store, Mockito.times(2)).get(table,
Entity.EntityType.TABLE, TableEntity.class);
+ }
+
+ @Test
+ void testNormalizeMetadataIdentifierHandlesMissingOrFailedStoredTable()
throws IOException {
+ CatalogManager catalogManager = foldingCatalogManager();
+ EntityStore store = Mockito.mock(EntityStore.class);
+ NameIdentifier table = NameIdentifier.of("metalake", "catalog", "schema",
"orders");
+ Mockito.when(store.get(table, Entity.EntityType.TABLE, TableEntity.class))
+ .thenThrow(new NoSuchEntityException("missing"));
+ NameIdentifier column = NameIdentifier.of("metalake", "catalog", "schema",
"ORDERS", "MyCol");
+ Assertions.assertEquals(
+ NameIdentifier.of("metalake", "catalog", "schema", "orders", "mycol"),
+ CapabilityHelpers.normalizeMetadataIdentifier(
+ column, Capability.Scope.COLUMN, catalogManager, () -> store));
+
+ IOException failure = new IOException("store down");
+ Mockito.reset(store);
+ Mockito.when(store.get(table, Entity.EntityType.TABLE,
TableEntity.class)).thenThrow(failure);
+ RuntimeException error =
+ Assertions.assertThrows(
+ RuntimeException.class,
+ () ->
+ CapabilityHelpers.normalizeMetadataIdentifier(
+ column, Capability.Scope.COLUMN, catalogManager, () ->
store));
+ Assertions.assertSame(failure, error.getCause());
+ Assertions.assertTrue(error.getMessage().contains(table.toString()));
+ }
+
+ private static CatalogManager foldingCatalogManager() {
+ CatalogManager catalogManager = Mockito.mock(CatalogManager.class);
+ BaseCatalog<?> catalog = Mockito.mock(BaseCatalog.class);
+ Mockito.when(catalog.capability())
+ .thenReturn(
+ new Capability() {
+ @Override
+ public CapabilityResult caseSensitiveOnName(Scope scope) {
+ return CapabilityResult.unsupported("case insensitive");
+ }
+ });
+ CatalogTestUtils.mockDoWithCatalog(catalogManager, catalog);
+ return catalogManager;
+ }
}
diff --git a/core/src/test/java/org/apache/gravitino/tag/TestTagManager.java
b/core/src/test/java/org/apache/gravitino/tag/TestTagManager.java
index 37a2f19ea1..3a4312faf8 100644
--- a/core/src/test/java/org/apache/gravitino/tag/TestTagManager.java
+++ b/core/src/test/java/org/apache/gravitino/tag/TestTagManager.java
@@ -46,6 +46,7 @@ import java.io.File;
import java.io.IOException;
import java.time.Instant;
import java.util.Arrays;
+import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.UUID;
@@ -63,17 +64,29 @@ import org.apache.gravitino.EntityStore;
import org.apache.gravitino.EntityStoreFactory;
import org.apache.gravitino.GravitinoEnv;
import org.apache.gravitino.MetadataObject;
+import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
import org.apache.gravitino.catalog.CatalogDispatcher;
+import org.apache.gravitino.catalog.CatalogManager;
+import org.apache.gravitino.catalog.CatalogTestUtils;
+import org.apache.gravitino.catalog.FilesetDispatcher;
import org.apache.gravitino.catalog.FunctionDispatcher;
+import org.apache.gravitino.catalog.ModelDispatcher;
import org.apache.gravitino.catalog.SchemaDispatcher;
import org.apache.gravitino.catalog.TableDispatcher;
+import org.apache.gravitino.catalog.TopicDispatcher;
import org.apache.gravitino.catalog.ViewDispatcher;
+import org.apache.gravitino.connector.BaseCatalog;
+import org.apache.gravitino.connector.capability.Capability;
+import org.apache.gravitino.connector.capability.CapabilityResult;
+import org.apache.gravitino.exceptions.IllegalNamespaceException;
+import org.apache.gravitino.exceptions.NoSuchMetadataObjectException;
import org.apache.gravitino.exceptions.NoSuchMetalakeException;
import org.apache.gravitino.exceptions.NoSuchTagException;
import org.apache.gravitino.exceptions.NotFoundException;
import org.apache.gravitino.exceptions.TagAlreadyAssociatedException;
import org.apache.gravitino.exceptions.TagAlreadyExistsException;
+import org.apache.gravitino.file.Fileset;
import org.apache.gravitino.function.FunctionDefinition;
import org.apache.gravitino.function.FunctionDefinitions;
import org.apache.gravitino.function.FunctionImpl;
@@ -86,10 +99,13 @@ import org.apache.gravitino.meta.AuditInfo;
import org.apache.gravitino.meta.BaseMetalake;
import org.apache.gravitino.meta.CatalogEntity;
import org.apache.gravitino.meta.ColumnEntity;
+import org.apache.gravitino.meta.FilesetEntity;
import org.apache.gravitino.meta.FunctionEntity;
+import org.apache.gravitino.meta.ModelEntity;
import org.apache.gravitino.meta.SchemaEntity;
import org.apache.gravitino.meta.SchemaVersion;
import org.apache.gravitino.meta.TableEntity;
+import org.apache.gravitino.meta.TopicEntity;
import org.apache.gravitino.meta.ViewEntity;
import org.apache.gravitino.metalake.MetalakeDispatcher;
import org.apache.gravitino.rel.Column;
@@ -104,6 +120,9 @@ import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+import org.mockito.AdditionalAnswers;
import org.mockito.Mockito;
public class TestTagManager {
@@ -135,6 +154,18 @@ public class TestTagManager {
private static final ViewDispatcher viewDispatcher =
mock(ViewDispatcher.class);
private static final FunctionDispatcher functionDispatcher =
mock(FunctionDispatcher.class);
+ private static final CatalogManager catalogManager =
mock(CatalogManager.class);
+ private static final BaseCatalog<?> backendCatalog = mock(BaseCatalog.class);
+ private static final Capability CASE_INSENSITIVE_CAPABILITY =
+ new Capability() {
+ @Override
+ public CapabilityResult caseSensitiveOnName(Scope scope) {
+ return CapabilityResult.unsupported("case insensitive");
+ }
+ };
+
+ private static CatalogManager previousCatalogManager;
+
private static EntityStore entityStore;
private static IdGenerator idGenerator;
@@ -266,6 +297,11 @@ public class TestTagManager {
entityStore.put(function, false /* overwritten */);
tagManager = new TagManager(idGenerator, entityStore);
+ previousCatalogManager =
+ (CatalogManager) FieldUtils.readField(GravitinoEnv.getInstance(),
"catalogManager", true);
+ FieldUtils.writeField(GravitinoEnv.getInstance(), "catalogManager",
catalogManager, true);
+ CatalogTestUtils.mockDoWithCatalog(catalogManager, backendCatalog);
+ when(backendCatalog.capability()).thenReturn(Capability.DEFAULT);
FieldUtils.writeField(
GravitinoEnv.getInstance(), "internalMetalakeDispatcher",
metalakeDispatcher, true);
@@ -289,7 +325,9 @@ public class TestTagManager {
}
@AfterAll
- public static void tearDown() throws IOException {
+ public static void tearDown() throws IOException, IllegalAccessException {
+ FieldUtils.writeField(
+ GravitinoEnv.getInstance(), "catalogManager", previousCatalogManager,
true);
if (entityStore != null) {
entityStore.close();
entityStore = null;
@@ -341,6 +379,293 @@ public class TestTagManager {
Arrays.stream(tagManager.listTags(METALAKE)).forEach(n ->
tagManager.deleteTag(METALAKE, n));
}
+ @ParameterizedTest
+ @EnumSource(
+ value = Entity.EntityType.class,
+ names = {"SCHEMA", "TABLE", "COLUMN", "VIEW", "FUNCTION"})
+ void testTagOperationsNormalizeCaseInsensitiveNames(Entity.EntityType type) {
+ MetadataObject canonical = tagTarget(type, false);
+ MetadataObject request = tagTarget(type, true);
+ when(backendCatalog.capability()).thenReturn(CASE_INSENSITIVE_CAPABILITY);
+ try {
+ assertTagRoundTrip(request, canonical, canonical);
+ } finally {
+ when(backendCatalog.capability()).thenReturn(Capability.DEFAULT);
+ }
+ }
+
+ @ParameterizedTest
+ @EnumSource(
+ value = Entity.EntityType.class,
+ names = {"FILESET", "TOPIC", "MODEL"})
+ void testTagOperationsNormalizeOtherCatalogObjects(Entity.EntityType type)
+ throws IOException, IllegalAccessException {
+ NameIdentifier ident = NameIdentifier.of(METALAKE, CATALOG, SCHEMA,
"normalization_object");
+ AuditInfo audit =
AuditInfo.builder().withCreator("test").withCreateTime(Instant.now()).build();
+ Object dispatcher;
+ String dispatcherField;
+ switch (type) {
+ case FILESET:
+ entityStore.put(
+ FilesetEntity.builder()
+ .withId(idGenerator.nextId())
+ .withName(ident.name())
+ .withNamespace(ident.namespace())
+ .withFilesetType(Fileset.Type.EXTERNAL)
+ .withStorageLocations(Map.of("default",
"file:/tmp/normalization_object"))
+ .withAuditInfo(audit)
+ .build(),
+ false);
+ FilesetDispatcher filesetDispatcher = mock(FilesetDispatcher.class);
+ when(filesetDispatcher.filesetExists(any())).thenReturn(true);
+ dispatcher = filesetDispatcher;
+ dispatcherField = "internalFilesetDispatcher";
+ break;
+ case TOPIC:
+ entityStore.put(
+ TopicEntity.builder()
+ .withId(idGenerator.nextId())
+ .withName(ident.name())
+ .withNamespace(ident.namespace())
+ .withAuditInfo(audit)
+ .build(),
+ false);
+ TopicDispatcher topicDispatcher = mock(TopicDispatcher.class);
+ when(topicDispatcher.topicExists(any())).thenReturn(true);
+ dispatcher = topicDispatcher;
+ dispatcherField = "internalTopicDispatcher";
+ break;
+ case MODEL:
+ entityStore.put(
+ ModelEntity.builder()
+ .withId(idGenerator.nextId())
+ .withName(ident.name())
+ .withNamespace(ident.namespace())
+ .withLatestVersion(0)
+ .withAuditInfo(audit)
+ .build(),
+ false);
+ ModelDispatcher modelDispatcher = mock(ModelDispatcher.class);
+ when(modelDispatcher.modelExists(any())).thenReturn(true);
+ dispatcher = modelDispatcher;
+ dispatcherField = "internalModelDispatcher";
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported test target: " + type);
+ }
+ Object previousDispatcher =
+ FieldUtils.readField(GravitinoEnv.getInstance(), dispatcherField,
true);
+ FieldUtils.writeField(GravitinoEnv.getInstance(), dispatcherField,
dispatcher, true);
+ when(backendCatalog.capability()).thenReturn(CASE_INSENSITIVE_CAPABILITY);
+ try {
+ MetadataObject request =
+ NameIdentifierUtil.toMetadataObject(
+ NameIdentifier.of(
+ METALAKE,
+ CATALOG,
+ SCHEMA.toUpperCase(Locale.ROOT),
+ ident.name().toUpperCase(Locale.ROOT)),
+ type);
+ MetadataObject canonical = NameIdentifierUtil.toMetadataObject(ident,
type);
+ assertTagRoundTrip(request, canonical, canonical);
+ } finally {
+ when(backendCatalog.capability()).thenReturn(Capability.DEFAULT);
+ FieldUtils.writeField(GravitinoEnv.getInstance(), dispatcherField,
previousDispatcher, true);
+ entityStore.delete(ident, type);
+ }
+ }
+
+ @Test
+ void testCatalogAndMetalakeTagOperationsDoNotLoadCapabilities() {
+ Mockito.clearInvocations(catalogManager);
+ MetadataObject catalogObject =
+ NameIdentifierUtil.toMetadataObject(
+ NameIdentifier.of(METALAKE, CATALOG), Entity.EntityType.CATALOG);
+ assertTagRoundTrip(catalogObject, catalogObject, catalogObject);
+ MetadataObject metalakeObject =
+ NameIdentifierUtil.toMetadataObject(
+ NameIdentifier.of(METALAKE), Entity.EntityType.METALAKE);
+ // Metalake tags are unsupported: preserve the existing store failure
without loading a catalog.
+ Assertions.assertThrows(
+ IllegalNamespaceException.class,
+ () -> tagManager.listTagsForMetadataObject(METALAKE, metalakeObject));
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ tagManager.associateTagsForMetadataObject(
+ METALAKE, metalakeObject, new String[] {"normalization_tag"},
null));
+ Mockito.verifyNoInteractions(catalogManager);
+ }
+
+ @Test
+ void testTagOperationsPreserveCaseSensitiveNames() {
+ MetadataObject canonical = tagTarget(Entity.EntityType.TABLE, false);
+ MetadataObject differentlyCased = tagTarget(Entity.EntityType.TABLE, true);
+ Tag tag = tagManager.createTag(METALAKE, "case_sensitive_tag", null, null);
+ tagManager.associateTagsForMetadataObject(METALAKE, canonical, new
String[] {tag.name()}, null);
+
+ Assertions.assertThrows(
+ NoSuchMetadataObjectException.class,
+ () -> tagManager.listTagsForMetadataObject(METALAKE,
differentlyCased));
+ Assertions.assertThrows(
+ NotFoundException.class,
+ () -> tagManager.getTagForMetadataObject(METALAKE, differentlyCased,
tag.name()));
+ Assertions.assertThrows(
+ NoSuchMetadataObjectException.class,
+ () ->
+ tagManager.associateTagsForMetadataObject(
+ METALAKE, differentlyCased, null, new String[] {tag.name()}));
+ Assertions.assertArrayEquals(
+ new String[] {tag.name()},
tagManager.listTagsForMetadataObject(METALAKE, canonical));
+ }
+
+ @Test
+ void testTagOperationsFollowGlueStyleColumnCase() throws IOException {
+ // Glue folds schema and table names but keeps the case of column names.
+ NameIdentifier table = putTableWithColumn("glue_table", "MixedCol");
+ Capability glueStyle =
+ new Capability() {
+ @Override
+ public CapabilityResult caseSensitiveOnName(Scope scope) {
+ return scope == Scope.SCHEMA || scope == Scope.TABLE
+ ? CapabilityResult.unsupported("case insensitive")
+ : CapabilityResult.SUPPORTED;
+ }
+ };
+ when(backendCatalog.capability()).thenReturn(glueStyle);
+ try {
+ MetadataObject request =
+ NameIdentifierUtil.toMetadataObject(
+ NameIdentifier.of(
+ METALAKE, CATALOG, SCHEMA.toUpperCase(Locale.ROOT),
"GLUE_TABLE", "MixedCol"),
+ Entity.EntityType.COLUMN);
+ MetadataObject canonical =
+ NameIdentifierUtil.toMetadataObject(
+ NameIdentifier.of(METALAKE, CATALOG, SCHEMA, table.name(),
"MixedCol"),
+ Entity.EntityType.COLUMN);
+ assertTagRoundTrip(request, canonical, canonical);
+ } finally {
+ when(backendCatalog.capability()).thenReturn(Capability.DEFAULT);
+ entityStore.delete(table, Entity.EntityType.TABLE);
+ }
+ }
+
+ @Test
+ void testTagOperationsKeepExactStoredColumnName() throws IOException {
+ // A capability can fold column names that the source stored with another
case. The exact
+ // stored name must still find the column.
+ NameIdentifier table = putTableWithColumn("exact_table", "MyCol");
+ when(backendCatalog.capability()).thenReturn(CASE_INSENSITIVE_CAPABILITY);
+ try {
+ MetadataObject canonical =
+ NameIdentifierUtil.toMetadataObject(
+ NameIdentifier.of(METALAKE, CATALOG, SCHEMA, table.name(),
"MyCol"),
+ Entity.EntityType.COLUMN);
+ MetadataObject request =
+ NameIdentifierUtil.toMetadataObject(
+ NameIdentifier.of(
+ METALAKE, CATALOG, SCHEMA.toUpperCase(Locale.ROOT),
"EXACT_TABLE", "MyCol"),
+ Entity.EntityType.COLUMN);
+ assertTagRoundTrip(request, canonical, canonical);
+ } finally {
+ when(backendCatalog.capability()).thenReturn(Capability.DEFAULT);
+ entityStore.delete(table, Entity.EntityType.TABLE);
+ }
+ }
+
+ @ParameterizedTest
+ @EnumSource(
+ value = Entity.EntityType.class,
+ names = {"TABLE", "COLUMN"})
+ void testFailedCapabilityLookupLeavesTagsUnchanged(Entity.EntityType type) {
+ MetadataObject canonical = tagTarget(type, false);
+ Tag kept = tagManager.createTag(METALAKE, "kept_tag", null, null);
+ Tag added = tagManager.createTag(METALAKE, "added_tag", null, null);
+ tagManager.associateTagsForMetadataObject(
+ METALAKE, canonical, new String[] {kept.name()}, null);
+
+ when(backendCatalog.capability()).thenThrow(new
IllegalStateException("catalog unavailable"));
+ try {
+ RuntimeException addError =
+ Assertions.assertThrows(
+ RuntimeException.class,
+ () ->
+ tagManager.associateTagsForMetadataObject(
+ METALAKE, canonical, new String[] {added.name()}, null));
+ Assertions.assertInstanceOf(IllegalStateException.class,
addError.getCause());
+ Assertions.assertThrows(
+ RuntimeException.class,
+ () ->
+ tagManager.associateTagsForMetadataObject(
+ METALAKE, canonical, null, new String[] {kept.name()}));
+ } finally {
+ Mockito.doReturn(Capability.DEFAULT).when(backendCatalog).capability();
+ }
+ Assertions.assertArrayEquals(
+ new String[] {kept.name()},
tagManager.listTagsForMetadataObject(METALAKE, canonical));
+ }
+
+ @Test
+ void testExistenceChecksUseOriginalIdentifier() {
+ MetadataObject request = tagTarget(Entity.EntityType.TABLE, true);
+ MetadataObject canonical = tagTarget(Entity.EntityType.TABLE, false);
+ NameIdentifier requestedTable =
+ NameIdentifier.of(
+ METALAKE, CATALOG, SCHEMA.toUpperCase(Locale.ROOT),
TABLE.toUpperCase(Locale.ROOT));
+ when(backendCatalog.capability()).thenReturn(CASE_INSENSITIVE_CAPABILITY);
+ when(tableDispatcher.tableExists(any()))
+ .thenAnswer(invocation ->
requestedTable.equals(invocation.getArgument(0)));
+ try {
+ assertTagRoundTrip(request, request, canonical);
+ } finally {
+ when(backendCatalog.capability()).thenReturn(Capability.DEFAULT);
+ when(tableDispatcher.tableExists(any())).thenReturn(true);
+ }
+ }
+
+ @Test
+ void testFailedStoredColumnReadLeavesTagsUnchanged() throws IOException {
+ NameIdentifier table = putTableWithColumn("failed_column_read", "MyCol");
+ MetadataObject canonical =
+ NameIdentifierUtil.toMetadataObject(
+ NameIdentifier.of(METALAKE, CATALOG, SCHEMA, table.name(),
"MyCol"),
+ Entity.EntityType.COLUMN);
+ Tag kept = tagManager.createTag(METALAKE, "kept_tag", null, null);
+ Tag added = tagManager.createTag(METALAKE, "added_tag", null, null);
+ tagManager.associateTagsForMetadataObject(
+ METALAKE, canonical, new String[] {kept.name()}, null);
+ EntityStore failingStore = mock(EntityStore.class,
AdditionalAnswers.delegatesTo(entityStore));
+ IOException failure = new IOException("stored columns unavailable");
+ Mockito.doThrow(failure)
+ .when(failingStore)
+ .get(table, Entity.EntityType.TABLE, TableEntity.class);
+ TagManager failingManager = new TagManager(idGenerator, failingStore);
+ when(backendCatalog.capability()).thenReturn(CASE_INSENSITIVE_CAPABILITY);
+ try {
+ RuntimeException error =
+ Assertions.assertThrows(
+ RuntimeException.class,
+ () ->
+ failingManager.associateTagsForMetadataObject(
+ METALAKE, canonical, new String[] {added.name()}, null));
+ Assertions.assertSame(failure, error.getCause());
+ Assertions.assertThrows(
+ RuntimeException.class,
+ () ->
+ failingManager.associateTagsForMetadataObject(
+ METALAKE, canonical, null, new String[] {kept.name()}));
+ } finally {
+ when(backendCatalog.capability()).thenReturn(Capability.DEFAULT);
+ }
+ try {
+ Assertions.assertArrayEquals(
+ new String[] {kept.name()},
tagManager.listTagsForMetadataObject(METALAKE, canonical));
+ Mockito.verify(failingStore, Mockito.never()).relationOperations();
+ } finally {
+ entityStore.delete(table, Entity.EntityType.TABLE);
+ }
+ }
+
@Test
public void testCreateAndGetTag() {
Tag tag = tagManager.createTag(METALAKE, "tag1", null, null);
@@ -981,4 +1306,79 @@ public class TestTagManager {
Assertions.assertTrue(
e3.getMessage().contains("Failed to get tag for metadata object " +
nonExistentObject));
}
+
+ private static NameIdentifier putTableWithColumn(String tableName, String
columnName)
+ throws IOException {
+ AuditInfo audit =
AuditInfo.builder().withCreator("test").withCreateTime(Instant.now()).build();
+ NameIdentifier ident = NameIdentifier.of(METALAKE, CATALOG, SCHEMA,
tableName);
+ entityStore.put(
+ TableEntity.builder()
+ .withId(idGenerator.nextId())
+ .withName(tableName)
+ .withNamespace(ident.namespace())
+ .withColumns(
+ Lists.newArrayList(
+ ColumnEntity.builder()
+ .withId(idGenerator.nextId())
+ .withName(columnName)
+ .withPosition(0)
+ .withDataType(Types.IntegerType.get())
+ .withNullable(true)
+ .withAutoIncrement(false)
+ .withAuditInfo(audit)
+ .build()))
+ .withAuditInfo(audit)
+ .build(),
+ false);
+ return ident;
+ }
+
+ private static MetadataObject tagTarget(Entity.EntityType type, boolean
upperCase) {
+ String schema = upperCase ? SCHEMA.toUpperCase(Locale.ROOT) : SCHEMA;
+ String table = upperCase ? TABLE.toUpperCase(Locale.ROOT) : TABLE;
+ NameIdentifier ident;
+ switch (type) {
+ case SCHEMA:
+ ident = NameIdentifier.of(METALAKE, CATALOG, schema);
+ break;
+ case TABLE:
+ ident = NameIdentifier.of(METALAKE, CATALOG, schema, table);
+ break;
+ case COLUMN:
+ String column = upperCase ? COLUMN.toUpperCase(Locale.ROOT) : COLUMN;
+ ident = NameIdentifier.of(METALAKE, CATALOG, schema, table, column);
+ break;
+ case VIEW:
+ case FUNCTION:
+ String name = type == Entity.EntityType.VIEW ? VIEW : FUNCTION;
+ name = upperCase ? name.toUpperCase(Locale.ROOT) : name;
+ ident = NameIdentifier.of(METALAKE, CATALOG, schema, name);
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported test target: " + type);
+ }
+ return NameIdentifierUtil.toMetadataObject(ident, type);
+ }
+
+ private static void assertTagRoundTrip(
+ MetadataObject request, MetadataObject alias, MetadataObject
storedObject) {
+ Tag tag = tagManager.createTag(METALAKE, "normalization_tag", null, null);
+ String[] names = {tag.name()};
+ tagManager.associateTagsForMetadataObject(METALAKE, alias, names, null);
+ Assertions.assertArrayEquals(names,
tagManager.listTagsForMetadataObject(METALAKE, request));
+ Assertions.assertEquals(
+ tag.name(), tagManager.listTagsInfoForMetadataObject(METALAKE,
request)[0].name());
+ Assertions.assertEquals(
+ tag.name(), tagManager.getTagForMetadataObject(METALAKE, request,
tag.name()).name());
+ Assertions.assertArrayEquals(
+ new MetadataObject[] {storedObject},
+ tagManager.listMetadataObjectsForTag(METALAKE, tag.name()));
+
+ Assertions.assertEquals(
+ 0, tagManager.associateTagsForMetadataObject(METALAKE, request, null,
names).length);
+ Assertions.assertArrayEquals(
+ names, tagManager.associateTagsForMetadataObject(METALAKE, request,
names, null));
+ tagManager.associateTagsForMetadataObject(METALAKE, alias, null, names);
+ Assertions.assertEquals(0,
tagManager.listTagsInfoForMetadataObject(METALAKE, alias).length);
+ }
}
diff --git
a/server-common/src/main/java/org/apache/gravitino/server/authorization/MetadataIdConverter.java
b/server-common/src/main/java/org/apache/gravitino/server/authorization/MetadataIdConverter.java
index 220acba819..516b0a6bb0 100644
---
a/server-common/src/main/java/org/apache/gravitino/server/authorization/MetadataIdConverter.java
+++
b/server-common/src/main/java/org/apache/gravitino/server/authorization/MetadataIdConverter.java
@@ -101,8 +101,9 @@ public class MetadataIdConverter {
return ident;
}
- Capability capability = CapabilityHelpers.getCapability(ident,
catalogManager);
- return CapabilityHelpers.applyCaseSensitive(ident, scope, capability);
+ // Tag operations use the same helper, so authorization looks up the same
stored object.
+ return CapabilityHelpers.normalizeMetadataIdentifier(
+ ident, scope, catalogManager, () ->
GravitinoEnv.getInstance().entityStore());
}
private static Long extractIdFromEntity(Entity entity) {
diff --git
a/server-common/src/test/java/org/apache/gravitino/server/authorization/TestMetadataIdConverter.java
b/server-common/src/test/java/org/apache/gravitino/server/authorization/TestMetadataIdConverter.java
index c2065bcbcc..88c08a4da8 100644
---
a/server-common/src/test/java/org/apache/gravitino/server/authorization/TestMetadataIdConverter.java
+++
b/server-common/src/test/java/org/apache/gravitino/server/authorization/TestMetadataIdConverter.java
@@ -30,6 +30,7 @@ import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import java.io.IOException;
import java.time.Instant;
+import java.util.Collections;
import java.util.Optional;
import org.apache.commons.lang3.reflect.FieldUtils;
import org.apache.gravitino.Catalog;
@@ -41,7 +42,9 @@ import org.apache.gravitino.MetadataObjects;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
import org.apache.gravitino.catalog.CatalogManager;
+import org.apache.gravitino.connector.BaseCatalog;
import org.apache.gravitino.connector.capability.Capability;
+import org.apache.gravitino.connector.capability.CapabilityResult;
import org.apache.gravitino.exceptions.NoSuchCatalogException;
import org.apache.gravitino.file.Fileset;
import org.apache.gravitino.meta.AuditInfo;
@@ -56,11 +59,13 @@ import org.apache.gravitino.meta.SchemaVersion;
import org.apache.gravitino.meta.TableEntity;
import org.apache.gravitino.meta.TopicEntity;
import org.apache.gravitino.rel.types.Types;
+import org.apache.gravitino.utils.ThrowableFunction;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestInstance;
import org.mockito.MockedStatic;
+import org.mockito.Mockito;
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class TestMetadataIdConverter {
@@ -232,6 +237,58 @@ public class TestMetadataIdConverter {
}
}
+ @Test
+ void testColumnAuthorizationKeepsExactStoredColumnName() throws Exception {
+ CatalogManager catalogs = mock(CatalogManager.class);
+ BaseCatalog<?> catalog = mock(BaseCatalog.class);
+ when(catalog.capability())
+ .thenReturn(
+ new Capability() {
+ @Override
+ public CapabilityResult caseSensitiveOnName(Scope scope) {
+ return CapabilityResult.unsupported("lowercase");
+ }
+ });
+ Mockito.doAnswer(
+ invocation -> {
+ ThrowableFunction<BaseCatalog<?>, Object> operation =
invocation.getArgument(1);
+ return operation.apply(catalog);
+ })
+ .when(catalogs)
+ .doWithCatalog(any(), any());
+ EntityStore store = mock(EntityStore.class);
+ TableEntity table = mock(TableEntity.class);
+ ColumnEntity column = mock(ColumnEntity.class);
+ when(column.name()).thenReturn("MyCol");
+ when(table.columns()).thenReturn(Collections.singletonList(column));
+ when(store.get(
+ NameIdentifier.of("metalake", "catalog", "schema", "orders"),
+ Entity.EntityType.TABLE,
+ TableEntity.class))
+ .thenReturn(table);
+ GravitinoEnv env = mock(GravitinoEnv.class);
+ when(env.catalogManager()).thenReturn(catalogs);
+ when(env.entityStore()).thenReturn(store);
+ try (MockedStatic<GravitinoEnv> mocked = mockStatic(GravitinoEnv.class)) {
+ mocked.when(GravitinoEnv::getInstance).thenReturn(env);
+ NameIdentifier exact = NameIdentifier.of("metalake", "catalog",
"schema", "orders", "MyCol");
+ NameIdentifier folded =
+ NameIdentifier.of("metalake", "catalog", "schema", "orders",
"othercol");
+ Assertions.assertEquals(
+ exact,
+ MetadataIdConverter.normalizeCaseSensitive(
+ NameIdentifier.of("metalake", "catalog", "SCHEMA", "ORDERS",
"MyCol"),
+ Capability.Scope.COLUMN,
+ catalogs));
+ Assertions.assertEquals(
+ folded,
+ MetadataIdConverter.normalizeCaseSensitive(
+ NameIdentifier.of("metalake", "catalog", "SCHEMA", "ORDERS",
"OtherCol"),
+ Capability.Scope.COLUMN,
+ catalogs));
+ }
+ }
+
private void initTestNameIdentifier() {
ident1 = NameIdentifier.of("metalake");
ident2 = NameIdentifier.of("metalake", "catalog");
