dennishuo commented on code in PR #2223:
URL: https://github.com/apache/polaris/pull/2223#discussion_r2253065317
##########
service/common/src/main/java/org/apache/polaris/service/admin/PolarisAdminService.java:
##########
@@ -1679,14 +1680,29 @@ public boolean grantPrivilegeOnNamespaceToRole(
PolarisAuthorizableOperation.ADD_NAMESPACE_GRANT_TO_CATALOG_ROLE;
authorizeGrantOnNamespaceOperationOrThrow(op, catalogName, namespace,
catalogRoleName);
+ CatalogEntity catalogEntity =
+ findCatalogByName(catalogName)
+ .orElseThrow(() -> new NotFoundException("Parent catalog %s not
found", catalogName));
PolarisEntity catalogRoleEntity =
findCatalogRoleByName(catalogName, catalogRoleName)
.orElseThrow(() -> new NotFoundException("CatalogRole %s not
found", catalogRoleName));
PolarisResolvedPathWrapper resolvedPathWrapper =
resolutionManifest.getResolvedPath(namespace);
if (resolvedPathWrapper == null
|| !resolvedPathWrapper.isFullyResolvedNamespace(catalogName,
namespace)) {
- throw new NotFoundException("Namespace %s not found", namespace);
+ if (resolutionManifest.getIsPassthroughFacade()) {
Review Comment:
We may also want to add a
[FeatureConfiguration](https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java)
to gate this path; even if federation is still "experimental" so we can change
default behaviors, this persistence-backfill is definitely something people
might still potentially decide to disable.
I'd make the feature configuration be a service-level config (no
`catalogConfig`) because the gate really is more of something the service owner
would want to be able to prevent catalog-users from using than a self opt-in
per-catalog.
Maybe something like `ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS`,
though maybe you can come up with a better config name :)
##########
service/common/src/main/java/org/apache/polaris/service/admin/PolarisAdminService.java:
##########
@@ -2034,6 +2136,74 @@ private boolean grantPrivilegeOnTableLikeToRole(
.isSuccess();
}
+ /**
+ * Creates and persists the missing synthetic table-like entity and its
parent namespace entities
+ * for external catalogs.
+ *
+ * @param catalogEntity the external passthrough facade catalog entity.
+ * @param identifier the path of the table-like entity(including the
namespace).
+ * @param subTypes the expected subtypes of the table-like entity
+ * @param existingPathWrapper the partially resolved path currently stored
in the metastore.
+ * @return the resolved path wrapper
+ */
+ private PolarisResolvedPathWrapper createSyntheticTableLikeEntities(
+ CatalogEntity catalogEntity,
+ TableIdentifier identifier,
+ List<PolarisEntitySubType> subTypes,
+ PolarisResolvedPathWrapper existingPathWrapper) {
+
+ Namespace namespace = identifier.namespace();
+ PolarisResolvedPathWrapper resolvedNamespacePathWrapper =
+ !namespace.isEmpty()
+ ? createSyntheticNamespaceEntities(catalogEntity, namespace,
existingPathWrapper)
+ : existingPathWrapper;
+
+ if (resolvedNamespacePathWrapper == null
+ || (!namespace.isEmpty()
+ && !resolvedNamespacePathWrapper.isFullyResolvedNamespace(
+ catalogEntity.getName(), namespace))) {
+ throw new RuntimeException(
+ String.format(
+ "Failed to create synthetic namespace entities for namespace %s
in catalog %s",
+ namespace.toString(), catalogEntity.getName()));
+ }
+
+ // TODO: Instead of creating a synthetic table-like entity, rely on
external catalog mediated
+ // backfill.
+ PolarisEntity parentNamespaceEntity =
resolvedNamespacePathWrapper.getRawLeafEntity();
+ PolarisEntity syntheticTableEntity =
+ new PolarisEntity.Builder()
+
.setId(metaStoreManager.generateNewEntityId(getCurrentPolarisContext()).getId())
+ .setCatalogId(parentNamespaceEntity.getCatalogId())
+ .setParentId(parentNamespaceEntity.getId())
+ .setType(PolarisEntityType.TABLE_LIKE)
+ .setSubType(subTypes.get(0))
Review Comment:
Unfortunately, we probably don't want to do this. Looks like the current
behavior will be to create a `GENERIC_TABLE` for the synthetic entity, which
will cause problems once we want to start leaning more on the JIT-created
entities for other things like actually having the metadata location, etc.
It might need to be ugly at first -- it's even better to just special-case
ICEBERG_TABLE vs VIEW (and ignore GENERIC_TABLE at first) for now. Maybe
extract a helper to hide the ugliness that takes `subTypes` and basically does:
if (subTypes.contains(ICEBERG_TABLE)) {
return ICEBERG_TABLE;
} else if (subTypes.contains(VIEW) {
return VIEW;
} else {
throw new IllegalStateException(...)
}
For now that is "correct" because our federation catalog factory (last I
checked) only supports Iceberg remote catalogs, and nothing for GENERIC_TABLE.
Once we also support GENERIC_TABLE federation, we'd need the intended type
at the callsite anyways to even know whether the "remote catalog for mediating
the backfill" should be instantiated via an Iceberg RESTCatalog factory or a
different factory for GenericCatalogs.
That might be a big hurdle because of ICEBERG_TABLE and GENERIC_TABLE living
in the same "type" namespace, but that will be a separate workstream anyways.
We can document this issue in another `// TODO:` comment.
##########
service/common/src/main/java/org/apache/polaris/service/admin/PolarisAdminService.java:
##########
@@ -1730,6 +1746,79 @@ public boolean revokePrivilegeOnNamespaceFromRole(
.isSuccess();
}
+ /**
+ * Creates and persists the missing synthetic namespace entities for
external catalogs.
+ *
+ * @param catalogEntity the external passthrough facade catalog entity.
+ * @param namespace the expected fully resolved namespace to be created.
+ * @param existingPath the partially resolved path currently stored in the
metastore.
+ * @return the fully resolved path wrapper.
+ */
+ private PolarisResolvedPathWrapper createSyntheticNamespaceEntities(
+ CatalogEntity catalogEntity, Namespace namespace,
PolarisResolvedPathWrapper existingPath) {
+
+ if (existingPath == null) {
+ throw new IllegalStateException(
+ String.format("Catalog entity %s does not exist.",
catalogEntity.getName()));
+ }
+
+ List<PolarisEntity> completePath = new
ArrayList<>(existingPath.getRawFullPath());
+ PolarisEntity currentParent = existingPath.getRawLeafEntity();
+
+ String[] allNamespaceLevels = namespace.levels();
+ int matchingLevel = -1;
+ for (PolarisEntity entity : completePath.subList(1, completePath.size())) {
+ if (entity.getName().equals(allNamespaceLevels[matchingLevel + 1])) {
+ matchingLevel++;
+ } else {
+ break;
+ }
+ }
+
+ for (int i = matchingLevel + 1; i < allNamespaceLevels.length; i++) {
+ String namespacePart = allNamespaceLevels[i];
+
+ // TODO: Instead of creating synthetic entitties, rely on external
catalog mediated backfill.
+ PolarisEntity syntheticNamespace =
+ new PolarisEntity.Builder()
Review Comment:
We should try to use the
[NamespaceEntity.Builder](https://github.com/apache/polaris/blob/d6ae2c89de2178113665ffa73ab773b0108da4fe/polaris-core/src/main/java/org/apache/polaris/core/entity/NamespaceEntity.java#L69)
if possible.
Also, it's necessary to call
[setParentNamespace](https://github.com/apache/polaris/blob/d6ae2c89de2178113665ffa73ab773b0108da4fe/polaris-core/src/main/java/org/apache/polaris/core/entity/NamespaceEntity.java#L82C20-L82C38)
because `listGrants*` relies on it.
##########
service/common/src/main/java/org/apache/polaris/service/admin/PolarisAdminService.java:
##########
@@ -1730,6 +1746,79 @@ public boolean revokePrivilegeOnNamespaceFromRole(
.isSuccess();
}
+ /**
+ * Creates and persists the missing synthetic namespace entities for
external catalogs.
+ *
+ * @param catalogEntity the external passthrough facade catalog entity.
+ * @param namespace the expected fully resolved namespace to be created.
+ * @param existingPath the partially resolved path currently stored in the
metastore.
+ * @return the fully resolved path wrapper.
+ */
+ private PolarisResolvedPathWrapper createSyntheticNamespaceEntities(
+ CatalogEntity catalogEntity, Namespace namespace,
PolarisResolvedPathWrapper existingPath) {
+
+ if (existingPath == null) {
+ throw new IllegalStateException(
+ String.format("Catalog entity %s does not exist.",
catalogEntity.getName()));
+ }
+
+ List<PolarisEntity> completePath = new
ArrayList<>(existingPath.getRawFullPath());
+ PolarisEntity currentParent = existingPath.getRawLeafEntity();
+
+ String[] allNamespaceLevels = namespace.levels();
+ int matchingLevel = -1;
+ for (PolarisEntity entity : completePath.subList(1, completePath.size())) {
Review Comment:
Some code comments would be nice here to explain why the `subList` starts at
1 (is it because 0 is the CatalogEntity?)
Also this all might be more readable if we shift-right by one, instead of
needing `matchingLevel + 1` in all the places that use it. Basically make it a
*count* instead of the level:
int numMatchingLevels = 0;
....
... (allNamespaceLevels[numMatchingLevels])...
...
for (int i = numMatchingLevels; ...
...
...Arrays.copyOf(allNamespaceLevels, i)...
Though the semantics of "matchingLevel" does make sense in the normal case,
relying on `matchingLevel == -1` for `i = matchingLevel + 1` in the case where
there are 0 matches is a bit jarring when reading the code.
##########
service/common/src/main/java/org/apache/polaris/service/admin/PolarisAdminService.java:
##########
@@ -2034,6 +2136,74 @@ private boolean grantPrivilegeOnTableLikeToRole(
.isSuccess();
}
+ /**
+ * Creates and persists the missing synthetic table-like entity and its
parent namespace entities
+ * for external catalogs.
+ *
+ * @param catalogEntity the external passthrough facade catalog entity.
+ * @param identifier the path of the table-like entity(including the
namespace).
+ * @param subTypes the expected subtypes of the table-like entity
+ * @param existingPathWrapper the partially resolved path currently stored
in the metastore.
+ * @return the resolved path wrapper
+ */
+ private PolarisResolvedPathWrapper createSyntheticTableLikeEntities(
+ CatalogEntity catalogEntity,
+ TableIdentifier identifier,
+ List<PolarisEntitySubType> subTypes,
+ PolarisResolvedPathWrapper existingPathWrapper) {
+
+ Namespace namespace = identifier.namespace();
+ PolarisResolvedPathWrapper resolvedNamespacePathWrapper =
+ !namespace.isEmpty()
+ ? createSyntheticNamespaceEntities(catalogEntity, namespace,
existingPathWrapper)
+ : existingPathWrapper;
+
+ if (resolvedNamespacePathWrapper == null
+ || (!namespace.isEmpty()
+ && !resolvedNamespacePathWrapper.isFullyResolvedNamespace(
+ catalogEntity.getName(), namespace))) {
+ throw new RuntimeException(
+ String.format(
+ "Failed to create synthetic namespace entities for namespace %s
in catalog %s",
+ namespace.toString(), catalogEntity.getName()));
+ }
+
+ // TODO: Instead of creating a synthetic table-like entity, rely on
external catalog mediated
+ // backfill.
+ PolarisEntity parentNamespaceEntity =
resolvedNamespacePathWrapper.getRawLeafEntity();
+ PolarisEntity syntheticTableEntity =
+ new PolarisEntity.Builder()
Review Comment:
We should try to use the
[IcebergTableLikeEntity.Builder](https://github.com/apache/polaris/blob/d6ae2c89de2178113665ffa73ab773b0108da4fe/polaris-core/src/main/java/org/apache/polaris/core/entity/table/IcebergTableLikeEntity.java#L76)
here instead of the general builder. I understand it might be a bit messy due
to the `metadataLocation` argument in the builder constructor, but as far as I
can tell it's graceful enough to accept `""` for the metadataLocation, and
actually this is kind of working as intended for the Builder to help enforce
the existence of expected fields in the entity type.
Once we have the remote-mediated backfill, we'll get to actually plug in the
real `metadataLocation` to the constructor which will benefit everyone.
Also, it's important to
[setTableIdentifier](https://github.com/apache/polaris/blob/d6ae2c89de2178113665ffa73ab773b0108da4fe/polaris-core/src/main/java/org/apache/polaris/core/entity/table/IcebergTableLikeEntity.java#L93C20-L93C38)
because `listGrants*` depends on it (otherwise, normally a leaf entity doesn't
have a representation of its parent)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@polaris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
