jngli commented on code in PR #14862:
URL: https://github.com/apache/gravitino/pull/14862#discussion_r4611445022
##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/LocationProvider.java:
##########
@@ -253,7 +265,78 @@ public interface TableLocationProvider {
* #provisionTableLocation(TableLocationContext)}, it is called concurrently
and must be
* thread-safe.
*
+ * <p>The result reports whether this call released anything. The catalog
only logs it; it does
+ * not change the outcome of the drop.
+ *
* @param context the table that is gone and the context needed to release
its location
+ * @return true if the provider released an allocation for the location,
false if there was
+ * nothing to release, such as a location it does not recognize or one
already released
+ */
+ boolean unprovisionTableLocation(TableLocationContext context);
+
+ /**
+ * Provisions the location of a schema that is being created. The default
implementation returns
+ * null.
+ *
+ * <p>The context carries the properties of the creation request, including
a caller-supplied
+ * {@code location}, if any. As with tables, what happens to a supplied
location is the provider's
+ * decision: return a location to place the schema there; return null or a
blank string to leave
+ * the location to the caller, in which case the schema is created with the
properties of the
+ * request as they are, including any {@code location} the caller supplied;
or throw to refuse the
+ * creation.
+ *
+ * <p>A location that is returned is stored verbatim as the schema's {@code
location} property.
+ * Because the built-in provider derives the location of a table from its
schema's {@code
+ * location}, recording it also keeps the schema consistent with the tables
placed under it.
+ *
+ * <p>It is called before the schema is stored, and before the catalog finds
out whether a schema
+ * of that name already exists, so it can be called for a name that is
taken; the creation then
+ * fails with {@link SchemaAlreadyExistsException}. A provider that knows
the schema exists may
+ * throw that itself. As with tables, there is no compensating unprovision
for a creation that
+ * fails after this method returned: a provider answering idempotently may
have returned the
+ * allocation of the schema that exists, and releasing it would take it from
that schema.
Review Comment:
> a provider answering idempotently may have returned the allocation of the
schema that exists, and releasing it would take it from that schema.
The provisionSchemaLocation API is not idempotent right? It's expected to
fail when schema already exists
##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/LocationProvider.java:
##########
@@ -68,6 +72,14 @@
* requests. A failure on the provisioning path fails the table creation; one
on the drop path is
* logged at WARN and nothing else happens.
*
+ * <p><b>Schemas.</b> Services that allocate storage usually do so
hierarchically: a schema is
+ * registered first, with the owner and the storage domain it belongs to, and
the paths of its
+ * tables are derived from it. A provider takes part in that by overriding
{@link
+ * #provisionSchemaLocation(SchemaLocationContext)} and {@link
+ * #unprovisionSchemaLocation(SchemaLocationContext)}. Both default to doing
nothing, so a provider
+ * that only places tables is not involved in schemas. They are called
concurrently as well, and a
Review Comment:
>They are called concurrently as well, and a failure in either fails the
schema request.
Please clarify which apis are called concurrently
##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/LocationProvider.java:
##########
@@ -253,7 +265,78 @@ public interface TableLocationProvider {
* #provisionTableLocation(TableLocationContext)}, it is called concurrently
and must be
* thread-safe.
*
+ * <p>The result reports whether this call released anything. The catalog
only logs it; it does
+ * not change the outcome of the drop.
+ *
* @param context the table that is gone and the context needed to release
its location
+ * @return true if the provider released an allocation for the location,
false if there was
+ * nothing to release, such as a location it does not recognize or one
already released
+ */
+ boolean unprovisionTableLocation(TableLocationContext context);
+
+ /**
+ * Provisions the location of a schema that is being created. The default
implementation returns
+ * null.
+ *
+ * <p>The context carries the properties of the creation request, including
a caller-supplied
+ * {@code location}, if any. As with tables, what happens to a supplied
location is the provider's
+ * decision: return a location to place the schema there; return null or a
blank string to leave
+ * the location to the caller, in which case the schema is created with the
properties of the
+ * request as they are, including any {@code location} the caller supplied;
or throw to refuse the
+ * creation.
+ *
+ * <p>A location that is returned is stored verbatim as the schema's {@code
location} property.
+ * Because the built-in provider derives the location of a table from its
schema's {@code
+ * location}, recording it also keeps the schema consistent with the tables
placed under it.
+ *
+ * <p>It is called before the schema is stored, and before the catalog finds
out whether a schema
+ * of that name already exists, so it can be called for a name that is
taken; the creation then
+ * fails with {@link SchemaAlreadyExistsException}. A provider that knows
the schema exists may
+ * throw that itself. As with tables, there is no compensating unprovision
for a creation that
Review Comment:
> As with tables, there is no compensating unprovision for a creation that
fails after this method returned
What does this mean?
##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/LocationProvider.java:
##########
@@ -253,7 +265,78 @@ public interface TableLocationProvider {
* #provisionTableLocation(TableLocationContext)}, it is called concurrently
and must be
* thread-safe.
*
+ * <p>The result reports whether this call released anything. The catalog
only logs it; it does
+ * not change the outcome of the drop.
+ *
* @param context the table that is gone and the context needed to release
its location
+ * @return true if the provider released an allocation for the location,
false if there was
+ * nothing to release, such as a location it does not recognize or one
already released
+ */
+ boolean unprovisionTableLocation(TableLocationContext context);
+
+ /**
+ * Provisions the location of a schema that is being created. The default
implementation returns
+ * null.
+ *
+ * <p>The context carries the properties of the creation request, including
a caller-supplied
+ * {@code location}, if any. As with tables, what happens to a supplied
location is the provider's
+ * decision: return a location to place the schema there; return null or a
blank string to leave
+ * the location to the caller, in which case the schema is created with the
properties of the
+ * request as they are, including any {@code location} the caller supplied;
or throw to refuse the
+ * creation.
+ *
+ * <p>A location that is returned is stored verbatim as the schema's {@code
location} property.
+ * Because the built-in provider derives the location of a table from its
schema's {@code
+ * location}, recording it also keeps the schema consistent with the tables
placed under it.
+ *
+ * <p>It is called before the schema is stored, and before the catalog finds
out whether a schema
+ * of that name already exists, so it can be called for a name that is
taken; the creation then
+ * fails with {@link SchemaAlreadyExistsException}. A provider that knows
the schema exists may
+ * throw that itself. As with tables, there is no compensating unprovision
for a creation that
+ * fails after this method returned: a provider answering idempotently may
have returned the
+ * allocation of the schema that exists, and releasing it would take it from
that schema.
+ *
+ * @param context the schema being created, with the properties of the
creation request
+ * @return the location to store on the schema, or null or blank to leave
the properties of the
+ * request unchanged
+ * @throws SchemaAlreadyExistsException if the provider knows a schema of
that name exists
+ */
+ @Nullable
+ default String provisionSchemaLocation(SchemaLocationContext context)
+ throws SchemaAlreadyExistsException {
+ return null;
+ }
+
+ /**
+ * Unprovisions the location of a schema that is being dropped. The default
implementation does
+ * nothing and returns false.
+ *
+ * <p>It is called after the tables of the schema are dropped and before the
schema itself is
Review Comment:
Too much details, suggest to remove as the implementation may change in the
future to avoid out of dated doc
##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java:
##########
@@ -443,12 +500,18 @@ private boolean dropOrPurgeTable(NameIdentifier ident,
boolean purge, Schema sch
// Use the same effective ownership as the format, including legacy Lance
registrations.
boolean external =
LancePropertiesUtils.isExternalTable(context.tableProperties());
if (dropped && (purge || !external)) {
- TableLocationProvider provider = tableLocationProvider;
+ LocationProvider provider = locationProvider;
try {
- provider.unprovisionTableLocation(context);
+ if (!provider.unprovisionTableLocation(context)) {
+ LOG.debug(
Review Comment:
```suggestion
LOG.warn(
```
##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java:
##########
@@ -218,18 +251,42 @@ public boolean dropSchema(NameIdentifier ident, boolean
cascade) throws NonEmpty
"Schema %s is not empty, cannot drop it without cascade", ident);
}
+ Schema schema = loadSchema(ident);
+
// Drop all tables under the schema first if cascade is true. This goes
through the same path
// as the catalog level dropTable, so that the location of each table is
unprovisioned and its
// cached format invalidated. The schema is resolved once for the whole
cascade rather than
- // once per table: every table here has the same parent, and the provider
may not ask for it at
- // all.
- if (tableIdents.length > 0) {
- Schema cascadedSchema = loadSchema(ident);
- for (NameIdentifier tableIdent : tableIdents) {
- dropOrPurgeTable(tableIdent, false /* purge */, cascadedSchema);
- }
+ // once per table: every table here has the same parent.
+ boolean dataRetained = false;
+ for (NameIdentifier tableIdent : tableIdents) {
+ dataRetained |=
+ dropOrPurgeTable(tableIdent, false /* purge */, schema)
+ == TableDropOutcome.DROPPED_DATA_RETAINED;
}
+ if (dataRetained) {
+ LOG.warn(
+ "Schema {} had external tables whose data was left in place, so its
location '{}' is "
+ + "not handed back to location provider '{}'.",
+ ident,
+ schema.properties().get(Schema.PROPERTY_LOCATION),
+ locationProvider.name());
+ } else {
+ boolean released =
+ locationProvider.unprovisionSchemaLocation(
+ SchemaLocationContext.builder()
+ .withSchemaIdentifier(ident)
+ .withSchemaProperties(schema.properties())
+ .withCatalogProperties(catalogProperties)
+ .build());
+ if (!released) {
+ LOG.debug(
Review Comment:
```suggestion
LOG.warn(
```
##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java:
##########
@@ -218,19 +275,40 @@ public boolean dropSchema(NameIdentifier ident, boolean
cascade) throws NonEmpty
"Schema %s is not empty, cannot drop it without cascade", ident);
}
+ TableLocationProvider provider = tableLocationProvider;
+ boolean schemaHooks = provider instanceof SupportsSchemaLocations;
+ Schema schema = null;
+ if (schemaHooks || tableIdents.length > 0) {
+ try {
+ schema = loadSchema(ident);
+ } catch (NoSuchSchemaException e) {
+ return false;
+ }
+ }
+
// Drop all tables under the schema first if cascade is true. This goes
through the same path
// as the catalog level dropTable, so that the location of each table is
unprovisioned and its
// cached format invalidated. The schema is resolved once for the whole
cascade rather than
// once per table: every table here has the same parent, and the provider
may not ask for it at
// all.
- if (tableIdents.length > 0) {
- Schema cascadedSchema = loadSchema(ident);
- for (NameIdentifier tableIdent : tableIdents) {
- dropOrPurgeTable(tableIdent, false /* purge */, cascadedSchema);
- }
+ List<NameIdentifier> tablesWithRetainedData = Lists.newArrayList();
+ for (NameIdentifier tableIdent : tableIdents) {
+ dropOrPurgeTable(tableIdent, false /* purge */, schema,
tablesWithRetainedData);
}
- return schemaOps.dropSchema(ident, cascade);
+ boolean dropped = schemaOps.dropSchema(ident, cascade);
+ if (dropped && schemaHooks) {
+ unprovisionSchemaLocation(
Review Comment:
Thanks for the details. Let's leave it for now.
##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/SchemaLocationContext.java:
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.lakehouse.generic;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Maps;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.gravitino.NameIdentifier;
+
+/**
+ * The context passed to a {@link SupportsSchemaLocations} provider on both of
its callbacks: when a
+ * schema is being created, and when a schema is being dropped.
+ *
+ * <p>As with {@link TableLocationContext}, one type serves both callbacks so
that a new input
+ * becomes an additional accessor rather than a signature change. Build
instances through {@link
+ * #builder()}.
+ */
+public class SchemaLocationContext {
+
+ private final NameIdentifier schemaIdentifier;
+
+ private final Map<String, String> schemaProperties;
+
+ private final Map<String, String> catalogProperties;
Review Comment:
What kind of properties a location provider requires from catalog? I've same
question for TableLocationContext
What's service endpoint or credential reference used for location provider?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]