jngli commented on code in PR #14862:
URL: https://github.com/apache/gravitino/pull/14862#discussion_r4611445022


##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/LocationProvider.java:
##########
@@ -253,7 +265,78 @@ public interface TableLocationProvider {
    * #provisionTableLocation(TableLocationContext)}, it is called concurrently 
and must be
    * thread-safe.
    *
+   * <p>The result reports whether this call released anything. The catalog 
only logs it; it does
+   * not change the outcome of the drop.
+   *
    * @param context the table that is gone and the context needed to release 
its location
+   * @return true if the provider released an allocation for the location, 
false if there was
+   *     nothing to release, such as a location it does not recognize or one 
already released
+   */
+  boolean unprovisionTableLocation(TableLocationContext context);
+
+  /**
+   * Provisions the location of a schema that is being created. The default 
implementation returns
+   * null.
+   *
+   * <p>The context carries the properties of the creation request, including 
a caller-supplied
+   * {@code location}, if any. As with tables, what happens to a supplied 
location is the provider's
+   * decision: return a location to place the schema there; return null or a 
blank string to leave
+   * the location to the caller, in which case the schema is created with the 
properties of the
+   * request as they are, including any {@code location} the caller supplied; 
or throw to refuse the
+   * creation.
+   *
+   * <p>A location that is returned is stored verbatim as the schema's {@code 
location} property.
+   * Because the built-in provider derives the location of a table from its 
schema's {@code
+   * location}, recording it also keeps the schema consistent with the tables 
placed under it.
+   *
+   * <p>It is called before the schema is stored, and before the catalog finds 
out whether a schema
+   * of that name already exists, so it can be called for a name that is 
taken; the creation then
+   * fails with {@link SchemaAlreadyExistsException}. A provider that knows 
the schema exists may
+   * throw that itself. As with tables, there is no compensating unprovision 
for a creation that
+   * fails after this method returned: a provider answering idempotently may 
have returned the
+   * allocation of the schema that exists, and releasing it would take it from 
that schema.

Review Comment:
   > a provider answering idempotently may have returned the allocation of the 
schema that exists, and releasing it would take it from that schema.
   
   The provisionSchemaLocation API is not idempotent right? It's expected to 
fail when schema already exists



##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/LocationProvider.java:
##########
@@ -68,6 +72,14 @@
  * requests. A failure on the provisioning path fails the table creation; one 
on the drop path is
  * logged at WARN and nothing else happens.
  *
+ * <p><b>Schemas.</b> Services that allocate storage usually do so 
hierarchically: a schema is
+ * registered first, with the owner and the storage domain it belongs to, and 
the paths of its
+ * tables are derived from it. A provider takes part in that by overriding 
{@link
+ * #provisionSchemaLocation(SchemaLocationContext)} and {@link
+ * #unprovisionSchemaLocation(SchemaLocationContext)}. Both default to doing 
nothing, so a provider
+ * that only places tables is not involved in schemas. They are called 
concurrently as well, and a

Review Comment:
   >They are called concurrently as well, and a failure in either fails the 
schema request.
   
   Please clarify which apis are called concurrently



##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/LocationProvider.java:
##########
@@ -253,7 +265,78 @@ public interface TableLocationProvider {
    * #provisionTableLocation(TableLocationContext)}, it is called concurrently 
and must be
    * thread-safe.
    *
+   * <p>The result reports whether this call released anything. The catalog 
only logs it; it does
+   * not change the outcome of the drop.
+   *
    * @param context the table that is gone and the context needed to release 
its location
+   * @return true if the provider released an allocation for the location, 
false if there was
+   *     nothing to release, such as a location it does not recognize or one 
already released
+   */
+  boolean unprovisionTableLocation(TableLocationContext context);
+
+  /**
+   * Provisions the location of a schema that is being created. The default 
implementation returns
+   * null.
+   *
+   * <p>The context carries the properties of the creation request, including 
a caller-supplied
+   * {@code location}, if any. As with tables, what happens to a supplied 
location is the provider's
+   * decision: return a location to place the schema there; return null or a 
blank string to leave
+   * the location to the caller, in which case the schema is created with the 
properties of the
+   * request as they are, including any {@code location} the caller supplied; 
or throw to refuse the
+   * creation.
+   *
+   * <p>A location that is returned is stored verbatim as the schema's {@code 
location} property.
+   * Because the built-in provider derives the location of a table from its 
schema's {@code
+   * location}, recording it also keeps the schema consistent with the tables 
placed under it.
+   *
+   * <p>It is called before the schema is stored, and before the catalog finds 
out whether a schema
+   * of that name already exists, so it can be called for a name that is 
taken; the creation then
+   * fails with {@link SchemaAlreadyExistsException}. A provider that knows 
the schema exists may
+   * throw that itself. As with tables, there is no compensating unprovision 
for a creation that

Review Comment:
   > As with tables, there is no compensating unprovision for a creation that 
fails after this method returned
   
   What does this mean?



##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/LocationProvider.java:
##########
@@ -253,7 +265,78 @@ public interface TableLocationProvider {
    * #provisionTableLocation(TableLocationContext)}, it is called concurrently 
and must be
    * thread-safe.
    *
+   * <p>The result reports whether this call released anything. The catalog 
only logs it; it does
+   * not change the outcome of the drop.
+   *
    * @param context the table that is gone and the context needed to release 
its location
+   * @return true if the provider released an allocation for the location, 
false if there was
+   *     nothing to release, such as a location it does not recognize or one 
already released
+   */
+  boolean unprovisionTableLocation(TableLocationContext context);
+
+  /**
+   * Provisions the location of a schema that is being created. The default 
implementation returns
+   * null.
+   *
+   * <p>The context carries the properties of the creation request, including 
a caller-supplied
+   * {@code location}, if any. As with tables, what happens to a supplied 
location is the provider's
+   * decision: return a location to place the schema there; return null or a 
blank string to leave
+   * the location to the caller, in which case the schema is created with the 
properties of the
+   * request as they are, including any {@code location} the caller supplied; 
or throw to refuse the
+   * creation.
+   *
+   * <p>A location that is returned is stored verbatim as the schema's {@code 
location} property.
+   * Because the built-in provider derives the location of a table from its 
schema's {@code
+   * location}, recording it also keeps the schema consistent with the tables 
placed under it.
+   *
+   * <p>It is called before the schema is stored, and before the catalog finds 
out whether a schema
+   * of that name already exists, so it can be called for a name that is 
taken; the creation then
+   * fails with {@link SchemaAlreadyExistsException}. A provider that knows 
the schema exists may
+   * throw that itself. As with tables, there is no compensating unprovision 
for a creation that
+   * fails after this method returned: a provider answering idempotently may 
have returned the
+   * allocation of the schema that exists, and releasing it would take it from 
that schema.
+   *
+   * @param context the schema being created, with the properties of the 
creation request
+   * @return the location to store on the schema, or null or blank to leave 
the properties of the
+   *     request unchanged
+   * @throws SchemaAlreadyExistsException if the provider knows a schema of 
that name exists
+   */
+  @Nullable
+  default String provisionSchemaLocation(SchemaLocationContext context)
+      throws SchemaAlreadyExistsException {
+    return null;
+  }
+
+  /**
+   * Unprovisions the location of a schema that is being dropped. The default 
implementation does
+   * nothing and returns false.
+   *
+   * <p>It is called after the tables of the schema are dropped and before the 
schema itself is

Review Comment:
   Too much details, suggest to remove as the implementation may change in the 
future to avoid out of dated doc



##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java:
##########
@@ -443,12 +500,18 @@ private boolean dropOrPurgeTable(NameIdentifier ident, 
boolean purge, Schema sch
     // Use the same effective ownership as the format, including legacy Lance 
registrations.
     boolean external = 
LancePropertiesUtils.isExternalTable(context.tableProperties());
     if (dropped && (purge || !external)) {
-      TableLocationProvider provider = tableLocationProvider;
+      LocationProvider provider = locationProvider;
       try {
-        provider.unprovisionTableLocation(context);
+        if (!provider.unprovisionTableLocation(context)) {
+          LOG.debug(

Review Comment:
   ```suggestion
             LOG.warn(
   ```



##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java:
##########
@@ -218,18 +251,42 @@ public boolean dropSchema(NameIdentifier ident, boolean 
cascade) throws NonEmpty
           "Schema %s is not empty, cannot drop it without cascade", ident);
     }
 
+    Schema schema = loadSchema(ident);
+
     // Drop all tables under the schema first if cascade is true. This goes 
through the same path
     // as the catalog level dropTable, so that the location of each table is 
unprovisioned and its
     // cached format invalidated. The schema is resolved once for the whole 
cascade rather than
-    // once per table: every table here has the same parent, and the provider 
may not ask for it at
-    // all.
-    if (tableIdents.length > 0) {
-      Schema cascadedSchema = loadSchema(ident);
-      for (NameIdentifier tableIdent : tableIdents) {
-        dropOrPurgeTable(tableIdent, false /* purge */, cascadedSchema);
-      }
+    // once per table: every table here has the same parent.
+    boolean dataRetained = false;
+    for (NameIdentifier tableIdent : tableIdents) {
+      dataRetained |=
+          dropOrPurgeTable(tableIdent, false /* purge */, schema)
+              == TableDropOutcome.DROPPED_DATA_RETAINED;
     }
 
+    if (dataRetained) {
+      LOG.warn(
+          "Schema {} had external tables whose data was left in place, so its 
location '{}' is "
+              + "not handed back to location provider '{}'.",
+          ident,
+          schema.properties().get(Schema.PROPERTY_LOCATION),
+          locationProvider.name());
+    } else {
+      boolean released =
+          locationProvider.unprovisionSchemaLocation(
+              SchemaLocationContext.builder()
+                  .withSchemaIdentifier(ident)
+                  .withSchemaProperties(schema.properties())
+                  .withCatalogProperties(catalogProperties)
+                  .build());
+      if (!released) {
+        LOG.debug(

Review Comment:
   ```suggestion
           LOG.warn(
   ```



##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java:
##########
@@ -218,19 +275,40 @@ public boolean dropSchema(NameIdentifier ident, boolean 
cascade) throws NonEmpty
           "Schema %s is not empty, cannot drop it without cascade", ident);
     }
 
+    TableLocationProvider provider = tableLocationProvider;
+    boolean schemaHooks = provider instanceof SupportsSchemaLocations;
+    Schema schema = null;
+    if (schemaHooks || tableIdents.length > 0) {
+      try {
+        schema = loadSchema(ident);
+      } catch (NoSuchSchemaException e) {
+        return false;
+      }
+    }
+
     // Drop all tables under the schema first if cascade is true. This goes 
through the same path
     // as the catalog level dropTable, so that the location of each table is 
unprovisioned and its
     // cached format invalidated. The schema is resolved once for the whole 
cascade rather than
     // once per table: every table here has the same parent, and the provider 
may not ask for it at
     // all.
-    if (tableIdents.length > 0) {
-      Schema cascadedSchema = loadSchema(ident);
-      for (NameIdentifier tableIdent : tableIdents) {
-        dropOrPurgeTable(tableIdent, false /* purge */, cascadedSchema);
-      }
+    List<NameIdentifier> tablesWithRetainedData = Lists.newArrayList();
+    for (NameIdentifier tableIdent : tableIdents) {
+      dropOrPurgeTable(tableIdent, false /* purge */, schema, 
tablesWithRetainedData);
     }
 
-    return schemaOps.dropSchema(ident, cascade);
+    boolean dropped = schemaOps.dropSchema(ident, cascade);
+    if (dropped && schemaHooks) {
+      unprovisionSchemaLocation(

Review Comment:
   Thanks for the details.  Let's leave it for now.



##########
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/SchemaLocationContext.java:
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.lakehouse.generic;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Maps;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.gravitino.NameIdentifier;
+
+/**
+ * The context passed to a {@link SupportsSchemaLocations} provider on both of 
its callbacks: when a
+ * schema is being created, and when a schema is being dropped.
+ *
+ * <p>As with {@link TableLocationContext}, one type serves both callbacks so 
that a new input
+ * becomes an additional accessor rather than a signature change. Build 
instances through {@link
+ * #builder()}.
+ */
+public class SchemaLocationContext {
+
+  private final NameIdentifier schemaIdentifier;
+
+  private final Map<String, String> schemaProperties;
+
+  private final Map<String, String> catalogProperties;

Review Comment:
   What kind of properties a location provider requires from catalog? I've same 
question for TableLocationContext
   
   What's service endpoint or credential reference used for location provider?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to