This is an automated email from the ASF dual-hosted git repository.
diqiu50 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 2d7607f77b [#13353] fix(connector): Declare shared cloud credential
properties on every catalog (#13354)
2d7607f77b is described below
commit 2d7607f77bc15ddd6848cc481907e4546a094b93
Author: MaSai <[email protected]>
AuthorDate: Mon Sep 21 20:28:45 2026 +0800
[#13353] fix(connector): Declare shared cloud credential properties on
every catalog (#13354)
### What changes were proposed in this pull request?
Declare the shared S3, OSS, Azure, GCS, COS, and AWS credential property
entries on every catalog from `BaseCatalogPropertiesMetadata`. A catalog
that already declares a key keeps its own entry.
- Add `AWSPropertiesMetadata` for `aws-access-key-id` (`hidden=false`)
and `aws-secret-access-key` (`hidden=true`). Glue references these
entries. `aws-region`, `aws-glue-catalog-id`, and `aws-glue-endpoint`
stay on the Glue catalog. `aws-region` is required and is not merged
into other catalogs.
- `FallbackPropertiesMetadata` also registers the AWS entries, alongside
the cloud entries it already had. That path is only used when a catalog
does not implement properties metadata for an entity type.
- JDBC `jdbc-user` / `jdbc-password` and Paimon REST/DLF keys stay on
their own catalogs. Engine runtime keys are not registered.
### Why are the changes needed?
Fix: #13353
`HiddenPropertyMaskUtils` fuzzy-masks a key only when the catalog has
not declared it and the name looks sensitive. Glue stores
`aws-access-key-id` (declared, cleartext) and can also have
`s3-access-key-id`. Iceberg already declares `s3-access-key-id` with
`hidden=false`, so the same name was cleartext there and `******` on
Glue. Declaring the shared cloud entries makes those official keys
follow their existing `hidden` flag on every catalog.
### Does this PR introduce _any_ user-facing change?
Catalogs that did not already declare these keys (Glue, JDBC, Kafka,
Model, Hudi, and the generic catalog) now treat the shared cloud keys as
known properties. Keys with `hidden=false` whose names contain words
such as `access`, `token`, or `account` are returned in cleartext
instead of being fuzzy-masked. Secret keys such as
`s3-secret-access-key` stay masked as `******`. No new property is
required. Hive, Iceberg, Paimon, and Fileset already declared the
S3/OSS/Azure/GCS/COS entries; they gain the AWS access-key pair.
### How was this patch tested?
- `./gradlew :core:test --tests
org.apache.gravitino.connector.TestBaseCatalogPropertiesMetadata --tests
org.apache.gravitino.secret.TestSecretPropertyOperationDispatcher
-PskipITs`
- `./gradlew :common:test --tests
org.apache.gravitino.credential.TestCredentialPropertiesUtils -PskipITs`
- `./gradlew :catalogs:catalog-glue:test --tests
org.apache.gravitino.catalog.glue.TestGlueCatalogPropertiesMetadata
-PskipITs`
---------
Co-authored-by: Cursor <[email protected]>
---
.../gravitino/catalog/glue/GlueConstants.java | 11 ++-
.../apache/gravitino/storage/AWSProperties.java | 31 +++++++++
.../fileset/FilesetCatalogPropertiesMetadata.java | 11 ---
.../catalog/fileset/FilesetPropertiesMetadata.java | 12 +---
.../fileset/FilesetSchemaPropertiesMetadata.java | 12 +---
.../TestFilesetCloudPropertiesMetadata.java | 14 ++++
.../glue/GlueCatalogPropertiesMetadata.java | 21 +-----
.../glue/TestGlueCatalogPropertiesMetadata.java | 8 +++
.../hive/HiveCatalogPropertiesMetadata.java | 10 ---
.../iceberg/IcebergCatalogPropertiesMetadata.java | 10 ---
.../paimon/PaimonCatalogPropertiesMetadata.java | 10 ---
.../cloud/storage/AWSPropertiesMetadata.java | 59 ++++++++++++++++
.../cloud/storage/CloudPropertiesMetadata.java | 54 +++++++++++++++
.../connector/BaseCatalogPropertiesMetadata.java | 25 ++++++-
.../secret/FallbackPropertiesMetadata.java | 29 ++++----
.../cloud/storage/TestCloudPropertiesMetadata.java | 39 +++++++++++
.../TestBaseCatalogPropertiesMetadata.java | 80 ++++++++++++++++++++++
.../TestSecretPropertyOperationDispatcher.java | 8 +++
design-docs/gravitino-glue-catalog.md | 18 ++---
19 files changed, 351 insertions(+), 111 deletions(-)
diff --git
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/catalog/glue/GlueConstants.java
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/catalog/glue/GlueConstants.java
index 2cd1f32cf3..7e1c6d4791 100644
---
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/catalog/glue/GlueConstants.java
+++
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/catalog/glue/GlueConstants.java
@@ -18,6 +18,8 @@
*/
package org.apache.gravitino.catalog.glue;
+import org.apache.gravitino.storage.AWSProperties;
+
/** Constant keys for the AWS Glue Data Catalog connector configuration and
table properties. */
public final class GlueConstants {
@@ -34,11 +36,14 @@ public final class GlueConstants {
*/
public static final String AWS_GLUE_CATALOG_ID = "aws-glue-catalog-id";
- /** AWS access key ID for static credential authentication (optional,
sensitive). */
- public static final String AWS_ACCESS_KEY_ID = "aws-access-key-id";
+ /**
+ * AWS access key ID for static credential authentication (optional, not
hidden). This is an
+ * account identifier, not a secret; {@link #AWS_SECRET_ACCESS_KEY} is the
hidden half.
+ */
+ public static final String AWS_ACCESS_KEY_ID =
AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID;
/** AWS secret access key for static credential authentication (optional,
sensitive). */
- public static final String AWS_SECRET_ACCESS_KEY = "aws-secret-access-key";
+ public static final String AWS_SECRET_ACCESS_KEY =
AWSProperties.GRAVITINO_AWS_SECRET_ACCESS_KEY;
/**
* Custom Glue endpoint URL (optional). Used for VPC endpoints or LocalStack
testing. Example:
diff --git
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/AWSProperties.java
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/AWSProperties.java
new file mode 100644
index 0000000000..e42842b4f6
--- /dev/null
+++
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/AWSProperties.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.storage;
+
+/** Property names for AWS static credentials shared across catalogs. */
+public final class AWSProperties {
+
+ /** AWS access key ID. Not hidden. */
+ public static final String GRAVITINO_AWS_ACCESS_KEY_ID = "aws-access-key-id";
+
+ /** AWS secret access key. Hidden. */
+ public static final String GRAVITINO_AWS_SECRET_ACCESS_KEY =
"aws-secret-access-key";
+
+ private AWSProperties() {}
+}
diff --git
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogPropertiesMetadata.java
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogPropertiesMetadata.java
index b71a3a34ac..346170a276 100644
---
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogPropertiesMetadata.java
@@ -33,11 +33,6 @@ import com.google.common.collect.ImmutableMap;
import java.util.Map;
import org.apache.gravitino.catalog.hadoop.fs.FileSystemProvider;
import org.apache.gravitino.catalog.hadoop.fs.LocalFileSystemProvider;
-import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
-import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
import org.apache.gravitino.connector.BaseCatalogPropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
@@ -227,12 +222,6 @@ public class FilesetCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetad
// The following two are about authentication.
.putAll(KERBEROS_PROPERTY_ENTRIES)
.putAll(AUTHENTICATION_PROPERTY_ENTRIES)
- // CREDENTIAL_PROPERTY_ENTRIES are registered in
BaseCatalogPropertiesMetadata.
- .putAll(S3PropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(COSPropertiesMetadata.PROPERTY_ENTRIES)
.build();
@Override
diff --git
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetPropertiesMetadata.java
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetPropertiesMetadata.java
index bc722f5060..8f01268adf 100644
---
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetPropertiesMetadata.java
+++
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetPropertiesMetadata.java
@@ -28,11 +28,7 @@ import static
org.apache.gravitino.file.Fileset.PROPERTY_SCHEMA_PLACEHOLDER;
import com.google.common.collect.ImmutableMap;
import java.util.Map;
-import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
-import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
+import org.apache.gravitino.cloud.storage.CloudPropertiesMetadata;
import org.apache.gravitino.connector.BasePropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
@@ -79,11 +75,7 @@ public class FilesetPropertiesMetadata extends
BasePropertiesMetadata {
false /* hidden */))
.putAll(KERBEROS_PROPERTY_ENTRIES)
.putAll(AUTHENTICATION_PROPERTY_ENTRIES)
- .putAll(S3PropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(COSPropertiesMetadata.PROPERTY_ENTRIES);
+ .putAll(CloudPropertiesMetadata.STORAGE_PROPERTY_ENTRIES);
return builder.build();
}
}
diff --git
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetSchemaPropertiesMetadata.java
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetSchemaPropertiesMetadata.java
index 7515d90556..4cd4d285d3 100644
---
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetSchemaPropertiesMetadata.java
+++
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetSchemaPropertiesMetadata.java
@@ -25,11 +25,7 @@ import static
org.apache.gravitino.file.Fileset.PROPERTY_MULTIPLE_LOCATIONS_PREF
import com.google.common.collect.ImmutableMap;
import java.util.Map;
-import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
-import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
+import org.apache.gravitino.cloud.storage.CloudPropertiesMetadata;
import org.apache.gravitino.connector.BasePropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
@@ -72,11 +68,7 @@ public class FilesetSchemaPropertiesMetadata extends
BasePropertiesMetadata {
false /* reserved */))
.putAll(KERBEROS_PROPERTY_ENTRIES)
.putAll(AUTHENTICATION_PROPERTY_ENTRIES)
- .putAll(S3PropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(COSPropertiesMetadata.PROPERTY_ENTRIES)
+ .putAll(CloudPropertiesMetadata.STORAGE_PROPERTY_ENTRIES)
.build();
@Override
diff --git
a/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCloudPropertiesMetadata.java
b/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCloudPropertiesMetadata.java
index 515d6849ae..fe20c37d71 100644
---
a/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCloudPropertiesMetadata.java
+++
b/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCloudPropertiesMetadata.java
@@ -26,6 +26,7 @@ import com.google.common.collect.ImmutableMap;
import java.util.Map;
import org.apache.gravitino.connector.HiddenPropertyMaskUtils;
import org.apache.gravitino.connector.PropertiesMetadata;
+import org.apache.gravitino.storage.AWSProperties;
import org.apache.gravitino.storage.AzureProperties;
import org.apache.gravitino.storage.COSProperties;
import org.apache.gravitino.storage.GCSProperties;
@@ -89,4 +90,17 @@ public class TestFilesetCloudPropertiesMetadata {
HiddenPropertyMaskUtils.MASKED_VALUE,
response.get(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
}
+
+ @Test
+ void testAwsAccessKeyIsDeclaredOnlyOnTheFilesetCatalog() {
+ assertTrue(
+ new FilesetCatalogPropertiesMetadata()
+ .containsProperty(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID));
+ assertFalse(
+ new FilesetSchemaPropertiesMetadata()
+ .containsProperty(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID));
+ assertFalse(
+ new FilesetPropertiesMetadata()
+ .containsProperty(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID));
+ }
}
diff --git
a/catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogPropertiesMetadata.java
b/catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogPropertiesMetadata.java
index 1b485374ee..5e3f6ea92e 100644
---
a/catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogPropertiesMetadata.java
@@ -33,6 +33,7 @@ import static
org.apache.gravitino.connector.PropertyEntry.stringRequiredPropert
import com.google.common.collect.ImmutableMap;
import java.util.Map;
+import org.apache.gravitino.cloud.storage.AWSPropertiesMetadata;
import org.apache.gravitino.connector.BaseCatalogPropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
@@ -57,24 +58,8 @@ public class GlueCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetadata
true /* immutable */,
null /* defaultValue */,
false /* hidden */))
- .put(
- AWS_ACCESS_KEY_ID,
- stringOptionalPropertyEntry(
- AWS_ACCESS_KEY_ID,
- "AWS access key ID for static credential authentication."
- + " When omitted the default credential chain is used.",
- false /* immutable */,
- null /* defaultValue */,
- false /* hidden */))
- .put(
- AWS_SECRET_ACCESS_KEY,
- stringOptionalPropertyEntry(
- AWS_SECRET_ACCESS_KEY,
- "AWS secret access key paired with aws-access-key-id."
- + " When omitted the default credential chain is used.",
- false /* immutable */,
- null /* defaultValue */,
- true /* hidden */))
+ .put(AWS_ACCESS_KEY_ID, AWSPropertiesMetadata.AWS_ACCESS_KEY_ID)
+ .put(AWS_SECRET_ACCESS_KEY,
AWSPropertiesMetadata.AWS_SECRET_ACCESS_KEY)
.put(
AWS_GLUE_ENDPOINT,
stringOptionalPropertyEntry(
diff --git
a/catalogs/catalog-glue/src/test/java/org/apache/gravitino/catalog/glue/TestGlueCatalogPropertiesMetadata.java
b/catalogs/catalog-glue/src/test/java/org/apache/gravitino/catalog/glue/TestGlueCatalogPropertiesMetadata.java
index 328b02bb0e..eed35fa2fc 100644
---
a/catalogs/catalog-glue/src/test/java/org/apache/gravitino/catalog/glue/TestGlueCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-glue/src/test/java/org/apache/gravitino/catalog/glue/TestGlueCatalogPropertiesMetadata.java
@@ -30,8 +30,10 @@ import static
org.apache.gravitino.catalog.glue.GlueConstants.TABLE_FORMAT_FILTE
import static org.apache.gravitino.catalog.glue.GlueConstants.WAREHOUSE;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import org.apache.gravitino.cloud.storage.AWSPropertiesMetadata;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -76,6 +78,12 @@ class TestGlueCatalogPropertiesMetadata {
// Access key ID is an identifier, same as s3-access-key-id; only the
secret is hidden.
assertFalse(metadata.isHiddenProperty(AWS_ACCESS_KEY_ID));
assertTrue(metadata.isHiddenProperty(AWS_SECRET_ACCESS_KEY));
+ // Same PropertyEntry instances as the shared definition, not a Glue-local
copy.
+ assertSame(
+ AWSPropertiesMetadata.AWS_ACCESS_KEY_ID,
metadata.propertyEntries().get(AWS_ACCESS_KEY_ID));
+ assertSame(
+ AWSPropertiesMetadata.AWS_SECRET_ACCESS_KEY,
+ metadata.propertyEntries().get(AWS_SECRET_ACCESS_KEY));
}
@Test
diff --git
a/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
b/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
index 889e450ea1..2577bdf944 100644
---
a/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
@@ -23,11 +23,6 @@ import static
org.apache.gravitino.catalog.hive.HiveConstants.HIVE_DEFAULT_CATAL
import com.google.common.collect.ImmutableMap;
import java.util.Map;
-import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
-import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
import org.apache.gravitino.connector.BaseCatalogPropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
import org.apache.gravitino.hive.ClientPropertiesMetadata;
@@ -128,11 +123,6 @@ public class HiveCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetadata
DEFAULT_LIST_ALL_TABLES,
false /* hidden */,
false /* reserved */))
- .putAll(S3PropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(COSPropertiesMetadata.PROPERTY_ENTRIES)
.putAll(CLIENT_PROPERTIES_METADATA.propertyEntries())
.build();
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogPropertiesMetadata.java
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogPropertiesMetadata.java
index 4383b8672c..407af67a33 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogPropertiesMetadata.java
@@ -29,11 +29,6 @@ import com.google.common.collect.Maps;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
-import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
import org.apache.gravitino.connector.BaseCatalogPropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
import org.apache.gravitino.iceberg.common.authentication.AuthenticationConfig;
@@ -140,11 +135,6 @@ public class IcebergCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetad
false /* hidden */));
HashMap<String, PropertyEntry<?>> result = Maps.newHashMap();
result.putAll(Maps.uniqueIndex(propertyEntries, PropertyEntry::getName));
- result.putAll(S3PropertiesMetadata.PROPERTY_ENTRIES);
- result.putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES);
- result.putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES);
- result.putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES);
- result.putAll(COSPropertiesMetadata.PROPERTY_ENTRIES);
result.putAll(KerberosConfig.KERBEROS_PROPERTY_ENTRIES);
result.putAll(AuthenticationConfig.AUTHENTICATION_PROPERTY_ENTRIES);
PROPERTIES_METADATA = ImmutableMap.copyOf(result);
diff --git
a/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
b/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
index 5398d71d3b..2092d53ced 100644
---
a/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
@@ -32,11 +32,6 @@ import java.util.Map;
import
org.apache.gravitino.catalog.lakehouse.paimon.authentication.AuthenticationConfig;
import
org.apache.gravitino.catalog.lakehouse.paimon.authentication.kerberos.KerberosConfig;
import org.apache.gravitino.catalog.lakehouse.paimon.utils.CatalogUtils;
-import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
-import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
import org.apache.gravitino.connector.BaseCatalogPropertiesMetadata;
import org.apache.gravitino.connector.PropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
@@ -217,11 +212,6 @@ public class PaimonCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetada
result.putAll(Maps.uniqueIndex(propertyEntries, PropertyEntry::getName));
result.putAll(KerberosConfig.KERBEROS_PROPERTY_ENTRIES);
result.putAll(AuthenticationConfig.AUTHENTICATION_PROPERTY_ENTRIES);
- result.putAll(S3PropertiesMetadata.PROPERTY_ENTRIES);
- result.putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES);
- result.putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES);
- result.putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES);
- result.putAll(COSPropertiesMetadata.PROPERTY_ENTRIES);
result.putAll(REST_PROPERTY_ENTRIES);
PROPERTIES_METADATA = ImmutableMap.copyOf(result);
}
diff --git
a/core/src/main/java/org/apache/gravitino/cloud/storage/AWSPropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/cloud/storage/AWSPropertiesMetadata.java
new file mode 100644
index 0000000000..4c8cb94df4
--- /dev/null
+++
b/core/src/main/java/org/apache/gravitino/cloud/storage/AWSPropertiesMetadata.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.cloud.storage;
+
+import static
org.apache.gravitino.connector.PropertyEntry.stringOptionalPropertyEntry;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.Map;
+import org.apache.gravitino.connector.PropertyEntry;
+import org.apache.gravitino.storage.AWSProperties;
+
+/** Shared AWS credential {@link PropertyEntry} definitions for catalog
properties metadata. */
+public final class AWSPropertiesMetadata {
+
+ /** AWS access key ID. Not hidden. */
+ public static final PropertyEntry<String> AWS_ACCESS_KEY_ID =
+ stringOptionalPropertyEntry(
+ AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID,
+ "AWS access key ID for static credential authentication."
+ + " When omitted the default credential chain is used.",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */);
+
+ /** AWS secret access key. Hidden. */
+ public static final PropertyEntry<String> AWS_SECRET_ACCESS_KEY =
+ stringOptionalPropertyEntry(
+ AWSProperties.GRAVITINO_AWS_SECRET_ACCESS_KEY,
+ "AWS secret access key paired with aws-access-key-id."
+ + " When omitted the default credential chain is used.",
+ false /* immutable */,
+ null /* defaultValue */,
+ true /* hidden */);
+
+ /** AWS credential keys merged into every catalog's properties metadata. */
+ public static final Map<String, PropertyEntry<?>> PROPERTY_ENTRIES =
+ ImmutableMap.<String, PropertyEntry<?>>builder()
+ .put(AWS_ACCESS_KEY_ID.getName(), AWS_ACCESS_KEY_ID)
+ .put(AWS_SECRET_ACCESS_KEY.getName(), AWS_SECRET_ACCESS_KEY)
+ .build();
+
+ private AWSPropertiesMetadata() {}
+}
diff --git
a/core/src/main/java/org/apache/gravitino/cloud/storage/CloudPropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/cloud/storage/CloudPropertiesMetadata.java
new file mode 100644
index 0000000000..384ef4ed76
--- /dev/null
+++
b/core/src/main/java/org/apache/gravitino/cloud/storage/CloudPropertiesMetadata.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.cloud.storage;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.Map;
+import org.apache.gravitino.connector.PropertyEntry;
+
+/**
+ * Combined cloud {@link PropertyEntry} maps so callers do not repeat the same
{@code putAll} list.
+ */
+public final class CloudPropertiesMetadata {
+
+ /**
+ * S3, OSS, Azure, GCS, and COS entries. Fileset and schema metadata use
this set. It does not
+ * include the AWS access-key pair, which is a Glue catalog property.
+ */
+ public static final Map<String, PropertyEntry<?>> STORAGE_PROPERTY_ENTRIES =
+ ImmutableMap.<String, PropertyEntry<?>>builder()
+ .putAll(S3PropertiesMetadata.PROPERTY_ENTRIES)
+ .putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
+ .putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
+ .putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES)
+ .putAll(COSPropertiesMetadata.PROPERTY_ENTRIES)
+ .build();
+
+ /**
+ * Storage entries plus the AWS access-key pair. Merged into every catalog.
A catalog that already
+ * declares a key keeps its own entry.
+ */
+ public static final Map<String, PropertyEntry<?>> ALL_PROPERTY_ENTRIES =
+ ImmutableMap.<String, PropertyEntry<?>>builder()
+ .putAll(STORAGE_PROPERTY_ENTRIES)
+ .putAll(AWSPropertiesMetadata.PROPERTY_ENTRIES)
+ .build();
+
+ private CloudPropertiesMetadata() {}
+}
diff --git
a/core/src/main/java/org/apache/gravitino/connector/BaseCatalogPropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/connector/BaseCatalogPropertiesMetadata.java
index 5e7e660504..bee325129b 100644
---
a/core/src/main/java/org/apache/gravitino/connector/BaseCatalogPropertiesMetadata.java
+++
b/core/src/main/java/org/apache/gravitino/connector/BaseCatalogPropertiesMetadata.java
@@ -29,9 +29,12 @@ import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Maps;
import java.util.Collections;
+import java.util.HashSet;
import java.util.Map;
+import java.util.Set;
import org.apache.gravitino.Catalog;
import org.apache.gravitino.annotation.Evolving;
+import org.apache.gravitino.cloud.storage.CloudPropertiesMetadata;
@Evolving
public abstract class BaseCatalogPropertiesMetadata extends
BasePropertiesMetadata {
@@ -102,21 +105,37 @@ public abstract class BaseCatalogPropertiesMetadata
extends BasePropertiesMetada
true /* hidden */)),
PropertyEntry::getName);
+ /**
+ * Cloud credential keys merged into every catalog. A catalog that already
declares a key wins.
+ */
+ private static final Map<String, PropertyEntry<?>> CLOUD_PROPERTY_ENTRIES =
+ CloudPropertiesMetadata.ALL_PROPERTY_ENTRIES;
+
@Override
public Map<String, PropertyEntry<?>> propertyEntries() {
if (propertyEntries == null) {
synchronized (this) {
if (propertyEntries == null) {
- // Reuse BasePropertiesMetadata (specific + BASIC +
CredentialConfig), then add
- // catalog-only entries.
+ // Reuse BasePropertiesMetadata (specific + BASIC +
CredentialConfig), then add shared
+ // cloud credential keys and catalog-only entries.
Map<String, PropertyEntry<?>> base = buildBasePropertyEntries();
ImmutableMap.Builder<String, PropertyEntry<?>> builder =
ImmutableMap.builder();
builder.putAll(base);
+ // Track keys already placed so a later collision reports "Property
metadata already
+ // exists" instead of Guava's "Multiple entries with same key" from
builder.build().
+ Set<String> placed = new HashSet<>(base.keySet());
+
+ CLOUD_PROPERTY_ENTRIES.forEach(
+ (name, entry) -> {
+ if (placed.add(name)) {
+ builder.put(name, entry);
+ }
+ });
BASIC_CATALOG_PROPERTY_ENTRIES.forEach(
(name, entry) -> {
Preconditions.checkArgument(
- !base.containsKey(name), "Property metadata already
exists: " + name);
+ placed.add(name), "Property metadata already exists: " +
name);
builder.put(name, entry);
});
propertyEntries = builder.build();
diff --git
a/core/src/main/java/org/apache/gravitino/secret/FallbackPropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/secret/FallbackPropertiesMetadata.java
index ba76bdd687..e7c334d269 100644
---
a/core/src/main/java/org/apache/gravitino/secret/FallbackPropertiesMetadata.java
+++
b/core/src/main/java/org/apache/gravitino/secret/FallbackPropertiesMetadata.java
@@ -18,13 +18,8 @@
*/
package org.apache.gravitino.secret;
-import com.google.common.collect.ImmutableMap;
import java.util.Map;
-import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
-import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
-import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
+import org.apache.gravitino.cloud.storage.CloudPropertiesMetadata;
import org.apache.gravitino.connector.BasePropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
@@ -32,23 +27,23 @@ import org.apache.gravitino.connector.PropertyEntry;
* Fallback {@link org.apache.gravitino.connector.PropertiesMetadata} when a
catalog does not
* support properties metadata for an entity type ({@link
UnsupportedOperationException}).
*
- * <p>Registers the shared base + credential-vending + cloud-storage property
entries so officially
- * non-hidden keys (for example {@code credential-providers}, {@code
s3-access-key-id}) are not
- * fuzzy-recovered into {@code getSecrets}. Undeclared sensitive-named keys
still use fuzzy
- * recovery; declared hidden secrets (for example {@code
s3-secret-access-key}) remain recoverable.
+ * <p>Registers shared base + credential-vending + cloud-storage entries
({@link
+ * CloudPropertiesMetadata#STORAGE_PROPERTY_ENTRIES}) so officially non-hidden
keys (for example
+ * {@code credential-providers}, {@code s3-access-key-id}) are not
fuzzy-recovered into {@code
+ * getSecrets}. Undeclared sensitive-named keys still use fuzzy recovery;
declared hidden secrets
+ * (for example {@code s3-secret-access-key}) remain recoverable.
+ *
+ * <p>Does not include the AWS access-key pair. That pair is a Glue catalog
property merged only via
+ * {@link org.apache.gravitino.connector.BaseCatalogPropertiesMetadata},
matching fileset and schema
+ * metadata which also omit it. A catalog path that hits this fallback
therefore treats {@code
+ * aws-access-key-id} as undeclared (fuzzy mask / recover), the same as a
fileset or schema.
*/
final class FallbackPropertiesMetadata extends BasePropertiesMetadata {
static final FallbackPropertiesMetadata INSTANCE = new
FallbackPropertiesMetadata();
private static final Map<String, PropertyEntry<?>> CLOUD_PROPERTY_ENTRIES =
- ImmutableMap.<String, PropertyEntry<?>>builder()
- .putAll(S3PropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES)
- .putAll(COSPropertiesMetadata.PROPERTY_ENTRIES)
- .build();
+ CloudPropertiesMetadata.STORAGE_PROPERTY_ENTRIES;
private FallbackPropertiesMetadata() {}
diff --git
a/core/src/test/java/org/apache/gravitino/cloud/storage/TestCloudPropertiesMetadata.java
b/core/src/test/java/org/apache/gravitino/cloud/storage/TestCloudPropertiesMetadata.java
index 7704948ae7..87234c9793 100644
---
a/core/src/test/java/org/apache/gravitino/cloud/storage/TestCloudPropertiesMetadata.java
+++
b/core/src/test/java/org/apache/gravitino/cloud/storage/TestCloudPropertiesMetadata.java
@@ -19,13 +19,17 @@
package org.apache.gravitino.cloud.storage;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import org.apache.gravitino.catalog.glue.GlueConstants;
import org.apache.gravitino.credential.CredentialConstants;
import org.apache.gravitino.credential.config.CredentialConfig;
import org.apache.gravitino.secret.SecretPropertyUtils;
+import org.apache.gravitino.storage.AWSProperties;
import org.apache.gravitino.storage.AzureProperties;
import org.apache.gravitino.storage.COSProperties;
+import org.apache.gravitino.storage.GCSProperties;
import org.apache.gravitino.storage.OSSProperties;
import org.apache.gravitino.storage.S3Properties;
import org.junit.jupiter.api.Test;
@@ -104,4 +108,39 @@ public class TestCloudPropertiesMetadata {
SecretPropertyUtils.isSensitivePropertyKey(
AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_NAME));
}
+
+ @Test
+ void testAwsCredentialPropertiesAreDeclared() {
+ var metadata = AWSPropertiesMetadata.PROPERTY_ENTRIES;
+
assertTrue(metadata.containsKey(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID));
+
assertTrue(metadata.containsKey(AWSProperties.GRAVITINO_AWS_SECRET_ACCESS_KEY));
+
assertFalse(metadata.get(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID).isHidden());
+
assertFalse(metadata.get(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID).isRequired());
+
assertTrue(metadata.get(AWSProperties.GRAVITINO_AWS_SECRET_ACCESS_KEY).isHidden());
+
assertFalse(metadata.get(AWSProperties.GRAVITINO_AWS_SECRET_ACCESS_KEY).isRequired());
+ }
+
+ @Test
+ void testCombinedCloudEntriesKeepAwsOffStorageMetadata() {
+ var storage = CloudPropertiesMetadata.STORAGE_PROPERTY_ENTRIES;
+ var all = CloudPropertiesMetadata.ALL_PROPERTY_ENTRIES;
+
+ assertTrue(storage.containsKey(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+ assertTrue(storage.containsKey(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID));
+
assertTrue(storage.containsKey(AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_NAME));
+
assertTrue(storage.containsKey(GCSProperties.GRAVITINO_GCS_SERVICE_ACCOUNT_FILE));
+ assertTrue(storage.containsKey(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID));
+
assertFalse(storage.containsKey(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID));
+ assertFalse(storage.containsKey(GlueConstants.AWS_REGION));
+
+ assertTrue(all.containsKey(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID));
+ assertTrue(all.containsKey(AWSProperties.GRAVITINO_AWS_SECRET_ACCESS_KEY));
+ assertSame(
+
S3PropertiesMetadata.PROPERTY_ENTRIES.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID),
+ all.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+ assertSame(
+
AWSPropertiesMetadata.PROPERTY_ENTRIES.get(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID),
+ all.get(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID));
+
assertFalse(all.get(AWSProperties.GRAVITINO_AWS_ACCESS_KEY_ID).isRequired());
+ }
}
diff --git
a/core/src/test/java/org/apache/gravitino/connector/TestBaseCatalogPropertiesMetadata.java
b/core/src/test/java/org/apache/gravitino/connector/TestBaseCatalogPropertiesMetadata.java
index 3e7c801e2f..4baa7a51f6 100644
---
a/core/src/test/java/org/apache/gravitino/connector/TestBaseCatalogPropertiesMetadata.java
+++
b/core/src/test/java/org/apache/gravitino/connector/TestBaseCatalogPropertiesMetadata.java
@@ -18,12 +18,15 @@
*/
package org.apache.gravitino.connector;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import com.google.common.collect.ImmutableMap;
import java.util.Collections;
import java.util.Map;
import org.apache.gravitino.credential.CredentialConstants;
+import org.apache.gravitino.storage.S3Properties;
import org.junit.jupiter.api.Test;
public class TestBaseCatalogPropertiesMetadata {
@@ -43,4 +46,81 @@ public class TestBaseCatalogPropertiesMetadata {
assertFalse(metadata.isHiddenProperty(CredentialConstants.CREDENTIAL_PROVIDERS));
assertFalse(metadata.isHiddenProperty(CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS));
}
+
+ @Test
+ void testSharedCloudCredentialKeysAreDeclaredForAllCatalogs() {
+
assertTrue(metadata.containsProperty(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+
assertTrue(metadata.containsProperty(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
+
assertFalse(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+
assertTrue(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
+ }
+
+ @Test
+ void testConnectorCredentialKeysAreDeclaredForAllCatalogs() {
+ assertTrue(metadata.containsProperty("aws-access-key-id"));
+ assertFalse(metadata.isHiddenProperty("aws-access-key-id"));
+ assertTrue(metadata.isHiddenProperty("aws-secret-access-key"));
+ assertFalse(metadata.containsProperty("jdbc-user"));
+ assertFalse(metadata.containsProperty("jdbc-password"));
+ assertFalse(metadata.containsProperty("token-provider"));
+ assertFalse(metadata.containsProperty("gcs.oauth2.token"));
+ assertFalse(metadata.containsProperty("s3.session-token"));
+ assertFalse(metadata.containsProperty("jdbc.user"));
+ }
+
+ @Test
+ void testRuntimeCopiedS3AccessKeyUsesSharedCloudMetadata() {
+ PropertiesMetadata glueLikeMetadata =
+ new BaseCatalogPropertiesMetadata() {
+ @Override
+ protected Map<String, PropertyEntry<?>> specificPropertyEntries() {
+ return ImmutableMap.of(
+ "aws-access-key-id",
+ PropertyEntry.stringOptionalPropertyEntry(
+ "aws-access-key-id", "AWS access key ID", false, null,
false),
+ "aws-secret-access-key",
+ PropertyEntry.stringOptionalPropertyEntry(
+ "aws-secret-access-key", "AWS secret access key", false,
null, true));
+ }
+ };
+
+ Map<String, String> properties =
+ ImmutableMap.of(
+ "aws-access-key-id",
+ "AKIAEXAMPLE",
+ S3Properties.GRAVITINO_S3_ACCESS_KEY_ID,
+ "AKIAEXAMPLE",
+ S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY,
+ "secret-value");
+ Map<String, String> masked =
+ HiddenPropertyMaskUtils.maskHiddenProperties(properties,
glueLikeMetadata);
+
+ assertEquals("AKIAEXAMPLE", masked.get("aws-access-key-id"));
+ assertEquals("AKIAEXAMPLE",
masked.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+ assertEquals(
+ HiddenPropertyMaskUtils.MASKED_VALUE,
+ masked.get(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
+ }
+
+ @Test
+ void testCatalogDeclarationWinsOverSharedCloudEntry() {
+ PropertiesMetadata catalogMetadata =
+ new BaseCatalogPropertiesMetadata() {
+ @Override
+ protected Map<String, PropertyEntry<?>> specificPropertyEntries() {
+ return ImmutableMap.of(
+ S3Properties.GRAVITINO_S3_ACCESS_KEY_ID,
+ PropertyEntry.stringOptionalPropertyEntry(
+ S3Properties.GRAVITINO_S3_ACCESS_KEY_ID,
+ "Catalog-owned S3 access key ID",
+ false,
+ null,
+ true));
+ }
+ };
+
+
assertTrue(catalogMetadata.containsProperty(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+
assertTrue(catalogMetadata.isHiddenProperty(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+
assertFalse(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+ }
}
diff --git
a/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyOperationDispatcher.java
b/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyOperationDispatcher.java
index be0d093cb0..7a6f10325f 100644
---
a/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyOperationDispatcher.java
+++
b/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyOperationDispatcher.java
@@ -197,5 +197,13 @@ public class TestSecretPropertyOperationDispatcher extends
TestOperationDispatch
SecretPropertyUtils.shouldRecoverSensitiveNamedSecret("s3-secret-access-key",
metadata));
Assertions.assertTrue(
SecretPropertyUtils.shouldRecoverSensitiveNamedSecret("custom-token",
metadata));
+ // AWS access-key pair is catalog-only (BaseCatalogPropertiesMetadata),
not in this entity
+ // fallback — same exclusion as fileset/schema STORAGE_PROPERTY_ENTRIES.
+ Assertions.assertFalse(metadata.containsProperty("aws-access-key-id"));
+ Assertions.assertFalse(metadata.containsProperty("aws-secret-access-key"));
+ Assertions.assertTrue(
+
SecretPropertyUtils.shouldRecoverSensitiveNamedSecret("aws-access-key-id",
metadata));
+ Assertions.assertTrue(
+
SecretPropertyUtils.shouldRecoverSensitiveNamedSecret("aws-secret-access-key",
metadata));
}
}
diff --git a/design-docs/gravitino-glue-catalog.md
b/design-docs/gravitino-glue-catalog.md
index 1b420ab432..1dc1b19456 100644
--- a/design-docs/gravitino-glue-catalog.md
+++ b/design-docs/gravitino-glue-catalog.md
@@ -124,15 +124,15 @@ Extend existing Hive and Iceberg catalogs with
`metastore-type=glue` / `catalog-
Glue is a separate AWS service from S3. The Glue region and credentials may
differ from S3 storage credentials, so Glue properties use their own `aws-*`
namespace:
-| Property | Required | Default | Description |
-|---|---|---|---|
-| `aws-region` | Yes | — | AWS region for the Glue Data Catalog |
-| `aws-access-key-id` | No | Default credential chain | AWS access key for
Glue API authentication. **Sensitive**: not visible to catalog readers via
Gravitino API. |
-| `aws-secret-access-key` | No | Default credential chain | AWS secret key for
Glue API authentication. **Sensitive**: not visible to catalog readers via
Gravitino API. |
-| `aws-glue-catalog-id` | Yes | — | Glue catalog ID. Required because an AWS
account can have multiple Glue catalogs (e.g., default catalog and federated S3
Tables catalog). |
-| `aws-glue-endpoint` | No | AWS default regional endpoint | Custom Glue
endpoint URL (for VPC endpoints or LocalStack testing). |
-| `default-table-format` | No | `iceberg` | Default format for tables created
via Gravitino's `createTable()` API. Accepted values: `iceberg`, `hive`. |
-| `table-type-filter` | No | `all` | Comma-separated list of table types
exposed by `listTables()` and `loadTable()`. Accepted values: `all`, `hive`,
`iceberg`, `delta`, `parquet`. Use to restrict visible table types for
backwards compatibility with existing systems that cannot handle mixed-format
catalogs. |
+| Property | Required | Default |
Description
|
+| ----------------------- | -------- | ----------------------------- |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| `aws-region` | Yes | — | AWS
region for the Glue Data Catalog
|
+| `aws-access-key-id` | No | Default credential chain | AWS
access key for Glue API authentication. Visible in cleartext to catalog
readers.
|
+| `aws-secret-access-key` | No | Default credential chain | AWS
secret key for Glue API authentication. Hidden from catalog readers; returned
as `******`.
|
+| `aws-glue-catalog-id` | Yes | — | Glue
catalog ID. Required because an AWS account can have multiple Glue catalogs
(e.g., default catalog and federated S3 Tables catalog).
|
+| `aws-glue-endpoint` | No | AWS default regional endpoint | Custom
Glue endpoint URL (for VPC endpoints or LocalStack testing).
|
+| `default-table-format` | No | `iceberg` | Default
format for tables created via Gravitino's `createTable()` API. Accepted values:
`iceberg`, `hive`.
|
+| `table-type-filter` | No | `all` |
Comma-separated list of table types exposed by `listTables()` and
`loadTable()`. Accepted values: `all`, `hive`, `iceberg`, `delta`, `parquet`.
Use to restrict visible table types for backwards compatibility with existing
systems that cannot handle mixed-format catalogs. |
**Authentication priority**: Static credentials (`aws-access-key-id` +
`aws-secret-access-key`) → Default credential chain (environment variables,
instance profile, container credentials). STS AssumeRole (`aws-role-arn`) is a
future enhancement — static credentials are sufficient for the initial release,
including cross-account access.
