This is an automated email from the ASF dual-hosted git repository.
diqiu50 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 8094542c01 [#13203] improvement(core): Register official cloud and
credential catalog properties in PropertiesMetadata (#13204)
8094542c01 is described below
commit 8094542c017f7f26285c930023f99bc77787ec91
Author: MaSai <[email protected]>
AuthorDate: Thu Sep 17 10:19:16 2026 +0800
[#13203] improvement(core): Register official cloud and credential catalog
properties in PropertiesMetadata (#13204)
### What changes were proposed in this pull request?
This PR registers official Gravitino cloud-storage and
credential-vending property keys in `PropertiesMetadata`, and applies
name-based fuzzy masking only to undeclared keys.
**A. Shared cloud metadata**
- Expand `S3PropertiesMetadata`, `OSSPropertiesMetadata`, and
`AzurePropertiesMetadata` with non-secret credential-config keys
(endpoint, region, role ARN, external ID, etc.).
- Add `COSPropertiesMetadata` for Tencent COS credential-config keys.
- Add `s3/oss/adls-token-expire-in-secs` to
`CredentialConfig.CREDENTIAL_PROPERTY_ENTRIES`.
**B. Catalog metadata merges**
- Merge `CredentialConfig.CREDENTIAL_PROPERTY_ENTRIES` into Iceberg,
Hive, and Paimon catalog properties metadata.
- Merge missing Azure/GCS shared metadata into Paimon.
**C. Fuzzy masking rule**
- Add `SecretPropertyUtils.isSensitivePropertyKey()` for credential-like
property names.
- Update `OperationDispatcher.getHiddenPropertyNames()` to mask
undeclared sensitive-named keys as `******`, but skip fuzzy matching
when `metadata.containsProperty(key)` is true.
- Declared non-secret keys such as `credential-providers`,
`azure-storage-account-name`, and `s3-token-expire-in-secs` remain
visible in `properties()` responses.
**Tests**
- Add `TestCloudPropertiesMetadata` in `core`.
- Extend `TestPaimonCatalogPropertiesMetadata` and
`TestSecretPropertyUtils`.
### Why are the changes needed?
Official property keys defined in constants/docs were missing from
`PropertiesMetadata`. Undeclared keys either bypass metadata-based
masking or get incorrectly masked by name-based fuzzy matching —
breaking the Web UI catalog edit flow for keys like
`credential-providers`, `azure-storage-account-name`, and
`s3-token-expire-in-secs`.
Fix: #13203
### Does this PR introduce _any_ user-facing change?
Yes, but only in masking/visibility behavior:
1. Official non-secret cloud/credential keys are now declared and remain
visible in `properties()` responses.
2. Undeclared properties whose names look credential-like are masked as
`******` on entity read paths.
3. Secret keys continue to be masked via the existing `hidden` flag or
secret URNs.
4. No property keys are added or removed from catalogs — only metadata
registration and masking rules.
### How was this patch tested?
```bash
./gradlew :core:test --tests
org.apache.gravitino.secret.TestSecretPropertyUtils \
--tests org.apache.gravitino.cloud.storage.TestCloudPropertiesMetadata
-PskipITs
./gradlew :catalogs:catalog-lakehouse-paimon:test \
--tests
org.apache.gravitino.catalog.lakehouse.paimon.TestPaimonCatalogPropertiesMetadata
-PskipITs
./gradlew spotlessApply
```
---------
Co-authored-by: Cursor <[email protected]>
---
.../apache/gravitino/secret/SupportsSecrets.java | 6 +-
.../CloudStorageCredentialPropertyKeys.java | 18 +-
.../TestCloudStorageCredentialPropertyKeys.java | 13 +-
.../fileset/FilesetCatalogPropertiesMetadata.java | 3 +-
.../catalog/fileset/FilesetPropertiesMetadata.java | 2 -
.../fileset/FilesetSchemaPropertiesMetadata.java | 2 -
.../fileset/TestFilesetCatalogCredential.java | 14 +-
.../TestFilesetCloudPropertiesMetadata.java | 10 +-
.../glue/GlueCatalogPropertiesMetadata.java | 2 +-
.../glue/TestGlueCatalogPropertiesMetadata.java | 5 +-
.../hive/HiveCatalogPropertiesMetadata.java | 2 +
.../catalog/hive/TestHiveCatalogOperations.java | 15 +-
.../iceberg/IcebergCatalogPropertiesMetadata.java | 2 +
.../TestIcebergCatalogPropertiesMetadata.java | 10 ++
.../paimon/PaimonCatalogPropertiesMetadata.java | 4 +-
.../TestPaimonCatalogPropertiesMetadata.java | 11 +-
.../hadoop/TestBaseGVFSOperationsSecrets.java | 48 ++++++
.../cloud/storage/AzurePropertiesMetadata.java | 16 ++
.../cloud/storage/COSPropertiesMetadata.java | 42 ++++-
.../cloud/storage/OSSPropertiesMetadata.java | 34 +++-
.../cloud/storage/S3PropertiesMetadata.java | 61 ++++++-
.../connector/BaseCatalogPropertiesMetadata.java | 17 +-
.../connector/BasePropertiesMetadata.java | 44 +++--
.../connector/HiddenPropertyMaskUtils.java | 2 +-
.../credential/config/CredentialConfig.java | 40 +++++
.../secret/FallbackPropertiesMetadata.java | 59 +++++++
.../secret/SecretPropertyOperationDispatcher.java | 189 ++++++++++++++-------
.../gravitino/secret/SecretPropertyUtils.java | 72 +++++++-
.../gravitino/TestBasePropertiesMetadata.java | 12 ++
.../cloud/storage/TestCloudPropertiesMetadata.java | 107 ++++++++++++
.../TestBaseCatalogPropertiesMetadata.java | 46 +++++
.../connector/TestHiddenPropertyMaskUtils.java | 16 +-
.../TestSecretPropertyOperationDispatcher.java | 28 +++
.../gravitino/secret/TestSecretPropertyUtils.java | 72 ++++++++
.../trino/connector/TestGravitinoConnector.java | 71 +++++---
35 files changed, 931 insertions(+), 164 deletions(-)
diff --git a/api/src/main/java/org/apache/gravitino/secret/SupportsSecrets.java
b/api/src/main/java/org/apache/gravitino/secret/SupportsSecrets.java
index af9563cf7d..1d0fd56856 100644
--- a/api/src/main/java/org/apache/gravitino/secret/SupportsSecrets.java
+++ b/api/src/main/java/org/apache/gravitino/secret/SupportsSecrets.java
@@ -30,8 +30,10 @@ import java.util.Map;
* also be delivered via {@link
org.apache.gravitino.credential.SupportsCredentials}).
* <li>Stored plaintext for property keys whose names look sensitive
(contain {@code secret},
* {@code password}, {@code token}, {@code credential}, {@code access},
or {@code account},
- * case-insensitive), so mistyped / undeclared credential properties
remain usable after API
- * responses mask them as {@code ******}.
+ * case-insensitive) when the key is undeclared or declared {@code
hidden}, so mistyped /
+ * undeclared credential properties remain usable after API responses
mask them as {@code
+ * ******}. Declared non-hidden configuration keys are not returned even
when the name
+ * matches.
* </ul>
*
* <p>Properties that are only declared {@code hidden} in property metadata,
and whose names do
diff --git
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/CloudStorageCredentialPropertyKeys.java
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/CloudStorageCredentialPropertyKeys.java
index a850e2c370..97bae3cef1 100644
---
a/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/CloudStorageCredentialPropertyKeys.java
+++
b/catalogs/catalog-common/src/main/java/org/apache/gravitino/storage/CloudStorageCredentialPropertyKeys.java
@@ -26,26 +26,30 @@ import java.util.Set;
import javax.annotation.Nullable;
/**
- * Gravitino property keys for cloud static credentials.
+ * Gravitino property keys for cloud static <em>secret</em> credentials.
*
* <p>GVFS must not consume these keys from REST catalog/schema/fileset {@code
properties()}
- * responses (which may be masked). Clients may still supply credentials via
local Hadoop {@code
- * Configuration} or {@code getCredentials()} when credential vending is
enabled.
+ * responses (which may be masked as {@code ******}). Secret plaintext is
recovered via {@code
+ * getSecrets()}. Non-secret identifiers such as {@code s3-access-key-id} /
{@code
+ * gcs-service-account-file} remain in {@code properties()} and are merged
normally. Clients may
+ * also supply credentials via local Hadoop {@code Configuration} or {@code
getCredentials()} when
+ * credential vending is enabled.
*/
public final class CloudStorageCredentialPropertyKeys {
/** Placeholder returned for masked hidden properties in REST responses. */
public static final String MASKED_PROPERTY_VALUE = "******";
+ /**
+ * Secret-bearing static credential keys only. Access key IDs and GCS
service-account file paths
+ * are intentionally excluded: they are declared non-hidden and must stay
available from {@code
+ * properties()}.
+ */
private static final Set<String> STATIC_CREDENTIAL_KEYS =
ImmutableSet.of(
- S3Properties.GRAVITINO_S3_ACCESS_KEY_ID,
S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY,
- OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID,
OSSProperties.GRAVITINO_OSS_ACCESS_KEY_SECRET,
AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_KEY,
- GCSProperties.GRAVITINO_GCS_SERVICE_ACCOUNT_FILE,
- COSProperties.GRAVITINO_COS_ACCESS_KEY_ID,
COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET);
private CloudStorageCredentialPropertyKeys() {}
diff --git
a/catalogs/catalog-common/src/test/java/org/apache/gravitino/storage/TestCloudStorageCredentialPropertyKeys.java
b/catalogs/catalog-common/src/test/java/org/apache/gravitino/storage/TestCloudStorageCredentialPropertyKeys.java
index ee54fcf8ae..768890dd8f 100644
---
a/catalogs/catalog-common/src/test/java/org/apache/gravitino/storage/TestCloudStorageCredentialPropertyKeys.java
+++
b/catalogs/catalog-common/src/test/java/org/apache/gravitino/storage/TestCloudStorageCredentialPropertyKeys.java
@@ -34,19 +34,23 @@ public class TestCloudStorageCredentialPropertyKeys {
S3Properties.GRAVITINO_S3_ENDPOINT,
"https://s3.amazonaws.com";,
S3Properties.GRAVITINO_S3_ACCESS_KEY_ID,
- CloudStorageCredentialPropertyKeys.MASKED_PROPERTY_VALUE,
+ "AKIATEST",
S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY,
"secret",
OSSProperties.GRAVITINO_OSS_REGION,
- "cn-hangzhou");
+ "cn-hangzhou",
+ "masked-any-key",
+ CloudStorageCredentialPropertyKeys.MASKED_PROPERTY_VALUE);
Map<String, String> filtered =
CloudStorageCredentialPropertyKeys.omitStaticCredentialProperties(input);
assertEquals("https://s3.amazonaws.com";,
filtered.get(S3Properties.GRAVITINO_S3_ENDPOINT));
assertEquals("cn-hangzhou",
filtered.get(OSSProperties.GRAVITINO_OSS_REGION));
- assertFalse(filtered.containsKey(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+ // Access key ID is non-hidden plaintext in properties(); GVFS keeps it.
+ assertEquals("AKIATEST",
filtered.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
assertFalse(filtered.containsKey(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
+ assertFalse(filtered.containsKey("masked-any-key"));
}
@Test
@@ -54,6 +58,9 @@ public class TestCloudStorageCredentialPropertyKeys {
assertTrue(
CloudStorageCredentialPropertyKeys.isStaticCredentialKey(
COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET));
+ assertFalse(
+ CloudStorageCredentialPropertyKeys.isStaticCredentialKey(
+ COSProperties.GRAVITINO_COS_ACCESS_KEY_ID));
assertFalse(
CloudStorageCredentialPropertyKeys.isStaticCredentialKey(
COSProperties.GRAVITINO_COS_REGION));
diff --git
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogPropertiesMetadata.java
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogPropertiesMetadata.java
index 141e085a3f..b71a3a34ac 100644
---
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogPropertiesMetadata.java
@@ -40,7 +40,6 @@ import
org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
import org.apache.gravitino.connector.BaseCatalogPropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
-import org.apache.gravitino.credential.config.CredentialConfig;
public class FilesetCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetadata {
@@ -228,7 +227,7 @@ public class FilesetCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetad
// The following two are about authentication.
.putAll(KERBEROS_PROPERTY_ENTRIES)
.putAll(AUTHENTICATION_PROPERTY_ENTRIES)
- .putAll(CredentialConfig.CREDENTIAL_PROPERTY_ENTRIES)
+ // CREDENTIAL_PROPERTY_ENTRIES are registered in
BaseCatalogPropertiesMetadata.
.putAll(S3PropertiesMetadata.PROPERTY_ENTRIES)
.putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
.putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
diff --git
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetPropertiesMetadata.java
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetPropertiesMetadata.java
index d75abd29ef..bc722f5060 100644
---
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetPropertiesMetadata.java
+++
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetPropertiesMetadata.java
@@ -35,7 +35,6 @@ import
org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
import org.apache.gravitino.connector.BasePropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
-import org.apache.gravitino.credential.config.CredentialConfig;
public class FilesetPropertiesMetadata extends BasePropertiesMetadata {
@@ -80,7 +79,6 @@ public class FilesetPropertiesMetadata extends
BasePropertiesMetadata {
false /* hidden */))
.putAll(KERBEROS_PROPERTY_ENTRIES)
.putAll(AUTHENTICATION_PROPERTY_ENTRIES)
- .putAll(CredentialConfig.CREDENTIAL_PROPERTY_ENTRIES)
.putAll(S3PropertiesMetadata.PROPERTY_ENTRIES)
.putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
.putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
diff --git
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetSchemaPropertiesMetadata.java
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetSchemaPropertiesMetadata.java
index f31f7b34f5..7515d90556 100644
---
a/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetSchemaPropertiesMetadata.java
+++
b/catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetSchemaPropertiesMetadata.java
@@ -32,7 +32,6 @@ import
org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
import org.apache.gravitino.connector.BasePropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
-import org.apache.gravitino.credential.config.CredentialConfig;
public class FilesetSchemaPropertiesMetadata extends BasePropertiesMetadata {
@@ -73,7 +72,6 @@ public class FilesetSchemaPropertiesMetadata extends
BasePropertiesMetadata {
false /* reserved */))
.putAll(KERBEROS_PROPERTY_ENTRIES)
.putAll(AUTHENTICATION_PROPERTY_ENTRIES)
- .putAll(CredentialConfig.CREDENTIAL_PROPERTY_ENTRIES)
.putAll(S3PropertiesMetadata.PROPERTY_ENTRIES)
.putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
.putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
diff --git
a/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCatalogCredential.java
b/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCatalogCredential.java
index 1a1800d0b7..72f9326c52 100644
---
a/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCatalogCredential.java
+++
b/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCatalogCredential.java
@@ -97,26 +97,20 @@ public class TestFilesetCatalogCredential {
FilesetCatalogImpl catalog = newCatalog(properties);
Map<String, String> masked = catalog.properties();
- Assertions.assertEquals(
- HiddenPropertyMaskUtils.MASKED_VALUE,
masked.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+ Assertions.assertEquals("s3-ak",
masked.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
Assertions.assertEquals(
HiddenPropertyMaskUtils.MASKED_VALUE,
masked.get(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
- Assertions.assertEquals(
- HiddenPropertyMaskUtils.MASKED_VALUE,
- masked.get(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID));
+ Assertions.assertEquals("oss-ak",
masked.get(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID));
Assertions.assertEquals(
HiddenPropertyMaskUtils.MASKED_VALUE,
masked.get(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_SECRET));
Assertions.assertEquals(
- HiddenPropertyMaskUtils.MASKED_VALUE,
- masked.get(AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_NAME));
+ "abs-account",
masked.get(AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_NAME));
Assertions.assertEquals(
HiddenPropertyMaskUtils.MASKED_VALUE,
masked.get(AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_KEY));
- Assertions.assertEquals(
- HiddenPropertyMaskUtils.MASKED_VALUE,
- masked.get(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID));
+ Assertions.assertEquals("cos-ak",
masked.get(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID));
Assertions.assertEquals(
HiddenPropertyMaskUtils.MASKED_VALUE,
masked.get(COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET));
diff --git
a/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCloudPropertiesMetadata.java
b/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCloudPropertiesMetadata.java
index 4e121c3b04..515d6849ae 100644
---
a/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCloudPropertiesMetadata.java
+++
b/catalogs/catalog-fileset/src/test/java/org/apache/gravitino/catalog/fileset/TestFilesetCloudPropertiesMetadata.java
@@ -52,12 +52,12 @@ public class TestFilesetCloudPropertiesMetadata {
@ParameterizedTest
@MethodSource("filesetPropertiesMetadata")
void testCloudCredentialsAreHidden(PropertiesMetadata metadata) {
-
assertTrue(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+
assertFalse(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
assertTrue(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
-
assertTrue(metadata.isHiddenProperty(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID));
+
assertFalse(metadata.isHiddenProperty(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID));
assertTrue(metadata.isHiddenProperty(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_SECRET));
assertTrue(metadata.isHiddenProperty(AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_KEY));
-
assertTrue(metadata.isHiddenProperty(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID));
+
assertFalse(metadata.isHiddenProperty(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID));
assertTrue(metadata.isHiddenProperty(COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET));
}
@@ -84,9 +84,7 @@ public class TestFilesetCloudPropertiesMetadata {
HiddenPropertyMaskUtils.maskHiddenProperties(properties, metadata);
assertEquals("s3a://bucket/path",
response.get(FilesetCatalogPropertiesMetadata.LOCATION));
- assertEquals(
- HiddenPropertyMaskUtils.MASKED_VALUE,
- response.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+ assertEquals("AKIATEST",
response.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
assertEquals(
HiddenPropertyMaskUtils.MASKED_VALUE,
response.get(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
diff --git
a/catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogPropertiesMetadata.java
b/catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogPropertiesMetadata.java
index 7275861ab7..1b485374ee 100644
---
a/catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogPropertiesMetadata.java
@@ -65,7 +65,7 @@ public class GlueCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetadata
+ " When omitted the default credential chain is used.",
false /* immutable */,
null /* defaultValue */,
- true /* hidden */))
+ false /* hidden */))
.put(
AWS_SECRET_ACCESS_KEY,
stringOptionalPropertyEntry(
diff --git
a/catalogs/catalog-glue/src/test/java/org/apache/gravitino/catalog/glue/TestGlueCatalogPropertiesMetadata.java
b/catalogs/catalog-glue/src/test/java/org/apache/gravitino/catalog/glue/TestGlueCatalogPropertiesMetadata.java
index e2091542c9..328b02bb0e 100644
---
a/catalogs/catalog-glue/src/test/java/org/apache/gravitino/catalog/glue/TestGlueCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-glue/src/test/java/org/apache/gravitino/catalog/glue/TestGlueCatalogPropertiesMetadata.java
@@ -70,9 +70,12 @@ class TestGlueCatalogPropertiesMetadata {
}
@Test
- void testCredentialsAreOptional() {
+ void testCredentialsAreOptionalAndAccessKeyIdIsVisible() {
assertFalse(metadata.isRequiredProperty(AWS_ACCESS_KEY_ID));
assertFalse(metadata.isRequiredProperty(AWS_SECRET_ACCESS_KEY));
+ // Access key ID is an identifier, same as s3-access-key-id; only the
secret is hidden.
+ assertFalse(metadata.isHiddenProperty(AWS_ACCESS_KEY_ID));
+ assertTrue(metadata.isHiddenProperty(AWS_SECRET_ACCESS_KEY));
}
@Test
diff --git
a/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
b/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
index 754da95f78..889e450ea1 100644
---
a/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
@@ -24,6 +24,7 @@ import static
org.apache.gravitino.catalog.hive.HiveConstants.HIVE_DEFAULT_CATAL
import com.google.common.collect.ImmutableMap;
import java.util.Map;
import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
+import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
@@ -131,6 +132,7 @@ public class HiveCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetadata
.putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
.putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
.putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES)
+ .putAll(COSPropertiesMetadata.PROPERTY_ENTRIES)
.putAll(CLIENT_PROPERTIES_METADATA.propertyEntries())
.build();
diff --git
a/catalogs/catalog-hive/src/test/java/org/apache/gravitino/catalog/hive/TestHiveCatalogOperations.java
b/catalogs/catalog-hive/src/test/java/org/apache/gravitino/catalog/hive/TestHiveCatalogOperations.java
index ac18ac92e9..6bcf0814fb 100644
---
a/catalogs/catalog-hive/src/test/java/org/apache/gravitino/catalog/hive/TestHiveCatalogOperations.java
+++
b/catalogs/catalog-hive/src/test/java/org/apache/gravitino/catalog/hive/TestHiveCatalogOperations.java
@@ -79,6 +79,8 @@ import
org.apache.gravitino.rel.expressions.transforms.Transform;
import org.apache.gravitino.rel.indexes.Index;
import org.apache.gravitino.rel.types.Types;
import org.apache.gravitino.storage.AzureProperties;
+import org.apache.gravitino.storage.COSProperties;
+import org.apache.gravitino.storage.S3Properties;
import org.apache.gravitino.utils.ClientPool;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
import org.apache.thrift.TException;
@@ -92,7 +94,6 @@ class TestHiveCatalogOperations {
Map<String, PropertyEntry<?>> propertyEntryMap =
HIVE_PROPERTIES_METADATA.catalogPropertiesMetadata().propertyEntries();
- Assertions.assertEquals(26, propertyEntryMap.size());
Assertions.assertTrue(propertyEntryMap.containsKey(METASTORE_URIS));
Assertions.assertTrue(propertyEntryMap.containsKey(Catalog.PROPERTY_PACKAGE));
Assertions.assertTrue(propertyEntryMap.containsKey(BaseCatalog.CATALOG_OPERATION_IMPL));
@@ -102,6 +103,18 @@ class TestHiveCatalogOperations {
Assertions.assertTrue(propertyEntryMap.containsKey(IMPERSONATION_ENABLE));
Assertions.assertTrue(propertyEntryMap.containsKey(LIST_ALL_TABLES));
Assertions.assertTrue(propertyEntryMap.containsKey(DEFAULT_CATALOG));
+ Assertions.assertTrue(
+ propertyEntryMap.containsKey(
+
org.apache.gravitino.credential.CredentialConstants.CREDENTIAL_PROVIDERS));
+ Assertions.assertTrue(
+ propertyEntryMap.containsKey(
+
org.apache.gravitino.credential.CredentialConstants.COS_TOKEN_EXPIRE_IN_SECS));
+
Assertions.assertTrue(propertyEntryMap.containsKey(S3Properties.GRAVITINO_S3_ROLE_ARN));
+
Assertions.assertTrue(propertyEntryMap.containsKey(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID));
+ Assertions.assertFalse(
+
propertyEntryMap.get(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID).isHidden());
+ Assertions.assertTrue(
+
propertyEntryMap.get(COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET).isHidden());
Assertions.assertTrue(
propertyEntryMap.get(AzureProperties.GRAVITINO_AZURE_CLIENT_SECRET).isHidden());
Assertions.assertTrue(propertyEntryMap.get(METASTORE_URIS).isRequired());
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogPropertiesMetadata.java
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogPropertiesMetadata.java
index 8defa29fbd..4383b8672c 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogPropertiesMetadata.java
@@ -30,6 +30,7 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
+import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
@@ -143,6 +144,7 @@ public class IcebergCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetad
result.putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES);
result.putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES);
result.putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES);
+ result.putAll(COSPropertiesMetadata.PROPERTY_ENTRIES);
result.putAll(KerberosConfig.KERBEROS_PROPERTY_ENTRIES);
result.putAll(AuthenticationConfig.AUTHENTICATION_PROPERTY_ENTRIES);
PROPERTIES_METADATA = ImmutableMap.copyOf(result);
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalogPropertiesMetadata.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalogPropertiesMetadata.java
index 8ca4c3d85a..33921b1c54 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalogPropertiesMetadata.java
@@ -22,6 +22,8 @@ import com.google.common.collect.ImmutableMap;
import java.util.Map;
import org.apache.gravitino.iceberg.common.IcebergConfig;
import org.apache.gravitino.iceberg.common.cache.LocalTableMetadataCache;
+import org.apache.gravitino.storage.COSProperties;
+import org.apache.gravitino.storage.S3Properties;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -43,6 +45,14 @@ public class TestIcebergCatalogPropertiesMetadata {
metadata.isHiddenProperty(IcebergCatalogPropertiesMetadata.GRAVITINO_JDBC_PASSWORD));
}
+ @Test
+ void testCloudAccessKeyIdsAreVisibleAndSecretsAreHidden() {
+
Assertions.assertFalse(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+
Assertions.assertTrue(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
+
Assertions.assertFalse(metadata.isHiddenProperty(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID));
+
Assertions.assertTrue(metadata.isHiddenProperty(COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET));
+ }
+
@Test
void testTableMetadataCacheImplDefaultValue() {
Assertions.assertEquals(
diff --git
a/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
b/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
index deeb86d1da..5398d71d3b 100644
---
a/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogPropertiesMetadata.java
@@ -33,6 +33,7 @@ import
org.apache.gravitino.catalog.lakehouse.paimon.authentication.Authenticati
import
org.apache.gravitino.catalog.lakehouse.paimon.authentication.kerberos.KerberosConfig;
import org.apache.gravitino.catalog.lakehouse.paimon.utils.CatalogUtils;
import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
+import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
@@ -136,7 +137,7 @@ public class PaimonCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetada
"The access key ID for Aliyun DLF",
false /* immutable */,
null /* defaultValue */,
- true /* hidden */))
+ false /* hidden */))
.put(
PaimonConstants.GRAVITINO_DLF_ACCESS_KEY_SECRET,
stringOptionalPropertyEntry(
@@ -220,6 +221,7 @@ public class PaimonCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetada
result.putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES);
result.putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES);
result.putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES);
+ result.putAll(COSPropertiesMetadata.PROPERTY_ENTRIES);
result.putAll(REST_PROPERTY_ENTRIES);
PROPERTIES_METADATA = ImmutableMap.copyOf(result);
}
diff --git
a/catalogs/catalog-lakehouse-paimon/src/test/java/org/apache/gravitino/catalog/lakehouse/paimon/TestPaimonCatalogPropertiesMetadata.java
b/catalogs/catalog-lakehouse-paimon/src/test/java/org/apache/gravitino/catalog/lakehouse/paimon/TestPaimonCatalogPropertiesMetadata.java
index 433ece5279..cd8d37ddcb 100644
---
a/catalogs/catalog-lakehouse-paimon/src/test/java/org/apache/gravitino/catalog/lakehouse/paimon/TestPaimonCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-lakehouse-paimon/src/test/java/org/apache/gravitino/catalog/lakehouse/paimon/TestPaimonCatalogPropertiesMetadata.java
@@ -22,6 +22,7 @@ import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
import org.apache.gravitino.storage.AzureProperties;
+import org.apache.gravitino.storage.COSProperties;
import org.apache.gravitino.storage.GCSProperties;
import org.apache.gravitino.storage.OSSProperties;
import org.apache.gravitino.storage.S3Properties;
@@ -34,14 +35,16 @@ public class TestPaimonCatalogPropertiesMetadata {
@Test
void testSensitivePropertiesAreHidden() {
assertTrue(metadata.isHiddenProperty(PaimonCatalogPropertiesMetadata.GRAVITINO_JDBC_PASSWORD));
-
assertTrue(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+
assertFalse(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
assertTrue(metadata.isHiddenProperty(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
-
assertTrue(metadata.isHiddenProperty(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID));
+
assertFalse(metadata.isHiddenProperty(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_ID));
assertTrue(metadata.isHiddenProperty(OSSProperties.GRAVITINO_OSS_ACCESS_KEY_SECRET));
assertTrue(metadata.isHiddenProperty(AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_KEY));
- // REST bearer token and DLF credentials share the same hidden requirement
+
assertFalse(metadata.isHiddenProperty(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID));
+
assertTrue(metadata.isHiddenProperty(COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET));
+ // REST bearer token and DLF secrets are hidden; DLF access-key-id matches
OSS/S3 (visible).
assertTrue(metadata.isHiddenProperty(PaimonConstants.TOKEN));
-
assertTrue(metadata.isHiddenProperty(PaimonConstants.GRAVITINO_DLF_ACCESS_KEY_ID));
+
assertFalse(metadata.isHiddenProperty(PaimonConstants.GRAVITINO_DLF_ACCESS_KEY_ID));
assertTrue(metadata.isHiddenProperty(PaimonConstants.GRAVITINO_DLF_ACCESS_KEY_SECRET));
assertTrue(metadata.isHiddenProperty(PaimonConstants.GRAVITINO_DLF_SECURITY_TOKEN));
}
diff --git
a/clients/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/TestBaseGVFSOperationsSecrets.java
b/clients/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/TestBaseGVFSOperationsSecrets.java
index 5253d867fc..8b43487571 100644
---
a/clients/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/TestBaseGVFSOperationsSecrets.java
+++
b/clients/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/TestBaseGVFSOperationsSecrets.java
@@ -224,6 +224,54 @@ public class TestBaseGVFSOperationsSecrets {
assertFalse("******".equals(all.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID)));
}
+ @Test
+ public void testKeepsPlaintextAccessKeyIdFromRestProperties() throws
Exception {
+ Configuration conf = new Configuration();
+
conf.set(GravitinoVirtualFileSystemConfiguration.FS_GRAVITINO_CLIENT_METALAKE_KEY,
"ml");
+ conf.set(
+ GravitinoVirtualFileSystemConfiguration.FS_GRAVITINO_SERVER_URI_KEY,
+ "http://localhost:8090";);
+
+ Catalog catalog = mock(Catalog.class);
+ Schema schema = mock(Schema.class);
+ Fileset fileset = mock(Fileset.class);
+ SupportsSchemas schemas = mock(SupportsSchemas.class);
+ FilesetCatalog filesetCatalog = mock(FilesetCatalog.class);
+ SupportsSecrets catalogSecrets = mock(SupportsSecrets.class);
+
+ when(catalog.properties())
+ .thenReturn(
+ Map.of(
+ S3Properties.GRAVITINO_S3_ACCESS_KEY_ID,
+ "AKIATEST",
+ S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY,
+ "******",
+ "s3-endpoint",
+ "http://s3.example.com";));
+ when(catalog.supportsSecrets()).thenReturn(catalogSecrets);
+ when(catalogSecrets.getSecrets())
+ .thenReturn(Map.of(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY,
"secret-from-getSecrets"));
+ when(catalog.asSchemas()).thenReturn(schemas);
+ when(schemas.loadSchema("schema")).thenReturn(schema);
+ when(schema.properties()).thenReturn(Map.of());
+ when(schema.supportsSecrets()).thenReturn(null);
+ when(catalog.asFilesetCatalog()).thenReturn(filesetCatalog);
+ when(filesetCatalog.loadFileset(NameIdentifier.of("schema",
"fs"))).thenReturn(fileset);
+ when(fileset.properties()).thenReturn(Map.of());
+ when(fileset.supportsSecrets()).thenReturn(null);
+
+ GravitinoClient client = mock(GravitinoClient.class);
+ when(client.loadCatalog("catalog")).thenReturn(catalog);
+
+ TestOps ops = new TestOps(conf, client);
+ Map<String, String> all =
+ ops.getAllProperties(NameIdentifier.of("ml", "catalog", "schema",
"fs"));
+
+ assertEquals("AKIATEST", all.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID));
+ assertEquals("secret-from-getSecrets",
all.get(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY));
+ assertEquals("http://s3.example.com";, all.get("s3-endpoint"));
+ }
+
private static final class TestOps extends BaseGVFSOperations {
private final GravitinoClient client;
diff --git
a/core/src/main/java/org/apache/gravitino/cloud/storage/AzurePropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/cloud/storage/AzurePropertiesMetadata.java
index 3f1598b7c6..a4c3827191 100644
---
a/core/src/main/java/org/apache/gravitino/cloud/storage/AzurePropertiesMetadata.java
+++
b/core/src/main/java/org/apache/gravitino/cloud/storage/AzurePropertiesMetadata.java
@@ -46,6 +46,22 @@ public class AzurePropertiesMetadata {
false /* immutable */,
null /* defaultValue */,
true /* hidden */))
+ .put(
+ AzureProperties.GRAVITINO_AZURE_TENANT_ID,
+ stringOptionalPropertyEntry(
+ AzureProperties.GRAVITINO_AZURE_TENANT_ID,
+ "Azure Active Directory tenant ID",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ AzureProperties.GRAVITINO_AZURE_CLIENT_ID,
+ stringOptionalPropertyEntry(
+ AzureProperties.GRAVITINO_AZURE_CLIENT_ID,
+ "Azure Active Directory client ID",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
.put(
AzureProperties.GRAVITINO_AZURE_CLIENT_SECRET,
stringOptionalPropertyEntry(
diff --git
a/core/src/main/java/org/apache/gravitino/cloud/storage/COSPropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/cloud/storage/COSPropertiesMetadata.java
index 9f9378fdf5..06156b2550 100644
---
a/core/src/main/java/org/apache/gravitino/cloud/storage/COSPropertiesMetadata.java
+++
b/core/src/main/java/org/apache/gravitino/cloud/storage/COSPropertiesMetadata.java
@@ -39,7 +39,7 @@ public class COSPropertiesMetadata {
"COS access key ID (Tencent Cloud SecretId)",
false /* immutable */,
null /* defaultValue */,
- true /* hidden */))
+ false /* hidden */))
.put(
COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET,
stringOptionalPropertyEntry(
@@ -48,6 +48,46 @@ public class COSPropertiesMetadata {
false /* immutable */,
null /* defaultValue */,
true /* hidden */))
+ .put(
+ COSProperties.GRAVITINO_COS_REGION,
+ stringOptionalPropertyEntry(
+ COSProperties.GRAVITINO_COS_REGION,
+ "Tencent Cloud COS region",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ COSProperties.GRAVITINO_COS_ENDPOINT,
+ stringOptionalPropertyEntry(
+ COSProperties.GRAVITINO_COS_ENDPOINT,
+ "Tencent Cloud COS endpoint",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ COSProperties.GRAVITINO_COS_ROLE_ARN,
+ stringOptionalPropertyEntry(
+ COSProperties.GRAVITINO_COS_ROLE_ARN,
+ "Tencent Cloud COS role ARN for STS credential vending",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ COSProperties.GRAVITINO_COS_EXTERNAL_ID,
+ stringOptionalPropertyEntry(
+ COSProperties.GRAVITINO_COS_EXTERNAL_ID,
+ "Tencent Cloud COS external ID for cross-account AssumeRole",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ COSProperties.GRAVITINO_COS_APP_ID,
+ stringOptionalPropertyEntry(
+ COSProperties.GRAVITINO_COS_APP_ID,
+ "Tencent Cloud COS application ID",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
.build();
private COSPropertiesMetadata() {}
diff --git
a/core/src/main/java/org/apache/gravitino/cloud/storage/OSSPropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/cloud/storage/OSSPropertiesMetadata.java
index 43e1897786..09585f3b92 100644
---
a/core/src/main/java/org/apache/gravitino/cloud/storage/OSSPropertiesMetadata.java
+++
b/core/src/main/java/org/apache/gravitino/cloud/storage/OSSPropertiesMetadata.java
@@ -37,7 +37,7 @@ public class OSSPropertiesMetadata {
"OSS access key ID",
false /* immutable */,
null /* defaultValue */,
- true /* hidden */))
+ false /* hidden */))
.put(
OSSProperties.GRAVITINO_OSS_ACCESS_KEY_SECRET,
stringOptionalPropertyEntry(
@@ -46,6 +46,38 @@ public class OSSPropertiesMetadata {
false /* immutable */,
null /* defaultValue */,
true /* hidden */))
+ .put(
+ OSSProperties.GRAVITINO_OSS_REGION,
+ stringOptionalPropertyEntry(
+ OSSProperties.GRAVITINO_OSS_REGION,
+ "OSS service region",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ OSSProperties.GRAVITINO_OSS_ENDPOINT,
+ stringOptionalPropertyEntry(
+ OSSProperties.GRAVITINO_OSS_ENDPOINT,
+ "OSS service endpoint",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ OSSProperties.GRAVITINO_OSS_ROLE_ARN,
+ stringOptionalPropertyEntry(
+ OSSProperties.GRAVITINO_OSS_ROLE_ARN,
+ "OSS role ARN for STS credential vending",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ OSSProperties.GRAVITINO_OSS_EXTERNAL_ID,
+ stringOptionalPropertyEntry(
+ OSSProperties.GRAVITINO_OSS_EXTERNAL_ID,
+ "OSS external ID for cross-account AssumeRole",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
.build();
private OSSPropertiesMetadata() {}
diff --git
a/core/src/main/java/org/apache/gravitino/cloud/storage/S3PropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/cloud/storage/S3PropertiesMetadata.java
index 8879d4b551..f6771fe7ec 100644
---
a/core/src/main/java/org/apache/gravitino/cloud/storage/S3PropertiesMetadata.java
+++
b/core/src/main/java/org/apache/gravitino/cloud/storage/S3PropertiesMetadata.java
@@ -18,6 +18,7 @@
*/
package org.apache.gravitino.cloud.storage;
+import static
org.apache.gravitino.connector.PropertyEntry.booleanPropertyEntry;
import static
org.apache.gravitino.connector.PropertyEntry.stringOptionalPropertyEntry;
import com.google.common.collect.ImmutableMap;
@@ -37,7 +38,7 @@ public class S3PropertiesMetadata {
"S3 access key ID",
false /* immutable */,
null /* defaultValue */,
- true /* hidden */))
+ false /* hidden */))
.put(
S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY,
stringOptionalPropertyEntry(
@@ -46,6 +47,64 @@ public class S3PropertiesMetadata {
false /* immutable */,
null /* defaultValue */,
true /* hidden */))
+ .put(
+ S3Properties.GRAVITINO_S3_ENDPOINT,
+ stringOptionalPropertyEntry(
+ S3Properties.GRAVITINO_S3_ENDPOINT,
+ "Alternative S3 service endpoint",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ S3Properties.GRAVITINO_S3_REGION,
+ stringOptionalPropertyEntry(
+ S3Properties.GRAVITINO_S3_REGION,
+ "S3 service region",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ S3Properties.GRAVITINO_S3_ROLE_ARN,
+ stringOptionalPropertyEntry(
+ S3Properties.GRAVITINO_S3_ROLE_ARN,
+ "S3 IAM role ARN for STS credential vending",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ S3Properties.GRAVITINO_S3_EXTERNAL_ID,
+ stringOptionalPropertyEntry(
+ S3Properties.GRAVITINO_S3_EXTERNAL_ID,
+ "S3 external ID for cross-account AssumeRole",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ S3Properties.GRAVITINO_S3_STS_ENDPOINT,
+ stringOptionalPropertyEntry(
+ S3Properties.GRAVITINO_S3_STS_ENDPOINT,
+ "Alternative STS endpoint for S3-compatible storage",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
+ .put(
+ S3Properties.GRAVITINO_S3_PATH_STYLE_ACCESS,
+ booleanPropertyEntry(
+ S3Properties.GRAVITINO_S3_PATH_STYLE_ACCESS,
+ "Whether to use path-style S3 access",
+ false /* required */,
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */,
+ false /* reserved */))
+ .put(
+ S3Properties.GRAVITINO_S3_CREDS_PROVIDER,
+ stringOptionalPropertyEntry(
+ S3Properties.GRAVITINO_S3_CREDS_PROVIDER,
+ "S3 credentials provider class name",
+ false /* immutable */,
+ null /* defaultValue */,
+ false /* hidden */))
.build();
private S3PropertiesMetadata() {}
diff --git
a/core/src/main/java/org/apache/gravitino/connector/BaseCatalogPropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/connector/BaseCatalogPropertiesMetadata.java
index d304c57b77..5e7e660504 100644
---
a/core/src/main/java/org/apache/gravitino/connector/BaseCatalogPropertiesMetadata.java
+++
b/core/src/main/java/org/apache/gravitino/connector/BaseCatalogPropertiesMetadata.java
@@ -107,23 +107,16 @@ public abstract class BaseCatalogPropertiesMetadata
extends BasePropertiesMetada
if (propertyEntries == null) {
synchronized (this) {
if (propertyEntries == null) {
+ // Reuse BasePropertiesMetadata (specific + BASIC +
CredentialConfig), then add
+ // catalog-only entries.
+ Map<String, PropertyEntry<?>> base = buildBasePropertyEntries();
ImmutableMap.Builder<String, PropertyEntry<?>> builder =
ImmutableMap.builder();
- Map<String, PropertyEntry<?>> properties = specificPropertyEntries();
- builder.putAll(properties);
+ builder.putAll(base);
- // put the basic property entries
- BASIC_PROPERTY_ENTRIES.forEach(
- (name, entry) -> {
- Preconditions.checkArgument(
- !properties.containsKey(name), "Property metadata already
exists: " + name);
- builder.put(name, entry);
- });
-
- // put the basic catalog property entries
BASIC_CATALOG_PROPERTY_ENTRIES.forEach(
(name, entry) -> {
Preconditions.checkArgument(
- !properties.containsKey(name), "Property metadata already
exists: " + name);
+ !base.containsKey(name), "Property metadata already
exists: " + name);
builder.put(name, entry);
});
propertyEntries = builder.build();
diff --git
a/core/src/main/java/org/apache/gravitino/connector/BasePropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/connector/BasePropertiesMetadata.java
index 79c8c742ae..a169f2c641 100644
---
a/core/src/main/java/org/apache/gravitino/connector/BasePropertiesMetadata.java
+++
b/core/src/main/java/org/apache/gravitino/connector/BasePropertiesMetadata.java
@@ -27,6 +27,7 @@ import com.google.common.collect.Maps;
import java.util.List;
import java.util.Map;
import org.apache.gravitino.annotation.Evolving;
+import org.apache.gravitino.credential.config.CredentialConfig;
/**
* An abstract class representing a base properties metadata for entities.
Developers should extend
@@ -62,24 +63,43 @@ public abstract class BasePropertiesMetadata implements
PropertiesMetadata {
if (propertyEntries == null) {
synchronized (this) {
if (propertyEntries == null) {
- ImmutableMap.Builder<String, PropertyEntry<?>> builder =
ImmutableMap.builder();
- Map<String, PropertyEntry<?>> properties = specificPropertyEntries();
- builder.putAll(properties);
-
- BASIC_PROPERTY_ENTRIES.forEach(
- (name, entry) -> {
- Preconditions.checkArgument(
- !properties.containsKey(name), "Property metadata already
exists: " + name);
- builder.put(name, entry);
- });
-
- propertyEntries = builder.build();
+ propertyEntries = buildBasePropertyEntries();
}
}
}
return propertyEntries;
}
+ /**
+ * Builds specific + shared base property entries (including {@link
CredentialConfig}) without
+ * caching. Catalog metadata subclasses add catalog-only entries on top of
this map.
+ *
+ * @return an immutable property entry map
+ */
+ protected final Map<String, PropertyEntry<?>> buildBasePropertyEntries() {
+ ImmutableMap.Builder<String, PropertyEntry<?>> builder =
ImmutableMap.builder();
+ Map<String, PropertyEntry<?>> properties = specificPropertyEntries();
+ builder.putAll(properties);
+
+ BASIC_PROPERTY_ENTRIES.forEach(
+ (name, entry) -> {
+ Preconditions.checkArgument(
+ !properties.containsKey(name), "Property metadata already
exists: " + name);
+ builder.put(name, entry);
+ });
+
+ // Credential vending keys (e.g. credential-providers) are valid on schema
/ fileset / table as
+ // well as catalog. Register once so official non-hidden keys are not
fuzzy-masked.
+ CredentialConfig.CREDENTIAL_PROPERTY_ENTRIES.forEach(
+ (name, entry) -> {
+ Preconditions.checkArgument(
+ !properties.containsKey(name), "Property metadata already
exists: " + name);
+ builder.put(name, entry);
+ });
+
+ return builder.build();
+ }
+
/**
* Returns the specific property entries for the entity. Developers should
override this method to
* provide the specific property entries for their entities.
diff --git
a/core/src/main/java/org/apache/gravitino/connector/HiddenPropertyMaskUtils.java
b/core/src/main/java/org/apache/gravitino/connector/HiddenPropertyMaskUtils.java
index 9b174a9838..9788966b95 100644
---
a/core/src/main/java/org/apache/gravitino/connector/HiddenPropertyMaskUtils.java
+++
b/core/src/main/java/org/apache/gravitino/connector/HiddenPropertyMaskUtils.java
@@ -111,7 +111,7 @@ public final class HiddenPropertyMaskUtils {
keysToOmit.add(key);
} else if (hidden
|| SecretPropertyUtils.isSecretProperty(key, value)
- || SecretPropertyUtils.isSensitivePropertyKey(key)) {
+ || (!metadata.containsProperty(key) &&
SecretPropertyUtils.isSensitivePropertyKey(key))) {
keysToMask.add(key);
}
}
diff --git
a/core/src/main/java/org/apache/gravitino/credential/config/CredentialConfig.java
b/core/src/main/java/org/apache/gravitino/credential/config/CredentialConfig.java
index b08bda8569..2234b648d0 100644
---
a/core/src/main/java/org/apache/gravitino/credential/config/CredentialConfig.java
+++
b/core/src/main/java/org/apache/gravitino/credential/config/CredentialConfig.java
@@ -67,6 +67,46 @@ public class CredentialConfig extends Config {
DEFAULT_CREDENTIAL_CACHE_MAX_SIZE /* default value */,
false /* hidden */,
false /* reserved */))
+ .put(
+ CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS,
+ PropertyEntry.integerPropertyEntry(
+ CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS,
+ "S3 STS token expire time in seconds.",
+ false /* required */,
+ false /* immutable */,
+ null /* default value */,
+ false /* hidden */,
+ false /* reserved */))
+ .put(
+ CredentialConstants.OSS_TOKEN_EXPIRE_IN_SECS,
+ PropertyEntry.integerPropertyEntry(
+ CredentialConstants.OSS_TOKEN_EXPIRE_IN_SECS,
+ "OSS security token expire time in seconds.",
+ false /* required */,
+ false /* immutable */,
+ null /* default value */,
+ false /* hidden */,
+ false /* reserved */))
+ .put(
+ CredentialConstants.ADLS_TOKEN_EXPIRE_IN_SECS,
+ PropertyEntry.integerPropertyEntry(
+ CredentialConstants.ADLS_TOKEN_EXPIRE_IN_SECS,
+ "ADLS SAS token expire time in seconds.",
+ false /* required */,
+ false /* immutable */,
+ null /* default value */,
+ false /* hidden */,
+ false /* reserved */))
+ .put(
+ CredentialConstants.COS_TOKEN_EXPIRE_IN_SECS,
+ PropertyEntry.integerPropertyEntry(
+ CredentialConstants.COS_TOKEN_EXPIRE_IN_SECS,
+ "COS STS token expire time in seconds.",
+ false /* required */,
+ false /* immutable */,
+ null /* default value */,
+ false /* hidden */,
+ false /* reserved */))
.build();
public static final ConfigEntry<List<String>> CREDENTIAL_PROVIDERS =
diff --git
a/core/src/main/java/org/apache/gravitino/secret/FallbackPropertiesMetadata.java
b/core/src/main/java/org/apache/gravitino/secret/FallbackPropertiesMetadata.java
new file mode 100644
index 0000000000..ba76bdd687
--- /dev/null
+++
b/core/src/main/java/org/apache/gravitino/secret/FallbackPropertiesMetadata.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.secret;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.Map;
+import org.apache.gravitino.cloud.storage.AzurePropertiesMetadata;
+import org.apache.gravitino.cloud.storage.COSPropertiesMetadata;
+import org.apache.gravitino.cloud.storage.GCSPropertiesMetadata;
+import org.apache.gravitino.cloud.storage.OSSPropertiesMetadata;
+import org.apache.gravitino.cloud.storage.S3PropertiesMetadata;
+import org.apache.gravitino.connector.BasePropertiesMetadata;
+import org.apache.gravitino.connector.PropertyEntry;
+
+/**
+ * Fallback {@link org.apache.gravitino.connector.PropertiesMetadata} when a
catalog does not
+ * support properties metadata for an entity type ({@link
UnsupportedOperationException}).
+ *
+ * <p>Registers the shared base + credential-vending + cloud-storage property
entries so officially
+ * non-hidden keys (for example {@code credential-providers}, {@code
s3-access-key-id}) are not
+ * fuzzy-recovered into {@code getSecrets}. Undeclared sensitive-named keys
still use fuzzy
+ * recovery; declared hidden secrets (for example {@code
s3-secret-access-key}) remain recoverable.
+ */
+final class FallbackPropertiesMetadata extends BasePropertiesMetadata {
+
+ static final FallbackPropertiesMetadata INSTANCE = new
FallbackPropertiesMetadata();
+
+ private static final Map<String, PropertyEntry<?>> CLOUD_PROPERTY_ENTRIES =
+ ImmutableMap.<String, PropertyEntry<?>>builder()
+ .putAll(S3PropertiesMetadata.PROPERTY_ENTRIES)
+ .putAll(OSSPropertiesMetadata.PROPERTY_ENTRIES)
+ .putAll(AzurePropertiesMetadata.PROPERTY_ENTRIES)
+ .putAll(GCSPropertiesMetadata.PROPERTY_ENTRIES)
+ .putAll(COSPropertiesMetadata.PROPERTY_ENTRIES)
+ .build();
+
+ private FallbackPropertiesMetadata() {}
+
+ @Override
+ protected Map<String, PropertyEntry<?>> specificPropertyEntries() {
+ return CLOUD_PROPERTY_ENTRIES;
+ }
+}
diff --git
a/core/src/main/java/org/apache/gravitino/secret/SecretPropertyOperationDispatcher.java
b/core/src/main/java/org/apache/gravitino/secret/SecretPropertyOperationDispatcher.java
index d766eabc9f..8b22dc8516 100644
---
a/core/src/main/java/org/apache/gravitino/secret/SecretPropertyOperationDispatcher.java
+++
b/core/src/main/java/org/apache/gravitino/secret/SecretPropertyOperationDispatcher.java
@@ -20,12 +20,15 @@ package org.apache.gravitino.secret;
import java.io.IOException;
import java.util.Map;
+import javax.annotation.Nullable;
import javax.ws.rs.NotSupportedException;
import org.apache.gravitino.Entity;
import org.apache.gravitino.EntityStore;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.catalog.CatalogManager;
import org.apache.gravitino.catalog.OperationDispatcher;
+import org.apache.gravitino.connector.HasPropertyMetadata;
+import org.apache.gravitino.connector.PropertiesMetadata;
import org.apache.gravitino.exceptions.NoSuchCatalogException;
import org.apache.gravitino.exceptions.NoSuchEntityException;
import org.apache.gravitino.exceptions.NoSuchFilesetException;
@@ -40,12 +43,14 @@ import org.apache.gravitino.messaging.Topic;
import org.apache.gravitino.meta.BaseMetalake;
import org.apache.gravitino.meta.FilesetEntity;
import org.apache.gravitino.meta.SchemaEntity;
+import org.apache.gravitino.metalake.MetalakePropertiesMetadata;
import org.apache.gravitino.model.Model;
import org.apache.gravitino.model.ModelVersion;
import org.apache.gravitino.rel.Table;
import org.apache.gravitino.rel.View;
import org.apache.gravitino.storage.IdGenerator;
import org.apache.gravitino.utils.NameIdentifierUtil;
+import org.apache.gravitino.utils.ThrowableFunction;
/**
* Dispatches secrets requests for metalake, catalog, schema, fileset, table,
topic, view, model,
@@ -82,70 +87,70 @@ public class SecretPropertyOperationDispatcher extends
OperationDispatcher {
* @return secret plaintext properties; never null
*/
public Map<String, String> getSecrets(NameIdentifier identifier,
Entity.EntityType entityType) {
- Map<String, String> rawProperties = loadRawProperties(identifier,
entityType);
- return SecretPropertyUtils.buildSecrets(secretManager, rawProperties);
+ RawPropertiesAndMetadata loaded = loadRawPropertiesAndMetadata(identifier,
entityType);
+ return SecretPropertyUtils.buildSecrets(secretManager,
loaded.rawProperties, loaded.metadata);
}
- private Map<String, String> loadRawProperties(
+ /**
+ * Loads raw properties and matching properties metadata in one catalog
lease when possible.
+ *
+ * <p>If the catalog does not expose properties metadata for the entity type
({@link
+ * UnsupportedOperationException}), uses {@link FallbackPropertiesMetadata}
which still registers
+ * shared base / credential / cloud entries so official non-hidden keys are
not fuzzy-recovered,
+ * while undeclared sensitive-named keys keep fuzzy recovery.
+ */
+ private RawPropertiesAndMetadata loadRawPropertiesAndMetadata(
NameIdentifier identifier, Entity.EntityType entityType) {
switch (entityType) {
case METALAKE:
- return loadMetalakeRawProperties(identifier);
+ return new RawPropertiesAndMetadata(
+ loadMetalakeRawProperties(identifier), new
MetalakePropertiesMetadata());
case CATALOG:
- return loadCatalogRawProperties(identifier);
+ return doWithCatalog(
+ identifier,
+ wrapper -> {
+ wrapper.catalog().checkMetalakeInUse();
+ return new RawPropertiesAndMetadata(
+ wrapper.catalog().entity().getProperties(),
+ resolvePropertiesMetadata(
+ wrapper,
HasPropertyMetadata::catalogPropertiesMetadata));
+ },
+ NoSuchCatalogException.class);
case SCHEMA:
- return loadSchemaRawProperties(identifier);
+ return loadSchemaRawPropertiesAndMetadata(identifier);
case FILESET:
- return loadFilesetRawProperties(identifier);
+ return loadFilesetRawPropertiesAndMetadata(identifier);
case TABLE:
- return loadTableRawProperties(identifier);
+ return loadTableRawPropertiesAndMetadata(identifier);
case TOPIC:
- return loadTopicRawProperties(identifier);
+ return loadTopicRawPropertiesAndMetadata(identifier);
case VIEW:
- return loadViewRawProperties(identifier);
+ return loadViewRawPropertiesAndMetadata(identifier);
case MODEL:
- return loadModelRawProperties(identifier);
+ return loadModelRawPropertiesAndMetadata(identifier);
case MODEL_VERSION:
- return loadModelVersionRawProperties(identifier);
+ return loadModelVersionRawPropertiesAndMetadata(identifier);
default:
throw new NotSupportedException(
"Doesn't support secret property operations for entity type: " +
entityType);
}
}
- private Map<String, String> loadMetalakeRawProperties(NameIdentifier
identifier) {
- try {
- BaseMetalake entity = store.get(identifier, Entity.EntityType.METALAKE,
BaseMetalake.class);
- return entity.properties() == null ? Map.of() : entity.properties();
- } catch (NoSuchEntityException e) {
- throw new NoSuchMetalakeException(e, "Metalake %s does not exist",
identifier);
- } catch (IOException e) {
- throw new RuntimeException("Failed to load metalake entity " +
identifier, e);
- }
- }
-
- private Map<String, String> loadCatalogRawProperties(NameIdentifier
identifier) {
- return doWithCatalog(
- identifier,
- wrapper -> {
- wrapper.catalog().checkMetalakeInUse();
- return wrapper.catalog().entity().getProperties();
- },
- NoSuchCatalogException.class);
- }
-
- private Map<String, String> loadSchemaRawProperties(NameIdentifier
identifier) {
+ private RawPropertiesAndMetadata
loadSchemaRawPropertiesAndMetadata(NameIdentifier identifier) {
NameIdentifier catalogIdent =
NameIdentifierUtil.getCatalogIdentifier(identifier);
- doWithCatalog(
- catalogIdent,
- wrapper -> {
- wrapper.catalog().checkMetalakeInUse();
- return null;
- },
- NoSuchCatalogException.class);
+ PropertiesMetadata metadata =
+ doWithCatalog(
+ catalogIdent,
+ wrapper -> {
+ wrapper.catalog().checkMetalakeInUse();
+ return resolvePropertiesMetadata(
+ wrapper, HasPropertyMetadata::schemaPropertiesMetadata);
+ },
+ NoSuchCatalogException.class);
try {
SchemaEntity entity = store.get(identifier, Entity.EntityType.SCHEMA,
SchemaEntity.class);
- return entity.properties() == null ? Map.of() : entity.properties();
+ return new RawPropertiesAndMetadata(
+ entity.properties() == null ? Map.of() : entity.properties(),
metadata);
} catch (NoSuchEntityException e) {
throw new NoSuchSchemaException(e, "Schema %s does not exist",
identifier);
} catch (IOException e) {
@@ -153,18 +158,21 @@ public class SecretPropertyOperationDispatcher extends
OperationDispatcher {
}
}
- private Map<String, String> loadFilesetRawProperties(NameIdentifier
identifier) {
+ private RawPropertiesAndMetadata
loadFilesetRawPropertiesAndMetadata(NameIdentifier identifier) {
NameIdentifier catalogIdent =
NameIdentifierUtil.getCatalogIdentifier(identifier);
- doWithCatalog(
- catalogIdent,
- wrapper -> {
- wrapper.catalog().checkMetalakeInUse();
- return null;
- },
- NoSuchCatalogException.class);
+ PropertiesMetadata metadata =
+ doWithCatalog(
+ catalogIdent,
+ wrapper -> {
+ wrapper.catalog().checkMetalakeInUse();
+ return resolvePropertiesMetadata(
+ wrapper, HasPropertyMetadata::filesetPropertiesMetadata);
+ },
+ NoSuchCatalogException.class);
try {
FilesetEntity entity = store.get(identifier, Entity.EntityType.FILESET,
FilesetEntity.class);
- return entity.properties() == null ? Map.of() : entity.properties();
+ return new RawPropertiesAndMetadata(
+ entity.properties() == null ? Map.of() : entity.properties(),
metadata);
} catch (NoSuchEntityException e) {
throw new NoSuchFilesetException(e, "Fileset %s does not exist",
identifier);
} catch (IOException e) {
@@ -172,7 +180,7 @@ public class SecretPropertyOperationDispatcher extends
OperationDispatcher {
}
}
- private Map<String, String> loadTableRawProperties(NameIdentifier
identifier) {
+ private RawPropertiesAndMetadata
loadTableRawPropertiesAndMetadata(NameIdentifier identifier) {
NameIdentifier catalogIdent =
NameIdentifierUtil.getCatalogIdentifier(identifier);
return doWithCatalog(
catalogIdent,
@@ -181,52 +189,66 @@ public class SecretPropertyOperationDispatcher extends
OperationDispatcher {
// Load from the connector so we get the same raw property map that
EntityCombinedTable
// masks for API responses (including Flink connector options like
flink.password).
Table table = wrapper.doWithTableOps(ops ->
ops.loadTable(identifier));
- return table.properties() == null ? Map.of() : table.properties();
+ Map<String, String> raw = table.properties() == null ? Map.of() :
table.properties();
+ return new RawPropertiesAndMetadata(
+ raw,
+ resolvePropertiesMetadata(wrapper,
HasPropertyMetadata::tablePropertiesMetadata));
},
NoSuchCatalogException.class,
NoSuchTableException.class);
}
- private Map<String, String> loadTopicRawProperties(NameIdentifier
identifier) {
+ private RawPropertiesAndMetadata
loadTopicRawPropertiesAndMetadata(NameIdentifier identifier) {
NameIdentifier catalogIdent =
NameIdentifierUtil.getCatalogIdentifier(identifier);
return doWithCatalog(
catalogIdent,
wrapper -> {
wrapper.catalog().checkMetalakeInUse();
Topic topic = wrapper.doWithTopicOps(ops ->
ops.loadTopic(identifier));
- return topic.properties() == null ? Map.of() : topic.properties();
+ Map<String, String> raw = topic.properties() == null ? Map.of() :
topic.properties();
+ return new RawPropertiesAndMetadata(
+ raw,
+ resolvePropertiesMetadata(wrapper,
HasPropertyMetadata::topicPropertiesMetadata));
},
NoSuchCatalogException.class,
NoSuchTopicException.class);
}
- private Map<String, String> loadViewRawProperties(NameIdentifier identifier)
{
+ private RawPropertiesAndMetadata
loadViewRawPropertiesAndMetadata(NameIdentifier identifier) {
NameIdentifier catalogIdent =
NameIdentifierUtil.getCatalogIdentifier(identifier);
return doWithCatalog(
catalogIdent,
wrapper -> {
wrapper.catalog().checkMetalakeInUse();
View view = wrapper.doWithViewOps(ops -> ops.loadView(identifier));
- return view.properties() == null ? Map.of() : view.properties();
+ Map<String, String> raw = view.properties() == null ? Map.of() :
view.properties();
+ // View masking uses table properties metadata elsewhere in
OperationDispatcher.
+ return new RawPropertiesAndMetadata(
+ raw,
+ resolvePropertiesMetadata(wrapper,
HasPropertyMetadata::tablePropertiesMetadata));
},
NoSuchCatalogException.class,
NoSuchViewException.class);
}
- private Map<String, String> loadModelRawProperties(NameIdentifier
identifier) {
+ private RawPropertiesAndMetadata
loadModelRawPropertiesAndMetadata(NameIdentifier identifier) {
NameIdentifier catalogIdent =
NameIdentifierUtil.getCatalogIdentifier(identifier);
return doWithCatalog(
catalogIdent,
wrapper -> {
wrapper.catalog().checkMetalakeInUse();
Model model = wrapper.doWithModelOps(ops ->
ops.getModel(identifier));
- return model.properties() == null ? Map.of() : model.properties();
+ Map<String, String> raw = model.properties() == null ? Map.of() :
model.properties();
+ return new RawPropertiesAndMetadata(
+ raw,
+ resolvePropertiesMetadata(wrapper,
HasPropertyMetadata::modelPropertiesMetadata));
},
NoSuchCatalogException.class,
NoSuchModelException.class);
}
- private Map<String, String> loadModelVersionRawProperties(NameIdentifier
identifier) {
+ private RawPropertiesAndMetadata loadModelVersionRawPropertiesAndMetadata(
+ NameIdentifier identifier) {
NameIdentifier catalogIdent =
NameIdentifierUtil.getCatalogIdentifier(identifier);
NameIdentifier modelIdent =
NameIdentifier.of(identifier.namespace().levels());
String versionName = identifier.name();
@@ -242,9 +264,54 @@ public class SecretPropertyOperationDispatcher extends
OperationDispatcher {
modelVersion =
wrapper.doWithModelOps(ops -> ops.getModelVersion(modelIdent,
versionName));
}
- return modelVersion.properties() == null ? Map.of() :
modelVersion.properties();
+ Map<String, String> raw =
+ modelVersion.properties() == null ? Map.of() :
modelVersion.properties();
+ return new RawPropertiesAndMetadata(
+ raw,
+ resolvePropertiesMetadata(
+ wrapper,
HasPropertyMetadata::modelVersionPropertiesMetadata));
},
NoSuchCatalogException.class,
NoSuchModelVersionException.class);
}
+
+ private Map<String, String> loadMetalakeRawProperties(NameIdentifier
identifier) {
+ try {
+ BaseMetalake entity = store.get(identifier, Entity.EntityType.METALAKE,
BaseMetalake.class);
+ return entity.properties() == null ? Map.of() : entity.properties();
+ } catch (NoSuchEntityException e) {
+ throw new NoSuchMetalakeException(e, "Metalake %s does not exist",
identifier);
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to load metalake entity " +
identifier, e);
+ }
+ }
+
+ /**
+ * Returns properties metadata under the catalog connector classloader, or
{@link
+ * FallbackPropertiesMetadata#INSTANCE} when the catalog does not support it
for this entity type.
+ */
+ static PropertiesMetadata resolvePropertiesMetadata(
+ CatalogManager.CatalogWrapper wrapper,
+ ThrowableFunction<HasPropertyMetadata, PropertiesMetadata> getter) {
+ try {
+ return wrapper.doWithPropertiesMeta(getter);
+ } catch (UnsupportedOperationException e) {
+ return FallbackPropertiesMetadata.INSTANCE;
+ } catch (RuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private static final class RawPropertiesAndMetadata {
+ private final Map<String, String> rawProperties;
+ private final PropertiesMetadata metadata;
+
+ private RawPropertiesAndMetadata(
+ @Nullable Map<String, String> rawProperties, PropertiesMetadata
metadata) {
+ this.rawProperties = rawProperties == null ? Map.of() : rawProperties;
+ this.metadata = metadata;
+ }
+ }
}
diff --git
a/core/src/main/java/org/apache/gravitino/secret/SecretPropertyUtils.java
b/core/src/main/java/org/apache/gravitino/secret/SecretPropertyUtils.java
index 7e87473a5f..19f0b89507 100644
--- a/core/src/main/java/org/apache/gravitino/secret/SecretPropertyUtils.java
+++ b/core/src/main/java/org/apache/gravitino/secret/SecretPropertyUtils.java
@@ -28,6 +28,8 @@ import java.util.Map;
import java.util.regex.Pattern;
import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
+import org.apache.gravitino.connector.PropertiesMetadata;
+import org.apache.gravitino.connector.PropertyEntry;
/**
* Helpers for secret-related entity property handling and request validation.
@@ -45,6 +47,15 @@ public final class SecretPropertyUtils {
private static final Pattern SENSITIVE_PROPERTY_KEY_PATTERN =
Pattern.compile(".*(secret|password|token|credential|access|account).*");
+ /** Empty metadata: every property key is undeclared (used for historical
fuzzy recovery). */
+ private static final PropertiesMetadata EMPTY_PROPERTIES_METADATA =
+ new PropertiesMetadata() {
+ @Override
+ public Map<String, PropertyEntry<?>> propertyEntries() {
+ return Map.of();
+ }
+ };
+
private SecretPropertyUtils() {}
/**
@@ -65,6 +76,38 @@ public final class SecretPropertyUtils {
return
SENSITIVE_PROPERTY_KEY_PATTERN.matcher(key.toLowerCase(Locale.ROOT)).matches();
}
+ /**
+ * Returns whether a sensitive-named inline property should be recovered via
{@code getSecrets}.
+ *
+ * <p>Secret-manager URNs are always recovered separately. For inline
plaintext:
+ *
+ * <ul>
+ * <li>{@code metadata == null}: do <strong>not</strong> recover
(URN-only). Used when the
+ * catalog does not expose properties metadata for the entity type.
+ * <li>otherwise: recover only undeclared keys or declared {@code hidden}
keys. Declared
+ * non-hidden configuration (for example {@code credential-providers},
{@code
+ * s3-access-key-id}) stays in {@code properties()} and is excluded
here.
+ * </ul>
+ *
+ * <p>Callers that need historical fuzzy recovery without real metadata
should pass an empty
+ * {@link PropertiesMetadata} (every key is undeclared) — see the
two-argument {@link
+ * #buildSecrets(SecretManager, Map)}.
+ *
+ * @param key the property key
+ * @param metadata entity properties metadata, or null when unavailable
+ * @return true when the inline plaintext should be included in {@code
getSecrets}
+ */
+ public static boolean shouldRecoverSensitiveNamedSecret(
+ String key, @Nullable PropertiesMetadata metadata) {
+ if (!isSensitivePropertyKey(key)) {
+ return false;
+ }
+ if (metadata == null) {
+ return false;
+ }
+ return !metadata.containsProperty(key) || metadata.isHiddenProperty(key);
+ }
+
/**
* Returns whether a property value is a Gravitino secret URN for the given
key.
*
@@ -76,6 +119,23 @@ public final class SecretPropertyUtils {
return key != null && value != null && value.startsWith(URN_PREFIX) &&
value.endsWith(key);
}
+ /**
+ * Builds a map of plaintext secret properties for {@code getSecrets} with
historical fuzzy
+ * recovery for sensitive-named keys.
+ *
+ * <p>Delegates to {@link #buildSecrets(SecretManager, Map,
PropertiesMetadata)} with an empty
+ * properties metadata so every key is treated as undeclared. Prefer the
three-argument overload
+ * when real entity metadata is available.
+ *
+ * @param secretManager secret manager used to resolve URNs
+ * @param rawProperties raw entity properties (may be null)
+ * @return a new secret plaintext property map; never null
+ */
+ public static Map<String, String> buildSecrets(
+ SecretManager secretManager, @Nullable Map<String, String>
rawProperties) {
+ return buildSecrets(secretManager, rawProperties,
EMPTY_PROPERTIES_METADATA);
+ }
+
/**
* Builds a map of plaintext secret properties for {@code getSecrets}.
*
@@ -85,7 +145,10 @@ public final class SecretPropertyUtils {
* <li>Include every entry where {@link #isSecretProperty} is true,
resolving the secret URN via
* {@link SecretManager#readSecret}.
* <li>Include every entry whose key matches {@link
#isSensitivePropertyKey} and whose value is
- * not a secret URN, returning the stored plaintext.
+ * not a secret URN, when {@link #shouldRecoverSensitiveNamedSecret}
is true (undeclared or
+ * declared hidden). Declared non-hidden keys are excluded even when
the name looks
+ * sensitive. When {@code metadata} is {@code null}, sensitive-named
plaintext is not
+ * recovered (URN-only).
* </ol>
*
* <p>Declared {@code hidden} properties are <strong>not</strong> included
merely because they are
@@ -99,10 +162,13 @@ public final class SecretPropertyUtils {
*
* @param secretManager secret manager used to resolve URNs
* @param rawProperties raw entity properties (may be null)
+ * @param metadata entity properties metadata, or null when unavailable
(URN-only recovery)
* @return a new secret plaintext property map; never null
*/
public static Map<String, String> buildSecrets(
- SecretManager secretManager, @Nullable Map<String, String>
rawProperties) {
+ SecretManager secretManager,
+ @Nullable Map<String, String> rawProperties,
+ @Nullable PropertiesMetadata metadata) {
Preconditions.checkArgument(secretManager != null, "secretManager must not
be null");
if (rawProperties == null || rawProperties.isEmpty()) {
return Map.of();
@@ -116,7 +182,7 @@ public final class SecretPropertyUtils {
}
if (isSecretProperty(key, value)) {
secrets.put(key, secretManager.readSecret(SecretUrn.parse(value)));
- } else if (isSensitivePropertyKey(key)) {
+ } else if (shouldRecoverSensitiveNamedSecret(key, metadata)) {
secrets.put(key, value);
}
}
diff --git
a/core/src/test/java/org/apache/gravitino/TestBasePropertiesMetadata.java
b/core/src/test/java/org/apache/gravitino/TestBasePropertiesMetadata.java
index 44281d7067..c96e2637d9 100644
--- a/core/src/test/java/org/apache/gravitino/TestBasePropertiesMetadata.java
+++ b/core/src/test/java/org/apache/gravitino/TestBasePropertiesMetadata.java
@@ -19,6 +19,7 @@
package org.apache.gravitino;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -29,6 +30,7 @@ import java.util.List;
import java.util.Map;
import org.apache.gravitino.connector.BasePropertiesMetadata;
import org.apache.gravitino.connector.PropertyEntry;
+import org.apache.gravitino.credential.CredentialConstants;
import org.junit.jupiter.api.Test;
public class TestBasePropertiesMetadata extends BasePropertiesMetadata {
@@ -93,4 +95,14 @@ public class TestBasePropertiesMetadata extends
BasePropertiesMetadata {
assertThrows(IllegalArgumentException.class, () ->
metadata.getPropertyEntry(""));
assertTrue(emptyException.getMessage().contains("Property is not
defined"));
}
+
+ @Test
+ public void testCredentialPropertyEntriesAreDeclaredForAllEntities() {
+ TestBasePropertiesMetadata metadata = new TestBasePropertiesMetadata();
+
+
assertTrue(metadata.containsProperty(CredentialConstants.CREDENTIAL_PROVIDERS));
+
assertTrue(metadata.containsProperty(CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS));
+
assertFalse(metadata.isHiddenProperty(CredentialConstants.CREDENTIAL_PROVIDERS));
+
assertFalse(metadata.isHiddenProperty(CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS));
+ }
}
diff --git
a/core/src/test/java/org/apache/gravitino/cloud/storage/TestCloudPropertiesMetadata.java
b/core/src/test/java/org/apache/gravitino/cloud/storage/TestCloudPropertiesMetadata.java
new file mode 100644
index 0000000000..7704948ae7
--- /dev/null
+++
b/core/src/test/java/org/apache/gravitino/cloud/storage/TestCloudPropertiesMetadata.java
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.cloud.storage;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.apache.gravitino.credential.CredentialConstants;
+import org.apache.gravitino.credential.config.CredentialConfig;
+import org.apache.gravitino.secret.SecretPropertyUtils;
+import org.apache.gravitino.storage.AzureProperties;
+import org.apache.gravitino.storage.COSProperties;
+import org.apache.gravitino.storage.OSSProperties;
+import org.apache.gravitino.storage.S3Properties;
+import org.junit.jupiter.api.Test;
+
+public class TestCloudPropertiesMetadata {
+
+ @Test
+ void testS3CredentialConfigPropertiesAreDeclared() {
+ var metadata = S3PropertiesMetadata.PROPERTY_ENTRIES;
+ assertTrue(metadata.containsKey(S3Properties.GRAVITINO_S3_ENDPOINT));
+ assertTrue(metadata.containsKey(S3Properties.GRAVITINO_S3_REGION));
+ assertTrue(metadata.containsKey(S3Properties.GRAVITINO_S3_ROLE_ARN));
+ assertFalse(metadata.get(S3Properties.GRAVITINO_S3_ENDPOINT).isHidden());
+
assertFalse(metadata.get(S3Properties.GRAVITINO_S3_ACCESS_KEY_ID).isHidden());
+
assertTrue(metadata.get(S3Properties.GRAVITINO_S3_SECRET_ACCESS_KEY).isHidden());
+ }
+
+ @Test
+ void testOssCredentialConfigPropertiesAreDeclared() {
+ var metadata = OSSPropertiesMetadata.PROPERTY_ENTRIES;
+ assertTrue(metadata.containsKey(OSSProperties.GRAVITINO_OSS_ENDPOINT));
+ assertTrue(metadata.containsKey(OSSProperties.GRAVITINO_OSS_ROLE_ARN));
+ assertFalse(metadata.get(OSSProperties.GRAVITINO_OSS_REGION).isHidden());
+ }
+
+ @Test
+ void testAzureAdlsPropertiesAreDeclared() {
+ var metadata = AzurePropertiesMetadata.PROPERTY_ENTRIES;
+
assertTrue(metadata.containsKey(AzureProperties.GRAVITINO_AZURE_TENANT_ID));
+
assertTrue(metadata.containsKey(AzureProperties.GRAVITINO_AZURE_CLIENT_ID));
+
assertFalse(metadata.get(AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_NAME).isHidden());
+
assertTrue(metadata.get(AzureProperties.GRAVITINO_AZURE_CLIENT_SECRET).isHidden());
+ }
+
+ @Test
+ void testCosCredentialConfigPropertiesAreDeclared() {
+ var metadata = COSPropertiesMetadata.PROPERTY_ENTRIES;
+ assertTrue(metadata.containsKey(COSProperties.GRAVITINO_COS_REGION));
+ assertTrue(metadata.containsKey(COSProperties.GRAVITINO_COS_ENDPOINT));
+ assertTrue(metadata.containsKey(COSProperties.GRAVITINO_COS_ROLE_ARN));
+ assertTrue(metadata.containsKey(COSProperties.GRAVITINO_COS_EXTERNAL_ID));
+ assertTrue(metadata.containsKey(COSProperties.GRAVITINO_COS_APP_ID));
+
assertFalse(metadata.get(COSProperties.GRAVITINO_COS_ACCESS_KEY_ID).isHidden());
+
assertTrue(metadata.get(COSProperties.GRAVITINO_COS_ACCESS_KEY_SECRET).isHidden());
+ }
+
+ @Test
+ void testCredentialPropertyEntriesIncludeTokenExpireKeys() {
+ var metadata = CredentialConfig.CREDENTIAL_PROPERTY_ENTRIES;
+ assertTrue(metadata.containsKey(CredentialConstants.CREDENTIAL_PROVIDERS));
+
assertTrue(metadata.containsKey(CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS));
+
assertTrue(metadata.containsKey(CredentialConstants.ADLS_TOKEN_EXPIRE_IN_SECS));
+
assertTrue(metadata.containsKey(CredentialConstants.COS_TOKEN_EXPIRE_IN_SECS));
+
assertFalse(metadata.get(CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS).isHidden());
+ }
+
+ @Test
+ void testDeclaredSensitiveNamedCredentialKeysAreNonHidden() {
+ assertFalse(
+ CredentialConfig.CREDENTIAL_PROPERTY_ENTRIES
+ .get(CredentialConstants.CREDENTIAL_PROVIDERS)
+ .isHidden());
+ assertTrue(
+
SecretPropertyUtils.isSensitivePropertyKey(CredentialConstants.CREDENTIAL_PROVIDERS));
+ assertFalse(
+ CredentialConfig.CREDENTIAL_PROPERTY_ENTRIES
+ .get(CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS)
+ .isHidden());
+ assertTrue(
+
SecretPropertyUtils.isSensitivePropertyKey(CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS));
+ assertFalse(
+ AzurePropertiesMetadata.PROPERTY_ENTRIES
+ .get(AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_NAME)
+ .isHidden());
+ assertTrue(
+ SecretPropertyUtils.isSensitivePropertyKey(
+ AzureProperties.GRAVITINO_AZURE_STORAGE_ACCOUNT_NAME));
+ }
+}
diff --git
a/core/src/test/java/org/apache/gravitino/connector/TestBaseCatalogPropertiesMetadata.java
b/core/src/test/java/org/apache/gravitino/connector/TestBaseCatalogPropertiesMetadata.java
new file mode 100644
index 0000000000..3e7c801e2f
--- /dev/null
+++
b/core/src/test/java/org/apache/gravitino/connector/TestBaseCatalogPropertiesMetadata.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.connector;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.Collections;
+import java.util.Map;
+import org.apache.gravitino.credential.CredentialConstants;
+import org.junit.jupiter.api.Test;
+
+public class TestBaseCatalogPropertiesMetadata {
+
+ private final PropertiesMetadata metadata =
+ new BaseCatalogPropertiesMetadata() {
+ @Override
+ protected Map<String, PropertyEntry<?>> specificPropertyEntries() {
+ return Collections.emptyMap();
+ }
+ };
+
+ @Test
+ void testCredentialPropertyEntriesAreDeclaredForAllCatalogs() {
+
assertTrue(metadata.containsProperty(CredentialConstants.CREDENTIAL_PROVIDERS));
+
assertTrue(metadata.containsProperty(CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS));
+
assertFalse(metadata.isHiddenProperty(CredentialConstants.CREDENTIAL_PROVIDERS));
+
assertFalse(metadata.isHiddenProperty(CredentialConstants.S3_TOKEN_EXPIRE_IN_SECS));
+ }
+}
diff --git
a/core/src/test/java/org/apache/gravitino/connector/TestHiddenPropertyMaskUtils.java
b/core/src/test/java/org/apache/gravitino/connector/TestHiddenPropertyMaskUtils.java
index 27e323bb73..0a04e119ff 100644
---
a/core/src/test/java/org/apache/gravitino/connector/TestHiddenPropertyMaskUtils.java
+++
b/core/src/test/java/org/apache/gravitino/connector/TestHiddenPropertyMaskUtils.java
@@ -129,7 +129,13 @@ public class TestHiddenPropertyMaskUtils {
return ImmutableMap.of(
"aws-region",
PropertyEntry.stringOptionalPropertyEntry(
- "aws-region", "region", false, null, false));
+ "aws-region", "region", false, null, false),
+ "azure-storage-account-name",
+ PropertyEntry.stringOptionalPropertyEntry(
+ "azure-storage-account-name", "account", false, null,
false),
+ "credential-providers",
+ PropertyEntry.stringOptionalPropertyEntry(
+ "credential-providers", "providers", false, null, false));
}
};
@@ -148,18 +154,20 @@ public class TestHiddenPropertyMaskUtils {
"azure-storage-account-key",
"abs-key",
"azure-storage-account-name",
- "abs-account");
+ "abs-account",
+ "credential-providers",
+ "s3-token");
Map<String, String> masked =
HiddenPropertyMaskUtils.maskHiddenProperties(properties, metadata);
Assertions.assertEquals("us-east-2", masked.get("aws-region"));
Assertions.assertEquals("s3://bucket/prefix", masked.get("warehouse"));
+ Assertions.assertEquals("abs-account",
masked.get("azure-storage-account-name"));
+ Assertions.assertEquals("s3-token", masked.get("credential-providers"));
Assertions.assertEquals(HiddenPropertyMaskUtils.MASKED_VALUE,
masked.get("s3-access-key-id"));
Assertions.assertEquals(
HiddenPropertyMaskUtils.MASKED_VALUE,
masked.get("s3-secret-access-key"));
Assertions.assertEquals(HiddenPropertyMaskUtils.MASKED_VALUE,
masked.get("custom-token"));
Assertions.assertEquals(
HiddenPropertyMaskUtils.MASKED_VALUE,
masked.get("azure-storage-account-key"));
- Assertions.assertEquals(
- HiddenPropertyMaskUtils.MASKED_VALUE,
masked.get("azure-storage-account-name"));
}
}
diff --git
a/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyOperationDispatcher.java
b/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyOperationDispatcher.java
index 49fd8ad039..be0d093cb0 100644
---
a/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyOperationDispatcher.java
+++
b/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyOperationDispatcher.java
@@ -20,6 +20,7 @@ package org.apache.gravitino.secret;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
import com.google.common.collect.ImmutableMap;
import java.io.IOException;
@@ -32,6 +33,7 @@ import org.apache.gravitino.Configs;
import org.apache.gravitino.Entity;
import org.apache.gravitino.GravitinoEnv;
import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.catalog.CatalogManager;
import org.apache.gravitino.catalog.ModelOperationDispatcher;
import org.apache.gravitino.catalog.SchemaDispatcher;
import org.apache.gravitino.catalog.SchemaOperationDispatcher;
@@ -39,6 +41,7 @@ import org.apache.gravitino.catalog.TableOperationDispatcher;
import org.apache.gravitino.catalog.TestOperationDispatcher;
import org.apache.gravitino.catalog.TopicOperationDispatcher;
import org.apache.gravitino.catalog.ViewOperationDispatcher;
+import org.apache.gravitino.connector.HasPropertyMetadata;
import org.apache.gravitino.lock.LockManager;
import org.apache.gravitino.meta.AuditInfo;
import org.apache.gravitino.meta.BaseMetalake;
@@ -170,4 +173,29 @@ public class TestSecretPropertyOperationDispatcher extends
TestOperationDispatch
Assertions.assertEquals("s3cr3t", secrets.get("jdbc-password"));
Assertions.assertFalse(secrets.containsKey("visible"));
}
+
+ @Test
+ public void testResolvePropertiesMetadataFallsBackOnUnsupportedOperation() {
+ CatalogManager.CatalogWrapper wrapper =
mock(CatalogManager.CatalogWrapper.class);
+ try {
+ when(wrapper.doWithPropertiesMeta(org.mockito.ArgumentMatchers.any()))
+ .thenThrow(new UnsupportedOperationException("no metadata"));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ org.apache.gravitino.connector.PropertiesMetadata metadata =
+ SecretPropertyOperationDispatcher.resolvePropertiesMetadata(
+ wrapper, HasPropertyMetadata::catalogPropertiesMetadata);
+ Assertions.assertSame(FallbackPropertiesMetadata.INSTANCE, metadata);
+ // Official non-hidden keys stay out of getSecrets.
+ Assertions.assertFalse(
+
SecretPropertyUtils.shouldRecoverSensitiveNamedSecret("credential-providers",
metadata));
+ Assertions.assertFalse(
+
SecretPropertyUtils.shouldRecoverSensitiveNamedSecret("s3-access-key-id",
metadata));
+ // Declared hidden secrets and undeclared sensitive names still
fuzzy-recover.
+ Assertions.assertTrue(
+
SecretPropertyUtils.shouldRecoverSensitiveNamedSecret("s3-secret-access-key",
metadata));
+ Assertions.assertTrue(
+ SecretPropertyUtils.shouldRecoverSensitiveNamedSecret("custom-token",
metadata));
+ }
}
diff --git
a/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyUtils.java
b/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyUtils.java
index 7201aa3aa0..b0c9da8080 100644
---
a/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyUtils.java
+++
b/core/src/test/java/org/apache/gravitino/secret/TestSecretPropertyUtils.java
@@ -18,11 +18,14 @@
*/
package org.apache.gravitino.secret;
+import com.google.common.collect.ImmutableMap;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import org.apache.gravitino.Config;
+import org.apache.gravitino.connector.PropertiesMetadata;
+import org.apache.gravitino.connector.PropertyEntry;
import org.apache.gravitino.secret.memory.InMemorySecretsProvider;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -159,11 +162,80 @@ public class TestSecretPropertyUtils {
}
}
+ @Test
+ void testBuildSecretsNullMetadataIsUrnOnly() {
+ try (SecretManager sm = memorySecretManager()) {
+ Map<String, String> entityProps = new HashMap<>();
+ entityProps.put("s3-access-key-id", "AKIA");
+ entityProps.put("jdbc-password", "inline-secret");
+ Map<String, SecretBinding> bindings =
+ Map.of("custom-secret", new SecretBinding("memory", "custom-value"));
+ List<SecretMaterial> writes =
+ sm.assembleSecretMaterials(Map.of(), entityProps, "catalog", 42L,
bindings, Map.of());
+ sm.writeSecrets(writes);
+
+ Map<String, String> secrets = SecretPropertyUtils.buildSecrets(sm,
entityProps, null);
+ Assertions.assertEquals("custom-value", secrets.get("custom-secret"));
+ Assertions.assertFalse(secrets.containsKey("s3-access-key-id"));
+ Assertions.assertFalse(secrets.containsKey("jdbc-password"));
+ }
+ }
+
@Test
void testBuildSecretsNullAndEmpty() {
try (SecretManager sm = memorySecretManager()) {
Assertions.assertTrue(SecretPropertyUtils.buildSecrets(sm,
null).isEmpty());
Assertions.assertTrue(SecretPropertyUtils.buildSecrets(sm,
Map.of()).isEmpty());
+ Assertions.assertTrue(SecretPropertyUtils.buildSecrets(sm, null,
null).isEmpty());
+ }
+ }
+
+ @Test
+ void testBuildSecretsExcludesDeclaredNonHiddenSensitiveKeys() {
+ try (SecretManager sm = memorySecretManager()) {
+ PropertiesMetadata metadata =
+ new PropertiesMetadata() {
+ @Override
+ public Map<String, PropertyEntry<?>> propertyEntries() {
+ return ImmutableMap.of(
+ "credential-providers",
+ PropertyEntry.stringOptionalPropertyEntry(
+ "credential-providers", "providers", false, null, false),
+ "azure-storage-account-name",
+ PropertyEntry.stringOptionalPropertyEntry(
+ "azure-storage-account-name", "account", false, null,
false),
+ "s3-access-key-id",
+ PropertyEntry.stringOptionalPropertyEntry(
+ "s3-access-key-id", "ak", false, null, false),
+ "jdbc-password",
+ PropertyEntry.stringOptionalPropertyEntry(
+ "jdbc-password", "password", false, null, true),
+ "s3-secret-access-key",
+ PropertyEntry.stringOptionalPropertyEntry(
+ "s3-secret-access-key", "sk", false, null, true));
+ }
+ };
+ Map<String, String> entityProps =
+ Map.of(
+ "credential-providers",
+ "s3-token",
+ "azure-storage-account-name",
+ "abs-account",
+ "s3-access-key-id",
+ "AKIA",
+ "jdbc-password",
+ "inline-secret",
+ "s3-secret-access-key",
+ "super-secret",
+ "custom-token",
+ "tok");
+ Map<String, String> secrets = SecretPropertyUtils.buildSecrets(sm,
entityProps, metadata);
+ Assertions.assertFalse(secrets.containsKey("credential-providers"));
+
Assertions.assertFalse(secrets.containsKey("azure-storage-account-name"));
+ Assertions.assertFalse(secrets.containsKey("s3-access-key-id"));
+ Assertions.assertEquals("inline-secret", secrets.get("jdbc-password"));
+ Assertions.assertEquals("super-secret",
secrets.get("s3-secret-access-key"));
+ Assertions.assertEquals("tok", secrets.get("custom-token"));
}
}
diff --git
a/trino-connector/trino-connector/src/test/java/org/apache/gravitino/trino/connector/TestGravitinoConnector.java
b/trino-connector/trino-connector/src/test/java/org/apache/gravitino/trino/connector/TestGravitinoConnector.java
index e0a7cf4a79..9d889c5dc3 100644
---
a/trino-connector/trino-connector/src/test/java/org/apache/gravitino/trino/connector/TestGravitinoConnector.java
+++
b/trino-connector/trino-connector/src/test/java/org/apache/gravitino/trino/connector/TestGravitinoConnector.java
@@ -288,6 +288,11 @@ public abstract class TestGravitinoConnector extends
AbstractGravitinoConnectorT
"(?s)Create catalog failed. Create catalog failed due to the loading
process fails\\."
+ ".*unknown-direct-key.*");
assertThat(computeActual("show
catalogs").getOnlyColumnAsSet()).doesNotContain("memory1");
+ // Failed registration still leaves a Gravitino catalog entry visible to
catalog_status /
+ // system.catalog; drop it so later tests that assume only the seed
"memory" catalog stay
+ // deterministic on the shared QueryRunner.
+ assertUpdate(
+ "call gravitino.system.drop_catalog(catalog => 'memory1',
ignore_not_exist => true)");
assertUpdate(
"call gravitino.system.create_catalog("
@@ -301,7 +306,10 @@ public abstract class TestGravitinoConnector extends
AbstractGravitinoConnectorT
@Test
public void testSystemTable() throws Exception {
- MaterializedResult expectedResult = computeActual("select * from
gravitino.system.catalog");
+ // Scope to the seed catalog: other tests may briefly leave FAILED
registrations in the shared
+ // QueryRunner (see testCreateCatalog /
testCatalogStatusReportsARegistrationFailure).
+ MaterializedResult expectedResult =
+ computeActual("select * from gravitino.system.catalog where name =
'memory'");
assertEquals(expectedResult.getRowCount(), 1);
List<MaterializedRow> expectedRows = expectedResult.getMaterializedRows();
MaterializedRow row = expectedRows.get(0);
@@ -315,7 +323,8 @@ public abstract class TestGravitinoConnector extends
AbstractGravitinoConnectorT
MaterializedResult result =
computeActual(
"select metalake, catalog_name, trino_catalog_name, provider,
status, last_error,"
- + " failure_count from gravitino.system.catalog_status");
+ + " failure_count from gravitino.system.catalog_status"
+ + " where catalog_name = 'memory'");
assertEquals(result.getRowCount(), 1);
MaterializedRow row = result.getMaterializedRows().get(0);
assertEquals(row.getField(1), "memory");
@@ -331,7 +340,9 @@ public abstract class TestGravitinoConnector extends
AbstractGravitinoConnectorT
// page.getColumns() honors the requested order; a projection that sorted
or de-duplicated
// channels would return correct looking data in the wrong columns.
MaterializedResult result =
- computeActual("select status, catalog_name, status from
gravitino.system.catalog_status");
+ computeActual(
+ "select status, catalog_name, status from
gravitino.system.catalog_status"
+ + " where catalog_name = 'memory'");
assertEquals(result.getRowCount(), 1);
MaterializedRow row = result.getMaterializedRows().get(0);
assertEquals(row.getField(0), "REGISTERED");
@@ -344,9 +355,14 @@ public abstract class TestGravitinoConnector extends
AbstractGravitinoConnectorT
// count(*) projects no column at all, so the page is built with zero
channels and only its
// position count carries the answer. A table that sized its page from the
projected columns
// would report no rows here while returning them for every other query.
- assertEquals(computeScalar("select count(*) from
gravitino.system.catalog_status"), 1L);
+ // Filter to the seed catalog so leftover FAILED registrations from other
tests do not flake.
+ assertEquals(
+ computeScalar(
+ "select count(*) from gravitino.system.catalog_status where
catalog_name = 'memory'"),
+ 1L);
assertEquals(computeScalar("select count(*) from
gravitino.system.load_status"), 1L);
- assertEquals(computeScalar("select count(*) from
gravitino.system.catalog"), 1L);
+ assertEquals(
+ computeScalar("select count(*) from gravitino.system.catalog where
name = 'memory'"), 1L);
}
@Test
@@ -354,27 +370,30 @@ public abstract class TestGravitinoConnector extends
AbstractGravitinoConnectorT
// An unknown bypass key makes the inner connector reject its
configuration, so the catalog
// exists in Gravitino but its CREATE CATALOG in Trino fails: a real
registration failure
// rather than a simulated state.
- assertQueryFails(
- "call gravitino.system.create_catalog("
- + "catalog=>'memory_failed', provider=>'memory',"
- + " properties => Map(array['trino.bypass.unknown-direct-key'],
array['10']))",
- "(?s)Create catalog failed. Create catalog failed due to the loading
process fails\\..*");
- assertThat(computeActual("show
catalogs").getOnlyColumnAsSet()).doesNotContain("memory_failed");
-
- MaterializedResult result =
- computeActual(
- "select status, last_error, failure_count from
gravitino.system.catalog_status"
- + " where catalog_name = 'memory_failed'");
- assertEquals(result.getRowCount(), 1);
- MaterializedRow row = result.getMaterializedRows().get(0);
- assertEquals(row.getField(0), "FAILED");
- assertThat((String) row.getField(1)).contains("unknown-direct-key");
- assertEquals(row.getField(2), 1L);
-
- // Leave the shared query runner as it was found, or the load loop keeps
retrying this catalog
- // and the other status table tests see an extra row.
- assertUpdate(
- "call gravitino.system.drop_catalog(catalog => 'memory_failed',
ignore_not_exist => true)");
+ try {
+ assertQueryFails(
+ "call gravitino.system.create_catalog("
+ + "catalog=>'memory_failed', provider=>'memory',"
+ + " properties => Map(array['trino.bypass.unknown-direct-key'],
array['10']))",
+ "(?s)Create catalog failed. Create catalog failed due to the loading
process fails\\..*");
+ assertThat(computeActual("show catalogs").getOnlyColumnAsSet())
+ .doesNotContain("memory_failed");
+
+ MaterializedResult result =
+ computeActual(
+ "select status, last_error, failure_count from
gravitino.system.catalog_status"
+ + " where catalog_name = 'memory_failed'");
+ assertEquals(result.getRowCount(), 1);
+ MaterializedRow row = result.getMaterializedRows().get(0);
+ assertEquals(row.getField(0), "FAILED");
+ assertThat((String) row.getField(1)).contains("unknown-direct-key");
+ assertEquals(row.getField(2), 1L);
+ } finally {
+ // Leave the shared query runner as it was found, or the load loop keeps
retrying this catalog
+ // and the other status table tests see an extra row.
+ assertUpdate(
+ "call gravitino.system.drop_catalog(catalog => 'memory_failed',
ignore_not_exist => true)");
+ }
assertEquals(
computeActual(
"select 1 from gravitino.system.catalog_status"
