This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c212aa81dd4 [feature](iceberg) Add REST encrypted metadata read
support (#70343)
c212aa81dd4 is described below
commit c212aa81dd479f6cb821701c1d6f4298ce01e4a5
Author: Gabriel <[email protected]>
AuthorDate: Wed Nov 18 11:03:51 2026 +0800
[feature](iceberg) Add REST encrypted metadata read support (#70343)
### Background
Iceberg encrypted reads use table metadata to locate retained table keys
and decrypt manifest lists, manifests and file descriptors. Doris
currently limits this groundwork to HMS catalogs. The pinned Iceberg
REST operations retain the plaintext encryption manager, even when the
REST response contains an encrypted table's key chain.
### What problem does this PR solve?
Issue Number: close #70342
Related PR: #70192 (encrypted metadata groundwork)
PR Type: Feature
Problem Summary:
Adding REST to the catalog allowlist alone leaves encrypted metadata
unreadable. This change supplies lazy, read-only REST encryption
operations while retaining the SDK REST table, authentication,
credentials, refresh and scan capabilities. Most of the change is unit
tests and an independent REST encryption fixture/regression suite. REST
now follows the same staged scope as HMS; the common encrypted SQL
execution fence remains in place.
### Details
- **Change map:** 299 product/build/style lines, 895
unit/regression/fixture lines, 85 generated baseline/output lines and 38
documentation lines (insertions plus deletions).
- **Scope:** Iceberg v3 REST tables using the standard SDK encryption
manager and configured KMS. The initial file target remains Parquet
modular encryption with AES_GCM_V1; encrypted writes and unrestricted
encrypted SQL execution are outside this PR.
- **Design:** A package-access bridge decorates the SDK's loaded REST
table operations. Descriptor-aware FileIO decrypts encrypted metadata
lazily using the effective catalog/server/table configuration. A
standalone metadata holder avoids retaining weak FileIOTracker operation
keys; refresh replaces the manager's key chain and reuses the
table-owned KMS client. Idempotent close preserves shared catalog FileIO
ownership and retires table-specific resources. Transactions, output
files and deletion are rejected for encrypted REST tables.
- **Usage:** Configure an Iceberg REST catalog with the existing KMS
properties. There is no new setting, dependency, service or port. The
newly consumed outbound destination is `kms.endpoint`, controlled by
catalog configuration and CREATE_PRIV for CREATE CATALOG or ALTER_PRIV
for ALTER CATALOG. The provider declares it for the existing SSRF policy
while retaining its JDBC driver URL checks. REST server
configuration/table overrides remain inside the configured catalog's
trust boundary.
- **Default behavior:** `enable_encryption_read` remains false. Disabled
SQL reads fail with READ_DISABLED before KMS initialization. Enabling
the existing intent flag still fails with BACKEND_CAPABILITY_UNAVAILABLE
until protected key transport and backend admission are complete.
Plaintext REST reads retain the SDK path.
- **Compatibility:** No storage or FE/BE wire changes. The additive
security-surface composition method requires connector plugin API 33.0
under the repository's SPI contract; deploy the matching engine and
connector plugins together. The plugin API baseline is regenerated and
the unchanged metadata-method baseline is verified.
- **Behavior change:** REST joins HMS in the encrypted metadata target
matrix. Missing KMS configuration and retained keys without the
mandatory table-key property fail explicitly rather than selecting
plaintext crypto. Encrypted REST mutation paths fail as read-only.
### Testing
Rebase validation (2026-11-17, master `3659256fa7d5`): all 165 SPI tests
passed, including both API baselines and the 33.0 version pin. The
complete Iceberg connector suite ran 1,987 tests with zero
failures/errors and eleven existing skips. FE Checkstyle passed in all
21 reactor modules. Both pinned-1.12.0 fixture Java sources compiled
together, and the conflict resolution preserves the native-delete and
REST routes with separate namespaces and warehouses. The original two
commits, commit messages, generated regression output and SPI
security-policy composition are retained. BE builds and cluster
regression were not repeated for this rebase; CI is pending.
Validation recorded before this rebase:
- Local `build.sh --be --fe -j6`, ASAN: passed, followed by an isolated
current-worktree FE/BE cluster run.
- Connector Iceberg tests with Maven build cache disabled: 1,892 tests,
zero failures/errors, 10 existing skips. This includes real connector
HTTP REST loading, actual encrypted SDK manifests and descriptor
payloads, historical scans/refresh, shared/per-table ownership,
concurrent lazy initialization, missing/denied/wrong KMS keys, read-only
transactions and vended credential forwarding. The REST manager test
fails against the original SDK plaintext operations.
- Connector SPI own suite: 154 tests passed, including both API
baselines and the plugin version pin. Endpoint policy composition
preserves the JDBC policy and immutable endpoint lists.
- FE Checkstyle: zero violations during product build and final
connector test validation.
- New `test_iceberg_encrypted_rest_metadata`: output generated by the
regression framework, then compared successfully; 18 golden checks and
128 expected READ_DISABLED errors across cache, batch and COUNT
settings. The independent fixture checks two historical row sets,
encrypted Parquet/manifest/list magic and independent file keys, plus
missing KMS, bad credentials, denied decrypt and wrong-key failures.
- Existing `test_iceberg_encrypted_hms_metadata`: passed, including the
SDK/Spark reference checks.
- Restarted the local FE with the existing read-intent flag enabled: 16
REST SQL queries rejected with BACKEND_CAPABILITY_UNAVAILABLE, while
plaintext rows remained correct.
- Staged patch whitespace and secret scan: passed.
### Notes for Reviewers
The SQL regression deliberately verifies the staged admission contract;
successful decryption is exercised by the connector HTTP unit tests and
an independent SDK reference reader over real encrypted files. It does
not claim that this PR enables encrypted SQL scans. Start with
DorisRESTSessionCatalog and IcebergRestEncryptionFileIO for the
ownership and metadata-generation boundaries.
### Release note
Extend Iceberg encrypted metadata read groundwork to REST Catalog with
standard SDK encryption and configured KMS. The existing default-off
setting and encrypted SQL execution fence remain in effect.
### Check List (For Author)
- Behavior changed: Yes (see Details).
- Does this need documentation: Yes; connector documentation and a
regression reproduction command are included. User-facing documentation
can follow when encrypted SQL execution is admitted.
- Pre-submission status: Ready; local validation passed, CI pending.
### AI Disclosure
- Tool: OpenAI Codex.
- Generated: Implementation, unit/regression tests, connector
documentation and this description.
- Human review: Pending maintainer review; the agent performed a full
self-review and local validation.
---
.../encryption/IcebergEncryptionFixture.java | 173 ++++++-
fe/check/checkstyle/suppressions.xml | 2 +
fe/fe-connector/docs/iceberg-encryption-read.md | 38 +-
.../doris/connector/iceberg/IcebergConnector.java | 5 +-
.../iceberg/IcebergConnectorProvider.java | 5 +-
.../connector/iceberg/IcebergEncryptionRead.java | 5 +-
.../iceberg/IcebergRestEncryptionFileIO.java | 158 +++++++
.../connector/iceberg/IcebergScanPlanProvider.java | 6 +-
.../iceberg/rest/DorisRESTSessionCatalog.java | 106 +++++
...cebergConnectorProviderSecuritySurfaceTest.java | 2 +-
.../iceberg/IcebergEncryptionReadTest.java | 49 +-
.../iceberg/IcebergRestEncryptionTest.java | 501 +++++++++++++++++++++
.../iceberg/IcebergScanPlanProviderTest.java | 11 +-
.../connector/spi/ConnectorSecuritySurface.java | 10 +
.../connector/spi/ConnectorPluginSurfaceTest.java | 4 +-
.../spi/ConnectorSecuritySurfaceTest.java | 39 ++
.../test/resources/connector-plugin-surface.txt | 1 +
fe/fe-connector/pom.xml | 2 +-
.../test_iceberg_encrypted_rest_metadata.out | 84 ++++
.../test_iceberg_encrypted_rest_metadata.groovy | 116 +++++
20 files changed, 1276 insertions(+), 41 deletions(-)
diff --git
a/docker/thirdparties/docker-compose/iceberg/encryption/IcebergEncryptionFixture.java
b/docker/thirdparties/docker-compose/iceberg/encryption/IcebergEncryptionFixture.java
index fb046d8d11f..62d6cc94bb0 100644
---
a/docker/thirdparties/docker-compose/iceberg/encryption/IcebergEncryptionFixture.java
+++
b/docker/thirdparties/docker-compose/iceberg/encryption/IcebergEncryptionFixture.java
@@ -38,6 +38,7 @@ import java.util.concurrent.TimeUnit;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.BaseTable;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.DataFile;
import org.apache.iceberg.FileScanTask;
@@ -47,6 +48,7 @@ import org.apache.iceberg.ManifestFile;
import org.apache.iceberg.ManifestListFile;
import org.apache.iceberg.Schema;
import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.StaticTableOperations;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableMetadata;
import org.apache.iceberg.TableProperties;
@@ -69,6 +71,9 @@ import org.apache.iceberg.hive.HiveCatalog;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.io.DataWriter;
import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.rest.responses.LoadTableResponse;
+import org.apache.iceberg.rest.responses.LoadTableResponseParser;
import org.apache.iceberg.types.Types;
import org.apache.iceberg.util.ByteBuffers;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
@@ -79,6 +84,7 @@ import software.amazon.awssdk.services.kms.model.KmsException;
/** Test-only reference reader and fixture generator. Never install this class
in a Doris plugin. */
public final class IcebergEncryptionFixture {
private static final String DB = "iceberg_encryption_metadata_db";
+ private static final String REST_DB = "iceberg_encryption_rest_db";
private static final Schema SCHEMA = new
Schema(Types.NestedField.required(1, "id", Types.IntegerType.get()));
private static final String WAREHOUSE =
"s3://iceberg-encryption/warehouse";
private final String hms;
@@ -144,6 +150,25 @@ public final class IcebergEncryptionFixture {
reply(request, 500, "{\"error\":\"Native delete fixture
verification failed\"}");
}
});
+ server.createContext("/rest/reset", request -> {
+ try {
+ check("POST".equals(request.getRequestMethod()), "POST
required");
+ fixture.reset(REST_DB, WAREHOUSE + "/rest");
+ reply(request, 200, "{\"created\":true}");
+ } catch (Exception | LinkageError e) {
+ e.printStackTrace(System.err);
+ reply(request, 500, "{\"error\":\"REST encryption fixture
creation failed\"}");
+ }
+ });
+ server.createContext("/rest/verify", request -> {
+ try {
+ reply(request, 200, fixture.verifyRest());
+ } catch (Exception | LinkageError e) {
+ e.printStackTrace(System.err);
+ reply(request, 500, "{\"error\":\"REST encryption fixture
verification failed\"}");
+ }
+ });
+ server.createContext("/rest/v1/", fixture::restRequest);
// These AWS protocol errors isolate credentials/authorization from
storage access and KMS availability.
server.createContext("/kms/credentials", request -> reply(request, 400,
"{\"__type\":\"UnrecognizedClientException\",\"message\":\"Invalid fixture
credentials\"}"));
@@ -185,19 +210,23 @@ public final class IcebergEncryptionFixture {
}
private synchronized void reset() throws Exception {
+ reset(DB, WAREHOUSE);
+ }
+
+ private synchronized void reset(String db, String warehouse) throws
Exception {
try (HiveCatalog catalog = catalog(properties()); S3FileIO cleanup =
new S3FileIO()) {
cleanup.initialize(properties());
- Namespace namespace = Namespace.of(DB);
+ Namespace namespace = Namespace.of(db);
if (!catalog.namespaceExists(namespace)) {
- catalog.createNamespace(namespace, Map.of("location",
"file:/tmp/" + DB));
+ catalog.createNamespace(namespace, Map.of("location",
"file:/tmp/" + db));
}
for (String name : List.of("encrypted_v3", "plain_v3")) {
- TableIdentifier id = TableIdentifier.of(DB, name);
+ TableIdentifier id = TableIdentifier.of(db, name);
// HMS survives stop/up while Moto loses its keys. Purging via
old encrypted
// manifests would fail after dropping the HMS entry, leaking
the old objects.
// Only these dedicated fixture locations may be cleaned
without reading manifests.
catalog.dropTable(id, false);
- String location = WAREHOUSE + "/" + name;
+ String location = warehouse + "/" + name;
cleanup.deletePrefix(location + "/");
check(!cleanup.listPrefix(location +
"/").iterator().hasNext(), "Fixture cleanup is incomplete");
Map<String, String> tableProperties = new HashMap<>();
@@ -271,6 +300,136 @@ public final class IcebergEncryptionFixture {
}
}
+ private void restRequest(HttpExchange request) throws IOException {
+ String path = request.getRequestURI().getPath();
+ try {
+ check("GET".equals(request.getRequestMethod()) ||
"HEAD".equals(request.getRequestMethod()),
+ "Read-only REST fixture");
+ if (path.equals("/rest/v1/config")) {
+ reply(request, 200,
"{\"defaults\":{\"scan-planning-mode\":\"client\"},\"overrides\":{}}");
+ return;
+ }
+ if (path.equals("/rest/v1/namespaces")) {
+ reply(request, 200, "{\"namespaces\":[[\"" + REST_DB +
"\"]]}");
+ return;
+ }
+ String namespace = "/rest/v1/namespaces/" + REST_DB;
+ if (path.equals(namespace)) {
+ reply(request, 200, "{\"namespace\":[\"" + REST_DB +
"\"],\"properties\":{}}");
+ return;
+ }
+ if (path.equals(namespace + "/tables")) {
+ reply(request, 200, "{\"identifiers\":[{\"namespace\":[\"" +
REST_DB
+ + "\"],\"name\":\"plain_v3\"},{\"namespace\":[\"" +
REST_DB
+ + "\"],\"name\":\"encrypted_v3\"}]}");
+ return;
+ }
+ for (String name : List.of("plain_v3", "encrypted_v3")) {
+ if (path.equals(namespace + "/tables/" + name)) {
+ try (HiveCatalog catalog = catalog(properties())) {
+ Table table =
catalog.loadTable(TableIdentifier.of(REST_DB, name));
+ // Return trusted catalog metadata directly. A REST
client must not reopen metadata.json
+ // from the object store to reconstruct the encryption
key chain.
+ try (org.apache.iceberg.io.FileIO io = table.io()) {
+ reply(request, 200,
LoadTableResponseParser.toJson(LoadTableResponse.builder()
+ .withTableMetadata(((HasTableOperations)
table).operations().current()).build()));
+ }
+ }
+ return;
+ }
+ }
+ reply(request, 404, "{\"error\":{\"message\":\"Unknown fixture
table\","
+ + "\"type\":\"NoSuchTableException\",\"code\":404}}");
+ } catch (Exception | LinkageError e) {
+ e.printStackTrace(System.err);
+ reply(request, 500, "{\"error\":{\"message\":\"REST fixture
request failed\","
+ + "\"type\":\"RuntimeException\",\"code\":500}}");
+ }
+ }
+
+ private Map<String, String> restProperties() {
+ Map<String, String> props = new HashMap<>(properties());
+ props.put(CatalogProperties.URI, "http://127.0.0.1:8181/rest");
+ props.put("rest.auth.type", "none");
+ props.put("scan-planning-mode", "client");
+ return props;
+ }
+
+ private String verifyRest() throws Exception {
+ Map<String, String> props = restProperties();
+ List<List<Integer>> snapshots = restRows(props);
+ check(snapshots.equals(List.of(List.of(1, 2), List.of(1, 2, 3, 4))),
"REST historical rows differ");
+ for (String mode : List.of("missingKms", "credentials", "denied")) {
+ Map<String, String> failure = new HashMap<>(props);
+ if (mode.equals("missingKms")) {
+ failure.remove(CatalogProperties.ENCRYPTION_KMS_TYPE);
+ } else {
+ failure.put("kms.endpoint", "http://127.0.0.1:8181/kms/" +
mode);
+ }
+ boolean rejected = false;
+ try {
+ restRows(failure);
+ } catch (RuntimeException expected) {
+ rejected = true;
+ }
+ check(rejected, "REST metadata read must reject " + mode);
+ }
+ return
"{\"sdkVersion\":\"1.12.0\",\"catalog\":\"rest\",\"formatVersion\":3,"
+ +
"\"snapshots\":[[1,2],[1,2,3,4]],\"encryptedManifests\":true,"
+ +
"\"encryptedManifestLists\":true,\"missingKms\":\"REJECTED\","
+ +
"\"wrongCredentials\":\"REJECTED\",\"denied\":\"REJECTED\",\"wrongKey\":\"REJECTED\"}";
+ }
+
+ private List<List<Integer>> restRows(Map<String, String> props) throws
Exception {
+ try (RESTCatalog rest = new RESTCatalog(); KeyManagementClient client
= EncryptionUtil.createKmsClient(props)) {
+ rest.initialize("rest_reference", props);
+ Table loaded = rest.loadTable(TableIdentifier.of(REST_DB,
"encrypted_v3"));
+ TableMetadata metadata = ((HasTableOperations)
loaded).operations().current();
+ check(metadata.formatVersion() == 3 &&
!metadata.encryptionKeys().isEmpty(), "REST key chain is absent");
+ // The pinned SDK has no REST encryption operations. Build an
independent reference read view from
+ // the REST response and standard SDK crypto, rather than
depending on Doris's production adapter.
+ EncryptionManager manager = EncryptionUtil.createEncryptionManager(
+ metadata.encryptionKeys(), metadata.properties(), client);
+ Table view = new BaseTable(new StaticTableOperations(metadata,
+ EncryptingFileIO.combine(loaded.io(), manager)) {
+ @Override
+ public EncryptionManager encryption() {
+ return manager;
+ }
+ }, loaded.name());
+ List<List<Integer>> result = new ArrayList<>();
+ for (Snapshot snapshot : view.snapshots()) {
+ check(snapshot.keyId() != null, "REST manifest-list key ID is
absent");
+ try (InputStream input =
loaded.io().newInputFile(snapshot.manifestListLocation()).newStream()) {
+ check(Arrays.equals(input.readNBytes(4),
"AGS1".getBytes(StandardCharsets.US_ASCII)),
+ "REST manifest list is not encrypted");
+ }
+ for (ManifestFile manifest : snapshot.allManifests(view.io()))
{
+ check(manifest.keyMetadata() != null, "REST manifest
encryption metadata is absent");
+ try (InputStream input =
loaded.io().newInputFile(manifest.path()).newStream()) {
+ check(Arrays.equals(input.readNBytes(4),
"AGS1".getBytes(StandardCharsets.US_ASCII)),
+ "REST manifest is not encrypted");
+ }
+ }
+ result.add(rows(view, snapshot.snapshotId()));
+ }
+ Set<String> keys = new HashSet<>();
+ try (CloseableIterable<FileScanTask> tasks =
view.newScan().planFiles()) {
+ for (FileScanTask task : tasks) {
+ check(task.file().keyMetadata() != null, "REST data file
key metadata is absent");
+
keys.add(Base64.getEncoder().encodeToString(ByteBuffers.toByteArray(task.file().keyMetadata())));
+ try (InputStream input =
loaded.io().newInputFile(task.file().location()).newStream()) {
+ check(Arrays.equals(input.readNBytes(4),
"PARE".getBytes(StandardCharsets.US_ASCII)),
+ "REST payload is not native encrypted
Parquet");
+ }
+ }
+ }
+ check(keys.size() == 2, "REST data files need independent
encryption keys");
+ verifyWrongKey(view, metadata);
+ return result;
+ }
+ }
+
private void verifySpark(Table table) throws Exception {
List<Long> snapshots = new ArrayList<>();
for (Snapshot snapshot : table.snapshots()) {
@@ -417,6 +576,12 @@ public final class IcebergEncryptionFixture {
private static void reply(HttpExchange request, int status, String
response) throws IOException {
request.getRequestBody().close();
+ // REST existence probes use HEAD and must not send a JSON body.
+ if ("HEAD".equals(request.getRequestMethod())) {
+ request.sendResponseHeaders(status, -1);
+ request.close();
+ return;
+ }
byte[] bytes = response.getBytes(StandardCharsets.UTF_8);
request.getResponseHeaders().set("Content-Type",
"application/x-amz-json-1.1");
request.sendResponseHeaders(status, bytes.length);
diff --git a/fe/check/checkstyle/suppressions.xml
b/fe/check/checkstyle/suppressions.xml
index 595616afabc..9c00a229c4a 100644
--- a/fe/check/checkstyle/suppressions.xml
+++ b/fe/check/checkstyle/suppressions.xml
@@ -25,6 +25,8 @@ under the License.
<suppressions>
<!-- The SDK metadata codec is package-private; only this small access
bridge shares its package. -->
<suppress
files="org[\\/]apache[\\/]iceberg[\\/]encryption[\\/]DorisKeyMetadata\.java"
checks="ImportControl"/>
+ <!-- The protected REST factory returns a package-private SDK operations
type; all other checks apply. -->
+ <suppress
files="org[\\/]apache[\\/]iceberg[\\/]rest[\\/]DorisRESTSessionCatalog\.java"
checks="ImportControl"/>
<!-- Excludes test files from having Javadocs for classes and methods -->
<suppress files="[\\/]jmockit[\\/]" checks=".*" />
<suppress files="[\\/]test[\\/]" checks="MissingJavadocMethod" />
diff --git a/fe/fe-connector/docs/iceberg-encryption-read.md
b/fe/fe-connector/docs/iceberg-encryption-read.md
index cdf8ad1a11a..856ba663c85 100644
--- a/fe/fe-connector/docs/iceberg-encryption-read.md
+++ b/fe/fe-connector/docs/iceberg-encryption-read.md
@@ -23,7 +23,7 @@ The initial target matrix is:
| Dimension | Initial target | Current admission |
| --- | --- | --- |
-| Catalog | HMS with the SDK standard encryption manager and an
operator-configured KMS | Other catalogs are rejected |
+| Catalog | HMS or REST with the SDK standard encryption manager and an
operator-configured KMS | Other catalogs are rejected |
| Table format | Iceberg v3 | Other versions are rejected for encrypted reads |
| Data and delete files | Native Parquet modular encryption, `AES_GCM_V1` | No
encrypted ranges are emitted yet |
| Manifest/list files | SDK-managed encryption | No plaintext fallback is
allowed |
@@ -51,7 +51,7 @@ Errors contain a stable `Iceberg encryption [CLASSIFICATION]`
prefix:
| Classification | Meaning |
| --- | --- |
| `READ_DISABLED` | Encrypted reads were not enabled in the plugin
configuration |
-| `UNSUPPORTED_CATALOG` | The catalog is outside the initial HMS scope |
+| `UNSUPPORTED_CATALOG` | The catalog is outside the HMS/REST scope |
| `UNSUPPORTED_TABLE_FORMAT` | The encrypted table is not v3 |
| `UNSUPPORTED_ENCRYPTION_MANAGER` | Encryption detection observed a custom
SDK encryption manager |
| `BACKEND_CAPABILITY_UNAVAILABLE` | The engine cannot prove encrypted-read
support |
@@ -183,3 +183,37 @@ disabled streaming; normal-scan fallback would retain all
file tasks and ranges
before the dispatch guard rejects them. This deterministic rejection must not
trigger the manifest cache's SDK retry. Plaintext COUNT retains its existing
collapse and cache-failure fallback behavior.
+
+## REST Catalog encrypted metadata
+
+REST catalogs use the same default-off read admission and
v3/standard-encryption
+requirements as HMS. The SDK REST operations used by this connector do not
attach
+an encryption manager themselves; a small package-access adapter retains the
+original REST table, scan capabilities, REST credentials and refresh protocol
while
+adding lazy, read-only encrypted metadata FileIO. It uses the effective FileIO
+properties after SDK catalog defaults, server overrides and table config
merging.
+No KMS client is initialized merely to load a table or reject an encrypted
query.
+
+The table-owned adapter closes its KMS client once. It closes a table-specific
+FileIO, but leaves a shared Catalog FileIO to the catalog owner. Refresh
replaces
+the encrypted metadata generation without discarding retained historical keys.
+Missing KMS configuration or a removed table-key property is rejected
explicitly;
+retained keys must never make the SDK select a plaintext manager.
+Encrypted REST writes and maintenance operations remain unsupported, and the
+common SQL execution fence remains in place pending protected key transport and
+native execution admission. Changing the REST allowlist does not bypass that
fence.
+
+`test_iceberg_encrypted_rest_metadata` reuses the mandatory encryption
fixture's
+`/rest` endpoint and existing ports; no extra enable flag is required. Its
dedicated
+namespace and object locations do not reset the HMS regression's tables. An
+independent SDK reference view reads real encrypted Parquet and manifest/list
+files using metadata loaded through REST, checks historical rows, and verifies
+missing KMS, rejected credentials, denied decrypt and wrong-key failures. Doris
+SQL covers plaintext reads and counts, table-cache and batch/count settings,
+repeated encrypted admission, historical and metadata-table reads, and refresh.
+The fixture is a read-only protocol test service, not a production REST
catalog.
+
+```sh
+bash run-regression-test.sh --run -d external_table_p0/iceberg \
+ -s test_iceberg_encrypted_rest_metadata
+```
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
index 0d21e2a27bc..5ee253226e0 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
@@ -65,6 +65,7 @@ import org.apache.iceberg.hive.HiveCatalog;
import org.apache.iceberg.hive.HiveHadoopUtil;
import org.apache.iceberg.io.FileIO;
import org.apache.iceberg.io.SupportsStorageCredentials;
+import org.apache.iceberg.rest.DorisRESTSessionCatalog;
import org.apache.iceberg.rest.HTTPClient;
import org.apache.iceberg.rest.RESTCatalog;
import org.apache.iceberg.rest.RESTSessionCatalog;
@@ -1213,11 +1214,11 @@ public class IcebergConnector implements Connector {
/** Builds and initializes a bare Iceberg {@link RESTSessionCatalog}
(fresh REST client + OAuth2 token fetch). */
private RESTSessionCatalog newRestSessionCatalog(String catalogName,
Map<String, String> props,
Configuration conf) {
- RESTSessionCatalog sessionCatalog = new RESTSessionCatalog(config ->
HTTPClient.builder(config)
+ RESTSessionCatalog sessionCatalog = new DorisRESTSessionCatalog(config
-> HTTPClient.builder(config)
.uri(config.get(CatalogProperties.URI))
.withHeaders(RESTUtil.configHeaders(config))
.withObjectMapper(IcebergRestSchemaCompatibility.objectMapper())
- .build(), null);
+ .build(), IcebergConnector::restCatalogFileIO);
CatalogUtil.configureHadoopConf(sessionCatalog, conf);
sessionCatalog.initialize(catalogName, props);
return sessionCatalog;
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorProvider.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorProvider.java
index 4f9bff2f213..4b32918fe5d 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorProvider.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorProvider.java
@@ -91,8 +91,9 @@ public class IcebergConnectorProvider implements
ConnectorProvider {
public ConnectorSecuritySurface securitySurface() {
// iceberg.jdbc.driver_url is a jar the connector loads, but only live
on a jdbc-flavored catalog
// (iceberg.catalog.type = jdbc); on a REST/HMS/... catalog it is dead
config the gate must not touch.
- // The secrets are those of every metastore flavor: the engine masks
by key name alone.
+ // Keep secret masking and JDBC restrictions when REST decryption adds
a KMS endpoint check.
return
ConnectorSecuritySurface.ofJdbcDriverUrls("iceberg.catalog.type",
"iceberg.jdbc.driver_url")
- .withSensitiveKeys(MetaStoreProviders.sensitivePropertyKeys());
+ .withSensitiveKeys(MetaStoreProviders.sensitivePropertyKeys())
+ .withSsrfEndpoints("kms.endpoint");
}
}
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergEncryptionRead.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergEncryptionRead.java
index 45faf6b70f0..ed4be124b4b 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergEncryptionRead.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergEncryptionRead.java
@@ -48,8 +48,9 @@ final class IcebergEncryptionRead {
throw failure("READ_DISABLED", "Encrypted Iceberg reads are
disabled; enable_encryption_read in "
+ "iceberg.conf defaults to false");
}
- if (!IcebergCatalogProperties.TYPE_HMS.equals(catalog.getFlavor())) {
- throw failure("UNSUPPORTED_CATALOG", "Encrypted Iceberg reads
require an HMS catalog");
+ if (!IcebergCatalogProperties.TYPE_HMS.equals(catalog.getFlavor())
+ &&
!IcebergCatalogProperties.TYPE_REST.equals(catalog.getFlavor())) {
+ throw failure("UNSUPPORTED_CATALOG", "Encrypted Iceberg reads
require an HMS or REST catalog");
}
if (metadata == null || metadata.formatVersion() != 3) {
throw failure("UNSUPPORTED_TABLE_FORMAT", "Encrypted Iceberg reads
require table format v3");
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRestEncryptionFileIO.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRestEncryptionFileIO.java
new file mode 100644
index 00000000000..6adcbc43fe7
--- /dev/null
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRestEncryptionFileIO.java
@@ -0,0 +1,158 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.iceberg;
+
+import org.apache.doris.connector.spi.DorisConnectorException;
+
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.ManifestFile;
+import org.apache.iceberg.ManifestListFile;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptingFileIO;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.encryption.EncryptionUtil;
+import org.apache.iceberg.encryption.KeyManagementClient;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+/** Read-only REST file adapter; its KMS client has the same lifetime as the
loaded table's FileIO. */
+public final class IcebergRestEncryptionFileIO implements FileIO {
+ private final FileIO delegate;
+ private final AtomicReference<TableMetadata> metadata;
+ private final boolean ownsDelegate;
+ private transient KeyManagementClient kms;
+ private transient TableMetadata encryptionMetadata;
+ private transient EncryptingFileIO encrypted;
+ private boolean closed;
+
+ public IcebergRestEncryptionFileIO(FileIO delegate,
AtomicReference<TableMetadata> metadata,
+ boolean ownsDelegate) {
+ this.delegate = delegate;
+ this.metadata = metadata;
+ this.ownsDelegate = ownsDelegate;
+ }
+
+ public EncryptionManager encryptionManager() {
+ return encrypted().encryptionManager();
+ }
+
+ private synchronized EncryptingFileIO encrypted() {
+ if (closed) {
+ throw new IllegalStateException("Iceberg REST table FileIO is
closed");
+ }
+ TableMetadata current = metadata.get();
+ if (encrypted == null || encryptionMetadata != current) {
+ // Retained keys still describe encrypted snapshots. The SDK
factory selects plaintext when the
+ // current write-key property is missing, so reject that invalid
catalog state before any KMS work.
+ if
(!current.properties().containsKey(TableProperties.ENCRYPTION_TABLE_KEY)) {
+ throw new DorisConnectorException("Iceberg encryption
[MISSING_TABLE_KEY]");
+ }
+ Map<String, String> properties = delegate.properties();
+ // The SDK's reflective KMS factory otherwise fails with a null
implementation class name.
+ if (properties.get(CatalogProperties.ENCRYPTION_KMS_TYPE) == null
+ && properties.get(CatalogProperties.ENCRYPTION_KMS_IMPL)
== null) {
+ throw new DorisConnectorException("Iceberg encryption
[MISSING_KMS]");
+ }
+ // REST load/config may vend table-specific properties. Do not
replace them with catalog credentials,
+ // and do not initialize a KMS merely to reject an encrypted table
at the read admission boundary.
+ if (kms == null) {
+ kms = EncryptionUtil.createKmsClient(properties);
+ }
+ EncryptionManager manager = EncryptionUtil.createEncryptionManager(
+ current.encryptionKeys(), current.properties(), kms);
+ encrypted = EncryptingFileIO.combine(delegate, manager);
+ encryptionMetadata = current;
+ }
+ return encrypted;
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ return delegate.newInputFile(path);
+ }
+
+ @Override
+ public InputFile newInputFile(String path, long length) {
+ return delegate.newInputFile(path, length);
+ }
+
+ @Override
+ public InputFile newInputFile(DataFile file) {
+ return file.keyMetadata() == null ? delegate.newInputFile(file) :
encrypted().newInputFile(file);
+ }
+
+ @Override
+ public InputFile newInputFile(DeleteFile file) {
+ return file.keyMetadata() == null ? delegate.newInputFile(file) :
encrypted().newInputFile(file);
+ }
+
+ @Override
+ public InputFile newInputFile(ManifestFile file) {
+ return file.keyMetadata() == null ? delegate.newInputFile(file) :
encrypted().newInputFile(file);
+ }
+
+ @Override
+ public InputFile newInputFile(ManifestListFile file) {
+ return file.encryptionKeyID() == null ? delegate.newInputFile(file) :
encrypted().newInputFile(file);
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ throw new UnsupportedOperationException("Encrypted Iceberg REST tables
are read-only in Doris");
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ throw new UnsupportedOperationException("Encrypted Iceberg REST tables
are read-only in Doris");
+ }
+
+ @Override
+ public Map<String, String> properties() {
+ return delegate.properties();
+ }
+
+ FileIO delegate() {
+ return delegate;
+ }
+
+ @Override
+ public synchronized void close() {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ // The SDK also tracks this adapter. A table lease and catalog
retirement may both close it;
+ // neither may close a shared catalog FileIO, and an independently
owned delegate must still retire.
+ try (KeyManagementClient ownedKms = kms) {
+ if (ownsDelegate) {
+ delegate.close();
+ }
+ } finally {
+ encrypted = null;
+ encryptionMetadata = null;
+ kms = null;
+ }
+ }
+}
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java
index 0f929834e0c..8b243d31470 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java
@@ -2214,8 +2214,10 @@ public class IcebergScanPlanProvider implements
ConnectorScanPlanProvider {
FileIO fileIO = table.io();
Map<String, String> ioProps = new HashMap<>(fileIO.properties());
// REST credentials live outside properties(). Authentication
decorators must not hide them from BE.
- while (fileIO instanceof IcebergAuthenticatedFileIO) {
- fileIO = ((IcebergAuthenticatedFileIO) fileIO).delegate();
+ while (fileIO instanceof IcebergAuthenticatedFileIO || fileIO
instanceof IcebergRestEncryptionFileIO) {
+ fileIO = fileIO instanceof IcebergAuthenticatedFileIO
+ ? ((IcebergAuthenticatedFileIO) fileIO).delegate()
+ : ((IcebergRestEncryptionFileIO) fileIO).delegate();
}
if (fileIO instanceof SupportsStorageCredentials) {
for (StorageCredential storageCredential :
((SupportsStorageCredentials) fileIO).credentials()) {
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/iceberg/rest/DorisRESTSessionCatalog.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/iceberg/rest/DorisRESTSessionCatalog.java
new file mode 100644
index 00000000000..62b4614d4b3
--- /dev/null
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/iceberg/rest/DorisRESTSessionCatalog.java
@@ -0,0 +1,106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.iceberg.rest;
+
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.TableOperations;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.io.FileIO;
+
+import org.apache.doris.connector.iceberg.IcebergRestEncryptionFileIO;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
+import java.util.function.Supplier;
+
+/**
+ * Adds encrypted metadata reads to the pinned SDK's REST operations. This
package is required because the
+ * protected SDK factory returns package-private RESTTableOperations;
RESTTable and its scan capabilities stay intact.
+ */
+public final class DorisRESTSessionCatalog extends RESTSessionCatalog {
+ private final Function<RESTSessionCatalog, FileIO> catalogFileIO;
+
+ public DorisRESTSessionCatalog(Function<Map<String, String>, RESTClient>
clientBuilder,
+ Function<RESTSessionCatalog, FileIO> catalogFileIO) {
+ super(clientBuilder, null);
+ this.catalogFileIO = catalogFileIO;
+ }
+
+ @Override
+ protected RESTTableOperations newTableOps(RESTClient client, String path,
+ Supplier<Map<String, String>> readHeaders, Supplier<Map<String,
String>> mutationHeaders,
+ FileIO io, TableMetadata metadata, Set<Endpoint> endpoints,
Map<String, String> readQueryParams) {
+ if
(!metadata.properties().containsKey(TableProperties.ENCRYPTION_TABLE_KEY)
+ && metadata.encryptionKeys().isEmpty()) {
+ return super.newTableOps(client, path, readHeaders,
mutationHeaders, io, metadata,
+ endpoints, readQueryParams);
+ }
+ FileIO shared = Objects.requireNonNull(catalogFileIO.apply(this),
"REST catalog FileIO ownership");
+ return new EncryptedTableOperations(client, path, readHeaders,
mutationHeaders, io, metadata,
+ endpoints, readQueryParams, io != shared);
+ }
+
+ private static final class EncryptedTableOperations extends
RESTTableOperations {
+ private final AtomicReference<TableMetadata> readMetadata;
+ private final IcebergRestEncryptionFileIO encryptedIO;
+
+ private EncryptedTableOperations(RESTClient client, String path,
+ Supplier<Map<String, String>> readHeaders,
Supplier<Map<String, String>> mutationHeaders,
+ FileIO io, TableMetadata metadata, Set<Endpoint> endpoints,
Map<String, String> readQueryParams,
+ boolean ownsIO) {
+ super(client, path, readHeaders, mutationHeaders, io, metadata,
endpoints, readQueryParams);
+ // The SDK's FileIOTracker has weak operation keys and strong
FileIO values. Sharing only a metadata
+ // holder, rather than passing this::current into the FileIO,
keeps that tracker from retaining the key.
+ readMetadata = new AtomicReference<>(metadata);
+ encryptedIO = new IcebergRestEncryptionFileIO(io, readMetadata,
ownsIO);
+ }
+
+ @Override
+ public TableMetadata refresh() {
+ TableMetadata refreshed = super.refresh();
+ readMetadata.set(refreshed);
+ return refreshed;
+ }
+
+ @Override
+ public FileIO io() {
+ return encryptedIO;
+ }
+
+ @Override
+ public EncryptionManager encryption() {
+ return encryptedIO.encryptionManager();
+ }
+
+ @Override
+ public TableOperations temp(TableMetadata uncommittedMetadata) {
+ // Enforce read-only semantics at transaction creation, before a
caller can stage
+ // mutable metadata or initialize the table's KMS client for a
write.
+ throw new UnsupportedOperationException("Encrypted Iceberg REST
tables are read-only in Doris");
+ }
+
+ @Override
+ public void commit(TableMetadata base, TableMetadata updated) {
+ throw new UnsupportedOperationException("Encrypted Iceberg REST
tables are read-only in Doris");
+ }
+ }
+}
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorProviderSecuritySurfaceTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorProviderSecuritySurfaceTest.java
index 27a09c46cbd..17d2ec55757 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorProviderSecuritySurfaceTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorProviderSecuritySurfaceTest.java
@@ -37,7 +37,7 @@ public class IcebergConnectorProviderSecuritySurfaceTest {
Assertions.assertEquals(Collections.singletonList("iceberg.jdbc.driver_url"),
surface.jdbcDriverUrlKeys());
Assertions.assertEquals("iceberg.catalog.type",
surface.jdbcDriverUrlFlavorKey());
- Assertions.assertTrue(surface.ssrfEndpointKeys().isEmpty());
+ Assertions.assertEquals(Collections.singletonList("kms.endpoint"),
surface.ssrfEndpointKeys());
}
@Test
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergEncryptionReadTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergEncryptionReadTest.java
index 32a0afe94a8..fefffd41f0a 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergEncryptionReadTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergEncryptionReadTest.java
@@ -46,6 +46,8 @@ import org.apache.iceberg.util.JsonUtil;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.function.Executable;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
import java.util.List;
import java.util.Map;
@@ -56,15 +58,16 @@ public class IcebergEncryptionReadTest {
private static final Schema SCHEMA = new
Schema(Types.NestedField.required(1, "id", Types.LongType.get()));
private static final IcebergTableHandle HANDLE = new
IcebergTableHandle("db", "tbl");
- @Test
- public void encryptedReadsAreDisabledAcrossPlanningRoutes() throws
Exception {
+ @ParameterizedTest
+ @ValueSource(strings = {"hms", "rest"})
+ public void encryptedReadsAreDisabledAcrossPlanningRoutes(String flavor)
throws Exception {
for (boolean populated : new boolean[] {false, true}) {
for (boolean cached : new boolean[] {false, true}) {
IcebergTableCache cache = new IcebergTableCache(100, 10);
try (InMemoryCatalog catalog = catalog()) {
Table table = encryptedView(table(catalog, populated), 3);
RecordingIcebergCatalogOps ops = ops(table);
- IcebergScanPlanProvider provider = new
IcebergScanPlanProvider(properties("hms"),
+ IcebergScanPlanProvider provider = new
IcebergScanPlanProvider(properties(flavor),
ignored -> ops, context(Map.of()), null, cached ?
cache : null);
for (ConnectorStatementScope scope :
List.of(ConnectorStatementScope.NONE,
new TestStatementScope())) {
@@ -100,28 +103,31 @@ public class IcebergEncryptionReadTest {
for (String flavor : List.of("hms", "rest", "hadoop", "jdbc",
"glue", "dlf", "s3tables")) {
IcebergScanPlanProvider provider = new
IcebergScanPlanProvider(properties(flavor),
ops(table), context(Map.of("enable_encryption_read",
"true")));
- reject(flavor.equals("hms") ? "BACKEND_CAPABILITY_UNAVAILABLE"
: "UNSUPPORTED_CATALOG",
+ reject(flavor.equals("hms") || flavor.equals("rest")
+ ? "BACKEND_CAPABILITY_UNAVAILABLE" :
"UNSUPPORTED_CATALOG",
() -> provider.planScan(null, request(HANDLE, true)));
}
}
}
- @Test
- public void unsupportedTableVersionHasAnExplicitError() throws Exception {
+ @ParameterizedTest
+ @ValueSource(strings = {"hms", "rest"})
+ public void unsupportedTableVersionHasAnExplicitError(String flavor)
throws Exception {
try (InMemoryCatalog catalog = catalog()) {
Table table = encryptedView(table(catalog, false), 2);
- IcebergScanPlanProvider provider = new
IcebergScanPlanProvider(properties("hms"),
+ IcebergScanPlanProvider provider = new
IcebergScanPlanProvider(properties(flavor),
ops(table), context(Map.of("enable_encryption_read",
"true")));
reject("UNSUPPORTED_TABLE_FORMAT", () -> provider.planScan(null,
request(HANDLE, false)));
}
}
- @Test
- public void metadataBindingCannotBypassReadChecks() throws Exception {
+ @ParameterizedTest
+ @ValueSource(strings = {"hms", "rest"})
+ public void metadataBindingCannotBypassReadChecks(String flavor) throws
Exception {
try (InMemoryCatalog catalog = catalog()) {
Table table = encryptedView(table(catalog, true), 3);
IcebergConnectorMetadata metadata = new IcebergConnectorMetadata(
- ops(table), properties("hms"), context(Map.of()));
+ ops(table), properties(flavor), context(Map.of()));
reject("READ_DISABLED", () -> metadata.getTableSchema(null,
HANDLE));
reject("READ_DISABLED", () -> metadata.beginQuerySnapshot(null,
HANDLE));
IcebergTableHandle sys = IcebergTableHandle.forSystemTable("db",
"tbl", "files", -1L, null, -1L);
@@ -129,8 +135,9 @@ public class IcebergEncryptionReadTest {
}
}
- @Test
- public void
retainedEncryptionKeysAndEmptySnapshotPinsCannotBypassReadChecks() throws
Exception {
+ @ParameterizedTest
+ @ValueSource(strings = {"hms", "rest"})
+ public void
retainedEncryptionKeysAndEmptySnapshotPinsCannotBypassReadChecks(String flavor)
throws Exception {
try (InMemoryCatalog catalog = catalog()) {
Table table = table(catalog, false);
ObjectNode json = (ObjectNode) JsonUtil.mapper().readTree(
@@ -140,7 +147,7 @@ public class IcebergEncryptionReadTest {
key.put("encrypted-key-metadata", "AA==");
Table retained = new BaseTable(new StaticTableOperations(
TableMetadataParser.fromJson(json.toString()),
table.io()), "db.tbl");
- IcebergScanPlanProvider provider = new
IcebergScanPlanProvider(properties("hms"), ops(retained));
+ IcebergScanPlanProvider provider = new
IcebergScanPlanProvider(properties(flavor), ops(retained));
IcebergTableHandle empty = HANDLE.withSnapshot(-1L, null, 0L);
reject("READ_DISABLED", () -> provider.planScan(null,
request(empty, false)));
reject("READ_DISABLED", () -> provider.streamSplits(null,
request(empty, false)));
@@ -148,8 +155,9 @@ public class IcebergEncryptionReadTest {
}
}
- @Test
- public void customEncryptionManagerIsDetectedWithoutTableProperties()
throws Exception {
+ @ParameterizedTest
+ @ValueSource(strings = {"hms", "rest"})
+ public void customEncryptionManagerIsDetectedWithoutTableProperties(String
flavor) throws Exception {
try (InMemoryCatalog catalog = catalog()) {
Table table = table(catalog, false);
StaticTableOperations operations = new StaticTableOperations(
@@ -170,16 +178,17 @@ public class IcebergEncryptionReadTest {
}
};
Table custom = new BaseTable(operations, "db.tbl");
- reject("READ_DISABLED", () -> new
IcebergScanPlanProvider(properties("hms"), ops(custom))
+ reject("READ_DISABLED", () -> new
IcebergScanPlanProvider(properties(flavor), ops(custom))
.planScan(null, request(HANDLE, false)));
- reject("UNSUPPORTED_ENCRYPTION_MANAGER", () -> new
IcebergScanPlanProvider(properties("hms"),
+ reject("UNSUPPORTED_ENCRYPTION_MANAGER", () -> new
IcebergScanPlanProvider(properties(flavor),
ops(custom), context(Map.of("enable_encryption_read",
"true")))
.planScan(null, request(HANDLE, false)));
}
}
- @Test
- public void unverifiedBackendRejectsBeforeKmsManagerInitialization()
throws Exception {
+ @ParameterizedTest
+ @ValueSource(strings = {"hms", "rest"})
+ public void unverifiedBackendRejectsBeforeKmsManagerInitialization(String
flavor) throws Exception {
try (InMemoryCatalog catalog = catalog()) {
BaseTable encrypted = (BaseTable) encryptedView(table(catalog,
false), 3);
Table table = new BaseTable(new StaticTableOperations(
@@ -189,7 +198,7 @@ public class IcebergEncryptionReadTest {
throw new AssertionError("Unverified execution must not
initialize a KMS client");
}
}, "db.tbl");
- IcebergScanPlanProvider provider = new
IcebergScanPlanProvider(properties("hms"), ops(table),
+ IcebergScanPlanProvider provider = new
IcebergScanPlanProvider(properties(flavor), ops(table),
context(Map.of("enable_encryption_read", "true")));
reject("BACKEND_CAPABILITY_UNAVAILABLE", () ->
provider.planScan(null, request(HANDLE, false)));
}
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRestEncryptionTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRestEncryptionTest.java
new file mode 100644
index 00000000000..88648c2ad93
--- /dev/null
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRestEncryptionTest.java
@@ -0,0 +1,501 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.iceberg;
+
+import org.apache.doris.connector.spi.DorisConnectorException;
+
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.sun.net.httpserver.HttpServer;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.DataFiles;
+import org.apache.iceberg.FileMetadata;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.LocationProviders;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.TableMetadataParser;
+import org.apache.iceberg.TableOperations;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.encryption.EncryptingFileIO;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.encryption.KeyManagementClient;
+import org.apache.iceberg.encryption.NativeEncryptionOutputFile;
+import org.apache.iceberg.encryption.StandardEncryptionManager;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.inmemory.InMemoryFileIO;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.rest.responses.LoadTableResponse;
+import org.apache.iceberg.rest.responses.LoadTableResponseParser;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.JsonUtil;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.net.InetSocketAddress;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+class IcebergRestEncryptionTest {
+ private static final TableIdentifier ID = TableIdentifier.of("db",
"encrypted");
+
+ @ParameterizedTest
+ @ValueSource(booleans = {false, true})
+ void restTablesInitializeEncryptionLazilyAndCloseOwnedClients(boolean
tableConfig) throws Exception {
+ FixtureKms.initialized.set(0);
+ FixtureKms.closed.set(0);
+ try (Fixture fixture = new Fixture(tableConfig)) {
+ Table table = fixture.load();
+ Assertions.assertEquals(0, FixtureKms.initialized.get(), "loading
must not initialize KMS");
+ Assertions.assertInstanceOf(StandardEncryptionManager.class,
table.encryption());
+ Assertions.assertEquals(1, FixtureKms.initialized.get());
+ Assertions.assertEquals(tableConfig ? "table" : "catalog",
FixtureKms.scope.get());
+ Assertions.assertSame(table.encryption(), table.encryption());
+ FileIO rawCatalogIo = fixture.catalogIo;
+ TrackingFileIO rawTableIo = (TrackingFileIO)
((IcebergRestEncryptionFileIO) table.io()).delegate();
+ table.io().close();
+ table.io().close();
+ Assertions.assertEquals(tableConfig ? 1 : 0,
rawTableIo.closes.get(),
+ "only table-specific FileIO belongs to the table lease");
+ Assertions.assertThrows(IllegalStateException.class,
table::encryption);
+ Assertions.assertEquals(1, FixtureKms.closed.get(), "one owner
closes each KMS client once");
+ if (rawCatalogIo instanceof TrackingFileIO) {
+ Assertions.assertEquals(0, ((TrackingFileIO)
rawCatalogIo).closes.get(),
+ "closing one table must not close the shared catalog
FileIO");
+ }
+ }
+ Assertions.assertEquals(1, FixtureKms.closed.get());
+ }
+
+ @Test
+ void restReadsEncryptedManifestsFilesAndHistoricalSnapshotsAcrossRefresh()
throws Exception {
+ FixtureKms.initialized.set(0);
+ FixtureKms.denied.set(false);
+ FixtureKms.wrongKey.set(false);
+ try (MemoryEncryptedTable writer = new MemoryEncryptedTable()) {
+ writer.append();
+ long firstSnapshot =
writer.current().currentSnapshot().snapshotId();
+ writer.append();
+
Assertions.assertFalse(writer.current().encryptionKeys().isEmpty());
+ try (Fixture fixture = new Fixture(true, writer.current())) {
+ Table table = fixture.load();
+ Assertions.assertEquals(0, FixtureKms.initialized.get());
+ Assertions.assertEquals(2, filesAndPayloads(table, null));
+ Assertions.assertEquals(1, filesAndPayloads(table,
firstSnapshot));
+ EncryptionManager first = table.encryption();
+ writer.append();
+ fixture.metadata.set(writer.current());
+ table.refresh();
+ Assertions.assertEquals(3, filesAndPayloads(table, null));
+ Assertions.assertEquals(1, filesAndPayloads(table,
firstSnapshot));
+ Assertions.assertNotSame(first, table.encryption(), "refresh
must publish the new encrypted key chain");
+ Assertions.assertEquals(1, FixtureKms.initialized.get(),
"refresh reuses the table-owned KMS client");
+ Assertions.assertThrows(UnsupportedOperationException.class,
+ () ->
table.io().newOutputFile("memory://unsupported-write"));
+ Assertions.assertThrows(UnsupportedOperationException.class,
+ () ->
table.io().deleteFile("memory://unsupported-delete"));
+ }
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {false, true})
+ void restMetadataDecryptionRejectsDeniedOrWrongKeys(boolean wrongKey)
throws Exception {
+ FixtureKms.denied.set(false);
+ FixtureKms.wrongKey.set(false);
+ try (MemoryEncryptedTable writer = new MemoryEncryptedTable()) {
+ writer.append();
+ try (Fixture fixture = new Fixture(true, writer.current())) {
+ Table table = fixture.load();
+ FixtureKms.denied.set(!wrongKey);
+ FixtureKms.wrongKey.set(wrongKey);
+ RuntimeException failure =
Assertions.assertThrows(RuntimeException.class,
+ () -> filesAndPayloads(table, null));
+ Throwable cause = failure;
+ while (cause.getCause() != null) {
+ cause = cause.getCause();
+ }
+ if (wrongKey) {
+
Assertions.assertInstanceOf(javax.crypto.AEADBadTagException.class, cause);
+ } else {
+ Assertions.assertEquals("Fixture KMS denied decryption",
cause.getMessage());
+ }
+ }
+ } finally {
+ FixtureKms.denied.set(false);
+ FixtureKms.wrongKey.set(false);
+ }
+ }
+
+ @Test
+ void parallelManifestReadersInitializeOneClientAndCloseItOnce() throws
Exception {
+ FixtureKms.initialized.set(0);
+ FixtureKms.closed.set(0);
+ try (Fixture fixture = new Fixture(false)) {
+ Table table = fixture.load();
+ var workers = Executors.newFixedThreadPool(4);
+ try {
+ var results = new
ArrayList<java.util.concurrent.Future<EncryptionManager>>();
+ for (int i = 0; i < 16; i++) {
+ results.add(workers.submit(table::encryption));
+ }
+ for (var result : results) {
+ Assertions.assertSame(table.encryption(), result.get(10,
TimeUnit.SECONDS));
+ }
+ Assertions.assertEquals(1, FixtureKms.initialized.get());
+ } finally {
+ workers.shutdownNow();
+ Assertions.assertTrue(workers.awaitTermination(10,
TimeUnit.SECONDS));
+ }
+ }
+ Assertions.assertEquals(1, FixtureKms.closed.get());
+ }
+
+ @Test
+ void plaintextRestTablesDoNotAcquireAnEncryptionManager() throws Exception
{
+ FixtureKms.initialized.set(0);
+ ObjectNode json = (ObjectNode) JsonUtil.mapper().readTree(
+ TableMetadataParser.toJson(Fixture.emptyEncryptedMetadata()));
+ json.with("properties").remove("encryption.key-id");
+ try (Fixture fixture = new Fixture(false,
TableMetadataParser.fromJson(json.toString()))) {
+ Table table = fixture.load();
+ Assertions.assertSame(fixture.catalogIo, table.io());
+
Assertions.assertInstanceOf(org.apache.iceberg.encryption.PlaintextEncryptionManager.class,
+ table.encryption());
+ Assertions.assertEquals(0, FixtureKms.initialized.get());
+ }
+ }
+
+ @Test
+ void missingKmsConfigurationCannotFallBackToPlaintext() throws Exception {
+ FixtureKms.initialized.set(0);
+ try (Fixture fixture = new Fixture(false,
Fixture.emptyEncryptedMetadata(), false)) {
+ Table table = fixture.load();
+ DorisConnectorException failure =
Assertions.assertThrows(DorisConnectorException.class, table::encryption);
+ Assertions.assertEquals("Iceberg encryption [MISSING_KMS]",
failure.getMessage());
+ Assertions.assertEquals(0, FixtureKms.initialized.get());
+ }
+ }
+
+ @Test
+ void retainedKeysCannotFallBackToPlaintextWithoutACurrentWriteKey() throws
Exception {
+ try (MemoryEncryptedTable writer = new MemoryEncryptedTable()) {
+ writer.append();
+ ObjectNode json = (ObjectNode)
JsonUtil.mapper().readTree(TableMetadataParser.toJson(writer.current()));
+ json.with("properties").remove("encryption.key-id");
+ try (Fixture fixture = new Fixture(false,
TableMetadataParser.fromJson(json.toString()))) {
+ DorisConnectorException failure =
Assertions.assertThrows(DorisConnectorException.class,
+ fixture.load()::encryption);
+ Assertions.assertEquals("Iceberg encryption
[MISSING_TABLE_KEY]", failure.getMessage());
+ }
+ }
+ }
+
+ @Test
+ void encryptedRestTransactionsRejectBeforeInitializingKms() throws
Exception {
+ FixtureKms.initialized.set(0);
+ try (Fixture fixture = new Fixture(false)) {
+ Table table = fixture.load();
+ Assertions.assertThrows(UnsupportedOperationException.class,
table::newTransaction);
+ Assertions.assertEquals(0, FixtureKms.initialized.get());
+ }
+ }
+
+ private static int filesAndPayloads(Table table, Long snapshot) throws
Exception {
+ var scan = table.newScan();
+ if (snapshot != null) {
+ scan = scan.useSnapshot(snapshot);
+ }
+ int files = 0;
+ try (CloseableIterable<FileScanTask> tasks = scan.planFiles()) {
+ for (FileScanTask task : tasks) {
+ Assertions.assertNotNull(task.file().keyMetadata());
+ try (var input =
table.io().newInputFile(task.file()).newStream()) {
+ Assertions.assertArrayEquals(new byte[] {1, 2, 3, 4},
input.readAllBytes());
+ }
+ var delete =
FileMetadata.deleteFileBuilder(table.spec()).ofPositionDeletes()
+
.withPath(task.file().location()).withFormat("parquet").withRecordCount(1)
+ .withFileSizeInBytes(task.file().fileSizeInBytes())
+
.withEncryptionKeyMetadata(task.file().keyMetadata()).build();
+ try (var input = table.io().newInputFile(delete).newStream()) {
+ Assertions.assertArrayEquals(new byte[] {1, 2, 3, 4},
input.readAllBytes());
+ }
+ files++;
+ }
+ }
+ return files;
+ }
+
+ private static final class MemoryEncryptedTable implements
TableOperations, AutoCloseable {
+ private final InMemoryFileIO raw = new InMemoryFileIO();
+ private final StandardEncryptionManager manager = new
StandardEncryptionManager(
+ List.of(), "fixture-key", 16, new FixtureKms());
+ private final FileIO io = EncryptingFileIO.combine(raw, manager);
+ private TableMetadata metadata = TableMetadata.newTableMetadata(
+ new Schema(Types.NestedField.required(1, "id",
Types.IntegerType.get())),
+ PartitionSpec.unpartitioned(), SortOrder.unsorted(),
"memory://rest-" + UUID.randomUUID(),
+ Map.of("format-version", "3", "encryption.key-id",
"fixture-key"));
+
+ private void append() throws Exception {
+ String path = metadata.location() + "/" + UUID.randomUUID() +
".parquet";
+ NativeEncryptionOutputFile output =
manager.encrypt(raw.newOutputFile(path));
+ try (var stream = output.encryptingOutputFile().create()) {
+ stream.write(new byte[] {1, 2, 3, 4});
+ }
+ long length = raw.newInputFile(path).getLength();
+ DataFile data =
DataFiles.builder(PartitionSpec.unpartitioned()).withPath(path)
+
.withFormat("parquet").withFileSizeInBytes(length).withRecordCount(1)
+
.withEncryptionKeyMetadata(output.keyMetadata().copyWithLength(length)).build();
+ new BaseTable(this,
"fixture").newFastAppend().appendFile(data).commit();
+ }
+
+ @Override
+ public TableMetadata current() {
+ return metadata;
+ }
+
+ @Override
+ public TableMetadata refresh() {
+ return metadata;
+ }
+
+ @Override
+ public void commit(TableMetadata base, TableMetadata updated) {
+ Assertions.assertSame(metadata, base);
+ // REST refresh identifies a new metadata generation by its
metadata-location, as a real catalog does.
+ metadata = TableMetadata.buildFrom(updated).discardChanges()
+ .withMetadataLocation(updated.location() + "/metadata/" +
UUID.randomUUID() + ".metadata.json")
+ .build();
+ }
+
+ @Override
+ public FileIO io() {
+ return io;
+ }
+
+ @Override
+ public EncryptionManager encryption() {
+ return manager;
+ }
+
+ @Override
+ public String metadataFileLocation(String name) {
+ return metadata.location() + "/metadata/" + name;
+ }
+
+ @Override
+ public LocationProvider locationProvider() {
+ return LocationProviders.locationsFor(metadata.location(),
metadata.properties());
+ }
+
+ @Override
+ public void close() {
+ raw.close();
+ }
+ }
+
+ private static final class Fixture implements AutoCloseable {
+ private final HttpServer server;
+ private final IcebergConnector connector;
+ private final Catalog catalog;
+ private final FileIO catalogIo;
+ private final List<Table> loaded = new ArrayList<>();
+
+ private final AtomicReference<TableMetadata> metadata;
+ private final boolean tableConfig;
+
+ private Fixture(boolean tableConfig) throws Exception {
+ this(tableConfig, emptyEncryptedMetadata());
+ }
+
+ private static TableMetadata emptyEncryptedMetadata() throws Exception
{
+ try (InMemoryCatalog storage = new InMemoryCatalog()) {
+ storage.initialize("fixture", Map.of());
+ storage.createNamespace(Namespace.of("db"));
+ BaseTable table = (BaseTable) storage.createTable(ID,
+ new Schema(Types.NestedField.required(1, "id",
Types.IntegerType.get())),
+ PartitionSpec.unpartitioned(), null,
Map.of("format-version", "3"));
+ ObjectNode json = (ObjectNode) JsonUtil.mapper().readTree(
+
TableMetadataParser.toJson(table.operations().current()));
+ json.with("properties").put("encryption.key-id",
"fixture-key");
+ return TableMetadataParser.fromJson(json.toString());
+ }
+ }
+
+ private Fixture(boolean tableConfig, TableMetadata initialMetadata)
throws Exception {
+ this(tableConfig, initialMetadata, true);
+ }
+
+ private Fixture(boolean tableConfig, TableMetadata initialMetadata,
boolean configuredKms) throws Exception {
+ this.tableConfig = tableConfig;
+ metadata = new AtomicReference<>(initialMetadata);
+ server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0),
0);
+ server.createContext("/v1/", exchange -> {
+ try (exchange) {
+ exchange.getRequestBody().close();
+ byte[] body;
+ if
(exchange.getRequestURI().getPath().equals("/v1/config")) {
+ body =
"{\"defaults\":{},\"overrides\":{\"fixture.kms.scope\":\"catalog\"}}"
+ .getBytes(StandardCharsets.UTF_8);
+ } else {
+ LoadTableResponse.Builder load =
LoadTableResponse.builder().withTableMetadata(metadata.get());
+ if (this.tableConfig) {
+ load.addConfig("fixture.kms.scope", "table");
+ }
+ body =
LoadTableResponseParser.toJson(load.build()).getBytes(StandardCharsets.UTF_8);
+ }
+ exchange.getResponseHeaders().set("Content-Type",
"application/json");
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ }
+ });
+ server.start();
+ Map<String, String> properties = new
HashMap<>(Map.of("iceberg.catalog.type", "rest",
+ "iceberg.rest.uri", "http://127.0.0.1:" +
server.getAddress().getPort(),
+ "rest.auth.type", "none", "io-impl",
TrackingFileIO.class.getName()));
+ if (configuredKms) {
+ properties.put(CatalogProperties.ENCRYPTION_KMS_IMPL,
FixtureKms.class.getName());
+ }
+ connector = new IcebergConnector(properties, new
RecordingConnectorContext());
+ Method get =
IcebergConnector.class.getDeclaredMethod("getOrCreateCatalog");
+ get.setAccessible(true);
+ catalog = (Catalog) get.invoke(connector);
+ Field session =
IcebergConnector.class.getDeclaredField("restSessionCatalog");
+ session.setAccessible(true);
+ catalogIo =
IcebergConnector.restCatalogFileIO(session.get(connector));
+ }
+
+ private Table load() {
+ Table table = catalog.loadTable(ID);
+ loaded.add(table);
+ return table;
+ }
+
+ @Override
+ public void close() throws Exception {
+ try {
+ for (Table table : loaded) {
+ IcebergConnector.cachedTableCleanup(table, "rest").run();
+ }
+ connector.close();
+ } finally {
+ server.stop(0);
+ }
+ }
+ }
+
+ public static class FixtureKms implements KeyManagementClient {
+ private static final AtomicInteger initialized = new AtomicInteger();
+ private static final AtomicInteger closed = new AtomicInteger();
+ private static final AtomicReference<String> scope = new
AtomicReference<>();
+ private static final AtomicBoolean denied = new AtomicBoolean();
+ private static final AtomicBoolean wrongKey = new AtomicBoolean();
+
+ @Override
+ public void initialize(Map<String, String> properties) {
+ Assertions.assertTrue(properties.containsKey("fixture.kms.scope"));
+ scope.set(properties.get("fixture.kms.scope"));
+ initialized.incrementAndGet();
+ }
+
+ @Override
+ public ByteBuffer wrapKey(ByteBuffer key, String keyId) {
+ return key.duplicate();
+ }
+
+ @Override
+ public ByteBuffer unwrapKey(ByteBuffer key, String keyId) {
+ if (denied.get()) {
+ throw new IllegalStateException("Fixture KMS denied
decryption");
+ }
+ byte[] bytes = new byte[key.remaining()];
+ key.duplicate().get(bytes);
+ if (wrongKey.get()) {
+ bytes[0] ^= 1;
+ }
+ return ByteBuffer.wrap(bytes);
+ }
+
+ @Override
+ public void close() {
+ closed.incrementAndGet();
+ }
+ }
+
+ public static class TrackingFileIO implements FileIO {
+ private final InMemoryFileIO delegate = new InMemoryFileIO();
+ private final AtomicInteger closes = new AtomicInteger();
+ private Map<String, String> properties = Map.of();
+
+ @Override
+ public void initialize(Map<String, String> config) {
+ properties = Map.copyOf(config);
+ }
+
+ @Override
+ public Map<String, String> properties() {
+ return properties;
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ Assertions.assertFalse(path.endsWith(".metadata.json"), "REST
metadata must come from the catalog response");
+ return delegate.newInputFile(path);
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return delegate.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ delegate.deleteFile(path);
+ }
+
+ @Override
+ public void close() {
+ closes.incrementAndGet();
+ delegate.close();
+ }
+ }
+}
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java
index 10eacd58728..f528c82c6d3 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java
@@ -6283,14 +6283,19 @@ public class IcebergScanPlanProviderTest {
// --- T09: vended credentials (extractVendedToken + static/vended
location.* + URI threading) ---
- @Test
- public void extractVendedTokenMergesIoPropsAndStorageCredentials() {
+ @ParameterizedTest
+ @ValueSource(booleans = {false, true})
+ public void extractVendedTokenMergesIoPropsAndStorageCredentials(boolean
encryptedRest) {
FakeIcebergTable table = fakeTable("t1");
Map<String, String> ioProps = new HashMap<>();
ioProps.put("s3.endpoint", "ep");
StorageCredential cred =
StorageCredential.create("s3://b",
Collections.singletonMap("s3.access-key-id", "ak"));
- table.setIo(new VendedFileIO(ioProps,
Collections.singletonList(cred)));
+ FileIO io = new VendedFileIO(ioProps, Collections.singletonList(cred));
+ if (encryptedRest) {
+ io = new IcebergRestEncryptionFileIO(io, new
java.util.concurrent.atomic.AtomicReference<>(), false);
+ }
+ table.setIo(io);
Map<String, String> token =
IcebergScanPlanProvider.extractVendedToken(table, true);
diff --git
a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorSecuritySurface.java
b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorSecuritySurface.java
index 6f1cd913dac..af11ddfd5d9 100644
---
a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorSecuritySurface.java
+++
b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorSecuritySurface.java
@@ -21,8 +21,10 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
+import java.util.Set;
/**
* The security-sensitive property surface a connector exposes to the engine's
CREATE / ALTER CATALOG
@@ -147,6 +149,14 @@ public final class ConnectorSecuritySurface {
sensitiveKeys, Collections.unmodifiableList(merged));
}
+ /** Adds endpoint checks without dropping existing driver-url or
secret-masking declarations. */
+ public ConnectorSecuritySurface withSsrfEndpoints(String... keys) {
+ Set<String> endpoints = new LinkedHashSet<>(ssrfEndpointKeys);
+ endpoints.addAll(Arrays.asList(keys));
+ return new ConnectorSecuritySurface(Collections.unmodifiableList(new
ArrayList<>(endpoints)),
+ jdbcDriverUrlKeys, jdbcDriverUrlFlavorKey, sensitiveKeys,
sensitiveKeyPrefixes);
+ }
+
/** The endpoint property keys to check against the SSRF policy; empty
when none. */
public List<String> ssrfEndpointKeys() {
return ssrfEndpointKeys;
diff --git
a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
index f7fe2b195ab..dc747b20ca3 100644
---
a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
+++
b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
@@ -88,8 +88,8 @@ public class ConnectorPluginSurfaceTest {
Assertions.assertNotNull(in, "missing connector plugin API version
resource");
version.load(in);
}
- // Major 32 adds connector-declared secret catalog properties to the
security surface.
- Assertions.assertEquals("32.0", version.getProperty("api.version"));
+ // Endpoint composition extends the published major-32 surface and
needs a distinct version.
+ Assertions.assertEquals("33.0", version.getProperty("api.version"));
}
/** Root entry points plus provider/handle types returned to connector
plugins. */
diff --git
a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorSecuritySurfaceTest.java
b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorSecuritySurfaceTest.java
index cecb61eb91a..c61c39b45da 100644
---
a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorSecuritySurfaceTest.java
+++
b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorSecuritySurfaceTest.java
@@ -148,4 +148,43 @@ public class ConnectorSecuritySurfaceTest {
Assertions.assertThrows(UnsupportedOperationException.class,
() ->
withPrefix.sensitiveKeyPrefixes().get(0).markers().add("key"));
}
+
+ @Test
+ public void addingEndpointChecksPreservesTheExistingDriverPolicy() {
+ ConnectorSecuritySurface original =
ConnectorSecuritySurface.ofJdbcDriverUrls("catalog.type", "driver_url")
+ .withSensitiveKeys("catalog.token")
+ .withSensitiveKeyPrefix("catalog.auth.", "secret");
+ ConnectorSecuritySurface composed =
original.withSsrfEndpoints("kms.endpoint");
+ Assertions.assertEquals(List.of("driver_url"),
composed.jdbcDriverUrlKeys());
+ Assertions.assertEquals("catalog.type",
composed.jdbcDriverUrlFlavorKey());
+ Assertions.assertEquals(List.of("kms.endpoint"),
composed.ssrfEndpointKeys());
+ Assertions.assertEquals(List.of("catalog.token"),
composed.sensitiveKeys());
+ Assertions.assertEquals(original.sensitiveKeyPrefixes(),
composed.sensitiveKeyPrefixes());
+
Assertions.assertTrue(composed.sensitiveKeyPrefixes().get(0).covers("catalog.auth.client-secret"));
+ Assertions.assertTrue(original.ssrfEndpointKeys().isEmpty(),
"composing must not mutate the original surface");
+ Assertions.assertThrows(UnsupportedOperationException.class,
+ () -> composed.ssrfEndpointKeys().clear());
+ }
+
+ @Test
+ public void
composingEndpointChecksKeepsExistingEndpointsWithoutDuplicates() {
+ ConnectorSecuritySurface original =
ConnectorSecuritySurface.ofSsrfEndpoints("catalog.uri");
+ ConnectorSecuritySurface composed =
original.withSsrfEndpoints("catalog.uri", "kms.endpoint");
+ Assertions.assertEquals(List.of("catalog.uri", "kms.endpoint"),
composed.ssrfEndpointKeys());
+ Assertions.assertEquals(List.of("catalog.uri"),
original.ssrfEndpointKeys());
+ }
+
+ @Test
+ public void addingSecretsAfterEndpointChecksPreservesEveryDeclaration() {
+ ConnectorSecuritySurface surface =
ConnectorSecuritySurface.ofJdbcDriverUrls("catalog.type", "driver_url")
+ .withSsrfEndpoints("kms.endpoint")
+ .withSensitiveKeys("catalog.token")
+ .withSensitiveKeyPrefix("catalog.auth.", "secret");
+
+ Assertions.assertEquals(List.of("driver_url"),
surface.jdbcDriverUrlKeys());
+ Assertions.assertEquals("catalog.type",
surface.jdbcDriverUrlFlavorKey());
+ Assertions.assertEquals(List.of("kms.endpoint"),
surface.ssrfEndpointKeys());
+ Assertions.assertEquals(List.of("catalog.token"),
surface.sensitiveKeys());
+
Assertions.assertTrue(surface.sensitiveKeyPrefixes().get(0).covers("catalog.auth.client-secret"));
+ }
}
diff --git
a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
index a5375819f37..d7ec27b5b3e 100644
---
a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
+++
b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
@@ -105,6 +105,7 @@
org.apache.doris.connector.spi.ConnectorSecuritySurface#ssrfEndpointKeys():java.
org.apache.doris.connector.spi.ConnectorSecuritySurface#withSensitiveKeyPrefix(java.lang.String,java.lang.String[]):org.apache.doris.connector.spi.ConnectorSecuritySurface
org.apache.doris.connector.spi.ConnectorSecuritySurface#withSensitiveKeys(java.lang.String[]):org.apache.doris.connector.spi.ConnectorSecuritySurface
org.apache.doris.connector.spi.ConnectorSecuritySurface#withSensitiveKeys(java.util.Collection):org.apache.doris.connector.spi.ConnectorSecuritySurface
+org.apache.doris.connector.spi.ConnectorSecuritySurface#withSsrfEndpoints(java.lang.String[]):org.apache.doris.connector.spi.ConnectorSecuritySurface
org.apache.doris.connector.spi.ConnectorSession#allocateTransactionId():long
org.apache.doris.connector.spi.ConnectorSession#getBeExecVersion():int
org.apache.doris.connector.spi.ConnectorSession#getCatalogId():long
diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml
index 4be342ff5b9..c5569817f24 100644
--- a/fe/fe-connector/pom.xml
+++ b/fe/fe-connector/pom.xml
@@ -55,7 +55,7 @@ under the License.
of the latter two means bumping this property as well (and
fe-extension-spi means bumping
all five families).
-->
- <connector.plugin.api.version>32.0</connector.plugin.api.version>
+ <connector.plugin.api.version>33.0</connector.plugin.api.version>
</properties>
<modules>
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_encrypted_rest_metadata.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_encrypted_rest_metadata.out
new file mode 100644
index 00000000000..d74a821f331
--- /dev/null
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_encrypted_rest_metadata.out
@@ -0,0 +1,84 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !plain_rows --
+1
+2
+3
+4
+
+-- !plain_count --
+4
+
+-- !plain_rows --
+1
+2
+3
+4
+
+-- !plain_count --
+4
+
+-- !plain_rows --
+1
+2
+3
+4
+
+-- !plain_count --
+4
+
+-- !plain_rows --
+1
+2
+3
+4
+
+-- !plain_count --
+4
+
+-- !plain_after_refresh --
+1
+2
+3
+4
+
+-- !plain_rows --
+1
+2
+3
+4
+
+-- !plain_count --
+4
+
+-- !plain_rows --
+1
+2
+3
+4
+
+-- !plain_count --
+4
+
+-- !plain_rows --
+1
+2
+3
+4
+
+-- !plain_count --
+4
+
+-- !plain_rows --
+1
+2
+3
+4
+
+-- !plain_count --
+4
+
+-- !plain_after_refresh --
+1
+2
+3
+4
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_encrypted_rest_metadata.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_encrypted_rest_metadata.groovy
new file mode 100644
index 00000000000..d3caceb6dfe
--- /dev/null
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_encrypted_rest_metadata.groovy
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import groovy.json.JsonSlurper
+
+suite("test_iceberg_encrypted_rest_metadata", "p0,external") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableIcebergTest")))
{
+ return
+ }
+ String host = context.config.otherConfigs.get("externalEnvIp")
+ String kmsPort =
context.config.otherConfigs.get("iceberg_encryption_kms_port")
+ String fixturePort =
context.config.otherConfigs.get("iceberg_encryption_fixture_port")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ def fixtureRequest = { String method, String path ->
+ def connection = (HttpURLConnection) new
URL("http://${host}:${fixturePort}${path}").openConnection()
+ connection.setRequestMethod(method)
+ connection.setConnectTimeout(10000)
+ connection.setReadTimeout(120000)
+ try {
+ assertEquals(200, connection.responseCode, "REST encryption
fixture ${method} ${path} failed")
+ return connection.inputStream.withCloseable { new
JsonSlurper().parse(it) }
+ } finally {
+ connection.disconnect()
+ }
+ }
+ assertTrue(fixtureRequest("POST", "/rest/reset").created)
+ def verified = fixtureRequest("GET", "/rest/verify")
+ assertEquals("rest", verified.catalog)
+ assertEquals("1.12.0", verified.sdkVersion)
+ assertEquals(3, verified.formatVersion)
+ assertEquals([[1, 2], [1, 2, 3, 4]], verified.snapshots)
+ assertTrue(verified.encryptedManifests)
+ assertTrue(verified.encryptedManifestLists)
+ for (String failure : ["missingKms", "wrongCredentials", "denied",
"wrongKey"]) {
+ assertEquals("REJECTED", verified[failure])
+ }
+ def metadata = fixtureRequest("GET",
+
"/rest/v1/namespaces/iceberg_encryption_rest_db/tables/encrypted_v3").metadata
+ def snapshots = metadata.snapshots.collect { it["snapshot-id"] }
+ assertEquals(2, snapshots.size())
+
+ String previousBatch = sql("SHOW VARIABLES LIKE
'enable_external_table_batch_mode'")[0][1]
+ String previousCount = sql("SHOW VARIABLES LIKE
'enable_count_push_down_for_external_table'")[0][1]
+ try {
+ for (String cached : ["false", "true"]) {
+ String catalog = "encryption_rest_cache_${cached}"
+ sql "DROP CATALOG IF EXISTS ${catalog}"
+ sql """CREATE CATALOG ${catalog} PROPERTIES (
+ 'type' = 'iceberg', 'iceberg.catalog.type' = 'rest',
+ 'iceberg.rest.uri' = 'http://${host}:${fixturePort}/rest',
'rest.auth.type' = 'none',
+ 'io-impl' = 'org.apache.iceberg.aws.s3.S3FileIO',
+ 'encryption.kms-type' = 'aws', 'kms.endpoint' =
'http://${host}:${kmsPort}',
+ 'client.region' = 'us-east-1',
+ 'client.credentials-provider' =
'org.apache.doris.connector.iceberg.glue.ConfigurationAWSCredentialsProvider2x',
+ 'client.credentials-provider.glue.access_key' = 'admin',
+ 'client.credentials-provider.glue.secret_key' = 'password',
+ 's3.access_key' = 'admin', 's3.secret_key' = 'password',
+ 's3.endpoint' = 'http://${host}:${minioPort}', 's3.region' =
'us-east-1',
+ 'use_path_style' = 'true', 'meta.cache.iceberg.table.enable' =
'${cached}'
+ )"""
+ for (String batch : ["false", "true"]) {
+ sql "SET enable_external_table_batch_mode=${batch}"
+ for (String count : ["false", "true"]) {
+ sql "SET
enable_count_push_down_for_external_table=${count}"
+ order_qt_plain_rows "SELECT id FROM
${catalog}.iceberg_encryption_rest_db.plain_v3 ORDER BY id"
+ qt_plain_count "SELECT COUNT(*) FROM
${catalog}.iceberg_encryption_rest_db.plain_v3"
+ for (int attempt = 0; attempt < 2; attempt++) {
+ // REST admission follows the same default-off
contract as HMS and must run before
+ // encrypted metadata can enter COUNT, metadata-table
or lazy split paths.
+ for (String statement : [
+ "SELECT id FROM
${catalog}.iceberg_encryption_rest_db.encrypted_v3 ORDER BY id",
+ "SELECT COUNT(*) FROM
${catalog}.iceberg_encryption_rest_db.encrypted_v3",
+ "DESC
${catalog}.iceberg_encryption_rest_db.encrypted_v3"
+ ]) {
+ test {
+ sql statement
+ exception "Iceberg encryption [READ_DISABLED]"
+ }
+ }
+ for (String systemTable : ["files", "snapshots",
"position_deletes"]) {
+ test {
+ sql "SELECT COUNT(*) FROM
${catalog}.iceberg_encryption_rest_db.`encrypted_v3\$${systemTable}`"
+ exception "Iceberg encryption [READ_DISABLED]"
+ }
+ }
+ for (def snapshot : snapshots) {
+ test {
+ sql "SELECT id FROM
${catalog}.iceberg_encryption_rest_db.encrypted_v3 FOR VERSION AS OF
${snapshot}"
+ exception "Iceberg encryption [READ_DISABLED]"
+ }
+ }
+ }
+ }
+ }
+ sql "REFRESH CATALOG ${catalog}"
+ order_qt_plain_after_refresh "SELECT id FROM
${catalog}.iceberg_encryption_rest_db.plain_v3 ORDER BY id"
+ }
+ } finally {
+ sql "SET enable_external_table_batch_mode=${previousBatch}"
+ sql "SET enable_count_push_down_for_external_table=${previousCount}"
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]