Gabriel39 opened a new pull request, #70343: URL: https://github.com/apache/doris/pull/70343
### Background Iceberg encrypted reads use table metadata to locate retained table keys and decrypt manifest lists, manifests and file descriptors. Doris currently limits this groundwork to HMS catalogs. The pinned Iceberg REST operations retain the plaintext encryption manager, even when the REST response contains an encrypted table's key chain. ### What problem does this PR solve? Issue Number: close #70342 Related PR: #70192 (encrypted metadata groundwork) PR Type: Feature Problem Summary: Adding REST to the catalog allowlist alone leaves encrypted metadata unreadable. This change supplies lazy, read-only REST encryption operations while retaining the SDK REST table, authentication, credentials, refresh and scan capabilities. Most of the change is unit tests and an independent REST encryption fixture/regression suite. REST now follows the same staged scope as HMS; the common encrypted SQL execution fence remains in place. ### Details - **Change map:** 300 product/build/style lines, 901 unit/regression/fixture lines, 85 generated baseline/output lines and 38 documentation lines (insertions plus deletions). - **Scope:** Iceberg v3 REST tables using the standard SDK encryption manager and configured KMS. The initial file target remains Parquet modular encryption with AES_GCM_V1; encrypted writes and unrestricted encrypted SQL execution are outside this PR. - **Design:** A package-access bridge decorates the SDK's loaded REST table operations. Descriptor-aware FileIO decrypts encrypted metadata lazily using the effective catalog/server/table configuration. A standalone metadata holder avoids retaining weak FileIOTracker operation keys; refresh replaces the manager's key chain and reuses the table-owned KMS client. Idempotent close preserves shared catalog FileIO ownership and retires table-specific resources. Transactions, output files and deletion are rejected for encrypted REST tables. - **Usage:** Configure an Iceberg REST catalog with the existing KMS properties. There is no new setting, dependency, service or port. The newly consumed outbound destination is `kms.endpoint`, controlled by catalog configuration and CREATE_PRIV for CREATE CATALOG or ALTER_PRIV for ALTER CATALOG. The provider declares it for the existing SSRF policy while retaining its JDBC driver URL checks. REST server configuration/table overrides remain inside the configured catalog's trust boundary. - **Default behavior:** `enable_encryption_read` remains false. Disabled SQL reads fail with READ_DISABLED before KMS initialization. Enabling the existing intent flag still fails with BACKEND_CAPABILITY_UNAVAILABLE until protected key transport and backend admission are complete. Plaintext REST reads retain the SDK path. - **Compatibility:** No storage or FE/BE wire changes. The additive security-surface composition method requires connector plugin API 31.0 under the repository's SPI contract; deploy the matching engine and connector plugins together. The plugin API baseline is regenerated and the unchanged metadata-method baseline is verified. - **Behavior change:** REST joins HMS in the encrypted metadata target matrix. Missing KMS configuration and retained keys without the mandatory table-key property fail explicitly rather than selecting plaintext crypto. Encrypted REST mutation paths fail as read-only. ### Testing - Local `build.sh --be --fe -j6`, ASAN: passed, followed by an isolated current-worktree FE/BE cluster run. - Connector Iceberg tests with Maven build cache disabled: 1,892 tests, zero failures/errors, 10 existing skips. This includes real connector HTTP REST loading, actual encrypted SDK manifests and descriptor payloads, historical scans/refresh, shared/per-table ownership, concurrent lazy initialization, missing/denied/wrong KMS keys, read-only transactions and vended credential forwarding. The REST manager test fails against the original SDK plaintext operations. - Connector SPI own suite: 154 tests passed, including both API baselines and the plugin version pin. Endpoint policy composition preserves the JDBC policy and immutable endpoint lists. - FE Checkstyle: zero violations during product build and final connector test validation. - New `test_iceberg_encrypted_rest_metadata`: output generated by the regression framework, then compared successfully; 18 golden checks and 128 expected READ_DISABLED errors across cache, batch and COUNT settings. The independent fixture checks two historical row sets, encrypted Parquet/manifest/list magic and independent file keys, plus missing KMS, bad credentials, denied decrypt and wrong-key failures. - Existing `test_iceberg_encrypted_hms_metadata`: passed, including the SDK/Spark reference checks. - Restarted the local FE with the existing read-intent flag enabled: 16 REST SQL queries rejected with BACKEND_CAPABILITY_UNAVAILABLE, while plaintext rows remained correct. - Staged patch whitespace and secret scan: passed. ### Notes for Reviewers The SQL regression deliberately verifies the staged admission contract; successful decryption is exercised by the connector HTTP unit tests and an independent SDK reference reader over real encrypted files. It does not claim that this PR enables encrypted SQL scans. Start with DorisRESTSessionCatalog and IcebergRestEncryptionFileIO for the ownership and metadata-generation boundaries. ### Release note Extend Iceberg encrypted metadata read groundwork to REST Catalog with standard SDK encryption and configured KMS. The existing default-off setting and encrypted SQL execution fence remain in effect. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; connector documentation and a regression reproduction command are included. User-facing documentation can follow when encrypted SQL execution is admitted. - Pre-submission status: Ready; local validation passed, CI pending. ### AI Disclosure - Tool: OpenAI Codex. - Generated: Implementation, unit/regression tests, connector documentation and this description. - Human review: Pending maintainer review; the agent performed a full self-review and local validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
