Gabriel39 opened a new pull request, #70246: URL: https://github.com/apache/doris/pull/70246
### Background Iceberg decrypts encrypted manifests before native scan planning. Doris already has a per-file encryption Thrift descriptor, but the Iceberg planner does not populate its key and AAD fields. ### What problem does this PR solve? Issue Number: #70245 Related PR: #70192 (metadata planning and fixture), #70213 (file-key wire contract) PR Type: Feature Problem Summary: Extract each data file's StandardKeyMetadata into every scan split so native readers can receive the correct file key and AAD. Preserve plaintext files and reject malformed metadata without exposing binary payloads. Also fix the encrypted metadata fixture's first reset after KMS restart: retained HMS tables previously caused purge to fail while reading manifests encrypted with the lost key. ### Details - **Change map:** 174 production lines changed, 205 test lines, 29 documentation lines and 2 Checkstyle configuration lines; no generated files. Production changes add a bounded SDK metadata bridge, an immutable per-file carrier and split population; tests cover SDK interoperability and split ownership, plus a real fixture restart; documentation describes the stage boundary and lifecycle test. - **Scope:** Parquet data-file key transport. Encrypted SQL admission remains disabled; encrypted deletes and native reader integration are subsequent stages. - **Design:** Validate metadata size and encoded field lengths before the SDK codec can allocate. Parse one immutable snapshot, reuse the carrier only within a file, and copy mutable Thrift buffers for each split. Null per-file metadata remains plaintext. Errors contain static classifications without codec causes. - **Usage:** No new configuration. Existing encrypted-read admission and BE dispatch guards remain in place. - **Default behavior:** Encrypted SQL remains rejected. Plaintext ranges retain their previous wire shape. - **Compatibility:** Reuses existing optional Thrift fields and the Iceberg SDK's StandardKeyMetadata v1 codec; no protocol or plugin SPI change. - **Behavior change:** Native scan splits now carry per-file encryption parameters; the test fixture can reset retained tables after its KMS loses keys. Encrypted SQL admission is unchanged. - **Fixture fix:** Drop only the two dedicated fixture HMS entries without purge, then delete and verify their fixed object prefixes. Cleanup therefore needs no old KMS key and leaves no orphan objects in those prefixes. The restart regression retains HMS/storage, restarts KMS/fixture, and requires the first reset to succeed without retrying it. ### Testing - Locally: 286 FE tests across scan planning/ranges, SDK encrypted metadata I/O, admission and the new adapter; zero failures. Final adapter/split rerun: 5 tests, zero failures. - Locally: Maven reactor validation / FE Checkstyle passed. - Before the fix: the first fixture reset after restarting the disposable KMS returned HTTP 500. - After the fix: the committed lifecycle regression passed; SDK and Spark both read historical snapshots `[1,2]` and `[1,2,3,4]`, with two file keys and encrypted manifests/manifest lists. Missing KMS, invalid credentials, denied access and wrong keys remained rejected. - Shell syntax and diff whitespace checks passed. Full encrypted Doris SQL is intentionally unavailable at this stage. ### Release note None. This prepares file-key transport while encrypted query execution remains disabled, and makes the external regression fixture restartable. ### Check List (For Author) - Behavior changed: Yes (see Details: per-file transport and fixture reset) - Does this need documentation: No user-facing feature enablement; internal contract and fixture instructions updated. - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests and this description. - Human review: Pending maintainer review; agent self-review completed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
