Gabriel39 opened a new pull request, #70321: URL: https://github.com/apache/doris/pull/70321
### Background Iceberg data, position-delete and equality-delete files can each have a different native Parquet encryption key and AAD. The existing implementation rejects encrypted deletes during FE planning, and the native table/delete readers do not yet receive their independent descriptors. ### What problem does this PR solve? Issue Number: close #70316 Related PR: #70246 (file crypto), #70248 (native encrypted pages), #70314 (independent index/Bloom work) PR Type: Feature Problem Summary: Implement D8: preserve each Parquet delete file's own key/AAD from FE planning to the physical reader, including the native `position_deletes` metadata reader. Keep decrypted position/equality objects and converted equality keys out of shared split caches so a warm path cannot authorize a later split with another key. Reject encrypted ORC/Puffin descriptors and keep metadata-only aggregates from bypassing delete authentication. This branch is based directly on master and does not depend on #70314. ### Details - **Change map:** About 200 changed production lines connect immutable FE descriptors, a neutral TableReader encryption hook, delete-reader construction and cache bypass. Most changes are coverage: six BE scenarios with multiple key/file variants, FE planning tests, a Java reference fixture, SQL checks and generated golden results. Fixture documentation explains reproduction and the admission boundary. - **Scope:** Independent Parquet data/position/equality keys; mixed plaintext/encrypted files; native `position_deletes`; unsupported encrypted ORC/Puffin rejection; metadata COUNT protection. - **Design:** The generic TableReader receives a split-local descriptor through a protected hook. Iceberg supplies the data descriptor, while each delete reader uses only its own descriptor. Fresh Thrift buffers preserve carrier immutability. Encrypted decoded positions, equality filters and type-converted filters bypass shared caches; plaintext caching remains intact. No key bytes enter cache identities. - **Usage:** Existing physical-reader encryption descriptors support controlled native integrations. No configuration, dependency or wire-schema field is added. Reference fixtures reuse the existing encryption fixture service and its pinned Iceberg 1.12.0 jars. - **Default behavior:** Encrypted SQL admission remains closed pending protected key transport and cache integration. Encrypted files are reauthenticated per split, including after a successful read of the same path. Metadata aggregate rejection remains explicit. - **Compatibility:** Plaintext reads and delete semantics retain their existing behavior. The common plaintext FE carrier avoids an extra allocation. V3 coverage uses V2 snapshots upgraded to V3, preserving their existing Parquet position deletes. ### Testing Local verification of this branch: - `build.sh --be --fe -j 6` with ASAN: passed, including the GLIBC 2.17 gate and FE Checkstyle. - `run-be-ut.sh --run --filter='*Iceberg*:*FileEncryption*:*FileScanRange*:*Parquet*' -j 6`: 519 tests, 517 passed and 2 existing opt-in microbenchmarks skipped. A focused source selection was used; the full BE UT suite was not run. - BE coverage includes all eight data/position/equality encryption combinations with distinct keys, repeated splits, warmed paths followed by missing/wrong keys and wrong AAD, recovery through `abort_split`, independent converted equality filters, metadata-table row projection, unsupported encrypted formats and metadata COUNT rejection before I/O. Four initial scenarios failed on master before the implementation. - Iceberg connector module tests, with Maven build cache disabled and Checkstyle enabled: 1,877 tests, zero failures/errors, 10 existing skips. - Independent Java reference reads verify 16 V2/upgraded-V3 tables, actual encrypted/plaintext payload magic, distinct descriptor keys/AAD and exact rows after applying both delete types. - `test_iceberg_encryption_delete_reader`: passed generation and comparison modes through `run-regression-test.sh`, against this branch's FE and ASAN BE. The matrix covers eager/lazy planning and COUNT pushdown on/off: 80 golden SQL checks and 176 explicit encrypted-read rejections. Plaintext position-delete metadata remains readable when only data or equality files are encrypted. Equality deletes make these SQL COUNT cases follow normal scan/dispatch; dedicated FE/BE UTs cover early metadata-only COUNT rejection. - Existing `test_iceberg_encryption_native_reader`: passed against the same deployment. - The new SQL suite reproduced master's earlier `DELETE_READ_UNAVAILABLE` rejection before FE descriptor propagation. Expected results were generated by the regression framework; only trailing EOF whitespace was normalized. - Clang-format 16, changed-line clang-tidy, build hygiene, whitespace checks and secret scanning: passed. ### Notes for Reviewers The fixture service compiles Java at startup. Restart `encryption-fixture` from the updated checkout before running the new SQL suite; it reuses the existing services, ports and reference jars. These public synthetic file-key fixtures keep table metadata plaintext to isolate native-file planning from KMS/catalog admission. Successful encrypted SQL execution remains outside this PR. ### Release note None. This completes native encrypted-delete integration while retaining the SQL admission fence. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No; internal staged integration and fixture documentation are included. - Pre-submission status: Ready. Full CI remains to run; encrypted SQL admission and shared decrypted cache support remain separate work. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Implementation, tests, reference fixture and this description. - Human review: Pending maintainer review; the agent self-reviewed the complete diff and ran the verification above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
