Gabriel39 opened a new pull request, #70314: URL: https://github.com/apache/doris/pull/70314
### Background The native Iceberg encryption reader can authenticate Parquet footers, dictionaries and data pages. Its optional encrypted indexes and Bloom filters are still bypassed, so selective scans cannot use those modules safely. ### What problem does this PR solve? Issue Number: close #70313 Related PR: #70246 (file crypto), #70248 (native encrypted pages) PR Type: Feature Problem Summary: Implement D7: authenticate and consume ColumnIndex, OffsetIndex and split-block Bloom filters in the native reader. Indexed reads preserve the physical data-page ordinal after seeking past earlier pages, and compare index rectangles against serialized disk sizes. Authentication and malformed encrypted-module failures propagate; genuinely absent optional indexes retain conservative reads. A cache-hit header also initializes its disk size before index reconciliation, preserving plaintext indexed scans. ### Details - **Change map:** Production code: +313/-82 lines for authenticated metadata parsing, index loading, Bloom framing and indexed-page AAD integration. Tests: about 530 added lines for native reader scenarios, an independent Java fixture generator, SQL checks and goldens, plus four small binary fixture copies. The fixture README documents reproduction and the remaining integration boundary. - **Scope:** Encrypted ColumnIndex/OffsetIndex, independently authenticated Bloom header/bitset, page skipping and plaintext cache-hit compatibility. - **Design:** Reuse Parquet's GCM implementation with module-specific AAD bound to the file, row group and physical column. Keep physical page ordinals across seeks and exclude dictionary pages from that sequence. Bound module allocations and decoded Thrift containers. Preserve coalesced index I/O and one live decoded Bloom payload per leaf. - **Usage:** The existing physical reader encryption descriptor enables these modules in controlled reader integrations. No new configuration or Thrift field is introduced. - **Default behavior:** SQL encrypted-read admission remains closed. Encrypted plaintext page/footer caches remain bypassed, and dictionary pruning remains conservative because its existing optional-probe path suppresses reader errors. Ordinary dictionary decoding still authenticates. - **Compatibility:** Plaintext optional-index fallback is preserved. An absent encrypted index/Bloom filter is optional; a present module with an invalid range, framing or authentication tag fails the read. ### Testing Local verification on this branch: - `build.sh --be -j 6` with ASAN: passed, including the GLIBC 2.17 compatibility gate. `build.sh --fe -j 6`: passed. - `run-be-ut.sh --run --filter='*Parquet*:*FileEncryption*:*FileScanRange*' -j 6` with ASAN and PME-enabled dependencies: 380 tests, 378 passed and 2 existing opt-in microbenchmarks skipped. A focused test source selection was used; this is not a claim that the full BE UT suite ran. - New reader cases cover Page V1/V2, dictionary/plain pages, multiple row groups, physical page skipping, module/file/row-group/column binding, corrupt index/Bloom tags, oversized lengths, invalid ranges, absent Bloom filters and cached plaintext headers. - Parquet Java 1.15.2 generated encrypted/plaintext twins: 512 rows in six row groups, indexes and Bloom filters on both columns. Twelve complete physical-reader scans compare pruned/unpruned exact rows and assert effective page/Bloom pruning counters. - `run-regression-test.sh --run -d external_table_p0/iceberg -s test_iceberg_encryption_native_reader`: passed against this branch's ASAN BE. Six new golden queries compare the plaintext twin with pruning on/off; encrypted-only and mixed-file keyless scans remain rejected, including negative Bloom probes. Goldens were generated through the regression framework and then verified in comparison mode. - The new index tests failed against the previous disabled-index behavior. The invalid-range test also reproduced a silent fallback before the fix. SQL regression exposed the cached-header ordering issue before it was corrected and covered by a dedicated UT. - Clang-format 16, changed-line clang-tidy, build hygiene, whitespace checks and secret scanning: passed. ### Release note None. This advances the native encrypted reader implementation; encrypted Iceberg SQL admission remains separate work. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No; internal staged reader integration and test-fixture documentation are included. - Pre-submission status: Ready. Encrypted SQL admission and shared decrypted cache support are explicitly outside this PR. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Implementation, tests, fixtures generator and this description. - Human review: Pending maintainer review; the agent self-reviewed the full diff and ran the verification above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
