Gabriel39 opened a new pull request, #70697: URL: https://github.com/apache/doris/pull/70697
### Background Native Parquet data/delete readers and binary per-file key descriptors exist, but encrypted SQL must stay closed until both private RPC channels and the actual selected readers can prove support. ### What problem does this PR solve? Issue Number: None Related PR: #70675, #70693 PR Type: Enhancement Add default-off HMS v3 native encrypted reads with fresh backend capability leases, dispatch/fetch checks and end-to-end SQL coverage. The existing cache/lifetime and private transport changes remain independently reviewed in the related PRs. Until they merge, this draft also contains those prerequisite commits; the integration change starts at `55be6efea33e`. ### Details - Optional heartbeat and split-request capabilities default to unavailable for old nodes. Capability proofs expire, clear on failed/missing heartbeats and never survive FE image replay. Both selected private TLS providers and the live split listener must prove support. - Key-bearing scans use a distinct native format, including shared parameters and plaintext files with encrypted deletes. Assigned readers are checked before serialization and again before send; lazy responses check the actual caller and keep refusals terminal. A capability refusal does not blacklist a healthy backend. - Declared encrypted COUNT uses physical readers. Undeclared per-file keys in raw COUNT probes fail explicitly, and SDK byte-cache hits cannot bypass current descriptor/KMS validation. Plaintext metadata COUNT remains available. - Independent data/position/equality-delete keys, binary AAD, snapshots, native position-delete metadata, pruning and batch settings have SQL coverage. Encrypted JNI metadata, unsupported formats, custom transports without proof and encrypted writes remain outside admission. V3 planning remains synchronous. - Connector plugin API advances to 34.0; 33.0 is reserved by #70343. Protocol IDs were checked against master and branches 4.2, 4.1, 4.0 and 3.1. FE and its connector plugins must be deployed together. ### Testing - Complete RELEASE FE/BE build; FE Checkstyle reports zero violations. Final ASAN native subset: 181 tests in 13 suites. FE capability/heartbeat/dispatch/split/provider coverage: 125 tests. Connector cache/SPI/Iceberg suites: 2,349 tests, zero failures/errors and 11 existing skips. - Two enabled SQL matrix runs cover all eight key modes, table/page cache settings, COUNT/pruning/batch settings, SDK/Spark reference snapshots, joins and deletes. Both default-off admission suites pass. - Same-endpoint mock KMS revocation fails warmed SELECT and COUNT, then restoration recovers. Physical object-read failures, 40 concurrent reads, 10 cancellations, five timeouts and five query-memory refusals preserve subsequent reads. Mock failures validate protocol/error handling, not production AWS IAM. - Real authenticated fragment and split RPCs, binary descriptor round trips, trust/identity rejection and cache reconnect/recovery pass. Old/new FE/BE combinations preserve plaintext and reject unsupported encrypted execution. - Live-object/resource checks and exact fixture-key diagnostic audits are recorded with their limits in the validation note. No deterministic JVM zeroization, arbitrary transformed-key absence or production rolling-upgrade certification is claimed. - Matching RELEASE cost measurements use the same synthetic million-row fixture, cache definitions, fixed affinity and a predeclared 5% plaintext median limit. All six fixed-preconditioning conditions satisfy the unchanged numerical gate; earlier failing protocols, CPU/RSS, object access, pruning and KMS cost remain visible in `fe/fe-connector/docs/iceberg-encryption-validation.md`. Shared-host results require performance review. ### Release note Opt-in native Iceberg encrypted reads require HMS v3, the SDK standard encryption manager, supported Parquet encryption, authenticated private RPC and fresh capable nodes. The deployment switch remains false by default. ### Check List (For Author) - Behavior changed: Yes; see Details. - Documentation: Read/wire contracts and reproducible fixture guidance are included. - Readiness: Draft pending prerequisite merges, performance evidence review and maintainer/security review. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, documentation and this description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
