Gabriel39 opened a new pull request, #70697:
URL: https://github.com/apache/doris/pull/70697

   ### Background
   
   Native Parquet data/delete readers and binary per-file key descriptors 
exist, but encrypted SQL must stay closed until both private RPC channels and 
the actual selected readers can prove support.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #70675, #70693
   PR Type: Enhancement
   
   Add default-off HMS v3 native encrypted reads with fresh backend capability 
leases, dispatch/fetch checks and end-to-end SQL coverage. The existing 
cache/lifetime and private transport changes remain independently reviewed in 
the related PRs. Until they merge, this draft also contains those prerequisite 
commits; the integration change starts at `55be6efea33e`.
   
   ### Details
   
   - Optional heartbeat and split-request capabilities default to unavailable 
for old nodes. Capability proofs expire, clear on failed/missing heartbeats and 
never survive FE image replay. Both selected private TLS providers and the live 
split listener must prove support.
   - Key-bearing scans use a distinct native format, including shared 
parameters and plaintext files with encrypted deletes. Assigned readers are 
checked before serialization and again before send; lazy responses check the 
actual caller and keep refusals terminal. A capability refusal does not 
blacklist a healthy backend.
   - Declared encrypted COUNT uses physical readers. Undeclared per-file keys 
in raw COUNT probes fail explicitly, and SDK byte-cache hits cannot bypass 
current descriptor/KMS validation. Plaintext metadata COUNT remains available.
   - Independent data/position/equality-delete keys, binary AAD, snapshots, 
native position-delete metadata, pruning and batch settings have SQL coverage. 
Encrypted JNI metadata, unsupported formats, custom transports without proof 
and encrypted writes remain outside admission. V3 planning remains synchronous.
   - Connector plugin API advances to 34.0; 33.0 is reserved by #70343. 
Protocol IDs were checked against master and branches 4.2, 4.1, 4.0 and 3.1. FE 
and its connector plugins must be deployed together.
   
   ### Testing
   
   - Complete RELEASE FE/BE build; FE Checkstyle reports zero violations. Final 
ASAN native subset: 181 tests in 13 suites. FE 
capability/heartbeat/dispatch/split/provider coverage: 125 tests. Connector 
cache/SPI/Iceberg suites: 2,349 tests, zero failures/errors and 11 existing 
skips.
   - Two enabled SQL matrix runs cover all eight key modes, table/page cache 
settings, COUNT/pruning/batch settings, SDK/Spark reference snapshots, joins 
and deletes. Both default-off admission suites pass.
   - Same-endpoint mock KMS revocation fails warmed SELECT and COUNT, then 
restoration recovers. Physical object-read failures, 40 concurrent reads, 10 
cancellations, five timeouts and five query-memory refusals preserve subsequent 
reads. Mock failures validate protocol/error handling, not production AWS IAM.
   - Real authenticated fragment and split RPCs, binary descriptor round trips, 
trust/identity rejection and cache reconnect/recovery pass. Old/new FE/BE 
combinations preserve plaintext and reject unsupported encrypted execution.
   - Live-object/resource checks and exact fixture-key diagnostic audits are 
recorded with their limits in the validation note. No deterministic JVM 
zeroization, arbitrary transformed-key absence or production rolling-upgrade 
certification is claimed.
   - Matching RELEASE cost measurements use the same synthetic million-row 
fixture, cache definitions, fixed affinity and a predeclared 5% plaintext 
median limit. All six fixed-preconditioning conditions satisfy the unchanged 
numerical gate; earlier failing protocols, CPU/RSS, object access, pruning and 
KMS cost remain visible in 
`fe/fe-connector/docs/iceberg-encryption-validation.md`. Shared-host results 
require performance review.
   
   ### Release note
   
   Opt-in native Iceberg encrypted reads require HMS v3, the SDK standard 
encryption manager, supported Parquet encryption, authenticated private RPC and 
fresh capable nodes. The deployment switch remains false by default.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes; see Details.
   - Documentation: Read/wire contracts and reproducible fixture guidance are 
included.
   - Readiness: Draft pending prerequisite merges, performance evidence review 
and maintainer/security review.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6).
   - Generated: Implementation, tests, documentation and this description.
   - Human review: Pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to