Gabriel39 opened a new issue, #70715:
URL: https://github.com/apache/doris/issues/70715

   ### Background
   
   Iceberg v3 tables can use the SDK standard encryption scheme for manifests 
and independent Parquet data/delete files. Doris needs to preserve that key 
separation and current KMS authorization when planning and executing native 
scans.
   
   ### Current behavior
   
   Metadata integration, native Parquet decryption and binary file-key 
descriptors are separate pieces. An encryption-enabled Arrow build alone does 
not prove that the selected backend readers and both private RPC paths can 
safely execute encrypted scans. Native encrypted SQL therefore requires an 
explicit execution contract.
   
   ### Expected behavior
   
   An operator can opt into native encrypted reads for HMS v3 tables when the 
standard SDK manager, supported Parquet encryption, protected internal 
transports and fresh backend capabilities are available. Unsupported execution 
fails explicitly. Plaintext reads retain their existing behavior.
   
   ### Proposed design
   
   Use a deployment switch that defaults to false in the Iceberg plugin 
configuration. Require authenticated private fragment and split RPC, fresh 
process-local backend capability leases and a distinct native encrypted Parquet 
format. Carry each data/delete key and binary AAD independently, and recheck 
assigned workers before serialization/send and split callers before returning 
keys.
   
   Statement-owned metadata/manifest readers bound key lifetime and avoid 
cross-statement authorization reuse. Encrypted tables cannot enter or replay 
SQL result caches: an unchanged snapshot or metadata file does not attest 
current KMS permission. Keep an independent encrypted-write rejection through 
transaction refresh and final metadata CAS.
   
   Release-version inference cannot identify the actual readers/transports. 
Reusing a stable metadata token for authorization would skip current key 
checks; neither is an execution proof.
   
   ### Impact
   
   This affects native Iceberg scans of standard encrypted HMS v3 tables and 
their FE/BE private RPC paths. Existing deployments keep the read switch 
disabled unless configured explicitly. FE and connector plugins must be 
upgraded together for the connector SPI version change.
   
   ### Reproduction or evidence
   
   The implementation and validation are in #70697. Focused coverage includes 
`IcebergEncryptedWriteFenceTest`, `BackendFileEncryptionCapabilityTest`, real 
authenticated fragment/split RPC tests and the opt-in native-read SQL suites. 
Mock KMS failures exercise authorization/error propagation without claiming 
production cloud-provider policy acceptance.
   
   ### Scope and non-goals
   
   Support HMS v3, the SDK standard encryption manager and native Parquet 
AES_GCM_V1 data/position-delete/equality-delete reads. Encrypted writes, REST 
native encrypted execution, encrypted JNI metadata readers and encrypted V3 
lazy SQL execution are outside this feature's admission.
   
   ### Acceptance criteria
   
   - Preserve independent data/delete keys, binary AAD and snapshot/delete 
semantics in native SQL results.
   - Reject unsupported, missing, stale or downgraded reader/transport 
capabilities while preserving plaintext compatibility.
   - Authenticate/encrypt both private key-carrying channels and cover 
cached/reopened connections and trust/identity failures.
   - Revoke encryption capabilities immediately after failed heartbeats without 
bypassing liveness tolerance on followers.
   - With SQL caching explicitly enabled, retain a real plaintext cache-hit 
control and reject warmed encrypted COUNT/SUM after same-endpoint KMS 
revocation; recover after permission restoration.
   - Reject encrypted native writes before dispatch and after concurrent 
metadata changes; preserve plaintext writes and existing OCC checks.
   - Record lifecycle, compatibility, diagnostic and cost evidence together 
with its deployment and acceptance limits.
   
   ### Risks and compatibility
   
   Optional heartbeat/split fields default to unavailable for old nodes. Old 
readers must not receive encrypted tasks through a format they silently 
interpret as plaintext. Capability state must not survive FE images as a valid 
runtime proof. The deployment switch remains off by default; custom transport 
providers must explicitly establish the contract.
   
   ### Related
   
   #70697 implements the integration and remains a draft while its 
prerequisites and acceptance are reviewed. #70675 supplies cache/statement 
ownership, and #70693 supplies private RPC transport support.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to