Gabriel39 opened a new pull request, #70693: URL: https://github.com/apache/doris/pull/70693
### Background FE sends query fragments to BE using gRPC/BRPC. BE retrieves lazy external scan splits from FE over Thrift. OSS implementations expose TLS provider interfaces, while their private clients use plaintext and their servers reject TLS activation. ### What problem does this PR solve? Issue Number: None Related PR: #70675 (separate encrypted table/cache ownership hardening) PR Type: Enhancement Problem Summary: OSS deployments need authenticated, encrypted internal RPC before transporting private file-key descriptors. Implement mutual TLS for fragment channels and blocking Thrift clients/listeners, using the existing TLS configuration and explicit protocol exclusions. Preserve the original certified hostname after DNS resolution, keep reconnects encrypted, and bound authentication IO independently of normal RPC read timeouts. Connection and server lifecycle tests also cover failed initialization and repeated cleanup. ### Details - **Change map:** 930 changed production lines, 802 test/fixture lines, 97 documentation lines, and no generated files. Native TLS setup and bounded sockets live in `private_rpc_tls`; Java contexts and Thrift transports live in `tls`; OSS providers select them, and the shared Thrift server accepts a provider listener. Tests use ephemeral OpenSSL identities and real listeners. - **Before and after:** Included private protocols now require a trusted peer certificate and encrypt their traffic. Authentication failures propagate to callers. Java and native Thrift reconnects retain TLS, and a peer that stalls before authentication cannot inherit an unlimited RPC idle timeout. Cleanup handles servers whose TLS validation failed before start, as well as a server already joined during explicit shutdown. - **Configs and defaults:** `enable_tls=false` stays the default. Private TLS requires `verify_fail_if_no_peer_cert`, certificate/key/CA files, static refresh interval zero, and a positive Thrift connect timeout. Unsupported SAN allowlists and BE private-key passwords are rejected. See `docs/private-rpc-tls.md` for public-protocol exclusions, certificate rotation and dedicated-cluster-CA requirements. Native BRPC clients authenticate the configured CA trust domain; FE gRPC and Thrift clients additionally verify the target hostname/IP. - **Impact on existing users:** Default plaintext deployments keep their transport behavior. Operators can opt into OSS private RPC mutual TLS with coordinated node configuration. Existing encrypted Iceberg SQL capability/admission checks remain in place. ### Testing Local verification used the transport patch together with the separate cache/lifetime hardening in #70675. All 17 transport files are identical on this independent PR branch, and their preimages are unchanged by that cache patch. - Local repository `run-fe-ut.sh`: 11 tests passed, including real gRPC/Thrift authentication, hostname and CA rejection, plaintext control, reconnects in both transport modes, and stalled-handshake cleanup. - Local repository `run-be-ut.sh`: 158 ASAN tests in 11 suites passed, including 11 TLS tests and native encrypted Parquet/data/delete/cancellation cases. New lifecycle tests reproduced null cleanup and repeated-join failures before the fixes; stalled-peer tests failed before bounded authentication was added. - Full ASAN `build.sh --be --fe` passed with FE Checkstyle; clang-format 16, header hygiene and clang-tidy passed on the affected C++ files. The shipped BE ELF has no GLIBC requirement newer than 2.17. - An isolated local FE/BE cluster completed authenticated heartbeats, a BE fragment query returning count 100 and sum 4950, an HMS data scan returning IDs 1–4, and a lazy V2 delete scan returning 1,3,4,6,8. The FE `fetchSplitBatch` RPC counter confirmed BE-to-FE split fetches. - All four live private RPC endpoints accepted trusted identities and rejected missing/untrusted identities, wrong CA, wrong hostname and plaintext. Trusted connections succeeded again after rejection checks. - Independent SDK 1.12.0 and Spark reference reads agreed on historical snapshots with real encrypted manifests, manifest lists and file keys. Default encrypted SQL still rejected admission with `READ_DISABLED`. ### Release note OSS deployments can enable mutual TLS for internal gRPC/BRPC and blocking Thrift using the existing TLS settings. Private RPC mutual authentication, static certificate configuration and explicit exclusions for unsupported public protocols are required. The default remains disabled. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; operator and trust-boundary notes are included in `docs/private-rpc-tls.md`. - Pre-submission status: Ready for transport review; encrypted SQL capability negotiation and broader encrypted-read acceptance are separate work. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Implementation, tests, fixture script, documentation and this description. - Human review: Pending maintainer review; agent self-review and local verification completed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
