Gabriel39 opened a new issue, #70740:
URL: https://github.com/apache/doris/issues/70740

   ### Search before asking
   
   - [x] I searched existing issues. Related TLS/MySQL and Arrow Flight work 
does not describe the OSS private RPC transport design below.
   
   ### Description
   
   Provide built-in mutual TLS for the open-source private FE/BE RPC providers. 
The existing provider interfaces and TLS configuration do not by themselves 
provide a usable authenticated transport: the OSS clients use plaintext, and 
private listener activation is unsupported.
   
   This issue records the problem, implementation approach and design of 
#70693. The design baseline is commit 
[`f842904af888`](https://github.com/apache/doris/commit/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5).
 Target release: Doris 5.0. The implementation PR is open at the time of 
writing; this issue does not claim a released feature.
   
   #### Motivation and the Iceberg PME connection
   
   Private RPC carries execution plans, scan ranges, split responses, status 
reports and data exchanges. Deployments that require confidentiality and 
authenticated peers need protection on these connections, independently of 
client-facing MySQL or HTTP TLS.
   
   The encrypted Iceberg read design makes the requirement particularly 
concrete. Parquet Modular Encryption protects file contents. The proposed 
FE-to-BE reader descriptor, `TFileEncryptionInfo`, contains an **unwrapped file 
data-encryption key (DEK)** and an AAD prefix. It never contains the KMS master 
key. If an unwrapped DEK were sent over plaintext RPC, someone able to observe 
that connection could recover the key and decrypt the corresponding file if 
they also obtained its ciphertext.
   
   KMS is a key-management service; DEK is the key used to encrypt file data. 
In an envelope-encryption design, an upper-level key managed by KMS protects 
lower-level keys. Once FE has resolved a usable file DEK, FE-to-BE distribution 
needs its own protection. Iceberg PME does not itself mandate Doris TLS: this 
requirement follows from the chosen key-distribution architecture.
   
   **Scope boundary:** #70693 supplies a transport prerequisite. It does not 
implement KMS integration, complete native PME reading, or enable encrypted 
Iceberg SQL. Existing admission checks remain in place until backend capability 
negotiation, reader support and end-to-end acceptance are demonstrated. Key 
redaction and statement-scoped key/cache ownership are separate work, including 
#70675.
   
   ### Solution
   
   #### 1. Goals and security boundary
   
   - Encrypt the supported internal RPC traffic and require certificates from 
both sides.
   - Authenticate the configured cluster CA trust domain; never retry a failed 
TLS connection as plaintext.
   - Preserve the registered target hostname across DNS resolution, connection 
reuse and reopen where hostname verification is supported.
   - Bound authentication IO, propagate configuration/handshake failures 
through existing RPC failure contracts, and release resources on partial 
initialization.
   - Preserve disabled-TLS behavior and the existing policy of external 
services and legacy custom providers.
   
   TLS protects network traffic. It does not protect against a compromised 
participating process, privileged host access, or plaintext keys captured from 
process memory. A dedicated cluster CA is required: possession of a certificate 
issued by that CA is a cluster RPC credential. Certificate authentication does 
not introduce a per-node or per-role authorization system.
   
   #### 2. Transport coverage
   
   | Connection | Implementation | Policy |
   | --- | --- | --- |
   | FE to local BE private services | Java gRPC client to native BRPC listener 
| Mutual TLS when private `brpc` is included |
   | Local BE to local BE private services | Native BRPC channels, including 
exchanges and internal streaming paths | Same internal TLS policy |
   | FE/BE internal Thrift clients and FE, BE backend/heartbeat listeners | 
Java/native blocking TLS transports | Mutual TLS when `thrift` is included; 
includes BE-to-FE lazy split fetches and callbacks |
   | Brokers and external RPC functions | Existing external-purpose clients | 
Retain the existing OSS plaintext policy |
   | Independent remote Doris cluster | Isolated remote FE/gRPC/BRPC clients | 
Retain its external policy; do not inherit local private credentials |
   | Public MySQL, HTTP, Arrow Flight, BDBJE and other cloud control-plane 
channels | Their own providers | Outside this transport implementation |
   
   Representative local flows:
   
   ```mermaid
   flowchart LR
     FE["Local FE"] -->|"gRPC to BRPC: mTLS"| BE1["Local BE 1"]
     FE -->|"Thrift heartbeat: mTLS"| BE1
     BE1 -->|"Thrift split fetch and callbacks: mTLS"| FE
     BE1 <-->|"Private BRPC exchange: mTLS"| BE2["Local BE 2"]
   ```
   
   #### 3. Reuse the provider boundaries
   
   On FE, `PrivateRpcTls` centralizes TLS context construction and 
configuration validation. `OssBackendServiceClientChannelProvider` configures 
authenticated gRPC channels. `OssThriftClientTransportProvider`, 
`TlsThriftSocket` and `TlsAcceptedThriftSocket` implement blocking Thrift 
client/server authentication, and `OssFeServerStarterProvider` wires the 
listener.
   
   On BE, `util/private_rpc_tls.{h,cpp}` centralizes native BRPC options and 
Thrift TLS socket factories. The OSS client provider and BE server starter 
install them on the existing channel/listener paths. The BRPC listener requires 
TLS rather than accepting both encrypted and plaintext connections.
   
   Business RPC methods and service processing remain unchanged. Default 
provider overloads preserve existing custom-provider dispatch unless those 
providers opt into the new endpoint-purpose handling.
   
   #### 4. Identity, DNS and connection lifetime
   
   Connect to the current resolved address while retaining the logical 
registered hostname as the certificate identity. The native Thrift wrapper 
separates the connect address from the peer identity; Java transports preserve 
the original peer name. Reopen authenticates again, including after a DNS 
address change.
   
   FE gRPC and the Java/native Thrift clients verify the target hostname/IP. 
The pinned native BRPC client authenticates the CA chain but does **not** 
implement per-node hostname or role allowlists. This difference is part of the 
trust contract and must not be presented as uniform hostname verification.
   
   Native socket ownership retains the TLS factory/context until the socket is 
destroyed. Failed initialization closes owned sockets and publishes no reusable 
client. Unstarted and stopped listeners remain safe to destroy; repeated 
cleanup is handled.
   
   #### 5. Bounded handshakes and failure behavior
   
   An unlimited application RPC read timeout must not permit an unauthenticated 
TCP peer to occupy a worker indefinitely. Thrift authentication therefore uses 
a positive connect/handshake bound, preserves a smaller caller timeout, and 
restores normal RPC read/write timeouts after authentication. Accepted-socket 
authentication runs on request workers.
   
   BE snapshots and validates `thrift_connect_timeout_seconds`, including 
overflow-safe conversion to milliseconds. FE uses a positive 
`thrift_rpc_connect_timeout_ms`. A later invalid mutable timeout or 
missing/malformed certificate must return a failed acquisition rather than 
escape a worker as an uncaught construction exception.
   
   Thrift cache creation returns the existing failed `Status`; BRPC setup 
failure returns the cache's existing null-client result. Failed clients/stubs 
are not cached, and a later valid configuration can recover. Diagnostics use 
generic classifications rather than embedding PEM input, passwords or provider 
parser details.
   
   #### 6. Separate internal and external endpoint policies
   
   Endpoint purpose is explicit; it is not inferred from whether the 
destination address happens to look local. Thrift pools, gRPC proxies and 
native BRPC caches keep internal and external clients separate, even when 
host/port values coincide. Reuse, reconnect and eviction preserve that 
separation.
   
   Remote Doris metadata/transaction RPCs, fragment prepare/start, 
fetch/cancel, write channels and partition callbacks retain the remote external 
policy. Both tablet writers propagate the optional 
`TOlapTableSink.remote_endpoint` flag. The exchange tablet partitioner derives 
the same policy from the existing explicit remote master-address metadata; 
cloning preserves it.
   
   The new Thrift flag uses optional field 26 and defaults to false. Its 
identifier was compared against complete master and 4.2/4.1/4.0/3.1 
definitions. Existing partition metadata is reused for the exchange path, so it 
needs no additional wire field.
   
   Encrypted file-key descriptors are rejected before serialization to external 
plaintext fragment channels. Enabling local TLS alone is not evidence that an 
independent remote cluster can receive those keys safely.
   
   #### 7. Cross-cluster query graphs
   
   A direct remote-worker plan can make a remote BE initiate exchanges or 
status reports back into the local cluster. Changing FE's outbound channel 
cannot give that independent BE the local cluster's TLS identity.
   
   When OSS private BRPC or Thrift is enabled, remote Doris reads therefore use 
the existing default `use_arrow_flight=true` mode. External results are fetched 
into local workers, and local query exchanges stay inside the TLS cluster. 
Incompatible `use_arrow_flight=false` worker graphs, including remote-only 
graphs with reverse callbacks, are rejected before task dispatch with 
actionable guidance. Custom providers retain their own contract.
   
   This PR does not add TLS to the external Arrow Flight connection. Metadata 
access and remote tablet-write transport from local workers keep their existing 
external routing; this is not a claim that every remote write SQL scenario has 
passed end-to-end acceptance.
   
   #### 8. Configuration and activation
   
   Use the existing settings on every participating FE and BE, with a distinct 
node certificate/private key and the dedicated cluster CA:
   
   ```properties
   enable_tls=true
   tls_verify_mode=verify_fail_if_no_peer_cert
   tls_certificate_path=/path/to/node-certificate.pem
   tls_private_key_path=/path/to/node-private-key.pem
   tls_ca_certificate_path=/path/to/cluster-ca.pem
   tls_cert_refresh_interval_seconds=0
   ```
   
   For the private-RPC-only deployment described here:
   
   | Process | `tls_excluded_protocols` | Additional requirements |
   | --- | --- | --- |
   | FE | `http,mysql,arrowflight,bdbje` | Blocking Thrift server: 
`THREAD_POOL` or `SIMPLE`; positive `thrift_rpc_connect_timeout_ms` |
   | BE | `http,arrowflight` | Positive bounded 
`thrift_connect_timeout_seconds`; unencrypted PEM private key |
   
   `enable_tls=false` remains the default. Private TLS requires mutual 
verification and rejects missing identities, a nonzero certificate-refresh 
interval and a nonempty `tls_peer_cert_required_san_dns`. BE rejects a nonempty 
private-key password; FE retains its password setting. Certificates need 
client/server usage and SANs matching the addresses peers use.
   
   Deploy the code before activation, provision identities, then coordinate 
activation/restart across participating nodes. Plaintext and TLS-only peers 
cannot communicate; this implementation does not claim a rolling mixed 
plaintext/TLS transition. Identity/trust changes require restart. Keep the CA 
signing key off Doris nodes and restrict access to each node's private key. 
Rollback also requires coordinated configuration and restart.
   
   #### 9. Alternatives and design choices
   
   - **Make every existing pool use local TLS:** breaks plaintext brokers, RPC 
functions and remote Doris clusters. Explicit endpoint purpose and isolated 
caches preserve those contracts.
   - **Let every BE resolve wrapped keys through KMS:** could avoid FE-to-BE 
plaintext-DEK distribution, but changes KMS credentials, authorization and key 
ownership on BEs. It is a separate key-distribution design and would not itself 
encrypt query exchanges.
   - **Use an external encrypted network/tunnel:** a deployment can provide a 
different protected transport boundary. This PR implements native certificate 
authentication through the existing OSS provider interfaces and does not assume 
such infrastructure.
   
   #### 10. Validation and acceptance
   
   | Area | Required coverage |
   | --- | --- |
   | Authentication | Real gRPC/BRPC and Thrift listeners; trusted, absent and 
untrusted client certificates; plaintext rejection |
   | Identity | DNS-only certificates, wrong host rejection, cache reuse/reopen 
and changed resolved address |
   | Failure and cleanup | Invalid timeout, missing/malformed identity files, 
bounded idle-peer handshake, failed acquisition/recovery, partial listener 
startup and repeated shutdown |
   | Compatibility | Disabled TLS, legacy custom providers, broker 
heartbeat/filesystem consistency, external RPC validation/execution and 
same-address policy isolation |
   | Remote routing | Metadata and worker RPC paths, native write/stream 
transports, partition master retry, exchange partitioner and clone policy |
   | Default regression | Local TLS FE plus two TLS BEs against an independent 
plaintext FE/BE; ordered reads and directional broadcast/shuffle joins; 
direct-worker refusals |
   | Fixture lifecycle | Generated ephemeral identities/ports, sanitizer 
resources, owned/foreign PID handling, startup failure/cancellation, key 
deletion even if log archiving fails |
   
   The SQL suite is `rpc_p0/test_remote_doris_private_rpc_tls`, in 
`p0,nonConcurrent`, selected by the standard NonConcurrent profile. It has no 
test-specific enable flag or supplied remote endpoint/account settings. 
`privateRpcTlsPackageHome` optionally locates FE/BE packages and defaults to 
`DORIS_HOME`; both checkout `output/` and installed `fe/`/`be/` layouts work. 
Missing packages or failed startup fail the test rather than silently skip it. 
Diagnostic logs remain in the standard regression log tree; owned processes, 
keys and data are removed.
   
   Previously recorded local validation for the design baseline includes a full 
ASAN BE package build; 32 related ASAN native tests; nine helper tests executed 
by the default suite; six SQL result blocks and five expected refusals with 
zero skips; and trusted/missing/untrusted identity checks on seven private 
listeners. Both new exchange-partitioner tests fail with the policy initializer 
removed and pass with it, including a real plaintext `createPartition` RPC 
under enabled local TLS. The SQL replay used that fresh ASAN BE and a 
previously built FE package. Creating this issue did not rerun those tests.
   
   CI and release packaging/ABI validation remain separate requirements. Full 
remote auto-partition INSERT SELECT acceptance is not claimed: preexisting 
metadata/replica-ID write problems also reproduce with TLS disabled and are 
outside this transport change. Complete Iceberg PME SQL acceptance remains 
separate from the transport tests.
   
   #### References
   
   - Implementation: #70693; related key/cache ownership work: #70675.
   - [Operator guide at the design 
baseline](https://github.com/apache/doris/blob/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5/docs/private-rpc-tls.md).
   - [Native TLS 
implementation](https://github.com/apache/doris/blob/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5/be/src/util/private_rpc_tls.cpp)
 and [FE TLS 
implementation](https://github.com/apache/doris/blob/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5/fe/fe-core/src/main/java/org/apache/doris/tls/PrivateRpcTls.java).
   - [File-key descriptor 
contract](https://github.com/apache/doris/blob/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5/gensrc/thrift/PlanNodes.thrift#L327-L337).
   - [Default 
regression](https://github.com/apache/doris/blob/f842904af888d3aa3fb8ba32fa8f8ca5e6eca0a5/regression-test/suites/rpc_p0/test_remote_doris_private_rpc_tls.groovy).
   - Operator documentation: apache/doris-website#4666.
   
   ### Are you willing to submit PR?
   
   - [x] Yes. Implementation is proposed in #70693.
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to