Gabriel39 commented on code in PR #70693:
URL: https://github.com/apache/doris/pull/70693#discussion_r4637638458
##########
fe/fe-core/src/main/java/org/apache/doris/tls/server/OssFeServerStarterProvider.java:
##########
@@ -73,36 +77,51 @@ private OssThriftServerStarter(int port, ServerSocket
listener) {
@Override
public void start() throws Exception {
- if (Config.enable_tls &&
TlsProtocolSet.isProtocolIncluded(TlsProtocolSet.Protocol.THRIFT)) {
- throw new UnsupportedOperationException("FE thrift TLS
requires TLS module");
+ boolean tls =
PrivateRpcTls.enabled(TlsProtocolSet.Protocol.THRIFT);
+ if (tls &&
ThriftServer.THREADED_SELECTOR.equalsIgnoreCase(Config.thrift_server_type)) {
+ throw new UnsupportedOperationException("Private RPC TLS
requires a blocking Thrift server");
}
- FrontendServiceImpl service = new
FrontendServiceImpl(ExecuteEnv.getInstance());
- Logger feServiceLogger =
LogManager.getLogger(FrontendServiceImpl.class);
- FrontendService.Iface instance = (FrontendService.Iface)
Proxy.newProxyInstance(
- FrontendServiceImpl.class.getClassLoader(),
- FrontendServiceImpl.class.getInterfaces(),
- (proxy, method, args) -> {
- long begin = System.currentTimeMillis();
- String name = method.getName();
- if (MetricRepo.isInit) {
-
MetricRepo.THRIFT_COUNTER_RPC_ALL.getOrAdd(name).increase(1L);
- }
- feServiceLogger.debug("receive request for {}", name);
- try {
- return method.invoke(service, args);
- } finally {
- ConnectContext.remove();
- feServiceLogger.debug("finish process request for
{}", name);
+ ServerSocket transportListener = listener;
+ boolean started = false;
+ try {
+ if (tls) {
+ transportListener = PrivateRpcTls.thriftServerSocket(new
InetSocketAddress(
Review Comment:
Fixed in f93e5d1c86d18129b54115ea6d00f6a8a076fab7. The native writer now
supplies the local BE's built-in Thrift TLS policy, node identity and bounded
connect timeout, overriding sink-supplied values. The isolated Java plugin
authenticates that FE connection, verifies its hostname and never falls back to
plaintext. Reconnect and master redirection retain the same policy;
native-compatible PKCS#1 keys are supported using the existing pinned BC
version.
Real Thrift/TLS tests exercise initial allocation and the writer's actual
rotation callback, including a preallocated first block, plus redirection and
negative authentication/timeout cases. The callback tests reproduce the
original failure. All 27 selected MaxCompute tests and 33 related native ASAN
tests pass. The MaxCompute storage session is mocked; the FE callback uses a
real listener. A separate isolated-class-loader probe using only the deployed
plugin directory also authenticates with PKCS#1 and reuses its connection.
##########
regression-test/suites/rpc_p0/test_remote_doris_private_rpc_tls.groovy:
##########
@@ -0,0 +1,188 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import groovy.json.JsonSlurper
+import org.apache.doris.regression.suite.ConnectionInfo
+
+import java.nio.file.Files
+import java.sql.DriverManager
+import java.sql.SQLException
+import java.util.concurrent.TimeUnit
+
+suite("test_remote_doris_private_rpc_tls", "p0,nonConcurrent") {
+ def logRoot = new File(System.getProperty("DORIS_HOME"),
"regression-test/log").toPath()
+ Files.createDirectories(logRoot)
+ def logs = Files.createTempDirectory(logRoot, "private-rpc-tls-").toFile()
+ def helper = new File(context.file.parentFile,
"private_rpc_tls_fixture.py")
+ // Keep lifecycle and port-allocation checks in the default regression
entry point as well.
+ def fixtureTests = new ProcessBuilder("python3", "-m", "unittest",
"discover",
Review Comment:
Fixed in f93e5d1c86d18129b54115ea6d00f6a8a076fab7. The suite checks for
Linux before creating logs, running helper tests or launching engines, and logs
an explicit unsupported-host skip. The Linux NonConcurrent path remains enabled
by default, without a test-specific opt-in flag or supplied remote endpoint
settings.
A simulated-macOS admission test fails at the process boundary before the
fix and passes afterward. Normal default-profile Linux replay on the fresh
FE/ASAN BE packages passes all six result blocks, five expected refusals and
nine helper tests, with zero skipped scripts.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/runtime/PipelineExecutionTaskBuilder.java:
##########
@@ -81,12 +90,13 @@ private Map<BackendWorker, MultiFragmentsPipelineTask>
buildMultiFragmentTasks(
ByteString serializeFragments =
workerToSerializeFragments.get(worker);
Backend backend = worker.getBackend();
+ // Remote catalog workers keep their independent cluster's channel
policy.
fragmentTasks.put(
worker,
new MultiFragmentsPipelineTask(
coordinatorContext,
backend,
- backendServiceProxy,
+ isInternalWorker(worker) ? backendServiceProxy :
BackendServiceProxy.getInstance(false),
Review Comment:
Implemented in f93e5d1c86d18129b54115ea6d00f6a8a076fab7. BackendServiceProxy
now binds an immutable BackendRpcTarget from the address and worker catalog
identity. CoordinatorContext retains one binding per worker, and
MultiFragmentsPipelineTask and ResultReceiver share it for prepare, start,
cancel, fetch and failed-channel removal. The binding retains its policy and
proxy shard while the existing proxy cache continues to handle DNS turnover and
channel recreation; it does not pin a physical channel. Legacy constructors
remain available.
Complete execution-graph admission remains in the coordinator before
dispatch and fragment serialization. Binding itself opens no channel. A
repeated-fetch test reproduces the former re-selection, and a shared-worker
test verifies all five operations use the same binding; same-address
local/remote isolation and rejected-graph tests remain. The official 60-test
FE/BE Java selection and real default TLS/plaintext mixed-join regression pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]