Henry Robinson has uploaded a new patch set (#4). Change subject: IMPALA-4856: Port ImpalaInternalService to KRPC ......................................................................
IMPALA-4856: Port ImpalaInternalService to KRPC This patch ports the ImpalaInternalService to KRPC. * ImpalaInternalService is split into two KRPC services. The first, ImpalaInternalService, deals with control messages for plan fragment instance execution, cancellation and reporting. The second, DataStreamService, handles large-payload RPCs for transmitting runtime filters and row batches between hosts. The separation allows us to dedicate resources to each service, rather than have them compete for the same thread pool and queue space. * In the DataStreamService, all RPCs use 'native' protobuf. ImpalaInternalService RPCs remain wrappers of Thrift data structures. * This patch adds support for asynchronous RPCs to the RpcMgr and Rpc classes. Previously, Impala used fixed size thread pools + synchronous RPCs to achieve some parallelism for 'broadcast' RPCs like filter propagation, or a dedicated per-sender+receiver pair thread on the sender side in the DataStreamSender case. In this patch, the PublishFilter(), CancelPlanFragment() and TransmitData() RPCs are all sent asynchronously using KRPC's thread pools. * The TransmitData() protocol has changed to adapt to asynchronous RPCs, and to more properly handle the case where receiver queues are full. The full details are in data-stream-mgr.h. * As a result, DataStreamSender no longer creates a thread-per-connection on the sender side. * Both tuple transmission and runtime filter publication use sidecars to minimise the number of copies and serialization steps required. * A large portion of this patch is the replacement of TRowBatch with its Protobuf equivalent, RowBatchPb. The replacement is a literal port of the data structure, and row-batch-test, row-batch-list-test and row-batch-serialize-benchmark continue to execute without logic changes. * Simplify FindRecvr() logic in DataStreamManager. No-longer need to handle blocking sender-side, so no need for complex promise-based machinery. Instead, all senders with no receiver are added to a per-receiver list, which is processed when the receiver arrives. If it does not arrive promptly, the DataStreamManager cleans them up after FLAGS_datastream_sender_timeout_ms. * This patch also begins a clean-up of how ImpalaServer instances are created (by removing CreateImpalaServer), and clarifying the relationship between ExecEnv and ImpalaServer. ImpalaServer now follows the standard construct->Init()->Start()->Join() lifecycle that we use for other services. TESTING ------- * New tests added to rpc-mgr-test. TO DO ----- * Re-enable throughput and latency measurements per data-stream sender when that information is exposed from KRPC (KUDU-1738). * TLS and Kerberos are still not supported by KRPC in this patch. Change-Id: I95229290566a8ccffd80ed2d74c1c57cf1479238 --- M .clang-format M CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/benchmarks/bloom-filter-benchmark.cc M be/src/benchmarks/expr-benchmark.cc M be/src/benchmarks/row-batch-serialize-benchmark.cc M be/src/common/init.cc M be/src/common/status.cc M be/src/common/status.h M be/src/rpc/CMakeLists.txt M be/src/rpc/common.proto M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc.h M be/src/rpc/thrift-server-test.cc D be/src/runtime/backend-client.h M be/src/runtime/client-cache-types.h M be/src/runtime/client-cache.cc M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/data-stream-mgr.cc M be/src/runtime/data-stream-mgr.h M be/src/runtime/data-stream-recvr.cc M be/src/runtime/data-stream-recvr.h M be/src/runtime/data-stream-sender.cc M be/src/runtime/data-stream-sender.h M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/plan-fragment-executor.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/scheduling/request-pool-service.h M be/src/scheduling/scheduler-test-util.h M be/src/service/CMakeLists.txt A be/src/service/data_stream_service.proto A be/src/service/exec_control_service.proto M be/src/service/fe-support.cc M be/src/service/frontend.h M be/src/service/impala-internal-service.cc M be/src/service/impala-internal-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/fault-injection-util.h M be/src/testutil/in-process-servers.cc M be/src/testutil/in-process-servers.h M be/src/util/bloom-filter-test.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/hdfs-util-test.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/CMakeLists.txt M common/thrift/ImpalaInternalService.thrift M common/thrift/Results.thrift M common/thrift/generate_error_codes.py M tests/custom_cluster/test_breakpad.py M tests/custom_cluster/test_exchange_delays.py M tests/custom_cluster/test_rpc_timeout.py 67 files changed, 2,353 insertions(+), 1,646 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/5888/4 -- To view, visit http://gerrit.cloudera.org:8080/5888 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I95229290566a8ccffd80ed2d74c1c57cf1479238 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Henry Robinson <[email protected]> Gerrit-Reviewer: Anonymous Coward #168 Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Marcel Kornacker <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]>
