Henry Robinson has uploaded a new change for review. http://gerrit.cloudera.org:8080/5888
Change subject: IMPALA-4856: Port ImpalaInternalService to KRPC ...................................................................... IMPALA-4856: Port ImpalaInternalService to KRPC This patch ports the ImpalaInternalService to KRPC. * ImpalaInternalService is split into two KRPC services. The first, ImpalaInternalService, deals with control messages for plan fragment instance execution, cancellation and reporting. The second, DataStreamService, handles large-payload RPCs for transmitting runtime filters and row batches between hosts. The separation allows us to dedicate resources to each service, rather than have them compete for the same thread pool and queue space. * In the DataStreamService, all RPCs use 'native' protobuf. ImpalaInternalService RPCs remain wrappers of Thrift data structures. * This patch adds support for asynchronous RPCs to the RpcMgr and Rpc classes. Previously, Impala used fixed size thread pools + synchronous RPCs to achieve some parallelism for 'broadcast' RPCs like filter propagation, or a dedicated per-sender+receiver pair thread on the sender side in the DataStreamSender case. In this patch, the PublishFilter(), CancelPlanFragment() and TransmitData() RPCs are all sent asynchronously using KRPC's thread pools. * As a result, DataStreamSender no longer creates a thread-per-connection on the sender side. In this patch, the receiver-side still blocks if the receiver is unable to process a new row batch. A follow-on patch will change that to send notifications to the sender asynchronously, without blocking in the receiver threads. * A large portion of this patch is the replacement of TRowBatch with its Protobuf equivalent, RowBatchPb. The replacement is a literal port of the data structure, and row-batch-test, row-batch-list-test and row-batch-serialize-benchmark continue to execute without logic changes. * This patch also begins a clean-up of how ImpalaServer instances are created (by removing CreateImpalaServer), and clarifying the relationship between ExecEnv and ImpalaServer. ImpalaServer now follows the standard construct->Init()->Start()->Join() lifecycle that we use for other services. TESTING ------- * New tests added to rpc-mgr-test. TO DO ----- * Re-enable throughput and latency measurements per data-stream sender when that information is exposed from KRPC (KUDU-1738). Change-Id: I95229290566a8ccffd80ed2d74c1c57cf1479238 --- M .clang-format M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/benchmarks/bloom-filter-benchmark.cc M be/src/benchmarks/expr-benchmark.cc M be/src/benchmarks/row-batch-serialize-benchmark.cc M be/src/common/status.cc M be/src/common/status.h M be/src/rpc/CMakeLists.txt M be/src/rpc/common.proto M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc.h M be/src/rpc/thrift-server-test.cc D be/src/runtime/backend-client.h M be/src/runtime/client-cache-types.h M be/src/runtime/client-cache.cc M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/data-stream-mgr.cc M be/src/runtime/data-stream-mgr.h M be/src/runtime/data-stream-recvr.cc M be/src/runtime/data-stream-recvr.h M be/src/runtime/data-stream-sender.cc M be/src/runtime/data-stream-sender.h M be/src/runtime/data-stream-test.cc M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/plan-fragment-executor.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/scheduling/request-pool-service.h M be/src/scheduling/simple-scheduler-test-util.h M be/src/service/CMakeLists.txt M be/src/service/fe-support.cc M be/src/service/frontend.h M be/src/service/impala-internal-service.cc M be/src/service/impala-internal-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h A be/src/service/impala_internal_service.proto M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M be/src/testutil/in-process-servers.h M be/src/util/bloom-filter-test.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/hdfs-util-test.cc M common/thrift/CMakeLists.txt M common/thrift/ImpalaInternalService.thrift M common/thrift/Results.thrift 59 files changed, 1,400 insertions(+), 1,264 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/5888/1 -- To view, visit http://gerrit.cloudera.org:8080/5888 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I95229290566a8ccffd80ed2d74c1c57cf1479238 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Henry Robinson <he...@cloudera.com>
