Fang-Yu Rao has uploaded a new patch set (#19). ( )

Change subject: IMPALA-7984: Port runtime filter from Thrift RPC to KRPC

IMPALA-7984: Port runtime filter from Thrift RPC to KRPC

Previously the aggregation and propagation of a runtime filter in Impala is
implemented using Thrift RPC, which suffers from a disadvantage that the number
of connections in a cluster grows with both the number of queries and cluster
size. This patch ports the functions that implement the aggregation and
propagation of a runtime filter, i.e., UpdateFilter() and PublishFilter(),
respctively, to KRPC, which requires only one connection per direction between
every pair of hosts, thus reducing the number of connections in a cluster.

In addition, this patch also incorporates KRPC sidecar when the runtime filter
is a Bloom filter. KRPC sidecar eliminates the need for an extra copy of the
Bloom filter contents when a Bloom filter is serialized to be transmitted and
hence reduces the serialization overhead. Due to the incorporation of KRPC
sidecar, a SpinLock is also added to prevent a BloomFilter from being
deallocated before its associated KRPC call finishes.

Two related BE tests and are
also modified accordingly because of the changes to the signatures of some
functions in BloomFilter.

This patch has passed the exhaustive tests.

Change-Id: I6b394796d250286510e157ae326882bfc01d387a
M be/src/benchmarks/
M be/src/rpc/
M be/src/runtime/backend-client.h
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator-filter-state.h
M be/src/runtime/
M be/src/runtime/coordinator.h
M be/src/runtime/
M be/src/runtime/decimal-value.h
M be/src/runtime/decimal-value.inline.h
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/fragment-instance-state.h
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/query-state.h
M be/src/runtime/
M be/src/runtime/runtime-filter-bank.h
M be/src/runtime/runtime-filter.h
M be/src/runtime/timestamp-value.h
M be/src/scheduling/request-pool-service.h
M be/src/service/
M be/src/service/client-request-state.h
M be/src/service/
M be/src/service/data-stream-service.h
M be/src/service/frontend.h
M be/src/service/
M be/src/service/impala-internal-service.h
M be/src/service/
M be/src/service/impala-server.h
M be/src/util/
M be/src/util/
M be/src/util/bloom-filter.h
M be/src/util/
M be/src/util/
M be/src/util/min-max-filter.h
M common/protobuf/common.proto
M common/protobuf/data_stream_service.proto
M common/thrift/ImpalaInternalService.thrift
41 files changed, 1,104 insertions(+), 736 deletions(-)

  git pull ssh:// refs/changes/82/13882/19
To view, visit
To unsubscribe, visit

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b394796d250286510e157ae326882bfc01d387a
Gerrit-Change-Number: 13882
Gerrit-PatchSet: 19
Gerrit-Owner: Fang-Yu Rao <>
Gerrit-Reviewer: Fang-Yu Rao <>
Gerrit-Reviewer: Impala Public Jenkins <>
Gerrit-Reviewer: Michael Ho <>
Gerrit-Reviewer: Thomas Tauber-Marshall <>

Reply via email to