Fang-Yu Rao has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14975 )
Change subject: IMPALA-9154: Make runtime filter propagation asynchronous ...................................................................... IMPALA-9154: Make runtime filter propagation asynchronous This patch fixes a bug introduced by IMPALA-7984 that ports the functions implementing the aggregation and propagation of runtime filters from Thrift RPC to KRPC. Specifically, in IMPALA-7984, the propagation of an aggregated runtime filter was implemented using the synchronous KRPC. Hence, when there is a very limited number of KRPC threads for Impala's data stream service, e.g., 1, there will be a deadlock if the node running the Coordinator is trying to propagate the aggregated filter to the same node running the Coordinator since there is no available thread to receive the aggregated filter. This patch makes the propagation of an aggregated runtime filter asynchronous to address the issue described above. To prevent the memory consumed by the aggregated filter from being reclaimed when the aggregated filter is still referenced by some inflight KRPC's, we add an additional field in the class Coordinator::FilterState to keep track of the number of inflight KRPC's for the propagation of this aggregated filter to make sure that we will reclaim the memory only when all the associated KRPC's have completed. Moreover, when ReleaseExecResources() is invoked by the Coordinator to release all the resources associated with query execution, including the memory consumed by the aggregated runtime filters, we make sure the consumed memory by the aggregated filters is released only when the inflight KRPC's associated with each aggregated filter have finished. Testing: - Passed primitive_many_fragments.test with the database tpch30 in an Impala minicluster started with the parameter --impalad_args=--datastream_service_num_svc_threads=1. - Passed the exhaustive tests in the DEBUG build. Change-Id: Ifb6726d349be701f3a0602b2ad5a934082f188a0 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h 5 files changed, 150 insertions(+), 71 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/14975/2 -- To view, visit http://gerrit.cloudera.org:8080/14975 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifb6726d349be701f3a0602b2ad5a934082f188a0 Gerrit-Change-Number: 14975 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao <[email protected]> Gerrit-Reviewer: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Fang-Yu Rao <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Thomas Tauber-Marshall <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
