Sailesh Mukil has uploaded a new patch set (#3).
Change subject: IMPALA-4456: Change query_exec_state_lock_ to a reader-writer
lock
......................................................................
IMPALA-4456: Change query_exec_state_lock_ to a reader-writer lock
This patch includes a ported reader-writer lock implementation from
Google utils and is used to replace the mutex query_exec_state_lock_
with a RW lock.
The main offender, ReportExecStatus() contends heavily for this lock
on the coordinator for each update that a fragment instance sends as
a RPC. This causes the creation of a larger number of new client
connections on that node as a lot of connections are blocked on the
ReportExecStatus() RPC.
On running a highly concurrent workload, the number of clients
reduced from ~1500 to ~1100 clients per node.
Note that this may not seem like a lot just as in terms of clients,
however, the number of clients created as a result of
ReportExecStatus() has come down drastically. Due to this other long
running RPCs like TransmitData() have ended up creating more clients.
This also reduces lock contention which results in client calls to
complete faster.
Please refer the JIRA for more information.
Future work to build on this patch to help with scalability:
-IMPALA-4459: Consider making ReportExecStatus() RPC execute
asynchronously
-IMPALA-3977: TransmitData() should not block
Change-Id: I790a95e7179f07aa7ba188d5422c5e054353ba0b
---
M be/src/gutil/CMakeLists.txt
A be/src/gutil/read_write_lock.cc
A be/src/gutil/read_write_lock.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A be/src/util/rw_lock.h
9 files changed, 178 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/5021/3
--
To view, visit http://gerrit.cloudera.org:8080/5021
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I790a95e7179f07aa7ba188d5422c5e054353ba0b
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <[email protected]>
Gerrit-Reviewer: Henry Robinson <[email protected]>
