Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9123 )
Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates ...................................................................... IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates This adds a set of "prioritized" statestore topics that are small but are important to deliver in a timely manner. These are delivered more frequently by a separate thread pool to reduce the window for stale admission control and scheduling information. The contract between statestore and subscriber is changed so that the statestore can send concurrent Update() RPCs for disjoint sets of topics. This required changes to the subscriber implementation, which assumed that only one Update RPC would arrive at a time. It also changes the locking in the statestore so that the prioritized update threads don't get stuck behind the catalog threads holding 'topic_lock_'. Specifically, it uses a reader-writer lock to protect modification of the set of topics and a reader-writer lock per topic to allow the topic data to be read by multiple threads concurrently. Added metrics to monitor the per-topic update interval. Testing: Ran core tests. Inspected metrics on Impala daemons, saw that membership and request queue processing times had more samples recorded than the catalog topic, reflecting the increased frequency. Ran under thread sanitizer, made sure no data races were reported in Statestore or StatestoreSubscriber. Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210 Reviewed-on: http://gerrit.cloudera.org:8080/9123 Reviewed-by: Tim Armstrong <tarmstr...@cloudera.com> Tested-by: Impala Public Jenkins --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/impala-server.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M common/thrift/metrics.json M tests/custom_cluster/test_admission_controller.py M tests/statestore/test_statestore.py M www/statestore_subscribers.tmpl M www/statestore_topics.tmpl 15 files changed, 845 insertions(+), 485 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210 Gerrit-Change-Number: 9123 Gerrit-PatchSet: 17 Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>