[Impala-ASF-CR] IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9123 )

Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog 
topic updates
..

IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

This adds a set of "prioritized" statestore topics that are small but
are important to deliver in a timely manner. These are delivered more
frequently by a separate thread pool to reduce the window for stale
admission control and scheduling information.

The contract between statestore and subscriber is changed so that the
statestore can send concurrent Update() RPCs for disjoint sets of
topics. This required changes to the subscriber implementation, which
assumed that only one Update RPC would arrive at a time.

It also changes the locking in the statestore so that the prioritized
update threads don't get stuck behind the catalog threads holding
'topic_lock_'. Specifically, it uses a reader-writer lock to protect
modification of the set of topics and a reader-writer lock per topic to
allow the topic data to be read by multiple threads concurrently.

Added metrics to monitor the per-topic update interval.

Testing:
Ran core tests.

Inspected metrics on Impala daemons, saw that membership and request
queue processing times had more samples recorded than the catalog
topic, reflecting the increased frequency.

Ran under thread sanitizer, made sure no data races were reported in
Statestore or StatestoreSubscriber.

Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Reviewed-on: http://gerrit.cloudera.org:8080/9123
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins
---
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/scheduler-test-util.cc
M be/src/scheduling/scheduler.cc
M be/src/scheduling/scheduler.h
M be/src/service/impala-server.cc
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M common/thrift/metrics.json
M tests/custom_cluster/test_admission_controller.py
M tests/statestore/test_statestore.py
M www/statestore_subscribers.tmpl
M www/statestore_topics.tmpl
15 files changed, 845 insertions(+), 485 deletions(-)

Approvals:
  Tim Armstrong: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Gerrit-Change-Number: 9123
Gerrit-PatchSet: 17
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9123 )

Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog 
topic updates
..


Patch Set 16: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Gerrit-Change-Number: 9123
Gerrit-PatchSet: 16
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 14 Feb 2018 22:44:39 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9123 )

Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog 
topic updates
..


Patch Set 16:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1937/


--
To view, visit http://gerrit.cloudera.org:8080/9123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Gerrit-Change-Number: 9123
Gerrit-PatchSet: 16
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 14 Feb 2018 19:15:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9123 )

Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog 
topic updates
..


Patch Set 16:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1936/


--
To view, visit http://gerrit.cloudera.org:8080/9123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Gerrit-Change-Number: 9123
Gerrit-PatchSet: 16
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 14 Feb 2018 17:57:49 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9123 )

Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog 
topic updates
..


Patch Set 16: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Gerrit-Change-Number: 9123
Gerrit-PatchSet: 16
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 14 Feb 2018 17:57:39 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

[Dan Hecht (Code Review)]

Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9123 )

Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog 
topic updates
..


Patch Set 15: Code-Review+2

Thanks for the larger scale testing.


--
To view, visit http://gerrit.cloudera.org:8080/9123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Gerrit-Change-Number: 9123
Gerrit-PatchSet: 15
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 14 Feb 2018 17:36:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9123 )

Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog 
topic updates
..


Patch Set 15:

I was able to borrow a 140 node cluster and ran a workload with a few streams 
of concurrent queries. I looked at the statestored topics page, impalad and 
statestored metrics, "top" and "perf top" to make sure that resource 
consumption was as expected. Things looks good. The statestore does remain 
active consuming a moderate amount of CPU just to poll the 140 subscribers for 
updates every 100ms, but that is expected. One thing that is interesting is 
that the request-queue topic updates very frequently when running without 
mem_limits, because it tracks the actual memory consumption in that case, which 
tends to fluctuate a lot. Detailed notes follow.

On an idle 140 node cluster, statestored consumes ~40% cpu and ~500kb/s 
network. Perf shows that time is mainly spent checking topic versions.


 9.19%  impalad   [.] 
impala::Statestore::GetMinSubscriberTopicVersion(std::string const&, 
std::string*)
   7.50%  [kernel]  [k] find_busiest_group
   4.47%  impalad   [.] 
_ZN5boost9unordered6detail12mix64_policyImE10apply_hashINS_4hashISsEESsEEmRKT_RKT0_.isra.271
   1.82%  [kernel]  [k] find_next_bit
   1.69%  impalad   [.] 
impala::Statestore::Subscriber::LastTopicVersionProcessed(std::string const&) 
const
   1.68%  [kernel]  [k] _spin_lock
   1.27%  libc-2.12.so  [.] __memcmp_sse4_1
   1.25%  [kernel]  [k] cpumask_next_and
   0.98%  [kernel]  [k] thread_return
   0.80%  [kernel]  [k] smaps_pte_entry
   0.74%  libjvm.so [.] 
GenericTaskQueueSet, 64ul>, (unsigned short
   0.70%  [kernel]  [k] schedule
   0.60%  libc-2.12.so  [.] memcpy
   0.54%  [kernel]  [k] _spin_lock_irqsave
   0.54%  [kernel]  [k] ixgbe_poll

If I ran a light workload of queries with no mem_limits, the request-queue 
topic updated frequently with the current memory consumption and statestored 
CPU consumption increased to 50-60% with a light workload of queries with no 
mem_limits.  If I set a default pool mem_limit the topic version only 
increments rarely and there is no noticeable increase in load.

On an idle cluster, prioritized updates were delivered in a timely manner and 
took minimal time to process. Snapshot of metrics from an Impalad while idle 
after running some queries:

statestore-subscriber.connected trueWhether the Impala Daemon 
considers itself connected to the StateStore.
statestore-subscriber.last-recovery-timeN/A The local time that the 
last statestore recovery happened.
statestore-subscriber.topic-update-interval-timeLast (of 67983): 
0.100672. Min: 0, max: 2.43858, avg: 0.144938  The time (sec) between 
Statestore subscriber topic updates.
statestore-subscriber.topic-update-duration Last (of 34783): 
0.000256518. Min: 0, max: 1.06605, avg: 0.00530736 The time (sec) taken to 
process Statestore subscriber topic updates.
statestore-subscriber.heartbeat-interval-time   Last (of 3344): 
1.1. Min: 0, max: 1.82638, avg: 1.00039 The time (sec) between 
Statestore heartbeats.
statestore-subscriber.topic-impala-request-queue.update-intervalLast 
(of 33200): 0.100672. Min: 0, max: 2.43848, avg: 0.100697  Interval between 
topic updates for Topic impala-request-queue
statestore-subscriber.topic-impala-membership.update-interval   Last 
(of 33200): 0.100671. Min: 0, max: 2.43858, avg: 0.100696  Interval between 
topic updates for Topic impala-membership
statestore-subscriber.topic-impala-request-queue.processing-time-s  Last 
(of 33200): 0.000254981. Min: 0, max: 0.0113791, avg: 0.000155621  Statestore 
Subscriber Topic impala-request-queue Processing Time
statestore-subscriber.topic-impala-membership.processing-time-s Last 
(of 33200): 0.000177222. Min: 0, max: 0.0113666, avg: 0.000120381  Statestore 
Subscriber Topic impala-membership Processing Time
statestore-subscriber.topic-catalog-update.update-interval  Last (of 1583): 
2.0009. Min: 1, max: 2.20628, avg: 2.00069  Interval between topic updates 
for Topic catalog-update
statestore-subscriber.topic-catalog-update.processing-time-sLast 
(of 1583): 0.136297. Min: 0, max: 1.06605, avg: 0.113322   Statestore 
Subscriber Topic catalog-update Processing Time

[Impala-ASF-CR] IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9123 )

Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog 
topic updates
..


Patch Set 15: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/9123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Gerrit-Change-Number: 9123
Gerrit-PatchSet: 15
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 08 Feb 2018 00:29:47 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9123 )

Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog 
topic updates
..


Patch Set 15:

I discovered that henry had filed a related JIRA a while ago, so I added that 
to the title.

I'm going to do some testing on a larger cluster.


--
To view, visit http://gerrit.cloudera.org:8080/9123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Gerrit-Change-Number: 9123
Gerrit-PatchSet: 15
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 07 Feb 2018 17:50:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

[Tim Armstrong (Code Review)]

Hello Tianyi Wang, Dimitris Tsirogiannis, Alex Behm, Bikramjeet Vig, Dan Hecht,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9123

to look at the new patch set (#15).

Change subject: IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog 
topic updates
..

IMPALA-4953,IMPALA-6437: separate AC/scheduler from catalog topic updates

This adds a set of "prioritized" statestore topics that are small but
are important to deliver in a timely manner. These are delivered more
frequently by a separate thread pool to reduce the window for stale
admission control and scheduling information.

The contract between statestore and subscriber is changed so that the
statestore can send concurrent Update() RPCs for disjoint sets of
topics. This required changes to the subscriber implementation, which
assumed that only one Update RPC would arrive at a time.

It also changes the locking in the statestore so that the prioritized
update threads don't get stuck behind the catalog threads holding
'topic_lock_'. Specifically, it uses a reader-writer lock to protect
modification of the set of topics and a reader-writer lock per topic to
allow the topic data to be read by multiple threads concurrently.

Added metrics to monitor the per-topic update interval.

Testing:
Ran core tests.

Inspected metrics on Impala daemons, saw that membership and request
queue processing times had more samples recorded than the catalog
topic, reflecting the increased frequency.

Ran under thread sanitizer, made sure no data races were reported in
Statestore or StatestoreSubscriber.

Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
---
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/scheduler-test-util.cc
M be/src/scheduling/scheduler.cc
M be/src/scheduling/scheduler.h
M be/src/service/impala-server.cc
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M common/thrift/metrics.json
M tests/custom_cluster/test_admission_controller.py
M tests/statestore/test_statestore.py
M www/statestore_subscribers.tmpl
M www/statestore_topics.tmpl
15 files changed, 845 insertions(+), 485 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/9123/15
--
To view, visit http://gerrit.cloudera.org:8080/9123
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210
Gerrit-Change-Number: 9123
Gerrit-PatchSet: 15
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Tim Armstrong