Zoltan Martonka has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22867
Change subject: [WIP] Add option to send no-op heartbeat operations batched ...................................................................... [WIP] Add option to send no-op heartbeat operations batched Due to the periodically sent heartbeat messages, a Kudu cluster having thousands of tablets still uses significant CPU and networking even without any user activity. When we send multiple messages to the same host within a short time-frame, they can be batched to reduce CPU impact. This would result in fewer RPC calls and some fields can also be shared between the no-op messages. Measurement: I launched an AWS cluster with a 1 master, 4 tserver configuration (t3.xlarge). Then put 2000 tablets (RF=3) on them. cpu_stime seems to be decreased by 10-15% (the same number of no-op messages arrive). Here is one result over 500 sec for the flag being turned on or off (change in metrics over 500 sec): Metric: no_op_heartbeat_count, old: 920739, new: 919910, inc: -0.0900% Metric: heartbeat_batch_count, old: 0, new: 30664, inc: n/a% Metric: cpu_stime, old: 81208, new: 64079, inc: -21.093% Metric: cpu_utime, old: 176558, new: 170430, inc: -3.471% WIP cause: Concurrent write and reads need some performance measurement I am not sure if the heartbeat_batch_count and no_op_heartbeat_count metrics should be added to the final code. Change-Id: Ie92ba4de5eae00d56cd513cb644dce8fb6e14538 --- M src/kudu/client/client-test.cc M src/kudu/consensus/CMakeLists.txt M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_peers-test.cc M src/kudu/consensus/consensus_peers.cc M src/kudu/consensus/consensus_peers.h A src/kudu/consensus/multi_raft_batcher.cc A src/kudu/consensus/multi_raft_batcher.h M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/peer_manager.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h M src/kudu/consensus/raft_consensus_quorum-test.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/sys_catalog.h M src/kudu/tablet/tablet_replica-test-base.cc M src/kudu/tablet/tablet_replica.cc M src/kudu/tablet/tablet_replica.h M src/kudu/tserver/tablet_copy_source_session-test.cc M src/kudu/tserver/tablet_service.cc M src/kudu/tserver/tablet_service.h M src/kudu/tserver/ts_tablet_manager.cc M src/kudu/tserver/ts_tablet_manager.h 23 files changed, 765 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/67/22867/1 -- To view, visit http://gerrit.cloudera.org:8080/22867 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie92ba4de5eae00d56cd513cb644dce8fb6e14538 Gerrit-Change-Number: 22867 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Martonka <[email protected]>
