helifu has posted comments on this change. ( http://gerrit.cloudera.org:8080/13426 )
Change subject: KUDU-2797: the master aggregates tablet metrics ...................................................................... Patch Set 5: > (5 comments) > > Overall this looks great. Thanks for implementing this, Lifu! > > A high-level concern I have is that while the master aggregates the > count and size statistics for all replicas of each tablet, it only > reports on a random replica (the first in the hash table). One > downside to that approach is that the counts may be unstable if the > consensus group membership is unstable. Since counts, and all > logical stats (admittedly, not physical stats like on-disk size) > can only change when there are writes through the leader, I think > it would improve stability of the stats to prefer using the stats > reported by the most recent leader of each tablet. > > The downside to *only* accepting leader stats is that if some > tablets are only available for read (due to lack of quorum / server > downtime), and the master is restarted, then it would not be able > to use any stats for that tablet. So then a hybrid approach would > be needed, maybe something like: > > When reporting table stats (i.e., through the web UI, or eventually > to Impala), for each tablet: > - If a leader has ever reported into this master, use the stats for > the most recent leader that has reported (highest-term leader) > - If a leader has never reported, use the stats from the first > replica (same approach used now, which is basically a consistent > random selection) > > I think this would give us a reasonable level of consistency, > without which I worry we might run into difficult-to-debug behavior > if stats from potentially extremely lagging replicas might be used > for join strategy decisions. It should also be simple to implement > using the ConsensusStatePB reported by the tablet. > > What do you think? 1) Every tablet should report to master whatever it's leader or follower, since it can work with 'enum ReplicaSelection'(in client.h/*.java) in query engine; 2) We just expose the leader's statistics for display on web; 3) If the consensus group membership is unstable: a) there is something wrong with page presentation, maybe we can pick one at random, b) It should not affect the execution of the query engine when it's not LEADER_ONLY; -- To view, visit http://gerrit.cloudera.org:8080/13426 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74406ab7cca7c22fda455c328b8ee9989a6b2d99 Gerrit-Change-Number: 13426 Gerrit-PatchSet: 5 Gerrit-Owner: helifu <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy <[email protected]> Gerrit-Reviewer: helifu <[email protected]> Gerrit-Comment-Date: Tue, 11 Jun 2019 14:00:01 +0000 Gerrit-HasComments: No
