Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13426 )
Change subject: KUDU-2797 p2: the master aggregates tablet statistics ...................................................................... Patch Set 31: (3 comments) > Patch Set 31: > > > I think this looks fine; I'd like to try it out on a real cluster > > to get a feel for the scalability of this thing is. > > > > Have you done so yourself too? > > > > It's noteworthy that active tablet servers will now heartbeat much > > more frequently (compactions and writes) and send out much more > > data. > > Yes, I ran this feature on our test cluster(3 nodes) with one writer(a > separate server) and the target table has 365 tablets(hash, pre-replication). > It seems that there was no significant increase in network traffic. But, i > found a problem about the initial value for the "disk size". It's 365 * > 8MB/tablet = 2.8GB even though the table is empty. I ran this on a 12 node cluster and a RF=3 30-tablet table, and found it to be pretty quick with respect to updating values on time. I toyed with hitting the tables with `leader_step_down` commands and saw that the sizes jumped a little bit, but not much. I also tested on a larger, non-replicated table, and confirmed our suspicions about jumpier behavior: I ran loadgen with 450 tablets (RF=1) and saw that for a while, the table size was pegged at around 3.6 GiB (450 * 8MiB). This matches what Lifu saw, and digging into a tablet server a bit, this 8MiB base is coming from the tablet's wal-0000001 file's initial size of 8MiB. The size remained like this for a while, until the MRS flushed, after which the table size jumped to 6.06GiB. I tried restarting my cluster, and saw that, as expected, the count/size started at 0, and slowly climbed as tablet reports started coming in. I ran `top` on the master to look at how this might affect CPU usage by the masters. These masters were colocated with the tablet server, so not-surprisingly, a majority of the CPU usage was coming from the kudu-tserver, and not much from the kudu-master. I didn't test on a huge cluster with many tablet servers (--> many reports). I also didn't test things like failures to report. That said, the feature itself feels pretty nice in the hand. There are probably a bunch of ways we can improve upon this, but this patch alone is pretty neat! http://gerrit.cloudera.org:8080/#/c/13426/31/src/kudu/master/catalog_manager.cc File src/kudu/master/catalog_manager.cc: http://gerrit.cloudera.org:8080/#/c/13426/31/src/kudu/master/catalog_manager.cc@275 PS31, Line 275: table_stats I'd like to decouple stats from metrics. Could we just call this "table"? http://gerrit.cloudera.org:8080/#/c/13426/31/www/table.mustache File www/table.mustache: http://gerrit.cloudera.org:8080/#/c/13426/31/www/table.mustache@35 PS31, Line 35: Aggregated Live Row Count nit: maybe remove "Aggregated". I think the user expectation is already that it is a count for the entire table. http://gerrit.cloudera.org:8080/#/c/13426/31/www/table.mustache@36 PS31, Line 36: On-disk Size nit: Can you change this to "On-Disk Size (pre-replication)" or "On-Disk Size (leaders only)" or something like that? "Aggregated" makes it seem like the total size aggregated across all replicas. -- To view, visit http://gerrit.cloudera.org:8080/13426 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74406ab7cca7c22fda455c328b8ee9989a6b2d99 Gerrit-Change-Number: 13426 Gerrit-PatchSet: 31 Gerrit-Owner: helifu <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy <[email protected]> Gerrit-Reviewer: helifu <[email protected]> Gerrit-Comment-Date: Sat, 10 Aug 2019 02:39:33 +0000 Gerrit-HasComments: Yes
