Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/9551 )
Change subject: KUDU-2289: Tablet deletion should be throttled ...................................................................... Patch Set 3: (1 comment) > > Do you have any advice on how to test this? I'm not setting a > queue length, the idea being that DeleteTablets can wait in the > queue instead of making the master constantly retry them, so the > DeleteTablet RPCs will no be throttled. I can reproduce symptoms of > the original problem in CreateTableStressTest.CreateAndDeleteBigTable > by turning up the number of tablets in the big table-- there'll be > service-queue-full rejections of CreateTablet and DeleteTablet > calls in the logs-- and those go away for DeleteTablets when this > change is applied. DeleteTablet is an admin service RPC, so I > can't, for example, Ping the service to see if the queue is full. > > Hmm, good question. You could install metrics on the new thread > pool (see ThreadPoolMetrics) and look at the amount of time each > task spends queued; before your change it'd be small (just waiting > for a thread to start), and after it'd be large (need to wait for > prior delete tablets to finish). But that's a little bit brittle. Maybe, you could set --num_tablets_to_delete_simultaneously=1, run with a single tablet server, inject a delay into TabletServiceAdminImpl::DeleteTable() or around, create a table with multiple tablets, and then remove it, capturing the readings of the METRIC_handler_latency_kudu_tserver_TabletServerAdminService_DeleteTablet (e.g., see the DeleteTableITest.TestUnknownTabletsAreNotDeleted scenario). The idea is to make sure the metrics readings go up until reaching the total number of tablets, and that happens not faster than it should. The same might be done for --num_tablets_to_delete_simultaneously=2, etc. http://gerrit.cloudera.org:8080/#/c/9551/3/src/kudu/tserver/tablet_service.cc File src/kudu/tserver/tablet_service.cc: http://gerrit.cloudera.org:8080/#/c/9551/3/src/kudu/tserver/tablet_service.cc@807 PS3, Line 807: context, req, resp > Thanks Adar. Here's a relevant snippet from the comment on RpcContext::Resp Thank you for the explanation. -- To view, visit http://gerrit.cloudera.org:8080/9551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3819bf8a3acf8ea03a76cc6cacd92d85bb114998 Gerrit-Change-Number: 9551 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley <wdberke...@gmail.com> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Will Berkeley <wdberke...@gmail.com> Gerrit-Comment-Date: Tue, 20 Mar 2018 18:41:36 +0000 Gerrit-HasComments: Yes
