Yuqi Du has posted comments on this change. ( http://gerrit.cloudera.org:8080/18454 )
Change subject: [master] KUDU-3061 support auto rebalance tablet leaders across TServers ...................................................................... Patch Set 21: (2 comments) Thanks your crs. http://gerrit.cloudera.org:8080/#/c/18454/19//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18454/19//COMMIT_MSG@10 PS19, Line 10: which lead to load skew on some nodes. > Do you have any data to show how much imbalance it is? I'm not convinced t I take some time to find an issue about leader imbalance at our internal JIRA before. The issue is about user query kudu failed because of timeout. Because the cluster only 3 tservers so the replica is balanced. We can see leader imbalance, All tables' leader distribution: kudu cluster ksck $master_list -ksck_format=plain_full | grep LEADER | awk '{print$2}' | sort | uniq -c 60 (YYY-prod1.ZZZ.XXX.cloud:7050): 4 (tYYY-prod2.ZZZ.XXX.cloud:7050): 113 (YYY-prod3.ZZZ.XXX.cloud:7050): node prod2 only 4 leaders and node prod3 has 113 leaders, the good distribution is (59, 59, 59) or so. The table event_wos_p2_correction's leader distribution: kudu cluster ksck $master_list -ksck_format=plain_full --tables=event_wos_p2_correction | grep LEADER | awk '{print$2}' | sort | uniq -c 6 (YYY-prod1.ZZZ.XXX.cloud:7050): 12 (YYY-prod3.ZZZ.XXX.cloud:7050): We can see node2 has no leader, the good distribution is (6, 6, 6) or so. http://gerrit.cloudera.org:8080/#/c/18454/19//COMMIT_MSG@21 PS19, Line 21: maybe cause imbalanced load > Yes, I understand it's not easy to setup a proper measurement experiment an OK, I have found a history issue and reply an above cr, but cpu usage and memory cann't find. After a few days, I can make a workload to test some data about cpu and memory usage. -- To view, visit http://gerrit.cloudera.org:8080/18454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibfb60d8759a93b6a19238637c27df4f6b1cac918 Gerrit-Change-Number: 18454 Gerrit-PatchSet: 21 Gerrit-Owner: Yuqi Du <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Yifan Zhang <[email protected]> Gerrit-Reviewer: Yingchun Lai <[email protected]> Gerrit-Reviewer: Yuqi Du <[email protected]> Gerrit-Comment-Date: Tue, 26 Jul 2022 04:25:05 +0000 Gerrit-HasComments: Yes
