[jira] [Commented] (HBASE-3724) Load balancer improvements
[ https://issues.apache.org/jira/browse/HBASE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187954#comment-13187954 ] Zhihong Yu commented on HBASE-3724: --- HBASE-5219 tries to address Jinchao's comment above. Load balancer improvements -- Key: HBASE-3724 URL: https://issues.apache.org/jira/browse/HBASE-3724 Project: HBase Issue Type: Umbrella Reporter: stack Umbrella issue under which we hang all regions related to balancer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3724) Load balancer improvements
[ https://issues.apache.org/jira/browse/HBASE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186959#comment-13186959 ] gaojinchao commented on HBASE-3724: --- I found the balance ago in branch92 is invalid for our scenario. So I use this issue to hang all issues related to balance. If someone want to see it, it will be easy. Load balancer improvements -- Key: HBASE-3724 URL: https://issues.apache.org/jira/browse/HBASE-3724 Project: HBase Issue Type: Umbrella Reporter: stack Umbrella issue under which we hang all regions related to balancer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3724) Load balancer improvements
[ https://issues.apache.org/jira/browse/HBASE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186963#comment-13186963 ] Zhihong Yu commented on HBASE-3724: --- @Jinchao: Can you describe your scenario ? Then we will see which task can best accommodate your requirement. Load balancer improvements -- Key: HBASE-3724 URL: https://issues.apache.org/jira/browse/HBASE-3724 Project: HBase Issue Type: Umbrella Reporter: stack Umbrella issue under which we hang all regions related to balancer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3724) Load balancer improvements
[ https://issues.apache.org/jira/browse/HBASE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187441#comment-13187441 ] gaojinchao commented on HBASE-3724: --- @zhihong In my scenario: 1. One region server has more than 1,000 regions.(hdf hard disk capacity(12T) / 3 (replication) / region size = 2G). 2. One moment, Dozens of Regions per region server are working for put operation. I went through the balance code , I found current balnace algorithm is invalid for our scenarios. below scenarios: 1)When adds one new machine to our cluster, Maybe all of hot regions(is working) will move to this one. 2)When one RS restarts, Maybe all of hot regions(is working) will move to this machine. Load balancer improvements -- Key: HBASE-3724 URL: https://issues.apache.org/jira/browse/HBASE-3724 Project: HBase Issue Type: Umbrella Reporter: stack Umbrella issue under which we hang all regions related to balancer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3724) Load balancer improvements
[ https://issues.apache.org/jira/browse/HBASE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187444#comment-13187444 ] Zhihong Yu commented on HBASE-3724: --- @Jinchao: Are all regions created at time of table creation ? If not, the balancer code in 0.92 is able to move both hot and cold regions onto the new server. Load balancer improvements -- Key: HBASE-3724 URL: https://issues.apache.org/jira/browse/HBASE-3724 Project: HBase Issue Type: Umbrella Reporter: stack Umbrella issue under which we hang all regions related to balancer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3724) Load balancer improvements
[ https://issues.apache.org/jira/browse/HBASE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187456#comment-13187456 ] gaojinchao commented on HBASE-3724: --- for example: we have 10 nodes cluster. 1000 regions per node and 50 regions are hot. per the current algorithm. If we add a new machine to this cluster. all the 50 hot regions will be moved to the new machine? Load balancer improvements -- Key: HBASE-3724 URL: https://issues.apache.org/jira/browse/HBASE-3724 Project: HBase Issue Type: Umbrella Reporter: stack Umbrella issue under which we hang all regions related to balancer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3724) Load balancer improvements
[ https://issues.apache.org/jira/browse/HBASE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187458#comment-13187458 ] Zhihong Yu commented on HBASE-3724: --- If the 50 hot regions are created due to region splits, load balancer is able to distinguish them from cold regions and pick both hot and cold regions for the new server. If the hot regions were created due to pre-split, I have code in internal repository to use region load information in achieving balance. This code is to be open-sourced. Load balancer improvements -- Key: HBASE-3724 URL: https://issues.apache.org/jira/browse/HBASE-3724 Project: HBase Issue Type: Umbrella Reporter: stack Umbrella issue under which we hang all regions related to balancer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3724) Load balancer improvements
[ https://issues.apache.org/jira/browse/HBASE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187464#comment-13187464 ] gaojinchao commented on HBASE-3724: --- Thanks, I am looking forward to seeing this part of the code. Sorry,I didn't express clearly. In my scenario: 1.There is 10 nodes cluster. 1000 regions per node. 2.We add a new machine to the cluster. 3.Balance needs move 1000* 10/11= 909 regions to the new machine. each region server will move both 45 hot regions and 45 cold regions to the new one. in this case, all hot regions will move to this new one? Load balancer improvements -- Key: HBASE-3724 URL: https://issues.apache.org/jira/browse/HBASE-3724 Project: HBase Issue Type: Umbrella Reporter: stack Umbrella issue under which we hang all regions related to balancer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3724) Load balancer improvements
[ https://issues.apache.org/jira/browse/HBASE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187465#comment-13187465 ] Zhihong Yu commented on HBASE-3724: --- If I remember correctly, this scenario might be due to your limiting the total number of active regions on each region server. Please consider raising the limit because 0.92 has reduced footprint of each region in Master's heap. If you cannot increase the limit to close to 1000 regions, please consider offlining inactive regions during each day. Load balancer improvements -- Key: HBASE-3724 URL: https://issues.apache.org/jira/browse/HBASE-3724 Project: HBase Issue Type: Umbrella Reporter: stack Umbrella issue under which we hang all regions related to balancer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3724) Load balancer improvements
[ https://issues.apache.org/jira/browse/HBASE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187488#comment-13187488 ] gaojinchao commented on HBASE-3724: --- Yes, you are right. we can't increase the limit to close to 1000 regions. because another reason is when hot regions are too many, It will produce many small Hfile. I have a solution to deal with this case. I will delevop a new balance algorithm for our scenario. Load balancer improvements -- Key: HBASE-3724 URL: https://issues.apache.org/jira/browse/HBASE-3724 Project: HBase Issue Type: Umbrella Reporter: stack Umbrella issue under which we hang all regions related to balancer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
