hbase-0.94.6.1 balancer issue
Hi, all I'm evaluating hbase-0.94.6.1 and i have 48 regions on 2 node cluster. I was restarting on of RSs and after that tried to balance cluster by running balancer from shell. After running command regions were not distributed to second RS and i found this line i master log: 2013-04-12 16:45:15,589 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=2 *regions=1 *average=0.5 mostloaded=1 leastloaded=0 This look like to me that wrong number of regions is reported by balancer and that cause of skipping load balancing . In hbase shell i see all 48 tables that i have and everything else looks fine. Did someone else see this type of behavior ? Did something changed around balancer in hbase-0.94.6.1 ? Regards Samir
Re: hbase-0.94.6.1 balancer issue
Hi Samir, Regions are balancer per table. So if you have 48 regions within the same table, it should be split about 24 on each server. But if you have 48 tables with 1 region each, the for each table, the balancer will see only 1 region and will display the message you saw. Have you looked at the UI? What do you have in it? Can you please confirm if yo uhave 48 tables or 1 table? Thanks, JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, all I'm evaluating hbase-0.94.6.1 and i have 48 regions on 2 node cluster. I was restarting on of RSs and after that tried to balance cluster by running balancer from shell. After running command regions were not distributed to second RS and i found this line i master log: 2013-04-12 16:45:15,589 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=2 *regions=1 *average=0.5 mostloaded=1 leastloaded=0 This look like to me that wrong number of regions is reported by balancer and that cause of skipping load balancing . In hbase shell i see all 48 tables that i have and everything else looks fine. Did someone else see this type of behavior ? Did something changed around balancer in hbase-0.94.6.1 ? Regards Samir
Re: hbase-0.94.6.1 balancer issue
Hi Samir, Since regions are balanced per table, as soon as you will have more than one region in your table, balancer will start to balance the regions over the servers. You can split some of those tables and will you start to see HBase balance them. This is normal behavior for 0.94. I don't know for versions before that. Also, are you sure you need 48 tables? And not less tables with more CFs? JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, JM I have 48 tables and as you said it is 1 region per table since i did not reach splitting limit yet. So this is normal behavior in 0.94.6.1 version ? And at what point balancer will start redistribute regions to second server ? Thanks Samir On Fri, Apr 12, 2013 at 6:06 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Regions are balancer per table. So if you have 48 regions within the same table, it should be split about 24 on each server. But if you have 48 tables with 1 region each, the for each table, the balancer will see only 1 region and will display the message you saw. Have you looked at the UI? What do you have in it? Can you please confirm if yo uhave 48 tables or 1 table? Thanks, JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, all I'm evaluating hbase-0.94.6.1 and i have 48 regions on 2 node cluster. I was restarting on of RSs and after that tried to balance cluster by running balancer from shell. After running command regions were not distributed to second RS and i found this line i master log: 2013-04-12 16:45:15,589 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=2 *regions=1 *average=0.5 mostloaded=1 leastloaded=0 This look like to me that wrong number of regions is reported by balancer and that cause of skipping load balancing . In hbase shell i see all 48 tables that i have and everything else looks fine. Did someone else see this type of behavior ? Did something changed around balancer in hbase-0.94.6.1 ? Regards Samir
Re: hbase-0.94.6.1 balancer issue
Thanks for explaining Jean-Marc, We are using 0.90.4 for very long time and balancing was based on total number of regions.That is why i was surprised with balancer log on 0.94. Well i'm more ops guy then dev i handle what other develop :) Regards On Fri, Apr 12, 2013 at 6:24 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Since regions are balanced per table, as soon as you will have more than one region in your table, balancer will start to balance the regions over the servers. You can split some of those tables and will you start to see HBase balance them. This is normal behavior for 0.94. I don't know for versions before that. Also, are you sure you need 48 tables? And not less tables with more CFs? JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, JM I have 48 tables and as you said it is 1 region per table since i did not reach splitting limit yet. So this is normal behavior in 0.94.6.1 version ? And at what point balancer will start redistribute regions to second server ? Thanks Samir On Fri, Apr 12, 2013 at 6:06 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Regions are balancer per table. So if you have 48 regions within the same table, it should be split about 24 on each server. But if you have 48 tables with 1 region each, the for each table, the balancer will see only 1 region and will display the message you saw. Have you looked at the UI? What do you have in it? Can you please confirm if yo uhave 48 tables or 1 table? Thanks, JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, all I'm evaluating hbase-0.94.6.1 and i have 48 regions on 2 node cluster. I was restarting on of RSs and after that tried to balance cluster by running balancer from shell. After running command regions were not distributed to second RS and i found this line i master log: 2013-04-12 16:45:15,589 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=2 *regions=1 *average=0.5 mostloaded=1 leastloaded=0 This look like to me that wrong number of regions is reported by balancer and that cause of skipping load balancing . In hbase shell i see all 48 tables that i have and everything else looks fine. Did someone else see this type of behavior ? Did something changed around balancer in hbase-0.94.6.1 ? Regards Samir
Re: hbase-0.94.6.1 balancer issue
Samir, When you say And at what point balancer will start redistribute regions to second server, do you mean that when you look at the master's web UI you see that one region server has 0 region? That would be a problem. Else, that line you posted in your original message should be repeated for each table, and globally the regions should all be correctly distributed... unless there's an edge case where when you have only tables with 1 region it puts them all on the same server :) Thx, J-D On Fri, Apr 12, 2013 at 12:37 PM, Samir Ahmic ahmic.sa...@gmail.com wrote: Thanks for explaining Jean-Marc, We are using 0.90.4 for very long time and balancing was based on total number of regions.That is why i was surprised with balancer log on 0.94. Well i'm more ops guy then dev i handle what other develop :) Regards On Fri, Apr 12, 2013 at 6:24 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Since regions are balanced per table, as soon as you will have more than one region in your table, balancer will start to balance the regions over the servers. You can split some of those tables and will you start to see HBase balance them. This is normal behavior for 0.94. I don't know for versions before that. Also, are you sure you need 48 tables? And not less tables with more CFs? JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, JM I have 48 tables and as you said it is 1 region per table since i did not reach splitting limit yet. So this is normal behavior in 0.94.6.1 version ? And at what point balancer will start redistribute regions to second server ? Thanks Samir On Fri, Apr 12, 2013 at 6:06 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Regions are balancer per table. So if you have 48 regions within the same table, it should be split about 24 on each server. But if you have 48 tables with 1 region each, the for each table, the balancer will see only 1 region and will display the message you saw. Have you looked at the UI? What do you have in it? Can you please confirm if yo uhave 48 tables or 1 table? Thanks, JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, all I'm evaluating hbase-0.94.6.1 and i have 48 regions on 2 node cluster. I was restarting on of RSs and after that tried to balance cluster by running balancer from shell. After running command regions were not distributed to second RS and i found this line i master log: 2013-04-12 16:45:15,589 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=2 *regions=1 *average=0.5 mostloaded=1 leastloaded=0 This look like to me that wrong number of regions is reported by balancer and that cause of skipping load balancing . In hbase shell i see all 48 tables that i have and everything else looks fine. Did someone else see this type of behavior ? Did something changed around balancer in hbase-0.94.6.1 ? Regards Samir
Re: hbase-0.94.6.1 balancer issue
I have just created 50 tables and they got distributed on different nodes (8) at the create time. I ran the balancer manually and they are still correctly distributed all over the cluster. But Samir tried with only 2 nodes. I don't know if this might change the results or not JM. 2013/4/12 Jean-Daniel Cryans jdcry...@apache.org Samir, When you say And at what point balancer will start redistribute regions to second server, do you mean that when you look at the master's web UI you see that one region server has 0 region? That would be a problem. Else, that line you posted in your original message should be repeated for each table, and globally the regions should all be correctly distributed... unless there's an edge case where when you have only tables with 1 region it puts them all on the same server :) Thx, J-D On Fri, Apr 12, 2013 at 12:37 PM, Samir Ahmic ahmic.sa...@gmail.com wrote: Thanks for explaining Jean-Marc, We are using 0.90.4 for very long time and balancing was based on total number of regions.That is why i was surprised with balancer log on 0.94. Well i'm more ops guy then dev i handle what other develop :) Regards On Fri, Apr 12, 2013 at 6:24 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Since regions are balanced per table, as soon as you will have more than one region in your table, balancer will start to balance the regions over the servers. You can split some of those tables and will you start to see HBase balance them. This is normal behavior for 0.94. I don't know for versions before that. Also, are you sure you need 48 tables? And not less tables with more CFs? JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, JM I have 48 tables and as you said it is 1 region per table since i did not reach splitting limit yet. So this is normal behavior in 0.94.6.1 version ? And at what point balancer will start redistribute regions to second server ? Thanks Samir On Fri, Apr 12, 2013 at 6:06 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Regions are balancer per table. So if you have 48 regions within the same table, it should be split about 24 on each server. But if you have 48 tables with 1 region each, the for each table, the balancer will see only 1 region and will display the message you saw. Have you looked at the UI? What do you have in it? Can you please confirm if yo uhave 48 tables or 1 table? Thanks, JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, all I'm evaluating hbase-0.94.6.1 and i have 48 regions on 2 node cluster. I was restarting on of RSs and after that tried to balance cluster by running balancer from shell. After running command regions were not distributed to second RS and i found this line i master log: 2013-04-12 16:45:15,589 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=2 *regions=1 *average=0.5 mostloaded=1 leastloaded=0 This look like to me that wrong number of regions is reported by balancer and that cause of skipping load balancing . In hbase shell i see all 48 tables that i have and everything else looks fine. Did someone else see this type of behavior ? Did something changed around balancer in hbase-0.94.6.1 ? Regards Samir
Re: hbase-0.94.6.1 balancer issue
Thanks for the replies, Jean-Marc. HBASE-7060 is related to the scenario Samir described: Region load balancing by table does not handle the case where a table's region count is lower than the number of the RS in the cluster It was fixed in 0.94.3 Cheers On Fri, Apr 12, 2013 at 10:30 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: I have just created 50 tables and they got distributed on different nodes (8) at the create time. I ran the balancer manually and they are still correctly distributed all over the cluster. But Samir tried with only 2 nodes. I don't know if this might change the results or not JM. 2013/4/12 Jean-Daniel Cryans jdcry...@apache.org Samir, When you say And at what point balancer will start redistribute regions to second server, do you mean that when you look at the master's web UI you see that one region server has 0 region? That would be a problem. Else, that line you posted in your original message should be repeated for each table, and globally the regions should all be correctly distributed... unless there's an edge case where when you have only tables with 1 region it puts them all on the same server :) Thx, J-D On Fri, Apr 12, 2013 at 12:37 PM, Samir Ahmic ahmic.sa...@gmail.com wrote: Thanks for explaining Jean-Marc, We are using 0.90.4 for very long time and balancing was based on total number of regions.That is why i was surprised with balancer log on 0.94. Well i'm more ops guy then dev i handle what other develop :) Regards On Fri, Apr 12, 2013 at 6:24 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Since regions are balanced per table, as soon as you will have more than one region in your table, balancer will start to balance the regions over the servers. You can split some of those tables and will you start to see HBase balance them. This is normal behavior for 0.94. I don't know for versions before that. Also, are you sure you need 48 tables? And not less tables with more CFs? JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, JM I have 48 tables and as you said it is 1 region per table since i did not reach splitting limit yet. So this is normal behavior in 0.94.6.1 version ? And at what point balancer will start redistribute regions to second server ? Thanks Samir On Fri, Apr 12, 2013 at 6:06 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Regions are balancer per table. So if you have 48 regions within the same table, it should be split about 24 on each server. But if you have 48 tables with 1 region each, the for each table, the balancer will see only 1 region and will display the message you saw. Have you looked at the UI? What do you have in it? Can you please confirm if yo uhave 48 tables or 1 table? Thanks, JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, all I'm evaluating hbase-0.94.6.1 and i have 48 regions on 2 node cluster. I was restarting on of RSs and after that tried to balance cluster by running balancer from shell. After running command regions were not distributed to second RS and i found this line i master log: 2013-04-12 16:45:15,589 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=2 *regions=1 *average=0.5 mostloaded=1 leastloaded=0 This look like to me that wrong number of regions is reported by balancer and that cause of skipping load balancing . In hbase shell i see all 48 tables that i have and everything else looks fine. Did someone else see this type of behavior ? Did something changed around balancer in hbase-0.94.6.1 ? Regards Samir
Re: hbase-0.94.6.1 balancer issue
Hi, J-D Well at this moment i have that edge case with only one region per table:). Like i said i was using 0.90 for long time and regions were distributed evenly on all RSs regardless on region per table ratio. Here is what confused me (like i said i have 2 nodes cluster distributed mode): start-hbase -- tables(regions) are distributed evenlyon two RSs (As expected) stop one RS --- all tables(regions) are moved to remaining RS (as expected) start RS that was down --- run balancer --- LOG:2013-04-12 19:47:20,725 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=2 regions=1 average=0.5 mostloaded=1 leastloaded=0 all tables(regions) stayed on one server (this is what i did not expect ?) :) Here is is part of status 'detailed' from shell ater i start RS that was down and run balancer: hbase(main):001:0 status 'detailed' version 0.94.6.1 0 regionsInTransition master coprocessors: [] 2 live servers 172.17.33.2:60020 1365787755294 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=38, maxHeapMB=3487 172.17.33.3:60020 1365777858778 requestsPerSecond=0, numberOfOnlineRegions=49, usedHeapMB=53, maxHeapMB=3487 So because i have 1 regions per table regions were not rebalances after start RS that was down? Thanks Samir On Fri, Apr 12, 2013 at 7:17 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote: Samir, When you say And at what point balancer will start redistribute regions to second server, do you mean that when you look at the master's web UI you see that one region server has 0 region? That would be a problem. Else, that line you posted in your original message should be repeated for each table, and globally the regions should all be correctly distributed... unless there's an edge case where when you have only tables with 1 region it puts them all on the same server :) Thx, J-D On Fri, Apr 12, 2013 at 12:37 PM, Samir Ahmic ahmic.sa...@gmail.com wrote: Thanks for explaining Jean-Marc, We are using 0.90.4 for very long time and balancing was based on total number of regions.That is why i was surprised with balancer log on 0.94. Well i'm more ops guy then dev i handle what other develop :) Regards On Fri, Apr 12, 2013 at 6:24 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Since regions are balanced per table, as soon as you will have more than one region in your table, balancer will start to balance the regions over the servers. You can split some of those tables and will you start to see HBase balance them. This is normal behavior for 0.94. I don't know for versions before that. Also, are you sure you need 48 tables? And not less tables with more CFs? JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, JM I have 48 tables and as you said it is 1 region per table since i did not reach splitting limit yet. So this is normal behavior in 0.94.6.1 version ? And at what point balancer will start redistribute regions to second server ? Thanks Samir On Fri, Apr 12, 2013 at 6:06 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Samir, Regions are balancer per table. So if you have 48 regions within the same table, it should be split about 24 on each server. But if you have 48 tables with 1 region each, the for each table, the balancer will see only 1 region and will display the message you saw. Have you looked at the UI? What do you have in it? Can you please confirm if yo uhave 48 tables or 1 table? Thanks, JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, all I'm evaluating hbase-0.94.6.1 and i have 48 regions on 2 node cluster. I was restarting on of RSs and after that tried to balance cluster by running balancer from shell. After running command regions were not distributed to second RS and i found this line i master log: 2013-04-12 16:45:15,589 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=2 *regions=1 *average=0.5 mostloaded=1 leastloaded=0 This look like to me that wrong number of regions is reported by balancer and that cause of skipping load balancing . In hbase shell i see all 48 tables that i have and everything else looks fine. Did someone else see this type of behavior ? Did something changed around balancer in hbase-0.94.6.1 ? Regards Samir
Re: hbase-0.94.6.1 balancer issue
HBASE-7060 explains my case, i'm using 0.94.6.1 and looks like issue is still present. Thanks for replaying guys Cheers:)
Re: hbase-0.94.6.1 balancer issue
bq. looks like issue is still present. I think the issue expressed in HBASE-7060 is slightly different: bq. For example, the cluster has 100 RS, the table has 50 regions sitting on one RS, Note: one table had 50 regions On Fri, Apr 12, 2013 at 11:17 AM, Samir Ahmic ahmic.sa...@gmail.com wrote: HBASE-7060 explains my case, i'm using 0.94.6.1 and looks like issue is still present. Thanks for replaying guys Cheers:)
