Another question, would this help when we split or only when we balance? Thanks, Abe
On Mon, Feb 16, 2015 at 11:24 AM, Abe Weinograd <[email protected]> wrote: > balancer said "true" and it is not disabled. Thanks again for your help. > > Abe > > On Mon, Feb 16, 2015 at 11:23 AM, Ted Yu <[email protected]> wrote: > >> What was the output from 'balancer' command ? >> >> Was is possible that balancer was disabled ? >> >> Cheers >> >> On Mon, Feb 16, 2015 at 8:04 AM, Abe Weinograd <[email protected]> wrote: >> >> > Ok. I forced the balancer run and am not seeing anything after a few >> > minutes. Master logs isn't showing anything. Should I look at the RS >> ones >> > instead? >> > >> > On Mon, Feb 16, 2015 at 11:03 AM, Ted Yu <[email protected]> wrote: >> > >> > > You should see effect in the next balancer run. >> > > >> > > Cheers >> > > >> > > On Mon, Feb 16, 2015 at 7:52 AM, Abe Weinograd <[email protected]> >> wrote: >> > > >> > > > Excellent. If i trigger the balancer, should this start to help or >> only >> > > for >> > > > future region creation? >> > > > >> > > > Thanks, >> > > > Abe >> > > > >> > > > On Mon, Feb 16, 2015 at 9:35 AM, Ted Yu <[email protected]> >> wrote: >> > > > >> > > > > Yes. This setting should be modified on Master. >> > > > > >> > > > > Cheers >> > > > > >> > > > > On Mon, Feb 16, 2015 at 6:27 AM, Abe Weinograd <[email protected]> >> > wrote: >> > > > > >> > > > > > Thanks Ted. We are putting this in the hbase-site.xml for the >> > > Master? >> > > > > > >> > > > > > Abe >> > > > > > >> > > > > > On Fri, Feb 13, 2015 at 5:03 PM, Shahab Yunus < >> > > [email protected]> >> > > > > > wrote: >> > > > > > >> > > > > > > Thanks, we will try that and report back. >> > > > > > > >> > > > > > > Regards, >> > > > > > > Shahab >> > > > > > > >> > > > > > > On Fri, Feb 13, 2015 at 4:56 PM, Ted Yu <[email protected]> >> > > wrote: >> > > > > > > >> > > > > > > > You can make TableSkewCostFunction more prominent by >> increasing >> > > the >> > > > > > value >> > > > > > > > for config parameter: >> > > > > > > > >> > > > > > > > hbase.master.balancer.stochastic.tableSkewCost >> > > > > > > > >> > > > > > > > Its default is 35. >> > > > > > > > >> > > > > > > > See if raising to 100 or 200 helps. >> > > > > > > > >> > > > > > > > On Fri, Feb 13, 2015 at 1:09 PM, Shahab Yunus < >> > > > > [email protected]> >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > Yes, this sever hosts other regions from other tables as >> > well. >> > > > > > > > > >> > > > > > > > > Regards >> > > > > > > > > Shahab >> > > > > > > > > >> > > > > > > > > On Fri, Feb 13, 2015 at 1:45 PM, Ted Yu < >> [email protected] >> > > >> > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Interesting, server7.ec3.internal,60020,1423845018628 >> was >> > > > > > > consistently >> > > > > > > > > > chosen as destination for the table. >> > > > > > > > > > Did server7.ec3.internal,60020,1423845018628 host >> regions >> > > from >> > > > > > other >> > > > > > > > > table >> > > > > > > > > > ? >> > > > > > > > > > >> > > > > > > > > > Cheers >> > > > > > > > > > >> > > > > > > > > > On Fri, Feb 13, 2015 at 10:27 AM, Shahab Yunus < >> > > > > > > [email protected] >> > > > > > > > > >> > > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Table name is: >> > > > > > > > > > > MYTABLE_RECENT_4W_V2 >> > > > > > > > > > > >> > > > > > > > > > > Pastebin snippet 1: http://pastebin.com/dQzMhGyP >> > > > > > > > > > > Pastebin snippet 2: http://pastebin.com/Y7ZsNAgF >> > > > > > > > > > > >> > > > > > > > > > > This is the master log after invoking balancer command >> > from >> > > > > hbase >> > > > > > > > > shell. >> > > > > > > > > > > >> > > > > > > > > > > Regards, >> > > > > > > > > > > Shahab >> > > > > > > > > > > >> > > > > > > > > > > On Fri, Feb 13, 2015 at 12:00 PM, Ted Yu < >> > > > [email protected]> >> > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > bq. all the regions of this table were back on this >> > same >> > > > RS! >> > > > > > > > > > > > >> > > > > > > > > > > > Interesting. Please check master log around the time >> > this >> > > > RS >> > > > > > was >> > > > > > > > > > brought >> > > > > > > > > > > > online. You can pastebin the relevant snippet. >> > > > > > > > > > > > >> > > > > > > > > > > > Thanks >> > > > > > > > > > > > >> > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:55 AM, Shahab Yunus < >> > > > > > > > > [email protected]> >> > > > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > Hi Ted. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Yes, the cluster itself is balanced. On average >> 300 >> > > > regions >> > > > > > per >> > > > > > > > > node >> > > > > > > > > > on >> > > > > > > > > > > > 10 >> > > > > > > > > > > > > nodes. >> > > > > > > > > > > > > >> > > > > > > > > > > > > # of tables is 53 of varying sizes. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Balancer was invoked and it didn't do anything >> (i.e. >> > no >> > > > > > > movement >> > > > > > > > of >> > > > > > > > > > > > > regions) but we didn't check the master's logs. We >> > can >> > > do >> > > > > > that. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Interestingly, we restarted the RS which was >> holding >> > > all >> > > > > the >> > > > > > > > > regions >> > > > > > > > > > of >> > > > > > > > > > > > > this one table. The regions were nicely spread >> out to >> > > the >> > > > > > > > remaining >> > > > > > > > > > RS. >> > > > > > > > > > > > But >> > > > > > > > > > > > > when we brought back this RS, all the regions of >> this >> > > > table >> > > > > > > were >> > > > > > > > > back >> > > > > > > > > > > on >> > > > > > > > > > > > > this same RS! >> > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks. >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > Regards, >> > > > > > > > > > > > > Shahab >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Fri, Feb 13, 2015 at 11:46 AM, Ted Yu < >> > > > > > [email protected]> >> > > > > > > > > > wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > > > How many tables are there in your cluster ? >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Is the cluster balanced overall (in terms of >> number >> > > of >> > > > > > > regions >> > > > > > > > > per >> > > > > > > > > > > > > server) >> > > > > > > > > > > > > > but this table is not ? >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > What happens (check master log) when you issue >> > > > 'balancer' >> > > > > > > > command >> > > > > > > > > > > > through >> > > > > > > > > > > > > > shell ? >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Cheers >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:19 AM, Shahab Yunus < >> > > > > > > > > > > [email protected]> >> > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > CDH 5.3 >> > > > > > > > > > > > > > > HBase 98.6 >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > We are writing data to an HBase table through >> a >> > M/R >> > > > > job. >> > > > > > We >> > > > > > > > pre >> > > > > > > > > > > split >> > > > > > > > > > > > > the >> > > > > > > > > > > > > > > table before each job run. The problem is that >> > most >> > > > of >> > > > > > the >> > > > > > > > > > regions >> > > > > > > > > > > > end >> > > > > > > > > > > > > up >> > > > > > > > > > > > > > > on the same RS. This results in that one RS >> being >> > > > > > severely >> > > > > > > > > > > overloaded >> > > > > > > > > > > > > and >> > > > > > > > > > > > > > > subsequent M/R jobs failing trying to write to >> > the >> > > > > > regions >> > > > > > > on >> > > > > > > > > > that >> > > > > > > > > > > > RS. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > The balancer is on and the split policy is >> > default. >> > > > No >> > > > > > > > changes >> > > > > > > > > > > there. >> > > > > > > > > > > > > It >> > > > > > > > > > > > > > is >> > > > > > > > > > > > > > > a 10 node cluster. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > All other related properties are defaults too. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Any idea, how can we force balancing of the >> new >> > > > > regions? >> > > > > > Do >> > > > > > > > we >> > > > > > > > > > have >> > > > > > > > > > > > to >> > > > > > > > > > > > > > > consider compaction into the equation as well? >> > > > Thanks. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Regards, >> > > > > > > > > > > > > > > Shahab >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
