When a region splits, the two daughter regions stay on the same host as the parent region did.
It is through balancing that the daughter regions (may) be assigned to other server(s). Cheers On Wed, Feb 18, 2015 at 7:35 AM, Abe Weinograd <[email protected]> wrote: > Another question, would this help when we split or only when we balance? > > Thanks, > Abe > > On Mon, Feb 16, 2015 at 11:24 AM, Abe Weinograd <[email protected]> wrote: > > > balancer said "true" and it is not disabled. Thanks again for your help. > > > > Abe > > > > On Mon, Feb 16, 2015 at 11:23 AM, Ted Yu <[email protected]> wrote: > > > >> What was the output from 'balancer' command ? > >> > >> Was is possible that balancer was disabled ? > >> > >> Cheers > >> > >> On Mon, Feb 16, 2015 at 8:04 AM, Abe Weinograd <[email protected]> wrote: > >> > >> > Ok. I forced the balancer run and am not seeing anything after a few > >> > minutes. Master logs isn't showing anything. Should I look at the RS > >> ones > >> > instead? > >> > > >> > On Mon, Feb 16, 2015 at 11:03 AM, Ted Yu <[email protected]> wrote: > >> > > >> > > You should see effect in the next balancer run. > >> > > > >> > > Cheers > >> > > > >> > > On Mon, Feb 16, 2015 at 7:52 AM, Abe Weinograd <[email protected]> > >> wrote: > >> > > > >> > > > Excellent. If i trigger the balancer, should this start to help or > >> only > >> > > for > >> > > > future region creation? > >> > > > > >> > > > Thanks, > >> > > > Abe > >> > > > > >> > > > On Mon, Feb 16, 2015 at 9:35 AM, Ted Yu <[email protected]> > >> wrote: > >> > > > > >> > > > > Yes. This setting should be modified on Master. > >> > > > > > >> > > > > Cheers > >> > > > > > >> > > > > On Mon, Feb 16, 2015 at 6:27 AM, Abe Weinograd <[email protected]> > >> > wrote: > >> > > > > > >> > > > > > Thanks Ted. We are putting this in the hbase-site.xml for the > >> > > Master? > >> > > > > > > >> > > > > > Abe > >> > > > > > > >> > > > > > On Fri, Feb 13, 2015 at 5:03 PM, Shahab Yunus < > >> > > [email protected]> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Thanks, we will try that and report back. > >> > > > > > > > >> > > > > > > Regards, > >> > > > > > > Shahab > >> > > > > > > > >> > > > > > > On Fri, Feb 13, 2015 at 4:56 PM, Ted Yu < > [email protected]> > >> > > wrote: > >> > > > > > > > >> > > > > > > > You can make TableSkewCostFunction more prominent by > >> increasing > >> > > the > >> > > > > > value > >> > > > > > > > for config parameter: > >> > > > > > > > > >> > > > > > > > hbase.master.balancer.stochastic.tableSkewCost > >> > > > > > > > > >> > > > > > > > Its default is 35. > >> > > > > > > > > >> > > > > > > > See if raising to 100 or 200 helps. > >> > > > > > > > > >> > > > > > > > On Fri, Feb 13, 2015 at 1:09 PM, Shahab Yunus < > >> > > > > [email protected]> > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Yes, this sever hosts other regions from other tables as > >> > well. > >> > > > > > > > > > >> > > > > > > > > Regards > >> > > > > > > > > Shahab > >> > > > > > > > > > >> > > > > > > > > On Fri, Feb 13, 2015 at 1:45 PM, Ted Yu < > >> [email protected] > >> > > > >> > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Interesting, server7.ec3.internal,60020,1423845018628 > >> was > >> > > > > > > consistently > >> > > > > > > > > > chosen as destination for the table. > >> > > > > > > > > > Did server7.ec3.internal,60020,1423845018628 host > >> regions > >> > > from > >> > > > > > other > >> > > > > > > > > table > >> > > > > > > > > > ? > >> > > > > > > > > > > >> > > > > > > > > > Cheers > >> > > > > > > > > > > >> > > > > > > > > > On Fri, Feb 13, 2015 at 10:27 AM, Shahab Yunus < > >> > > > > > > [email protected] > >> > > > > > > > > > >> > > > > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > > > Table name is: > >> > > > > > > > > > > MYTABLE_RECENT_4W_V2 > >> > > > > > > > > > > > >> > > > > > > > > > > Pastebin snippet 1: http://pastebin.com/dQzMhGyP > >> > > > > > > > > > > Pastebin snippet 2: http://pastebin.com/Y7ZsNAgF > >> > > > > > > > > > > > >> > > > > > > > > > > This is the master log after invoking balancer > command > >> > from > >> > > > > hbase > >> > > > > > > > > shell. > >> > > > > > > > > > > > >> > > > > > > > > > > Regards, > >> > > > > > > > > > > Shahab > >> > > > > > > > > > > > >> > > > > > > > > > > On Fri, Feb 13, 2015 at 12:00 PM, Ted Yu < > >> > > > [email protected]> > >> > > > > > > > wrote: > >> > > > > > > > > > > > >> > > > > > > > > > > > bq. all the regions of this table were back on > this > >> > same > >> > > > RS! > >> > > > > > > > > > > > > >> > > > > > > > > > > > Interesting. Please check master log around the > time > >> > this > >> > > > RS > >> > > > > > was > >> > > > > > > > > > brought > >> > > > > > > > > > > > online. You can pastebin the relevant snippet. > >> > > > > > > > > > > > > >> > > > > > > > > > > > Thanks > >> > > > > > > > > > > > > >> > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:55 AM, Shahab Yunus < > >> > > > > > > > > [email protected]> > >> > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > > Hi Ted. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Yes, the cluster itself is balanced. On average > >> 300 > >> > > > regions > >> > > > > > per > >> > > > > > > > > node > >> > > > > > > > > > on > >> > > > > > > > > > > > 10 > >> > > > > > > > > > > > > nodes. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > # of tables is 53 of varying sizes. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Balancer was invoked and it didn't do anything > >> (i.e. > >> > no > >> > > > > > > movement > >> > > > > > > > of > >> > > > > > > > > > > > > regions) but we didn't check the master's logs. > We > >> > can > >> > > do > >> > > > > > that. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Interestingly, we restarted the RS which was > >> holding > >> > > all > >> > > > > the > >> > > > > > > > > regions > >> > > > > > > > > > of > >> > > > > > > > > > > > > this one table. The regions were nicely spread > >> out to > >> > > the > >> > > > > > > > remaining > >> > > > > > > > > > RS. > >> > > > > > > > > > > > But > >> > > > > > > > > > > > > when we brought back this RS, all the regions of > >> this > >> > > > table > >> > > > > > > were > >> > > > > > > > > back > >> > > > > > > > > > > on > >> > > > > > > > > > > > > this same RS! > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Regards, > >> > > > > > > > > > > > > Shahab > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Fri, Feb 13, 2015 at 11:46 AM, Ted Yu < > >> > > > > > [email protected]> > >> > > > > > > > > > wrote: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > How many tables are there in your cluster ? > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Is the cluster balanced overall (in terms of > >> number > >> > > of > >> > > > > > > regions > >> > > > > > > > > per > >> > > > > > > > > > > > > server) > >> > > > > > > > > > > > > > but this table is not ? > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > What happens (check master log) when you issue > >> > > > 'balancer' > >> > > > > > > > command > >> > > > > > > > > > > > through > >> > > > > > > > > > > > > > shell ? > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Cheers > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Fri, Feb 13, 2015 at 8:19 AM, Shahab Yunus > < > >> > > > > > > > > > > [email protected]> > >> > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > CDH 5.3 > >> > > > > > > > > > > > > > > HBase 98.6 > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > We are writing data to an HBase table > through > >> a > >> > M/R > >> > > > > job. > >> > > > > > We > >> > > > > > > > pre > >> > > > > > > > > > > split > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > table before each job run. The problem is > that > >> > most > >> > > > of > >> > > > > > the > >> > > > > > > > > > regions > >> > > > > > > > > > > > end > >> > > > > > > > > > > > > up > >> > > > > > > > > > > > > > > on the same RS. This results in that one RS > >> being > >> > > > > > severely > >> > > > > > > > > > > overloaded > >> > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > subsequent M/R jobs failing trying to write > to > >> > the > >> > > > > > regions > >> > > > > > > on > >> > > > > > > > > > that > >> > > > > > > > > > > > RS. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > The balancer is on and the split policy is > >> > default. > >> > > > No > >> > > > > > > > changes > >> > > > > > > > > > > there. > >> > > > > > > > > > > > > It > >> > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > a 10 node cluster. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > All other related properties are defaults > too. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Any idea, how can we force balancing of the > >> new > >> > > > > regions? > >> > > > > > Do > >> > > > > > > > we > >> > > > > > > > > > have > >> > > > > > > > > > > > to > >> > > > > > > > > > > > > > > consider compaction into the equation as > well? > >> > > > Thanks. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Regards, > >> > > > > > > > > > > > > > > Shahab > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
