Yes, this sever hosts other regions from other tables as well. Regards Shahab
On Fri, Feb 13, 2015 at 1:45 PM, Ted Yu <[email protected]> wrote: > Interesting, server7.ec3.internal,60020,1423845018628 was consistently > chosen as destination for the table. > Did server7.ec3.internal,60020,1423845018628 host regions from other table > ? > > Cheers > > On Fri, Feb 13, 2015 at 10:27 AM, Shahab Yunus <[email protected]> > wrote: > > > Table name is: > > MYTABLE_RECENT_4W_V2 > > > > Pastebin snippet 1: http://pastebin.com/dQzMhGyP > > Pastebin snippet 2: http://pastebin.com/Y7ZsNAgF > > > > This is the master log after invoking balancer command from hbase shell. > > > > Regards, > > Shahab > > > > On Fri, Feb 13, 2015 at 12:00 PM, Ted Yu <[email protected]> wrote: > > > > > bq. all the regions of this table were back on this same RS! > > > > > > Interesting. Please check master log around the time this RS was > brought > > > online. You can pastebin the relevant snippet. > > > > > > Thanks > > > > > > On Fri, Feb 13, 2015 at 8:55 AM, Shahab Yunus <[email protected]> > > > wrote: > > > > > > > Hi Ted. > > > > > > > > Yes, the cluster itself is balanced. On average 300 regions per node > on > > > 10 > > > > nodes. > > > > > > > > # of tables is 53 of varying sizes. > > > > > > > > Balancer was invoked and it didn't do anything (i.e. no movement of > > > > regions) but we didn't check the master's logs. We can do that. > > > > > > > > Interestingly, we restarted the RS which was holding all the regions > of > > > > this one table. The regions were nicely spread out to the remaining > RS. > > > But > > > > when we brought back this RS, all the regions of this table were back > > on > > > > this same RS! > > > > > > > > Thanks. > > > > > > > > > > > > Regards, > > > > Shahab > > > > > > > > On Fri, Feb 13, 2015 at 11:46 AM, Ted Yu <[email protected]> > wrote: > > > > > > > > > How many tables are there in your cluster ? > > > > > > > > > > Is the cluster balanced overall (in terms of number of regions per > > > > server) > > > > > but this table is not ? > > > > > > > > > > What happens (check master log) when you issue 'balancer' command > > > through > > > > > shell ? > > > > > > > > > > Cheers > > > > > > > > > > On Fri, Feb 13, 2015 at 8:19 AM, Shahab Yunus < > > [email protected]> > > > > > wrote: > > > > > > > > > > > CDH 5.3 > > > > > > HBase 98.6 > > > > > > > > > > > > We are writing data to an HBase table through a M/R job. We pre > > split > > > > the > > > > > > table before each job run. The problem is that most of the > regions > > > end > > > > up > > > > > > on the same RS. This results in that one RS being severely > > overloaded > > > > and > > > > > > subsequent M/R jobs failing trying to write to the regions on > that > > > RS. > > > > > > > > > > > > The balancer is on and the split policy is default. No changes > > there. > > > > It > > > > > is > > > > > > a 10 node cluster. > > > > > > > > > > > > All other related properties are defaults too. > > > > > > > > > > > > Any idea, how can we force balancing of the new regions? Do we > have > > > to > > > > > > consider compaction into the equation as well? Thanks. > > > > > > > > > > > > Regards, > > > > > > Shahab > > > > > > > > > > > > > > > > > > > > >
