Hmm...any workaround? I only want to do this: Rebalance the new regions *evenly* to all servers after manually adding splits, so later bulk insertions won't cause contention.
P.S. Looks like two of the region servers which had majority of the regions were down during Major compaction... I guess it had too much data. Jianshi On Tue, Sep 23, 2014 at 12:13 AM, Jianshi Huang <[email protected]> wrote: > Yes, I have access to Master UI, however logs/*.log cannot be opened or > downloaded, must be some security restrictions in the proxy... > > Jianshi > > On Tue, Sep 23, 2014 at 12:06 AM, Ted Yu <[email protected]> wrote: > >> Do you have access to Master UI ? >> >> <master-address>:60010/logs/ would show you list of log files. >> >> The you can view <master-address>:60010/logs/hbase-<user>-master-XXX.log >> >> Cheers >> >> On Mon, Sep 22, 2014 at 9:00 AM, Jianshi Huang <[email protected]> >> wrote: >> >> > Ah... I don't have access to HMaster logs... I need to ask the admin. >> > >> > Jianshi >> > >> > On Mon, Sep 22, 2014 at 11:49 PM, Ted Yu <[email protected]> wrote: >> > >> > > bq. assign per-table balancer class >> > > >> > > No that I know of. >> > > Can you pastebin master log involving output from balancer ? >> > > >> > > Cheers >> > > >> > > On Mon, Sep 22, 2014 at 8:29 AM, Jianshi Huang < >> [email protected]> >> > > wrote: >> > > >> > > > Hi Ted, >> > > > >> > > > I moved setBalancerRunning before balancer and run them twice. >> However >> > I >> > > > still got highly skewed region distribution. >> > > > >> > > > I guess it's because of the StochasticLoadBalancer, can I assign >> > > per-table >> > > > balancer class in HBase? >> > > > >> > > > >> > > > Jianshi >> > > > >> > > > On Mon, Sep 22, 2014 at 9:50 PM, Ted Yu <[email protected]> >> wrote: >> > > > >> > > > > admin.setBalancerRunning() call should precede the call to >> > > > > admin.balancer(). >> > > > > >> > > > > You can inspect master log to see whether regions are being moved >> off >> > > the >> > > > > heavily loaded server. >> > > > > >> > > > > Cheers >> > > > > >> > > > > On Mon, Sep 22, 2014 at 1:42 AM, Jianshi Huang < >> > > [email protected]> >> > > > > wrote: >> > > > > >> > > > > > Hi Ted and others, >> > > > > > >> > > > > > I did the following after adding splits (without data) to my >> table, >> > > > > however >> > > > > > the region is still very imbalanced (one region server has 221 >> > > regions >> > > > > and >> > > > > > other 50 region servers have about 4~8 regions each). >> > > > > > >> > > > > > admin.balancer() >> > > > > > admin.setBalancerRunning(true, true) >> > > > > > >> > > > > > The balancer class in my HBase cluster is >> > > > > > >> > > > > > org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer >> > > > > > >> > > > > > So, is this behavior expected? Can I assign different balancer >> > class >> > > to >> > > > > my >> > > > > > tables (I don't have HBase admin permission)? Which one should I >> > use? >> > > > > > >> > > > > > I just want HBase to evenly distribute the regions even they >> don't >> > > have >> > > > > > data (that's the purpose of pre-split I think). >> > > > > > >> > > > > > >> > > > > > Jianshi >> > > > > > >> > > > > > >> > > > > > On Sat, Sep 6, 2014 at 12:45 AM, Ted Yu <[email protected]> >> > wrote: >> > > > > > >> > > > > > > Yes. See the following method in HBaseAdmin: >> > > > > > > >> > > > > > > public boolean balancer() >> > > > > > > >> > > > > > > >> > > > > > > On Fri, Sep 5, 2014 at 9:38 AM, Jianshi Huang < >> > > > [email protected] >> > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Thanks Ted! >> > > > > > > > >> > > > > > > > Didn't know I still need to run the 'balancer' command. >> > > > > > > > >> > > > > > > > Is there a way to do it programmatically? >> > > > > > > > >> > > > > > > > Jianshi >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > On Sat, Sep 6, 2014 at 12:29 AM, Ted Yu < >> [email protected]> >> > > > wrote: >> > > > > > > > >> > > > > > > > > After splitting the region, you may need to run balancer >> to >> > > > spread >> > > > > > the >> > > > > > > > new >> > > > > > > > > regions out. >> > > > > > > > > >> > > > > > > > > Cheers >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On Fri, Sep 5, 2014 at 9:25 AM, Jianshi Huang < >> > > > > > [email protected] >> > > > > > > > >> > > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Hi Shahab, >> > > > > > > > > > >> > > > > > > > > > I see, that seems to be the right way... >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On Sat, Sep 6, 2014 at 12:21 AM, Shahab Yunus < >> > > > > > > [email protected]> >> > > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Shahab >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > -- >> > > > > > > > > > Jianshi Huang >> > > > > > > > > > >> > > > > > > > > > LinkedIn: jianshi >> > > > > > > > > > Twitter: @jshuang >> > > > > > > > > > Github & Blog: http://huangjs.github.com/ >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > Jianshi Huang >> > > > > > > > >> > > > > > > > LinkedIn: jianshi >> > > > > > > > Twitter: @jshuang >> > > > > > > > Github & Blog: http://huangjs.github.com/ >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > Jianshi Huang >> > > > > > >> > > > > > LinkedIn: jianshi >> > > > > > Twitter: @jshuang >> > > > > > Github & Blog: http://huangjs.github.com/ >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > Jianshi Huang >> > > > >> > > > LinkedIn: jianshi >> > > > Twitter: @jshuang >> > > > Github & Blog: http://huangjs.github.com/ >> > > > >> > > >> > >> > >> > >> > -- >> > Jianshi Huang >> > >> > LinkedIn: jianshi >> > Twitter: @jshuang >> > Github & Blog: http://huangjs.github.com/ >> > >> > > > > -- > Jianshi Huang > > LinkedIn: jianshi > Twitter: @jshuang > Github & Blog: http://huangjs.github.com/ > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
