https://issues.apache.org/jira/browse/HBASE-11368 related to the original issue too.
On Mon, Sep 22, 2014 at 10:18 AM, Ted Yu <yuzhih...@gmail.com> wrote: > As you noted in the FIXME, there're some factors which should be tackled by > balancer / assignment manager. > > Please continue digging up master log so that we can find the cause for > balancer not fulfilling your goal. > > Cheers > > On Mon, Sep 22, 2014 at 10:09 AM, Jianshi Huang <jianshi.hu...@gmail.com> > wrote: > > > Ok, I fixed this by manually reassign region servers to newly created > ones. > > > > def reassignRegionServer(admin: HBaseAdmin, regions: Seq[HRegionInfo], > > regionServers: Seq[ServerName]): Unit = { > > val rand = new Random() > > regions.foreach { r => > > val idx = rand.nextInt(regionServers.size) > > val server = regionServers(idx) > > // FIXME: what if selected region server is dead? > > admin.move(r.getEncodedNameAsBytes, > > server.getServerName.getBytes("UTF8")) > > } > > } > > > > er... > > > > Jianshi > > > > On Tue, Sep 23, 2014 at 12:24 AM, Jianshi Huang <jianshi.hu...@gmail.com > > > > wrote: > > > > > Hmm...any workaround? I only want to do this: > > > > > > Rebalance the new regions *evenly* to all servers after manually adding > > > splits, so later bulk insertions won't cause contention. > > > > > > P.S. > > > Looks like two of the region servers which had majority of the regions > > > were down during Major compaction... I guess it had too much data. > > > > > > > > > Jianshi > > > > > > On Tue, Sep 23, 2014 at 12:13 AM, Jianshi Huang < > jianshi.hu...@gmail.com > > > > > > wrote: > > > > > >> Yes, I have access to Master UI, however logs/*.log cannot be opened > or > > >> downloaded, must be some security restrictions in the proxy... > > >> > > >> Jianshi > > >> > > >> On Tue, Sep 23, 2014 at 12:06 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > >> > > >>> Do you have access to Master UI ? > > >>> > > >>> <master-address>:60010/logs/ would show you list of log files. > > >>> > > >>> The you can view > > <master-address>:60010/logs/hbase-<user>-master-XXX.log > > >>> > > >>> Cheers > > >>> > > >>> On Mon, Sep 22, 2014 at 9:00 AM, Jianshi Huang < > > jianshi.hu...@gmail.com> > > >>> wrote: > > >>> > > >>> > Ah... I don't have access to HMaster logs... I need to ask the > admin. > > >>> > > > >>> > Jianshi > > >>> > > > >>> > On Mon, Sep 22, 2014 at 11:49 PM, Ted Yu <yuzhih...@gmail.com> > > wrote: > > >>> > > > >>> > > bq. assign per-table balancer class > > >>> > > > > >>> > > No that I know of. > > >>> > > Can you pastebin master log involving output from balancer ? > > >>> > > > > >>> > > Cheers > > >>> > > > > >>> > > On Mon, Sep 22, 2014 at 8:29 AM, Jianshi Huang < > > >>> jianshi.hu...@gmail.com> > > >>> > > wrote: > > >>> > > > > >>> > > > Hi Ted, > > >>> > > > > > >>> > > > I moved setBalancerRunning before balancer and run them twice. > > >>> However > > >>> > I > > >>> > > > still got highly skewed region distribution. > > >>> > > > > > >>> > > > I guess it's because of the StochasticLoadBalancer, can I > assign > > >>> > > per-table > > >>> > > > balancer class in HBase? > > >>> > > > > > >>> > > > > > >>> > > > Jianshi > > >>> > > > > > >>> > > > On Mon, Sep 22, 2014 at 9:50 PM, Ted Yu <yuzhih...@gmail.com> > > >>> wrote: > > >>> > > > > > >>> > > > > admin.setBalancerRunning() call should precede the call to > > >>> > > > > admin.balancer(). > > >>> > > > > > > >>> > > > > You can inspect master log to see whether regions are being > > >>> moved off > > >>> > > the > > >>> > > > > heavily loaded server. > > >>> > > > > > > >>> > > > > Cheers > > >>> > > > > > > >>> > > > > On Mon, Sep 22, 2014 at 1:42 AM, Jianshi Huang < > > >>> > > jianshi.hu...@gmail.com> > > >>> > > > > wrote: > > >>> > > > > > > >>> > > > > > Hi Ted and others, > > >>> > > > > > > > >>> > > > > > I did the following after adding splits (without data) to > my > > >>> table, > > >>> > > > > however > > >>> > > > > > the region is still very imbalanced (one region server has > > 221 > > >>> > > regions > > >>> > > > > and > > >>> > > > > > other 50 region servers have about 4~8 regions each). > > >>> > > > > > > > >>> > > > > > admin.balancer() > > >>> > > > > > admin.setBalancerRunning(true, true) > > >>> > > > > > > > >>> > > > > > The balancer class in my HBase cluster is > > >>> > > > > > > > >>> > > > > > > > org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer > > >>> > > > > > > > >>> > > > > > So, is this behavior expected? Can I assign different > > balancer > > >>> > class > > >>> > > to > > >>> > > > > my > > >>> > > > > > tables (I don't have HBase admin permission)? Which one > > should > > >>> I > > >>> > use? > > >>> > > > > > > > >>> > > > > > I just want HBase to evenly distribute the regions even > they > > >>> don't > > >>> > > have > > >>> > > > > > data (that's the purpose of pre-split I think). > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > Jianshi > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > On Sat, Sep 6, 2014 at 12:45 AM, Ted Yu < > yuzhih...@gmail.com > > > > > >>> > wrote: > > >>> > > > > > > > >>> > > > > > > Yes. See the following method in HBaseAdmin: > > >>> > > > > > > > > >>> > > > > > > public boolean balancer() > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > On Fri, Sep 5, 2014 at 9:38 AM, Jianshi Huang < > > >>> > > > jianshi.hu...@gmail.com > > >>> > > > > > > > >>> > > > > > > wrote: > > >>> > > > > > > > > >>> > > > > > > > Thanks Ted! > > >>> > > > > > > > > > >>> > > > > > > > Didn't know I still need to run the 'balancer' command. > > >>> > > > > > > > > > >>> > > > > > > > Is there a way to do it programmatically? > > >>> > > > > > > > > > >>> > > > > > > > Jianshi > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > On Sat, Sep 6, 2014 at 12:29 AM, Ted Yu < > > >>> yuzhih...@gmail.com> > > >>> > > > wrote: > > >>> > > > > > > > > > >>> > > > > > > > > After splitting the region, you may need to run > > balancer > > >>> to > > >>> > > > spread > > >>> > > > > > the > > >>> > > > > > > > new > > >>> > > > > > > > > regions out. > > >>> > > > > > > > > > > >>> > > > > > > > > Cheers > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > On Fri, Sep 5, 2014 at 9:25 AM, Jianshi Huang < > > >>> > > > > > jianshi.hu...@gmail.com > > >>> > > > > > > > > > >>> > > > > > > > > wrote: > > >>> > > > > > > > > > > >>> > > > > > > > > > Hi Shahab, > > >>> > > > > > > > > > > > >>> > > > > > > > > > I see, that seems to be the right way... > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > On Sat, Sep 6, 2014 at 12:21 AM, Shahab Yunus < > > >>> > > > > > > shahab.yu...@gmail.com> > > >>> > > > > > > > > > wrote: > > >>> > > > > > > > > > > > >>> > > > > > > > > > > Shahab > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > -- > > >>> > > > > > > > > > Jianshi Huang > > >>> > > > > > > > > > > > >>> > > > > > > > > > LinkedIn: jianshi > > >>> > > > > > > > > > Twitter: @jshuang > > >>> > > > > > > > > > Github & Blog: http://huangjs.github.com/ > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > -- > > >>> > > > > > > > Jianshi Huang > > >>> > > > > > > > > > >>> > > > > > > > LinkedIn: jianshi > > >>> > > > > > > > Twitter: @jshuang > > >>> > > > > > > > Github & Blog: http://huangjs.github.com/ > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > -- > > >>> > > > > > Jianshi Huang > > >>> > > > > > > > >>> > > > > > LinkedIn: jianshi > > >>> > > > > > Twitter: @jshuang > > >>> > > > > > Github & Blog: http://huangjs.github.com/ > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > -- > > >>> > > > Jianshi Huang > > >>> > > > > > >>> > > > LinkedIn: jianshi > > >>> > > > Twitter: @jshuang > > >>> > > > Github & Blog: http://huangjs.github.com/ > > >>> > > > > > >>> > > > > >>> > > > >>> > > > >>> > > > >>> > -- > > >>> > Jianshi Huang > > >>> > > > >>> > LinkedIn: jianshi > > >>> > Twitter: @jshuang > > >>> > Github & Blog: http://huangjs.github.com/ > > >>> > > > >>> > > >> > > >> > > >> > > >> -- > > >> Jianshi Huang > > >> > > >> LinkedIn: jianshi > > >> Twitter: @jshuang > > >> Github & Blog: http://huangjs.github.com/ > > >> > > > > > > > > > > > > -- > > > Jianshi Huang > > > > > > LinkedIn: jianshi > > > Twitter: @jshuang > > > Github & Blog: http://huangjs.github.com/ > > > > > > > > > > > -- > > Jianshi Huang > > > > LinkedIn: jianshi > > Twitter: @jshuang > > Github & Blog: http://huangjs.github.com/ > > > -- Bharath Vissapragada <http://www.cloudera.com>