Hi Sergey, I think this is what is happening: the hbase shell should be caching the region locations, if the balancer is on and you run those split commands it is very likely you will be getting that error since the regions have moved to other RSs. The easiest solution I think is to run the splits on different hbase shell processes so every time it will use a new connection. You could also try to close the connections or purge the cached locations from the hbase shell process.
thanks, esteban. -- Cloudera, Inc. On Fri, Apr 17, 2015 at 12:43 PM, Sergey Belousov <[email protected] > wrote: > Thank you Esteban for reply > > see inline > > On Apr 17, 2015 12:50 PM, "Esteban Gutierrez" <[email protected]> > wrote: > > > > Sergey, > > > > My first question would be if you are turning off the HBase balancer > before > > splitting this region and how long between locating the region and then > > splitting the region has passed. > > I do not disable balancer before. > > Also, are you splitting multiple regions > > per run of your ruby script or just one? > > I do create multiple splits for the table at the time. But it all > sequential through shell > > like > > split 'table', 'splitkey1' > sleep(2) > split 'table', 'splitkey1' > sleep(2) > split 'table', 'splitkey1' > etc.. > > > > > > thanks! > > esteban. > > > > -- > > Cloudera, Inc. > > > > > > On Fri, Apr 17, 2015 at 9:41 AM, Sergey Belousov < > [email protected]> > > wrote: > > > > > Hi all > > > > > > I was looking into doing some splitting for the table (sort of in > > > production so I can not use SPLITS on create time) using 'split' > command > > > from HBase shell. (0.98.9-hadoop2) > > > I have simple JRuby script where I just run 'split' > > > 'table_name','split_key' command using hbase shell /tmp/split_table.rb > > > > > > Time from time I have this error coming up > > > > > > ERROR: org.apache.hadoop.hbase.NotServingRegionException: Region > > > > > > [****************eaten by > > > mouse***************],1429257507107.82bfbd974d36db11075e4ef1da7abfed. > > > is not online on ******************,60020,1429256987509 > > > at > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2780) > > > at > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4337) > > > at > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.splitRegion(HRegionServer.java:4042) > > > at > > > > > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20170) > > > at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2029) > > > at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) > > > at > > > > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:112) > > > at > > > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:92) > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > > > > Even thou sleep(2) help mostly it's not guaranty (especially on live > > > cluster) and not the solution I would like. > > > > > > So my question is what would be proper way of checking that I am good > to > > > run next split command? > > > > > > > > > Thank you > > > SB > > > >
