When Master attempt to assign region to RS and assignment fails, there should be something in RS log file (check errors), that explains reason of a failure.
How many not-assigned region do you have? You can try to assign them manually in hbase shell On Tue, May 23, 2017 at 1:25 PM, jeff saremi <[email protected]> wrote: > Are dead region servers to blame? Is this possibly stale information in > the ZK? > > ________________________________ > From: Vladimir Rodionov <[email protected]> > Sent: Tuesday, May 23, 2017 12:20:16 PM > To: [email protected] > Subject: Re: Regions in Transition: FAILED_CLOSE status > > You should check RS logs to see why regions can not be assigned. > Get RS name from master log and check RS log > > -Vlad > > On Tue, May 23, 2017 at 11:47 AM, jeff saremi <[email protected]> > wrote: > > > Our write code throws exceptions like the following: > > > > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: > > Failed 10331 actions: NotServingRegionException: 10331 times,at > > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException( > > AsyncProcess.java:258) > > at org.apache.hadoop.hbase.client.AsyncProcess$ > BatchErrors.access$2000( > > AsyncProcess.java:238) > > at org.apache.hadoop.hbase.client.AsyncProcess. > > waitForAllPreviousOpsAndReset(AsyncProcess.java:1817) > > at org.apache.hadoop.hbase.client.BufferedMutatorImpl. > > backgroundFlushCommits(BufferedMutatorImpl.java:240) > > at org.apache.hadoop.hbase.client.BufferedMutatorImpl. > > mutate(BufferedMutatorImpl.java:146) > > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1028) > > at com.microsoft.bing.malta.hbaseClient11$$anon$2.run( > > ImageFeaturesHdfsToHbaseInjector.scala:115) > > at java.lang.Thread.run(Thread.java:745) > > > > > > ________________________________ > > From: jeff saremi <[email protected]> > > Sent: Tuesday, May 23, 2017 11:36:11 AM > > To: [email protected] > > Subject: Regions in Transition: FAILED_CLOSE status > > > > Why are a few hundred of our regions in this state? and what can we do to > > fix this? > > I have been running hbck a few times (is running one time enough?) to no > > avail. > > > > Internet search does not come up with anything useful either. > > > > I have restarted all masters and all region servers with no luck. > > > > Jeff > > >
