Hi there Ted- Thanks for the info. Very interesting...I will be trying this for sure.
Thanks alot, great information. Amanda On Wed, Dec 2, 2015 at 12:57 PM, Ted Yu <[email protected]> wrote: > From log snippet: > > Class > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion > not found > > I assume the above class is in the trafodion jar (just a guess) > This can explain why regions would be stuck in transition. > > For upgrade, consider replacing the trafodion jar instead of removing it. > > Cheers > > On Wed, Dec 2, 2015 at 12:51 PM, Amanda Moran <[email protected]> > wrote: > > > Hi there Ted- > > > > Thanks for response. Update: I copied my trafodion.jar files back to > > ../hbase/lib , did a restart and then hbase is back online... I then > > removed them again (just to make sure) did a restart .. and then its back > > in the region in transition. Seems like we should be able to remove our > > files and do a restart and have hbase be back online... Is this > incorrect? > > > > Log snippet: > > > > 2015-12-02 19:48:41,614 ERROR > > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed > open > > of region=hbase:meta,,1.1588230740, starting to roll back the global > > memstore size. > > java.lang.IllegalStateException: Could not instantiate a region instance. > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672) > > at > > > > > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356) > > at > > > > > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126) > > at > > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > > Class > > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion > not > > found > > at > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475) > > ... 10 more > > Caused by: java.lang.ClassNotFoundException: Class > > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion > not > > found > > at > > > > > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018) > > at > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110) > > ... 11 more > > 2015-12-02 19:48:41,627 INFO > > org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of > > region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', > > ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, > > expecting version 4 > > 2015-12-02 19:48:41,673 INFO > > org.apache.hadoop.hbase.regionserver.RSRpcServices: Open > > hbase:meta,,1.1588230740 > > 2015-12-02 19:48:41,679 ERROR > > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed > open > > of region=hbase:meta,,1.1588230740, starting to roll back the global > > memstore size. > > java.lang.IllegalStateException: Could not instantiate a region instance. > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672) > > at > > > > > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356) > > at > > > > > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126) > > at > > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > > Class > > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion > not > > found > > at > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475) > > ... 10 more > > Caused by: java.lang.ClassNotFoundException: Class > > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion > not > > found > > at > > > > > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018) > > at > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110) > > ... 11 more > > > > > > Let me know if you need more from the logs. > > > > THANKS! > > > > On Wed, Dec 2, 2015 at 12:37 PM, Ted Yu <[email protected]> wrote: > > > > > Do you mind pastebin snippet of region server log where the region > stuck > > in > > > transition was hosted ? > > > > > > This would give us some clue. > > > > > > Cheers > > > > > > On Wed, Dec 2, 2015 at 12:14 PM, Amanda Moran <[email protected]> > > > wrote: > > > > > > > Hi there All- > > > > > > > > I apologize if this issue has been raised before... I have done ALOT > of > > > > googling on this issues... and finally decided to just write here to > > the > > > > source :) > > > > > > > > Background: > > > > > > > > I work on the incubating Apache Trafodion project. > > > > > > > > I am currently using HBase 1.0 with Cloudera 5.4.5. > > > > > > > > I have tried many times to upgrade my cloudera cluster (to get the > > latest > > > > version of HBase) and have always failed due to my HBase master > failing > > > to > > > > come backup do to a Region suck in Transition. It is always the > > > hbase:meta. > > > > > > > > In the past I just assumed that the upgrade feature just didn't > > actually > > > > work... and so even though frustrating... I just let it go. > > > > > > > > But now I have come across this issue again, in a slightly different > > > way... > > > > and this makes me think it is an HBase problem. > > > > > > > > I am preparing for an upgrade, and because of this I decided to > remove > > > the > > > > "Trafodion".jar files that are located in the ../hbase/lib > directory. I > > > > thought this might be the cause of not being able to upgrade, > because I > > > was > > > > leaving the trafodion files in this directory. I removed them, then > > did a > > > > deploy client config (since some configs were stale), and then did a > > > > restart. Now I have a region stuck in transition, even before the > > > upgrade! > > > > > > > > Everything I have seen on the web, has not worked in the past. How > do I > > > get > > > > pass this issue? Do I have no choice but to do a full > > uninstall/reinstall > > > > and lose my data? > > > > > > > > Thanks all for any help! > > > > > > > > > > > > -- > > > > Thanks, > > > > > > > > Amanda Moran > > > > > > > > > > > > > > > -- > > Thanks, > > > > Amanda Moran > > > -- Thanks, Amanda Moran
