Hi there Ted- Just a quick updated.
SUCCESS! Actually just swapping out the files didn't work, BUT thanks to this email my co-worker (Narendra) mentioned we would might need to remove some of our custom settings as well. I changed the setting back to default... and SUCCESS the upgrade was successful. I was then able to reinstall Trafodion no issue. Thanks alot for the help! You led us down the right path! Amanda On Wed, Dec 2, 2015 at 1:03 PM, Amanda Moran <[email protected]> wrote: > Hi there Ted- > > Thanks for the info. Very interesting...I will be trying this for sure. > > Thanks alot, great information. > > Amanda > > On Wed, Dec 2, 2015 at 12:57 PM, Ted Yu <[email protected]> wrote: > >> From log snippet: >> >> Class >> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion >> not found >> >> I assume the above class is in the trafodion jar (just a guess) >> This can explain why regions would be stuck in transition. >> >> For upgrade, consider replacing the trafodion jar instead of removing it. >> >> Cheers >> >> On Wed, Dec 2, 2015 at 12:51 PM, Amanda Moran <[email protected]> >> wrote: >> >> > Hi there Ted- >> > >> > Thanks for response. Update: I copied my trafodion.jar files back to >> > ../hbase/lib , did a restart and then hbase is back online... I then >> > removed them again (just to make sure) did a restart .. and then its >> back >> > in the region in transition. Seems like we should be able to remove our >> > files and do a restart and have hbase be back online... Is this >> incorrect? >> > >> > Log snippet: >> > >> > 2015-12-02 19:48:41,614 ERROR >> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed >> open >> > of region=hbase:meta,,1.1588230740, starting to roll back the global >> > memstore size. >> > java.lang.IllegalStateException: Could not instantiate a region >> instance. >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486) >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793) >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765) >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721) >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672) >> > at >> > >> > >> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356) >> > at >> > >> > >> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126) >> > at >> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) >> > at >> > >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> > at >> > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> > at java.lang.Thread.run(Thread.java:745) >> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: >> > Class >> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion >> not >> > found >> > at >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112) >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475) >> > ... 10 more >> > Caused by: java.lang.ClassNotFoundException: Class >> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion >> not >> > found >> > at >> > >> > >> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018) >> > at >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110) >> > ... 11 more >> > 2015-12-02 19:48:41,627 INFO >> > org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening >> of >> > region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', >> > ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, >> > expecting version 4 >> > 2015-12-02 19:48:41,673 INFO >> > org.apache.hadoop.hbase.regionserver.RSRpcServices: Open >> > hbase:meta,,1.1588230740 >> > 2015-12-02 19:48:41,679 ERROR >> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed >> open >> > of region=hbase:meta,,1.1588230740, starting to roll back the global >> > memstore size. >> > java.lang.IllegalStateException: Could not instantiate a region >> instance. >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486) >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793) >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765) >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721) >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672) >> > at >> > >> > >> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356) >> > at >> > >> > >> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126) >> > at >> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) >> > at >> > >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> > at >> > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> > at java.lang.Thread.run(Thread.java:745) >> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: >> > Class >> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion >> not >> > found >> > at >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112) >> > at >> > >> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475) >> > ... 10 more >> > Caused by: java.lang.ClassNotFoundException: Class >> > org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion >> not >> > found >> > at >> > >> > >> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018) >> > at >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110) >> > ... 11 more >> > >> > >> > Let me know if you need more from the logs. >> > >> > THANKS! >> > >> > On Wed, Dec 2, 2015 at 12:37 PM, Ted Yu <[email protected]> wrote: >> > >> > > Do you mind pastebin snippet of region server log where the region >> stuck >> > in >> > > transition was hosted ? >> > > >> > > This would give us some clue. >> > > >> > > Cheers >> > > >> > > On Wed, Dec 2, 2015 at 12:14 PM, Amanda Moran <[email protected] >> > >> > > wrote: >> > > >> > > > Hi there All- >> > > > >> > > > I apologize if this issue has been raised before... I have done >> ALOT of >> > > > googling on this issues... and finally decided to just write here to >> > the >> > > > source :) >> > > > >> > > > Background: >> > > > >> > > > I work on the incubating Apache Trafodion project. >> > > > >> > > > I am currently using HBase 1.0 with Cloudera 5.4.5. >> > > > >> > > > I have tried many times to upgrade my cloudera cluster (to get the >> > latest >> > > > version of HBase) and have always failed due to my HBase master >> failing >> > > to >> > > > come backup do to a Region suck in Transition. It is always the >> > > hbase:meta. >> > > > >> > > > In the past I just assumed that the upgrade feature just didn't >> > actually >> > > > work... and so even though frustrating... I just let it go. >> > > > >> > > > But now I have come across this issue again, in a slightly different >> > > way... >> > > > and this makes me think it is an HBase problem. >> > > > >> > > > I am preparing for an upgrade, and because of this I decided to >> remove >> > > the >> > > > "Trafodion".jar files that are located in the ../hbase/lib >> directory. I >> > > > thought this might be the cause of not being able to upgrade, >> because I >> > > was >> > > > leaving the trafodion files in this directory. I removed them, then >> > did a >> > > > deploy client config (since some configs were stale), and then did a >> > > > restart. Now I have a region stuck in transition, even before the >> > > upgrade! >> > > > >> > > > Everything I have seen on the web, has not worked in the past. How >> do I >> > > get >> > > > pass this issue? Do I have no choice but to do a full >> > uninstall/reinstall >> > > > and lose my data? >> > > > >> > > > Thanks all for any help! >> > > > >> > > > >> > > > -- >> > > > Thanks, >> > > > >> > > > Amanda Moran >> > > > >> > > >> > >> > >> > >> > -- >> > Thanks, >> > >> > Amanda Moran >> > >> > > > > -- > Thanks, > > Amanda Moran > -- Thanks, Amanda Moran
