Could it be HBASE-3027? (Did you upgrade recently?) St.Ack
On Wed, Sep 22, 2010 at 8:21 PM, Matthew LeMieux <[email protected]> wrote: > That would be an awfully long GC pause. GC logging was not enabled on that > particular machine (it is now). > > The only exception in that part of the log file, > "java.io.FileNotFoundException: Parent path is not a directory" occurred > quite a few times, although HDFS seems to think of itself as being healthy. > > > The master did some of this for a while: > > 2010-09-22 22:06:26,563 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: > Split writer thread for region > user,\x00b\x7Ft,1281914827925.9c90ade2d19b8ce0b1613982eba2a673. got 1 to > process > 2010-09-22 22:06:26,580 INFO > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs > -- HDFS-200 > 2010-09-22 22:06:26,580 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: > Creating writer > path=hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/08b0e0f67fde365167627ad6fcbaa7b4/recovered.edits/0000000000077355008 > region=user,\x00W\x0D@,1281914368529.08b0e0f67fde365167627ad6fcbaa7b4. > 2010-09-22 22:06:26,580 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: > Split writer thread for region > user,\x00W\x0D@,1281914368529.08b0e0f67fde365167627ad6fcbaa7b4. Applied 1 > total edits to > user,\x00W\x0D@,1281914368529.08b0e0f67fde365167627ad6fcbaa7b4. in 40ms > > That was mixed in with several of these: > > 2010-09-22 22:06:26,922 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: > Split writer thread for region > user,\x01\xCB\xA8\x95,1281914501571.99a18a4a45748deb69fc6d7c804ffbe7. Got > while writing log entry to log > java.io.IOException: cannot get log writer > at > org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:551) > at org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1610) > at org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1586) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: org.apache.hadoop.ipc.RemoteException: > java.io.FileNotFoundException: Parent path is not a directory: > /hbase/user/99a18a4a45748deb69fc6d7c804ffbe7/recovered.edits > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.mkdirs(FSDirectory.java:933) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:154) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1153) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1002) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:407) > at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:512) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:968) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:962) > > And then a bunch of these: > > 2010-09-22 22:06:28,482 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: > Closed > hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/.META./1028785192/recovered.edits/0000000000077298364 > 2010-09-22 22:06:28,515 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: > Closed > hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/counters/692b9f45969bcc7ae2774b0e5fc17b12/recovered.edits/0000000000077298445 > 2010-09-22 22:06:29,431 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: > Closed > hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/9097ba3b5f7b255e19861dc2131c8477/recovered.edits/0000000000077298636 > 2010-09-22 22:06:29,448 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: > Closed > hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/dbc3619ce88ac8153334fb5f581c1575/recovered.edits/0000000000077298637 > 2010-09-22 22:06:29,464 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: > Closed > hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/0f03f4ef25d647784ce69cd1e354af94/recovered.edits/0000000000077298666 > ... > 2010-09-22 22:06:30,614 WARN > org.apache.hadoop.hbase.master.RegionServerOperationQueue: Failed processing: > ProcessServerShutdown of > domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559; putting onto > delayed todo queue > java.io.IOException: java.io.IOException: cannot get log writer > at > org.apache.hadoop.hbase.regionserver.wal.HLog.writeEditsBatchToRegions(HLog.java:1510) > at > org.apache.hadoop.hbase.regionserver.wal.HLog.splitLog(HLog.java:1335) > at > org.apache.hadoop.hbase.regionserver.wal.HLog.splitLog(HLog.java:1210) > at > org.apache.hadoop.hbase.master.ProcessServerShutdown.process(ProcessServerShutdown.java:299) > at > org.apache.hadoop.hbase.master.RegionServerOperationQueue.process(RegionServerOperationQueue.java:147) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:532) > Caused by: java.io.IOException: cannot get log writer > at > org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:551) > at org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1610) > at org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1586) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: org.apache.hadoop.ipc.RemoteException: > java.io.FileNotFoundException: Parent path is not a directory: > /hbase/user/2d225ad65207b0ab51b4604a9f889773/recovered.edits > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.mkdirs(FSDirectory.java:933) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:154) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1153) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1002) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:407) > at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:512) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:968) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:962) > > There were also quite a few messages that indicate the split was unsuccessful: > > 2010-09-22 22:06:45,430 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: > Split writer thread for region > user,\x00\x14w\xB0,1284527871890.752e96713b4408d77b750af06d0e2bad. got 57 to > process > 2010-09-22 22:06:45,432 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: > Found existing old edits file. It could be the result of a previous failed > split attempt. Deleting > hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/752e96713b4408d77b750af06d0e2bad/recovered.edits/0000000000077298635, > length=200866 > > Does this point to a particular problem? > > > Matthew > > > On Sep 22, 2010, at 3:52 PM, Jean-Daniel Cryans wrote: > >> It seem that the log splitting didn't succeed (can't tell why, small >> log snippet is too small). Did it get stuck on EOF or something like >> that? In any case, it looks like a bug. >> >>> * how do I avoid disruption when this sort of thing happens >> >> Your region server was unavailable for 3 minutes, probably a GC pause >> (did you enable GC logging?), so the master splits the logs after it >> was notified via a ZK watcher. The timeout is 1 minute, you can set >> that lower if you want to answer to region server failure faster (but >> be more affected by shorter GC pauses or similar). You can also lower >> the recovery time by setting a smaller hbase.regionserver.maxlogs >> (which is 32 by default), at the expense of flushing a lot more. >> >>> * what is the best procedure for recovering from such a thing (i.e., is >>> there something to be done other than simply restarting the region server?) >> >> It's fatal if the master isn't able to complete log splitting >> properly, so restarting is your best bet until we can figure how you >> got stuck there. >> >> J-D >> >> On Wed, Sep 22, 2010 at 3:41 PM, Matthew LeMieux <[email protected]> wrote: >>> I've just had another of those region server shutdowns >>> (RegionServerSuicide). >>> >>> * how do I avoid disruption when this sort of thing happens >>> * what is the best procedure for recovering from such a thing (i.e., is >>> there something to be done other than simply restarting the region server?) >>> >>> I've heard others talk of this event as being easy to recover from. I'd >>> like to understand how I can get into a configuration where that is the >>> case. Currently, an RS shutdown causes significant disruption. Clients >>> are unable to query HBase. Simply restarting the region server doesn't >>> improve things. In every case, I've needed to do a complete restart of the >>> entire cluster which isn't exactly super fast. A single region server >>> shutdown means a minimum of 1/2 hour down time, and sometimes that is more >>> like hours depending on the reason for the RS shutdown (once it took an >>> entire day). Is the experience similarly painful for others? >>> >>> Common error messages on the client are: java.lang.RuntimeException: >>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up >>> proxy to domU-12-31-39-18-1B-05.compute-1.internal/10.104.55.247:60020 >>> after attempts=1 >>> ... and ... the RS complains with a "NotServingRegion" exception. >>> >>> As you can see form the selected messages below, after I restart the >>> affected RS, the cluster is still unusable, and gives no indication that >>> this state still change. >>> >>> It is of course no problem just restart a region server if it goes down >>> (either with a script or manually), the real question is how do I avoid >>> downtime? >>> >>> Details are below... >>> >>> Thank you for your help, >>> >>> Matthew >>> >>> I'm running in EC2 on DFS from CDH3b2, using a version of HBase from >>> 20100830. >>> >>> START OF THE SHUTDOWN ON AFFECTED RS: >>> >>> 2010-09-22 22:00:39,887 DEBUG org.apache.hadoop.hbase.regionserver.Store: >>> Major compaction triggered on store pg_payload; time since last major >>> compaction 93578344ms >>> 2010-09-22 22:00:39,887 INFO org.apache.hadoop.hbase.regionserver.Store: >>> Started compaction of 1 file(s) in pg_payload of >>> userh,\x1D\xAA2f\xE0\x1F\x12:\xA4\x98r\xFF!\xB8\x1A`,1285083827570.6be256da616fc20a6ac78a41952104e8. >>> into >>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/userh/6be256da616fc20a6ac78a41952104e8/.tmp, >>> sequenceid=77142618 >>> 2010-09-22 22:03:23,317 DEBUG >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.57 GB, >>> free=396.14 MB, max=1.95 GB, blocks=27819, accesses=38513233, >>> hits=14159798, hitRatio=36.76%%, evictions=6494, evicted=20462105, >>> evictedPerRun=3150.924560546875 >>> 2010-09-22 22:03:23,361 INFO org.apache.zookeeper.ClientCnxn: Client >>> session timed out, have not heard from server in 181390ms for sessionid >>> 0x12b266801550065, closing socket connection and attempting reconnect >>> 2010-09-22 22:03:23,363 INFO org.apache.zookeeper.ClientCnxn: Client >>> session timed out, have not heard from server in 183256ms for sessionid >>> 0x12b266801550002, closing socket connection and attempting reconnect >>> 2010-09-22 22:03:23,451 FATAL >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server >>> serverName=domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559, >>> load=(requests=4458, regions=134, usedHeap=8872, maxHeap=9995): Unhandled >>> exception >>> org.apache.hadoop.hbase.YouAreDeadException: >>> org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; >>> currently processing >>> domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 as dead server >>> at >>> org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217) >>> at >>> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271) >>> at >>> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:753) >>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576) >>> at >>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919) >>> >>> THE RESTART ON THE AFFECTED RS: >>> >>> 2010-09-22 22:10:04,534 INFO >>> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020 >>> exiting >>> 2010-09-22 22:13:23,449 INFO >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>> 4628396768048238482 lease expired >>> 2010-09-22 22:13:23,449 INFO >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>> 891312872292456529 lease expired >>> 2010-09-22 22:13:23,449 INFO >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>> 2977737069391109815 lease expired >>> 2010-09-22 22:13:23,450 INFO org.apache.hadoop.hbase.Leases: >>> regionserver60020.leaseChecker closing leases >>> 2010-09-22 22:13:23,450 INFO org.apache.hadoop.hbase.Leases: >>> regionserver60020.leaseChecker closed leases >>> 2010-09-22 22:13:23,450 INFO >>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; >>> hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-9,5,main] >>> 2010-09-22 22:13:23,451 INFO >>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown >>> hook thread. >>> 2010-09-22 22:13:23,452 INFO >>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished. >>> Wed Sep 22 22:15:11 UTC 2010 Starting regionserver on domU-12-31-39-18-1B-05 >>> 2010-09-22 22:15:12,716 INFO >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> vmInputArguments=[-Xmx768m, -XX:+HeapDumpOnOutOfMemoryError, >>> -XX:+UseConcMarkSweepGC, -XX:+CMSIncrementalMode, >>> -XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcMarkSweepGC, >>> -XX:+CMSIncrementalMode, -Xmx10000m, -Dhbase.log.dir=/mnt/tmp/hbaselog, >>> -Dhbase.log.file=hbase-mlcamus-regionserver-domU-12-31-39-18-1B-05.log, >>> -Dhbase.home.dir=/home/mlcamus/apps/hbase/bin/.., -Dhbase.id.str=mlcamus, >>> -Dhbase.root.logger=INFO,DRFA] >>> ... >>> 2010-09-22 22:25:03,530 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:04,288 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:05,299 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:05,348 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:06,308 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:06,358 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:07,368 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:07,545 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:08,318 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:08,377 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:08,548 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:08,552 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; >>> user,^...@^tj�,1284527871890.bfee076d62bd372a70299f5854517145. >>> 2010-09-22 22:25:09,288 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:09,558 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> 2010-09-22 22:25:09,698 DEBUG >>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>> NotServingRegionException; .META.,,1 >>> ... (these messages continue) >>> >>> ON THE MASTER WHEN THE RS WAS SHUTTING DOWN: >>> >>> 2010-09-22 21:59:51,588 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> All 1 .META. region(s) scanned >>> 2010-09-22 22:00:22,931 INFO org.apache.hadoop.hbase.master.ServerManager: >>> 2 region servers, 0 dead, average load 134.0 >>> 2010-09-22 22:00:39,331 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> RegionManager.rootScanner scanning meta region {server: >>> 10.104.37.247:60020, regionname: -ROOT-,,0.70236052, startKey: <>} >>> 2010-09-22 22:00:39,333 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> RegionManager.rootScanner scan of 1 row(s) of meta region {server: >>> 10.104.37.247:60020, regionname: -ROOT-,,0.70236052, startKey: <>} complete >>> 2010-09-22 22:00:51,368 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> RegionManager.metaScanner scanning meta region {server: >>> 10.104.55.247:60020, regionname: .META.,,1.1028785192, startKey: <>} >>> 2010-09-22 22:00:51,801 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> RegionManager.metaScanner scan of 266 row(s) of meta region {server: >>> 10.104.55.247:60020, regionname: .META.,,1.1028785192, startKey: <>} >>> complete >>> 2010-09-22 22:00:51,801 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> All 1 .META. region(s) scanned >>> 2010-09-22 22:01:22,941 INFO org.apache.hadoop.hbase.master.ServerManager: >>> 2 region servers, 0 dead, average load 134.0 >>> 2010-09-22 22:01:39,341 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> RegionManager.rootScanner scanning meta region {server: >>> 10.104.37.247:60020, regionname: -ROOT-,,0.70236052, startKey: <>} >>> 2010-09-22 22:01:39,343 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> RegionManager.rootScanner scan of 1 row(s) of meta region {server: >>> 10.104.37.247:60020, regionname: -ROOT-,,0.70236052, startKey: <>} complete >>> 2010-09-22 22:01:51,371 INFO org.apache.hadoop.hbase.master.BaseScanner: >>> RegionManager.metaScanner scanning meta region {server: >>> 10.104.55.247:60020, regionname: .META.,,1.1028785192, startKey: <>} >>> 2010-09-22 22:02:22,951 INFO org.apache.hadoop.hbase.master.ServerManager: >>> 2 region servers, 0 dead, average load 134.0 >>> 2010-09-22 22:02:24,203 INFO org.apache.hadoop.hbase.master.ServerManager: >>> domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 znode expired >>> 2010-09-22 22:02:24,204 DEBUG org.apache.hadoop.hbase.master.ServerManager: >>> Added=domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 to dead >>> servers, added shutdown processing operation >>> 2010-09-22 22:02:24,204 INFO org.apache.hadoop.hbase.master.RegionManager: >>> META region whose startkey is removed from onlineMetaRegions >>> 2010-09-22 22:02:24,204 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo: >>> ProcessServerShutdown of >>> domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 >>> 2010-09-22 22:02:24,204 INFO >>> org.apache.hadoop.hbase.master.RegionServerOperation: Process shutdown of >>> server domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559: >>> logSplit: false, rootRescanned: false, numberOfMetaRegions: 1, >>> onlineMetaRegions.size(): 0 >>> 2010-09-22 22:02:24,242 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: >>> Splitting 35 hlog(s) in >>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/.logs/domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 >>> >>> ON THE MASTER AFTER GIVING A FEW MINUTES TO RECOVER: >>> >>> 2010-09-22 22:31:00,300 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: >>> 1, onlineMetaRegions.size(): 0 >>> 2010-09-22 22:31:00,300 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because not >>> all meta regions are online >>> 2010-09-22 22:31:00,301 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo: >>> ProcessRegionClose of >>> userh,R\x80\xBD\x0F\xCB\xEAf\x95"+\xE1\x9A)/\xD7\x19,1285081795205.bf1a30fb67c1b0ea421da1f458d83c44., >>> false, reassign: true >>> 2010-09-22 22:31:00,301 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: >>> 1, onlineMetaRegions.size(): 0 >>> 2010-09-22 22:31:00,301 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because not >>> all meta regions are online >>> 2010-09-22 22:31:10,310 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo: >>> ProcessRegionClose of >>> userh,R\x80\xBD\x0F\xCB\xEAf\x95"+\xE1\x9A)/\xD7\x19,1285081795205.bf1a30fb67c1b0ea421da1f458d83c44., >>> false, reassign: true >>> 2010-09-22 22:31:10,310 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: >>> 1, onlineMetaRegions.size(): 0 >>> 2010-09-22 22:31:10,310 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because not >>> all meta regions are online >>> 2010-09-22 22:31:10,311 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo: >>> ProcessRegionClose of >>> userh,\xE5\x1A\x93\xCEQ\xC9\x8E\x8DIlvh9\xE1\xC8O,1285067882016.8aecae7491b95dd76aefab0058a71cff., >>> false, reassign: true >>> 2010-09-22 22:31:10,311 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions: >>> 1, onlineMetaRegions.size(): 0 >>> 2010-09-22 22:31:10,311 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because not >>> all meta regions are online >>> 2010-09-22 22:31:20,320 DEBUG >>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo: >>> ProcessRegionClose of >>> userh,\xE5\x1A\x93\xCEQ\xC9\x8E\x8DIlvh9\xE1\xC8O,1285067882016.8aecae7491b95dd76aefab00 >>> >>> WHEN RUNNING "list" FROM "hbase shell": >>> >>> org.apache.hadoop.hbase.NotServingRegionException: >>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 >>> at >>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2217) >>> at >>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1808) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576) >>> at >>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919) >>> >>> >>> > >
