That would be an awfully long GC pause. GC logging was not enabled on that
particular machine (it is now).
The only exception in that part of the log file,
"java.io.FileNotFoundException: Parent path is not a directory" occurred quite
a few times, although HDFS seems to think of itself as being healthy.
The master did some of this for a while:
2010-09-22 22:06:26,563 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Split writer thread for region
user,\x00b\x7Ft,1281914827925.9c90ade2d19b8ce0b1613982eba2a673. got 1 to process
2010-09-22 22:06:26,580 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs --
HDFS-200
2010-09-22 22:06:26,580 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Creating writer
path=hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/08b0e0f67fde365167627ad6fcbaa7b4/recovered.edits/0000000000077355008
region=user,\x00W\x0D@,1281914368529.08b0e0f67fde365167627ad6fcbaa7b4.
2010-09-22 22:06:26,580 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Split writer thread for region
user,\x00W\x0D@,1281914368529.08b0e0f67fde365167627ad6fcbaa7b4. Applied 1 total
edits to user,\x00W\x0D@,1281914368529.08b0e0f67fde365167627ad6fcbaa7b4. in 40ms
That was mixed in with several of these:
2010-09-22 22:06:26,922 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog:
Split writer thread for region
user,\x01\xCB\xA8\x95,1281914501571.99a18a4a45748deb69fc6d7c804ffbe7. Got while
writing log entry to log
java.io.IOException: cannot get log writer
at
org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:551)
at org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1610)
at org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1586)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.ipc.RemoteException:
java.io.FileNotFoundException: Parent path is not a directory:
/hbase/user/99a18a4a45748deb69fc6d7c804ffbe7/recovered.edits
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.mkdirs(FSDirectory.java:933)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:154)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1153)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1002)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:407)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:512)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:968)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:962)
And then a bunch of these:
2010-09-22 22:06:28,482 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Closed
hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/.META./1028785192/recovered.edits/0000000000077298364
2010-09-22 22:06:28,515 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Closed
hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/counters/692b9f45969bcc7ae2774b0e5fc17b12/recovered.edits/0000000000077298445
2010-09-22 22:06:29,431 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Closed
hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/9097ba3b5f7b255e19861dc2131c8477/recovered.edits/0000000000077298636
2010-09-22 22:06:29,448 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Closed
hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/dbc3619ce88ac8153334fb5f581c1575/recovered.edits/0000000000077298637
2010-09-22 22:06:29,464 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Closed
hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/0f03f4ef25d647784ce69cd1e354af94/recovered.edits/0000000000077298666
...
2010-09-22 22:06:30,614 WARN
org.apache.hadoop.hbase.master.RegionServerOperationQueue: Failed processing:
ProcessServerShutdown of
domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559; putting onto
delayed todo queue
java.io.IOException: java.io.IOException: cannot get log writer
at
org.apache.hadoop.hbase.regionserver.wal.HLog.writeEditsBatchToRegions(HLog.java:1510)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.splitLog(HLog.java:1335)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.splitLog(HLog.java:1210)
at
org.apache.hadoop.hbase.master.ProcessServerShutdown.process(ProcessServerShutdown.java:299)
at
org.apache.hadoop.hbase.master.RegionServerOperationQueue.process(RegionServerOperationQueue.java:147)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:532)
Caused by: java.io.IOException: cannot get log writer
at
org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:551)
at org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1610)
at org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1586)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.ipc.RemoteException:
java.io.FileNotFoundException: Parent path is not a directory:
/hbase/user/2d225ad65207b0ab51b4604a9f889773/recovered.edits
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.mkdirs(FSDirectory.java:933)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:154)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1153)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1002)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:407)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:512)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:968)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:962)
There were also quite a few messages that indicate the split was unsuccessful:
2010-09-22 22:06:45,430 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Split writer thread for region
user,\x00\x14w\xB0,1284527871890.752e96713b4408d77b750af06d0e2bad. got 57 to
process
2010-09-22 22:06:45,432 WARN org.apache.hadoop.hbase.regionserver.wal.HLog:
Found existing old edits file. It could be the result of a previous failed
split attempt. Deleting
hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/752e96713b4408d77b750af06d0e2bad/recovered.edits/0000000000077298635,
length=200866
Does this point to a particular problem?
Matthew
On Sep 22, 2010, at 3:52 PM, Jean-Daniel Cryans wrote:
> It seem that the log splitting didn't succeed (can't tell why, small
> log snippet is too small). Did it get stuck on EOF or something like
> that? In any case, it looks like a bug.
>
>> * how do I avoid disruption when this sort of thing happens
>
> Your region server was unavailable for 3 minutes, probably a GC pause
> (did you enable GC logging?), so the master splits the logs after it
> was notified via a ZK watcher. The timeout is 1 minute, you can set
> that lower if you want to answer to region server failure faster (but
> be more affected by shorter GC pauses or similar). You can also lower
> the recovery time by setting a smaller hbase.regionserver.maxlogs
> (which is 32 by default), at the expense of flushing a lot more.
>
>> * what is the best procedure for recovering from such a thing (i.e., is
>> there something to be done other than simply restarting the region server?)
>
> It's fatal if the master isn't able to complete log splitting
> properly, so restarting is your best bet until we can figure how you
> got stuck there.
>
> J-D
>
> On Wed, Sep 22, 2010 at 3:41 PM, Matthew LeMieux <[email protected]> wrote:
>> I've just had another of those region server shutdowns (RegionServerSuicide).
>>
>> * how do I avoid disruption when this sort of thing happens
>> * what is the best procedure for recovering from such a thing (i.e., is
>> there something to be done other than simply restarting the region server?)
>>
>> I've heard others talk of this event as being easy to recover from. I'd
>> like to understand how I can get into a configuration where that is the
>> case. Currently, an RS shutdown causes significant disruption. Clients are
>> unable to query HBase. Simply restarting the region server doesn't improve
>> things. In every case, I've needed to do a complete restart of the entire
>> cluster which isn't exactly super fast. A single region server shutdown
>> means a minimum of 1/2 hour down time, and sometimes that is more like hours
>> depending on the reason for the RS shutdown (once it took an entire day).
>> Is the experience similarly painful for others?
>>
>> Common error messages on the client are: java.lang.RuntimeException:
>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up
>> proxy to domU-12-31-39-18-1B-05.compute-1.internal/10.104.55.247:60020 after
>> attempts=1
>> ... and ... the RS complains with a "NotServingRegion" exception.
>>
>> As you can see form the selected messages below, after I restart the
>> affected RS, the cluster is still unusable, and gives no indication that
>> this state still change.
>>
>> It is of course no problem just restart a region server if it goes down
>> (either with a script or manually), the real question is how do I avoid
>> downtime?
>>
>> Details are below...
>>
>> Thank you for your help,
>>
>> Matthew
>>
>> I'm running in EC2 on DFS from CDH3b2, using a version of HBase from
>> 20100830.
>>
>> START OF THE SHUTDOWN ON AFFECTED RS:
>>
>> 2010-09-22 22:00:39,887 DEBUG org.apache.hadoop.hbase.regionserver.Store:
>> Major compaction triggered on store pg_payload; time since last major
>> compaction 93578344ms
>> 2010-09-22 22:00:39,887 INFO org.apache.hadoop.hbase.regionserver.Store:
>> Started compaction of 1 file(s) in pg_payload of
>> userh,\x1D\xAA2f\xE0\x1F\x12:\xA4\x98r\xFF!\xB8\x1A`,1285083827570.6be256da616fc20a6ac78a41952104e8.
>> into
>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/userh/6be256da616fc20a6ac78a41952104e8/.tmp,
>> sequenceid=77142618
>> 2010-09-22 22:03:23,317 DEBUG
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.57 GB,
>> free=396.14 MB, max=1.95 GB, blocks=27819, accesses=38513233, hits=14159798,
>> hitRatio=36.76%%, evictions=6494, evicted=20462105,
>> evictedPerRun=3150.924560546875
>> 2010-09-22 22:03:23,361 INFO org.apache.zookeeper.ClientCnxn: Client session
>> timed out, have not heard from server in 181390ms for sessionid
>> 0x12b266801550065, closing socket connection and attempting reconnect
>> 2010-09-22 22:03:23,363 INFO org.apache.zookeeper.ClientCnxn: Client session
>> timed out, have not heard from server in 183256ms for sessionid
>> 0x12b266801550002, closing socket connection and attempting reconnect
>> 2010-09-22 22:03:23,451 FATAL
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server
>> serverName=domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559,
>> load=(requests=4458, regions=134, usedHeap=8872, maxHeap=9995): Unhandled
>> exception
>> org.apache.hadoop.hbase.YouAreDeadException:
>> org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected;
>> currently processing
>> domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 as dead server
>> at
>> org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217)
>> at
>> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271)
>> at
>> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:753)
>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)
>>
>> THE RESTART ON THE AFFECTED RS:
>>
>> 2010-09-22 22:10:04,534 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020 exiting
>> 2010-09-22 22:13:23,449 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>> 4628396768048238482 lease expired
>> 2010-09-22 22:13:23,449 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>> 891312872292456529 lease expired
>> 2010-09-22 22:13:23,449 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>> 2977737069391109815 lease expired
>> 2010-09-22 22:13:23,450 INFO org.apache.hadoop.hbase.Leases:
>> regionserver60020.leaseChecker closing leases
>> 2010-09-22 22:13:23,450 INFO org.apache.hadoop.hbase.Leases:
>> regionserver60020.leaseChecker closed leases
>> 2010-09-22 22:13:23,450 INFO
>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting;
>> hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-9,5,main]
>> 2010-09-22 22:13:23,451 INFO
>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown hook
>> thread.
>> 2010-09-22 22:13:23,452 INFO
>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished.
>> Wed Sep 22 22:15:11 UTC 2010 Starting regionserver on domU-12-31-39-18-1B-05
>> 2010-09-22 22:15:12,716 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> vmInputArguments=[-Xmx768m, -XX:+HeapDumpOnOutOfMemoryError,
>> -XX:+UseConcMarkSweepGC, -XX:+CMSIncrementalMode,
>> -XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcMarkSweepGC,
>> -XX:+CMSIncrementalMode, -Xmx10000m, -Dhbase.log.dir=/mnt/tmp/hbaselog,
>> -Dhbase.log.file=hbase-mlcamus-regionserver-domU-12-31-39-18-1B-05.log,
>> -Dhbase.home.dir=/home/mlcamus/apps/hbase/bin/.., -Dhbase.id.str=mlcamus,
>> -Dhbase.root.logger=INFO,DRFA]
>> ...
>> 2010-09-22 22:25:03,530 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:04,288 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:05,299 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:05,348 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:06,308 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:06,358 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:07,368 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:07,545 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:08,318 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:08,377 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:08,548 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:08,552 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException;
>> user,^...@^tj�,1284527871890.bfee076d62bd372a70299f5854517145.
>> 2010-09-22 22:25:09,288 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:09,558 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> 2010-09-22 22:25:09,698 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> NotServingRegionException; .META.,,1
>> ... (these messages continue)
>>
>> ON THE MASTER WHEN THE RS WAS SHUTTING DOWN:
>>
>> 2010-09-22 21:59:51,588 INFO org.apache.hadoop.hbase.master.BaseScanner: All
>> 1 .META. region(s) scanned
>> 2010-09-22 22:00:22,931 INFO org.apache.hadoop.hbase.master.ServerManager: 2
>> region servers, 0 dead, average load 134.0
>> 2010-09-22 22:00:39,331 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.104.37.247:60020,
>> regionname: -ROOT-,,0.70236052, startKey: <>}
>> 2010-09-22 22:00:39,333 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
>> 10.104.37.247:60020, regionname: -ROOT-,,0.70236052, startKey: <>} complete
>> 2010-09-22 22:00:51,368 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {server: 10.104.55.247:60020,
>> regionname: .META.,,1.1028785192, startKey: <>}
>> 2010-09-22 22:00:51,801 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of 266 row(s) of meta region {server:
>> 10.104.55.247:60020, regionname: .META.,,1.1028785192, startKey: <>} complete
>> 2010-09-22 22:00:51,801 INFO org.apache.hadoop.hbase.master.BaseScanner: All
>> 1 .META. region(s) scanned
>> 2010-09-22 22:01:22,941 INFO org.apache.hadoop.hbase.master.ServerManager: 2
>> region servers, 0 dead, average load 134.0
>> 2010-09-22 22:01:39,341 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.104.37.247:60020,
>> regionname: -ROOT-,,0.70236052, startKey: <>}
>> 2010-09-22 22:01:39,343 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
>> 10.104.37.247:60020, regionname: -ROOT-,,0.70236052, startKey: <>} complete
>> 2010-09-22 22:01:51,371 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {server: 10.104.55.247:60020,
>> regionname: .META.,,1.1028785192, startKey: <>}
>> 2010-09-22 22:02:22,951 INFO org.apache.hadoop.hbase.master.ServerManager: 2
>> region servers, 0 dead, average load 134.0
>> 2010-09-22 22:02:24,203 INFO org.apache.hadoop.hbase.master.ServerManager:
>> domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 znode expired
>> 2010-09-22 22:02:24,204 DEBUG org.apache.hadoop.hbase.master.ServerManager:
>> Added=domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 to dead
>> servers, added shutdown processing operation
>> 2010-09-22 22:02:24,204 INFO org.apache.hadoop.hbase.master.RegionManager:
>> META region whose startkey is removed from onlineMetaRegions
>> 2010-09-22 22:02:24,204 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo:
>> ProcessServerShutdown of
>> domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559
>> 2010-09-22 22:02:24,204 INFO
>> org.apache.hadoop.hbase.master.RegionServerOperation: Process shutdown of
>> server domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559:
>> logSplit: false, rootRescanned: false, numberOfMetaRegions: 1,
>> onlineMetaRegions.size(): 0
>> 2010-09-22 22:02:24,242 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
>> Splitting 35 hlog(s) in
>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/.logs/domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559
>>
>> ON THE MASTER AFTER GIVING A FEW MINUTES TO RECOVER:
>>
>> 2010-09-22 22:31:00,300 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
>> 1, onlineMetaRegions.size(): 0
>> 2010-09-22 22:31:00,300 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because not
>> all meta regions are online
>> 2010-09-22 22:31:00,301 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo:
>> ProcessRegionClose of
>> userh,R\x80\xBD\x0F\xCB\xEAf\x95"+\xE1\x9A)/\xD7\x19,1285081795205.bf1a30fb67c1b0ea421da1f458d83c44.,
>> false, reassign: true
>> 2010-09-22 22:31:00,301 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
>> 1, onlineMetaRegions.size(): 0
>> 2010-09-22 22:31:00,301 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because not
>> all meta regions are online
>> 2010-09-22 22:31:10,310 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo:
>> ProcessRegionClose of
>> userh,R\x80\xBD\x0F\xCB\xEAf\x95"+\xE1\x9A)/\xD7\x19,1285081795205.bf1a30fb67c1b0ea421da1f458d83c44.,
>> false, reassign: true
>> 2010-09-22 22:31:10,310 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
>> 1, onlineMetaRegions.size(): 0
>> 2010-09-22 22:31:10,310 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because not
>> all meta regions are online
>> 2010-09-22 22:31:10,311 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo:
>> ProcessRegionClose of
>> userh,\xE5\x1A\x93\xCEQ\xC9\x8E\x8DIlvh9\xE1\xC8O,1285067882016.8aecae7491b95dd76aefab0058a71cff.,
>> false, reassign: true
>> 2010-09-22 22:31:10,311 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
>> 1, onlineMetaRegions.size(): 0
>> 2010-09-22 22:31:10,311 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because not
>> all meta regions are online
>> 2010-09-22 22:31:20,320 DEBUG
>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo:
>> ProcessRegionClose of
>> userh,\xE5\x1A\x93\xCEQ\xC9\x8E\x8DIlvh9\xE1\xC8O,1285067882016.8aecae7491b95dd76aefab00
>>
>> WHEN RUNNING "list" FROM "hbase shell":
>>
>> org.apache.hadoop.hbase.NotServingRegionException:
>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2217)
>> at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1808)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)
>>
>>
>>
