If an old recovered.edits was already there from a previous failure from before your upgrade on Sept. 5th then it's sufficient to explain the problem you encountered.
J-D On Sun, Sep 26, 2010 at 8:09 AM, Matthew LeMieux <[email protected]> wrote: > I took a look at the bug (https://issues.apache.org/jira/browse/HBASE-3027). > > The issue I'm experiencing does not appear to be a result of deploying a new > version. I've been on the same version since Sept 5th. > > -Matthew > > On Sep 22, 2010, at 9:01 PM, Stack wrote: > >> Could it be HBASE-3027? (Did you upgrade recently?) >> St.Ack >> >> On Wed, Sep 22, 2010 at 8:21 PM, Matthew LeMieux <[email protected]> wrote: >>> That would be an awfully long GC pause. GC logging was not enabled on that >>> particular machine (it is now). >>> >>> The only exception in that part of the log file, >>> "java.io.FileNotFoundException: Parent path is not a directory" occurred >>> quite a few times, although HDFS seems to think of itself as being healthy. >>> >>> >>> The master did some of this for a while: >>> >>> 2010-09-22 22:06:26,563 DEBUG >>> org.apache.hadoop.hbase.regionserver.wal.HLog: Split writer thread for >>> region user,\x00b\x7Ft,1281914827925.9c90ade2d19b8ce0b1613982eba2a673. got >>> 1 to process >>> 2010-09-22 22:06:26,580 INFO >>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using >>> syncFs -- HDFS-200 >>> 2010-09-22 22:06:26,580 DEBUG >>> org.apache.hadoop.hbase.regionserver.wal.HLog: Creating writer >>> path=hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/08b0e0f67fde365167627ad6fcbaa7b4/recovered.edits/0000000000077355008 >>> region=user,\x00W\x0D@,1281914368529.08b0e0f67fde365167627ad6fcbaa7b4. >>> 2010-09-22 22:06:26,580 DEBUG >>> org.apache.hadoop.hbase.regionserver.wal.HLog: Split writer thread for >>> region user,\x00W\x0D@,1281914368529.08b0e0f67fde365167627ad6fcbaa7b4. >>> Applied 1 total edits to >>> user,\x00W\x0D@,1281914368529.08b0e0f67fde365167627ad6fcbaa7b4. in 40ms >>> >>> That was mixed in with several of these: >>> >>> 2010-09-22 22:06:26,922 FATAL >>> org.apache.hadoop.hbase.regionserver.wal.HLog: Split writer thread for >>> region >>> user,\x01\xCB\xA8\x95,1281914501571.99a18a4a45748deb69fc6d7c804ffbe7. Got >>> while writing log entry to log >>> java.io.IOException: cannot get log writer >>> at >>> org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:551) >>> at >>> org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1610) >>> at >>> org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1586) >>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:619) >>> Caused by: org.apache.hadoop.ipc.RemoteException: >>> java.io.FileNotFoundException: Parent path is not a directory: >>> /hbase/user/99a18a4a45748deb69fc6d7c804ffbe7/recovered.edits >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.mkdirs(FSDirectory.java:933) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:154) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1153) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1002) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:407) >>> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:512) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:968) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:396) >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:962) >>> >>> And then a bunch of these: >>> >>> 2010-09-22 22:06:28,482 DEBUG >>> org.apache.hadoop.hbase.regionserver.wal.HLog: Closed >>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/.META./1028785192/recovered.edits/0000000000077298364 >>> 2010-09-22 22:06:28,515 DEBUG >>> org.apache.hadoop.hbase.regionserver.wal.HLog: Closed >>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/counters/692b9f45969bcc7ae2774b0e5fc17b12/recovered.edits/0000000000077298445 >>> 2010-09-22 22:06:29,431 DEBUG >>> org.apache.hadoop.hbase.regionserver.wal.HLog: Closed >>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/9097ba3b5f7b255e19861dc2131c8477/recovered.edits/0000000000077298636 >>> 2010-09-22 22:06:29,448 DEBUG >>> org.apache.hadoop.hbase.regionserver.wal.HLog: Closed >>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/dbc3619ce88ac8153334fb5f581c1575/recovered.edits/0000000000077298637 >>> 2010-09-22 22:06:29,464 DEBUG >>> org.apache.hadoop.hbase.regionserver.wal.HLog: Closed >>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/0f03f4ef25d647784ce69cd1e354af94/recovered.edits/0000000000077298666 >>> ... >>> 2010-09-22 22:06:30,614 WARN >>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Failed >>> processing: ProcessServerShutdown of >>> domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559; putting onto >>> delayed todo queue >>> java.io.IOException: java.io.IOException: cannot get log writer >>> at >>> org.apache.hadoop.hbase.regionserver.wal.HLog.writeEditsBatchToRegions(HLog.java:1510) >>> at >>> org.apache.hadoop.hbase.regionserver.wal.HLog.splitLog(HLog.java:1335) >>> at >>> org.apache.hadoop.hbase.regionserver.wal.HLog.splitLog(HLog.java:1210) >>> at >>> org.apache.hadoop.hbase.master.ProcessServerShutdown.process(ProcessServerShutdown.java:299) >>> at >>> org.apache.hadoop.hbase.master.RegionServerOperationQueue.process(RegionServerOperationQueue.java:147) >>> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:532) >>> Caused by: java.io.IOException: cannot get log writer >>> at >>> org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:551) >>> at >>> org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1610) >>> at >>> org.apache.hadoop.hbase.regionserver.wal.HLog$1.call(HLog.java:1586) >>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:619) >>> Caused by: org.apache.hadoop.ipc.RemoteException: >>> java.io.FileNotFoundException: Parent path is not a directory: >>> /hbase/user/2d225ad65207b0ab51b4604a9f889773/recovered.edits >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.mkdirs(FSDirectory.java:933) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:154) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1153) >>> at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1002) >>> at >>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:407) >>> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:512) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:968) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:396) >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:962) >>> >>> There were also quite a few messages that indicate the split was >>> unsuccessful: >>> >>> 2010-09-22 22:06:45,430 DEBUG >>> org.apache.hadoop.hbase.regionserver.wal.HLog: Split writer thread for >>> region user,\x00\x14w\xB0,1284527871890.752e96713b4408d77b750af06d0e2bad. >>> got 57 to process >>> 2010-09-22 22:06:45,432 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: >>> Found existing old edits file. It could be the result of a previous failed >>> split attempt. Deleting >>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/user/752e96713b4408d77b750af06d0e2bad/recovered.edits/0000000000077298635, >>> length=200866 >>> >>> Does this point to a particular problem? >>> >>> >>> Matthew >>> >>> >>> On Sep 22, 2010, at 3:52 PM, Jean-Daniel Cryans wrote: >>> >>>> It seem that the log splitting didn't succeed (can't tell why, small >>>> log snippet is too small). Did it get stuck on EOF or something like >>>> that? In any case, it looks like a bug. >>>> >>>>> * how do I avoid disruption when this sort of thing happens >>>> >>>> Your region server was unavailable for 3 minutes, probably a GC pause >>>> (did you enable GC logging?), so the master splits the logs after it >>>> was notified via a ZK watcher. The timeout is 1 minute, you can set >>>> that lower if you want to answer to region server failure faster (but >>>> be more affected by shorter GC pauses or similar). You can also lower >>>> the recovery time by setting a smaller hbase.regionserver.maxlogs >>>> (which is 32 by default), at the expense of flushing a lot more. >>>> >>>>> * what is the best procedure for recovering from such a thing (i.e., is >>>>> there something to be done other than simply restarting the region >>>>> server?) >>>> >>>> It's fatal if the master isn't able to complete log splitting >>>> properly, so restarting is your best bet until we can figure how you >>>> got stuck there. >>>> >>>> J-D >>>> >>>> On Wed, Sep 22, 2010 at 3:41 PM, Matthew LeMieux <[email protected]> >>>> wrote: >>>>> I've just had another of those region server shutdowns >>>>> (RegionServerSuicide). >>>>> >>>>> * how do I avoid disruption when this sort of thing happens >>>>> * what is the best procedure for recovering from such a thing (i.e., is >>>>> there something to be done other than simply restarting the region >>>>> server?) >>>>> >>>>> I've heard others talk of this event as being easy to recover from. I'd >>>>> like to understand how I can get into a configuration where that is the >>>>> case. Currently, an RS shutdown causes significant disruption. Clients >>>>> are unable to query HBase. Simply restarting the region server doesn't >>>>> improve things. In every case, I've needed to do a complete restart of >>>>> the entire cluster which isn't exactly super fast. A single region >>>>> server shutdown means a minimum of 1/2 hour down time, and sometimes that >>>>> is more like hours depending on the reason for the RS shutdown (once it >>>>> took an entire day). Is the experience similarly painful for others? >>>>> >>>>> Common error messages on the client are: java.lang.RuntimeException: >>>>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >>>>> up proxy to domU-12-31-39-18-1B-05.compute-1.internal/10.104.55.247:60020 >>>>> after attempts=1 >>>>> ... and ... the RS complains with a "NotServingRegion" exception. >>>>> >>>>> As you can see form the selected messages below, after I restart the >>>>> affected RS, the cluster is still unusable, and gives no indication that >>>>> this state still change. >>>>> >>>>> It is of course no problem just restart a region server if it goes down >>>>> (either with a script or manually), the real question is how do I avoid >>>>> downtime? >>>>> >>>>> Details are below... >>>>> >>>>> Thank you for your help, >>>>> >>>>> Matthew >>>>> >>>>> I'm running in EC2 on DFS from CDH3b2, using a version of HBase from >>>>> 20100830. >>>>> >>>>> START OF THE SHUTDOWN ON AFFECTED RS: >>>>> >>>>> 2010-09-22 22:00:39,887 DEBUG org.apache.hadoop.hbase.regionserver.Store: >>>>> Major compaction triggered on store pg_payload; time since last major >>>>> compaction 93578344ms >>>>> 2010-09-22 22:00:39,887 INFO org.apache.hadoop.hbase.regionserver.Store: >>>>> Started compaction of 1 file(s) in pg_payload of >>>>> userh,\x1D\xAA2f\xE0\x1F\x12:\xA4\x98r\xFF!\xB8\x1A`,1285083827570.6be256da616fc20a6ac78a41952104e8. >>>>> into >>>>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/userh/6be256da616fc20a6ac78a41952104e8/.tmp, >>>>> sequenceid=77142618 >>>>> 2010-09-22 22:03:23,317 DEBUG >>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.57 GB, >>>>> free=396.14 MB, max=1.95 GB, blocks=27819, accesses=38513233, >>>>> hits=14159798, hitRatio=36.76%%, evictions=6494, evicted=20462105, >>>>> evictedPerRun=3150.924560546875 >>>>> 2010-09-22 22:03:23,361 INFO org.apache.zookeeper.ClientCnxn: Client >>>>> session timed out, have not heard from server in 181390ms for sessionid >>>>> 0x12b266801550065, closing socket connection and attempting reconnect >>>>> 2010-09-22 22:03:23,363 INFO org.apache.zookeeper.ClientCnxn: Client >>>>> session timed out, have not heard from server in 183256ms for sessionid >>>>> 0x12b266801550002, closing socket connection and attempting reconnect >>>>> 2010-09-22 22:03:23,451 FATAL >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region >>>>> server >>>>> serverName=domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559, >>>>> load=(requests=4458, regions=134, usedHeap=8872, maxHeap=9995): Unhandled >>>>> exception >>>>> org.apache.hadoop.hbase.YouAreDeadException: >>>>> org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; >>>>> currently processing >>>>> domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 as dead >>>>> server >>>>> at >>>>> org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217) >>>>> at >>>>> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271) >>>>> at >>>>> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:753) >>>>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919) >>>>> >>>>> THE RESTART ON THE AFFECTED RS: >>>>> >>>>> 2010-09-22 22:10:04,534 INFO >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020 >>>>> exiting >>>>> 2010-09-22 22:13:23,449 INFO >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>> 4628396768048238482 lease expired >>>>> 2010-09-22 22:13:23,449 INFO >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>> 891312872292456529 lease expired >>>>> 2010-09-22 22:13:23,449 INFO >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner >>>>> 2977737069391109815 lease expired >>>>> 2010-09-22 22:13:23,450 INFO org.apache.hadoop.hbase.Leases: >>>>> regionserver60020.leaseChecker closing leases >>>>> 2010-09-22 22:13:23,450 INFO org.apache.hadoop.hbase.Leases: >>>>> regionserver60020.leaseChecker closed leases >>>>> 2010-09-22 22:13:23,450 INFO >>>>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook >>>>> starting; hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-9,5,main] >>>>> 2010-09-22 22:13:23,451 INFO >>>>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown >>>>> hook thread. >>>>> 2010-09-22 22:13:23,452 INFO >>>>> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished. >>>>> Wed Sep 22 22:15:11 UTC 2010 Starting regionserver on >>>>> domU-12-31-39-18-1B-05 >>>>> 2010-09-22 22:15:12,716 INFO >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> vmInputArguments=[-Xmx768m, -XX:+HeapDumpOnOutOfMemoryError, >>>>> -XX:+UseConcMarkSweepGC, -XX:+CMSIncrementalMode, >>>>> -XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcMarkSweepGC, >>>>> -XX:+CMSIncrementalMode, -Xmx10000m, -Dhbase.log.dir=/mnt/tmp/hbaselog, >>>>> -Dhbase.log.file=hbase-mlcamus-regionserver-domU-12-31-39-18-1B-05.log, >>>>> -Dhbase.home.dir=/home/mlcamus/apps/hbase/bin/.., -Dhbase.id.str=mlcamus, >>>>> -Dhbase.root.logger=INFO,DRFA] >>>>> ... >>>>> 2010-09-22 22:25:03,530 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:04,288 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:05,299 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:05,348 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:06,308 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:06,358 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:07,368 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:07,545 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:08,318 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:08,377 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:08,548 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:08,552 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; >>>>> user,^...@^tj�,1284527871890.bfee076d62bd372a70299f5854517145. >>>>> 2010-09-22 22:25:09,288 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:09,558 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> 2010-09-22 22:25:09,698 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> NotServingRegionException; .META.,,1 >>>>> ... (these messages continue) >>>>> >>>>> ON THE MASTER WHEN THE RS WAS SHUTTING DOWN: >>>>> >>>>> 2010-09-22 21:59:51,588 INFO org.apache.hadoop.hbase.master.BaseScanner: >>>>> All 1 .META. region(s) scanned >>>>> 2010-09-22 22:00:22,931 INFO >>>>> org.apache.hadoop.hbase.master.ServerManager: 2 region servers, 0 dead, >>>>> average load 134.0 >>>>> 2010-09-22 22:00:39,331 INFO org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scanning meta region {server: >>>>> 10.104.37.247:60020, regionname: -ROOT-,,0.70236052, startKey: <>} >>>>> 2010-09-22 22:00:39,333 INFO org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {server: >>>>> 10.104.37.247:60020, regionname: -ROOT-,,0.70236052, startKey: <>} >>>>> complete >>>>> 2010-09-22 22:00:51,368 INFO org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scanning meta region {server: >>>>> 10.104.55.247:60020, regionname: .META.,,1.1028785192, startKey: <>} >>>>> 2010-09-22 22:00:51,801 INFO org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scan of 266 row(s) of meta region {server: >>>>> 10.104.55.247:60020, regionname: .META.,,1.1028785192, startKey: <>} >>>>> complete >>>>> 2010-09-22 22:00:51,801 INFO org.apache.hadoop.hbase.master.BaseScanner: >>>>> All 1 .META. region(s) scanned >>>>> 2010-09-22 22:01:22,941 INFO >>>>> org.apache.hadoop.hbase.master.ServerManager: 2 region servers, 0 dead, >>>>> average load 134.0 >>>>> 2010-09-22 22:01:39,341 INFO org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scanning meta region {server: >>>>> 10.104.37.247:60020, regionname: -ROOT-,,0.70236052, startKey: <>} >>>>> 2010-09-22 22:01:39,343 INFO org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {server: >>>>> 10.104.37.247:60020, regionname: -ROOT-,,0.70236052, startKey: <>} >>>>> complete >>>>> 2010-09-22 22:01:51,371 INFO org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scanning meta region {server: >>>>> 10.104.55.247:60020, regionname: .META.,,1.1028785192, startKey: <>} >>>>> 2010-09-22 22:02:22,951 INFO >>>>> org.apache.hadoop.hbase.master.ServerManager: 2 region servers, 0 dead, >>>>> average load 134.0 >>>>> 2010-09-22 22:02:24,203 INFO >>>>> org.apache.hadoop.hbase.master.ServerManager: >>>>> domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 znode >>>>> expired >>>>> 2010-09-22 22:02:24,204 DEBUG >>>>> org.apache.hadoop.hbase.master.ServerManager: >>>>> Added=domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 to >>>>> dead servers, added shutdown processing operation >>>>> 2010-09-22 22:02:24,204 INFO >>>>> org.apache.hadoop.hbase.master.RegionManager: META region whose startkey >>>>> is removed from onlineMetaRegions >>>>> 2010-09-22 22:02:24,204 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing >>>>> todo: ProcessServerShutdown of >>>>> domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 >>>>> 2010-09-22 22:02:24,204 INFO >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: Process shutdown of >>>>> server domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559: >>>>> logSplit: false, rootRescanned: false, numberOfMetaRegions: 1, >>>>> onlineMetaRegions.size(): 0 >>>>> 2010-09-22 22:02:24,242 INFO >>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 35 hlog(s) in >>>>> hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/.logs/domU-12-31-39-18-1B-05.compute-1.internal,60020,1284840302559 >>>>> >>>>> ON THE MASTER AFTER GIVING A FEW MINUTES TO RECOVER: >>>>> >>>>> 2010-09-22 22:31:00,300 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: >>>>> numberOfMetaRegions: 1, onlineMetaRegions.size(): 0 >>>>> 2010-09-22 22:31:00,300 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because >>>>> not all meta regions are online >>>>> 2010-09-22 22:31:00,301 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing >>>>> todo: ProcessRegionClose of >>>>> userh,R\x80\xBD\x0F\xCB\xEAf\x95"+\xE1\x9A)/\xD7\x19,1285081795205.bf1a30fb67c1b0ea421da1f458d83c44., >>>>> false, reassign: true >>>>> 2010-09-22 22:31:00,301 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: >>>>> numberOfMetaRegions: 1, onlineMetaRegions.size(): 0 >>>>> 2010-09-22 22:31:00,301 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because >>>>> not all meta regions are online >>>>> 2010-09-22 22:31:10,310 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing >>>>> todo: ProcessRegionClose of >>>>> userh,R\x80\xBD\x0F\xCB\xEAf\x95"+\xE1\x9A)/\xD7\x19,1285081795205.bf1a30fb67c1b0ea421da1f458d83c44., >>>>> false, reassign: true >>>>> 2010-09-22 22:31:10,310 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: >>>>> numberOfMetaRegions: 1, onlineMetaRegions.size(): 0 >>>>> 2010-09-22 22:31:10,310 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because >>>>> not all meta regions are online >>>>> 2010-09-22 22:31:10,311 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing >>>>> todo: ProcessRegionClose of >>>>> userh,\xE5\x1A\x93\xCEQ\xC9\x8E\x8DIlvh9\xE1\xC8O,1285067882016.8aecae7491b95dd76aefab0058a71cff., >>>>> false, reassign: true >>>>> 2010-09-22 22:31:10,311 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: >>>>> numberOfMetaRegions: 1, onlineMetaRegions.size(): 0 >>>>> 2010-09-22 22:31:10,311 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: Requeuing because >>>>> not all meta regions are online >>>>> 2010-09-22 22:31:20,320 DEBUG >>>>> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing >>>>> todo: ProcessRegionClose of >>>>> userh,\xE5\x1A\x93\xCEQ\xC9\x8E\x8DIlvh9\xE1\xC8O,1285067882016.8aecae7491b95dd76aefab00 >>>>> >>>>> WHEN RUNNING "list" FROM "hbase shell": >>>>> >>>>> org.apache.hadoop.hbase.NotServingRegionException: >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 >>>>> at >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2217) >>>>> at >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1808) >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> at >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919) >>>>> >>>>> >>>>> >>> >>> > >
