Looks like a case that is not handled by the ITHbase: if the regionserver dies, master tries to split his logs but its not configured to use the THLogKeys...
Should be easy enough to fix, I'll try to look into it next week. But yeah I guess this is not the *root* problem because its caused by regionserver failure, but a case that ITHBase should handle.. -clint On Thu, Sep 17, 2009 at 4:25 AM, Lucas Nazário dos Santos < [email protected]> wrote: > Thanks, Clint. In fact, I've enabled the indexing (and not transactions) > with the following properties inside the hbase-site.xml file as described > in > > http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/client/tableindexed/package-summary.html > . > > <property> > <name>hbase.regionserver.class</name> > <value>org.apache.hadoop.hbase.ipc.IndexedRegionInterface</value> > </property> > <property> > <name>hbase.regionserver.impl</name> > > > <value>org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer</value> > </property> > > Any clue on what should I do to solve the "wrong key class" exception? > > Lucas > > > > On Wed, Sep 16, 2009 at 8:25 PM, Andrew Purtell <[email protected]> > wrote: > > > Good catch! It looks like there is more than one problem here. > > > > - Andy > > > > > > > > > > ________________________________ > > From: Clint Morgan <[email protected]> > > To: [email protected] > > Sent: Wednesday, September 16, 2009 2:43:22 PM > > Subject: Re: HBase cracks on EC2 > > > > This looks like the problem to me: > > > > java.io.IOException: wrong key class: > > org.apache.hadoop.hbase.regionserver.HLogKey is not class > > org.apache.hadoop.hbase.regionserver.transactional.THLogKey > > at > > > > Have you enable transactional hbase? Looks like you are using regular > HBase > > to recover a THBase log... > > > > > > On Wed, Sep 16, 2009 at 11:32 AM, Lucas Nazário dos Santos < > > [email protected]> wrote: > > > > > Thanks, Andy. I'll give it a shot. > > > > > > > > > On Wed, Sep 16, 2009 at 3:18 PM, Andrew Purtell <[email protected]> > > > wrote: > > > > > > > Hi Lucas, > > > > > > > > The log fragment you sent is actually after the problem occurred, and > > it > > > > does not look like 1777 but there's really insufficient information > to > > > say > > > > for sure. > > > > > > > > I find that the hypervisor steals 20-50% of CPU time on m1.large or > > > > m1.xlarge instance types as reported by top. I don't recommend any > > > instance > > > > type other than c1.xlarge (0% stolen as reported by top). Also I > would > > > > recommend a cluster of 3 c1.xlarge instances to start. Also please > have > > a > > > > look at the troubleshooting wiki and be sure to follow the advice for > > > items > > > > #5, #6, and #7. > > > > > > > > Best regards, > > > > > > > > - Andy > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > From: Lucas Nazário dos Santos <[email protected]> > > > > To: [email protected] > > > > Sent: Wednesday, September 16, 2009 10:48:53 AM > > > > Subject: Re: HBase cracks on EC2 > > > > > > > > Thanks, Andy. > > > > > > > > It's only a test with one large instance (7.5Gb of memory and 4 EC2 > > > compute > > > > units). I've already tried a cluster with 3 small instances but the > > same > > > > error occurred (I know small instances are not recommended and that > is > > > why > > > > I'm trying large instances now). Can it be the number of instances > I'm > > > > using? > > > > > > > > Hadoop logs are clear. > > > > > > > > The line bellow caught my attention. I'm wondering if the problem can > > be > > > > related to the issue http://issues.apache.org/jira/browse/HBASE-1777 > . > > > > > > > > 2009-09-16 09:02:59,846 FATAL > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: > > > > Replay of hlog required. Forcing server shutdown > > > > java.io.IOException: Server not running, aborting > > > > at org.apache.hadoop.hbase. > > > > regionserver.HRegionServer.checkOpen(HRegionServer.java:2299) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.exists(HRegionServer.java:1775) > > > > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown > Source) > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > Source) > > > > at java.lang.reflect.Method.invoke(Unknown Source) > > > > at > > > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) > > > > at > > > > > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > > > 2009-09-16 09:02:59,863 INFO > > > > org.apache.hadoop.hbase.regionserver.CompactSplitThread: > > > > regionserver/10.209.207.80:60020.compactor exiting > > > > > > > > Lucas > > > > > > > > > > > > > > > > > > > > On Wed, Sep 16, 2009 at 1:37 PM, Andrew Purtell <[email protected] > > > > > > wrote: > > > > > > > > > Lucas, > > > > > > > > > > > [...] ServerManager: 0 region servers, 1 dead, average load > > > > > > > > > > Is this a pseudo distributed test? I.e. everything running on one > EC2 > > > > > instance? > > > > > > > > > > What instance type are you using? > > > > > > > > > > The reported exceptions indicate a possible problem with DFS. Are > > there > > > > > exceptions in the DataNode logs? Or the NameNode log? > > > > > > > > > > Best regards, > > > > > > > > > > - Andy > > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > From: Lucas Nazário dos Santos <[email protected]> > > > > > To: [email protected] > > > > > Sent: Wednesday, September 16, 2009 6:21:27 AM > > > > > Subject: HBase cracks on EC2 > > > > > > > > > > Hi, > > > > > > > > > > I'm trying to migrate my solution to Amazon EC2 but the following > > > errors > > > > > crack HBase. I'm using HBase and Hadoop 0.20.0. Does anybody know > > what > > > > may > > > > > be the cause of it? > > > > > > > > > > Lucas > > > > > > > > > > > > > > > > > > > > HBASE MASTER > > > > > 2009-09-16 09:02:41,271 INFO > > > > org.apache.hadoop.hbase.master.ServerManager: > > > > > 1 > > > > > region servers, 0 dead, average load 6.0 > > > > > 2009-09-16 09:02:41,453 INFO > > > org.apache.hadoop.hbase.master.BaseScanner: > > > > > RegionManager.rootScanner scanning meta region {server: > > > > > 10.209.207.80:60020, > > > > > regionname: -ROOT-,,0, startKey: <>} > > > > > 2009-09-16 09:02:41,472 INFO > > > org.apache.hadoop.hbase.master.BaseScanner: > > > > > RegionManager.rootScanner scan of 1 row(s) of meta region {server: > > > > > 10.209.207.80:60020, regionname: -ROOT-,,0, startKey: <>} complete > > > > > 2009-09-16 09:02:41,553 INFO > > > org.apache.hadoop.hbase.master.BaseScanner: > > > > > RegionManager.metaScanner scanning meta region {server: > > > > > 10.209.207.80:60020, > > > > > regionname: .META.,,1, startKey: <>} > > > > > 2009-09-16 09:02:41,604 INFO > > > org.apache.hadoop.hbase.master.BaseScanner: > > > > > RegionManager.metaScanner scan of 4 row(s) of meta region {server: > > > > > 10.209.207.80:60020, regionname: .META.,,1, startKey: <>} complete > > > > > 2009-09-16 09:02:41,605 INFO > > > org.apache.hadoop.hbase.master.BaseScanner: > > > > > All > > > > > 1 .META. region(s) scanned > > > > > 2009-09-16 09:03:01,905 INFO > > > > org.apache.hadoop.hbase.master.ServerManager: > > > > > domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743 znode > > > > expired > > > > > 2009-09-16 09:03:01,921 INFO > > > > org.apache.hadoop.hbase.master.RegionManager: > > > > > -ROOT- region unset (but not set to be reassigned) > > > > > 2009-09-16 09:03:01,921 INFO > > > > org.apache.hadoop.hbase.master.RegionManager: > > > > > META region removed from onlineMetaRegions > > > > > 2009-09-16 09:03:01,923 INFO > > > > > org.apache.hadoop.hbase.master.RegionServerOperation: process > > shutdown > > > of > > > > > server > domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743: > > > > > logSplit: false, rootRescanned: false, numberOfMetaRegions: 1, > > > > > onlineMetaRegions.size(): 0 > > > > > 2009-09-16 09:03:01,935 INFO > > org.apache.hadoop.hbase.regionserver.HLog: > > > > > Splitting 2 hlog(s) in > > > > > > > > > > > > > > > > > > > > hdfs://domU-12-31-39-07-CC-A2.compute-1.internal:9000/hbase/.logs/domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743 > > > > > 2009-09-16 09:03:01,943 WARN > > org.apache.hadoop.hbase.regionserver.HLog: > > > > > Exception processing > > > > > > > > > > > > > > > > > > > > hdfs://domU-12-31-39-07-CC-A2.compute-1.internal:9000/hbase/.logs/domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743/hlog.dat.1253103102168 > > > > > -- continuing. Possible DATA LOSS! > > > > > java.io.IOException: wrong key class: > > > > > org.apache.hadoop.hbase.regionserver.HLogKey is not class > > > > > org.apache.hadoop.hbase.regionserver.transactional.THLogKey > > > > > at > > > > > > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1824) > > > > > at > > > > > > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876) > > > > > at > > > > org.apache.hadoop.hbase.regionserver.HLog.splitLog(HLog.java:880) > > > > > at > > > > org.apache.hadoop.hbase.regionserver.HLog.splitLog(HLog.java:802) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.master.ProcessServerShutdown.process(ProcessServerShutdown.java:274) > > > > > at > > > > > > > > > org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:492) > > > > > at > > org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:426) > > > > > 2009-09-16 09:03:01,951 WARN > > org.apache.hadoop.hbase.regionserver.HLog: > > > > > Exception processing > > > > > > > > > > > > > > > > > > > > hdfs://domU-12-31-39-07-CC-A2.compute-1.internal:9000/hbase/.logs/domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743/hlog.dat.1253105692784 > > > > > -- continuing. Possible DATA LOSS! > > > > > java.io.IOException: wrong key class: > > > > > org.apache.hadoop.hbase.regionserver.HLogKey is not class > > > > > org.apache.hadoop.hbase.regionserver.transactional.THLogKey > > > > > at > > > > > > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1824) > > > > > at > > > > > > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876) > > > > > at > > > > org.apache.hadoop.hbase.regionserver.HLog.splitLog(HLog.java:880) > > > > > at > > > > org.apache.hadoop.hbase.regionserver.HLog.splitLog(HLog.java:802) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.master.ProcessServerShutdown.process(ProcessServerShutdown.java:274) > > > > > at > > > > > > > > > org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:492) > > > > > at > > org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:426) > > > > > 2009-09-16 09:03:01,956 INFO > > org.apache.hadoop.hbase.regionserver.HLog: > > > > > hlog > > > > > file splitting completed in 29 millis for > > > > > > > > > > > > > > > > > > > > hdfs://domU-12-31-39-07-CC-A2.compute-1.internal:9000/hbase/.logs/domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743 > > > > > 2009-09-16 09:03:01,956 INFO > > > > > org.apache.hadoop.hbase.master.RegionServerOperation: Log split > > > complete, > > > > > meta reassignment and scanning: > > > > > 2009-09-16 09:03:01,956 INFO > > > > > org.apache.hadoop.hbase.master.RegionServerOperation: > > > > ProcessServerShutdown > > > > > reassigning ROOT region > > > > > 2009-09-16 09:03:01,956 INFO > > > > org.apache.hadoop.hbase.master.RegionManager: > > > > > -ROOT- region unset (but not set to be reassigned) > > > > > 2009-09-16 09:03:01,956 INFO > > > > org.apache.hadoop.hbase.master.RegionManager: > > > > > ROOT inserted into regionsInTransition > > > > > 2009-09-16 09:03:01,956 INFO > > > > > org.apache.hadoop.hbase.master.RegionServerOperation: > > > > ProcessServerShutdown > > > > > setting to unassigned: {server: 10.209.207.80:60020, regionname: > > > > > .META.,,1, > > > > > startKey: <>} > > > > > 2009-09-16 09:03:41,273 INFO > > > > org.apache.hadoop.hbase.master.ServerManager: > > > > > 0 > > > > > region servers, 1 dead, average load > > > > > NaN[domU-12-31-39-07-CC-A2.compute-1.internal,60020,1253103101743] > > > > > > > > > > > > > > > > > > > > HBASE REGIONSERVER > > > > > 2009-09-16 09:02:56,571 WARN org.apache.hadoop.hdfs.DFSClient: > > > > > NotReplicatedYetException sleeping > > > > > /hbase/news/698339833/charset/3497317581208054515 retries left 1 > > > > > 2009-09-16 09:02:59,844 WARN org.apache.hadoop.hdfs.DFSClient: > > > > DataStreamer > > > > > Exception: org.apache.hadoop.ipc.RemoteException: > > java.io.IOException: > > > > File > > > > > /hbase/news/698339833/charset/3497317581208054515 could only be > > > > replicated > > > > > to 0 nodes, instead of 1 > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1256) > > > > > at > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422) > > > > > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown > > Source) > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > Source) > > > > > at java.lang.reflect.Method.invoke(Unknown Source) > > > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > > > > > at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > > > > > at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > > > > > at java.security.AccessController.doPrivileged(Native > Method) > > > > > at javax.security.auth.Subject.doAs(Unknown Source) > > > > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > > > > > > > > > > at org.apache.hadoop.ipc.Client.call(Client.java:739) > > > > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > > > > > at $Proxy1.addBlock(Unknown Source) > > > > > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown > Source) > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > Source) > > > > > at java.lang.reflect.Method.invoke(Unknown Source) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > > > > > at $Proxy1.addBlock(Unknown Source) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234) > > > > > > > > > > 2009-09-16 09:02:59,844 WARN org.apache.hadoop.hdfs.DFSClient: > Error > > > > > Recovery for block null bad datanode[0] nodes == null > > > > > 2009-09-16 09:02:59,844 WARN org.apache.hadoop.hdfs.DFSClient: > Could > > > not > > > > > get > > > > > block locations. Source file > > > > > "/hbase/news/698339833/charset/3497317581208054515" - Aborting... > > > > > 17581208054515" - Aborting... > > > > > 2009-09-16 09:02:59,846 FATAL > > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of > hlog > > > > > required. Forcing server shutdown > > > > > org.apache.hadoop.hbase.DroppedSnapshotException: region: > > > > > news,,1253103842893 > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:950) > > > > > at > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:843) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149) > > > > > Caused by: org.apache.hadoop.ipc.RemoteException: > > java.io.IOException: > > > > File > > > > > /hbase/news/698339833/charset/3497317581208054515 could only be > > > > replicated > > > > > to 0 nodes, instead of 1 > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1256) > > > > > at > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422) > > > > > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown > > Source) > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > Source) > > > > > at java.lang.reflect.Method.invoke(Unknown Source) > > > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > > > > > at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > > > > > at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > > > > > at java.security.AccessController.doPrivileged(Native > Method) > > > > > at javax.security.auth.Subject.doAs(Unknown Source) > > > > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > > > > > > > > > > at org.apache.hadoop.ipc.Client.call(Client.java:739) > > > > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > > > > > at $Proxy1.addBlock(Unknown Source) > > > > > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown > Source) > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > Source) > > > > > at java.lang.reflect.Method.invoke(Unknown Source) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > > > > > at $Proxy1.addBlock(Unknown Source) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234) > > > > > 2009-09-16 09:02:59,849 INFO > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of > metrics: > > > > > request=0.0, regions=6, stores=27, storefiles=36, > > storefileIndexSize=0, > > > > > memstoreSize=0, usedHeap=195, maxHeap=996, blockCacheSize=82678352, > > > > > blockCacheFree=126368384, blockCacheCount=1168, > blockCacheHitRatio=99 > > > > > 2009-09-16 09:02:59,849 INFO > > > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: > > > > > regionserver/10.209.207.80:60020.cacheFlusher exiting > > > > > 2009-09-16 09:02:59,849 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 1086 on 60020, call exists([...@466057, row= > > > > > http://www.bloomberg.com/news/commentary/marinis.html#, > > maxVersions=1, > > > > > timeRange=[0,9223372036854775807), families=ALL) from > > > > 10.209.207.80:47866: > > > > > error: java.io.IOException: Server not running, aborting > > > > > java.io.IOException: Server not running, aborting > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2299) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.exists(HRegionServer.java:1775) > > > > > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown > > Source) > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > Source) > > > > > at java.lang.reflect.Method.invoke(Unknown Source) > > > > > at > > > > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) > > > > > at > > > > > > > > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > > > > 2009-09-16 09:02:59,863 INFO > > > > > org.apache.hadoop.hbase.regionserver.CompactSplitThread: > > > > > regionserver/10.209.207.80:60020.compactor exiting > > > > > 2009-09-16 09:03:00,027 INFO org.apache.hadoop.ipc.HBaseServer: > > > Stopping > > > > > server on 60020 > > > > > 2009-09-16 09:03:00,028 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 1 on 60020: exiting > > > > > 2009-09-16 09:03:00,039 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 1092 on 60020: exiting > > > > > 2009-09-16 09:03:00,042 INFO org.apache.hadoop.ipc.HBaseServer: > > > Stopping > > > > > IPC > > > > > Server listener on 60020 > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 3 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 4 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 7 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 8 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 9 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 10 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 18 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 19 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 20 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 22 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 23 on 60020: exiting > > > > > 2009-09-16 09:03:00,043 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 24 on 60020: exiting > > > > > 2009-09-16 09:03:00,044 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 25 on 60020: exiting > > > > > 2009-09-16 09:03:00,044 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 26 on 60020: exiting > > > > > 2009-09-16 09:03:00,044 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 27 on 60020: exiting > > > > > 2009-09-16 09:03:00,044 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 29 on 60020: exiting > > > > > 2009-09-16 09:03:00,044 INFO org.apache.hadoop.ipc.HBaseServer: IPC > > > > Server > > > > > handler 30 on 60020: exiting > > > > > 2009-09-16 09:03:01,898 INFO > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread > > > exiting > > > > > 2009-09-16 09:03:01,898 INFO org.apache.zookeeper.ZooKeeper: > Closing > > > > > session: 0x123c2c3d8bb0001 > > > > > 2009-09-16 09:03:01,899 INFO org.apache.zookeeper.ClientCnxn: > Closing > > > > > ClientCnxn for session: 0x123c2c3d8bb0001 > > > > > 2009-09-16 09:03:01,905 INFO org.apache.zookeeper.ClientCnxn: > > Exception > > > > > while closing send thread for session 0x123c2c3d8bb0001 : Read > error > > rc > > > = > > > > > -1 > > > > > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > > > > > 2009-09-16 09:03:02,008 INFO org.apache.zookeeper.ClientCnxn: > > > > Disconnecting > > > > > ClientCnxn for session: 0x123c2c3d8bb0001 > > > > > 2009-09-16 09:03:02,008 INFO org.apache.zookeeper.ZooKeeper: > Session: > > > > > 0x123c2c3d8bb0001 closed > > > > > 2009-09-16 09:03:02,008 INFO org.apache.zookeeper.ClientCnxn: > > > EventThread > > > > > shut down > > > > > 2009-09-16 09:03:02,110 INFO > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/ > > > > > 10.209.207.80:60020 exiting > > > > > 2009-09-16 09:03:03,603 INFO org.apache.hadoop.hbase.Leases: > > > > > regionserver/10.209.207.80:60020.leaseChecker closing leases > > > > > 2009-09-16 09:03:03,603 INFO org.apache.hadoop.hbase.Leases: > > > > > regionserver/10.209.207.80:60020.leaseChecker closed leases > > > > > 2009-09-16 09:03:03,604 INFO > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Starting > shutdown > > > > > thread. > > > > > 2009-09-16 09:03:03,604 INFO > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread > > > > > complete > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
