Hi St.Ack,
Please see the attach file, and there are 3 RS/DN/TT + 1 MS/NN/JT in my
cluster. (Hadoop-0.20.2, HBase 0.20.6)
Thanks.
Shen
On Wed, Dec 29, 2010 at 1:34 PM, Stack <[email protected]> wrote:
> On Tue, Dec 28, 2010 at 6:59 PM, ChingShen <[email protected]>
> wrote:
> > But I want to know why the HBase master sends a MSG_REGIONSERVER_STOP op
> to
> > region sever to stop its services rather than reinitialize a new
> zookeeper
> > client or restart region server?
> >
>
> Can I see more regionserver log? If session expired, I don't a
> MSG_REGIONSERVER_STOP should be involved. And what version of hbase?
>
> Thanks,
> St.Ack
>
2010-12-13 18:09:18,954 DEBUG org.apache.hadoop.hbase.regionserver.LogRoller:
Hlog roll period 3600000ms elapsed
2010-12-13 18:09:21,693 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292231357240,
entries=60542, calcsize=18765414, filesize=11843170. New hlog
/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292234961106
2010-12-13 18:09:21,693 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Last
sequence written is empty. Deleting all old hlogs
2010-12-13 18:09:21,693 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292231349250 whose
highest sequence/edit id is 223790657
2010-12-13 18:09:21,695 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292231354175 whose
highest sequence/edit id is 223996332
2010-12-13 18:09:21,779 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292231357240 whose
highest sequence/edit id is 224056869
2010-12-13 18:23:21,827 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP
2010-12-13 18:23:23,091 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server
on 60020
2010-12-13 18:23:23,092 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020
2010-12-13 18:23:23,365 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 18 on 60020: exiting
2010-12-13 18:23:23,365 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 23 on 60020: exiting
2010-12-13 18:23:23,365 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2010-12-13 18:23:23,365 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 13 on 60020: exiting
2010-12-13 18:23:23,365 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 17 on 60020: exiting
2010-12-13 18:23:23,365 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 16 on 60020: exiting
2010-12-13 18:23:23,697 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 15 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 11 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 12 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 21 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 19 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 22 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 20 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 14 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2010-12-13 18:23:27,629 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 24 on 60020: exiting
2010-12-13 18:23:27,850 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2010-12-13 18:23:27,850 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2010-12-13 18:23:28,187 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.1.57:60020.cacheFlusher exiting
2010-12-13 18:23:28,541 INFO org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.1.57:60020.logFlusher exiting
2010-12-13 18:23:36,806 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.1.57:60020.majorCompactionChecker exiting
2010-12-13 18:23:36,981 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2010-12-13 18:23:38,265 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.1.57:60020.compactor exiting
2010-12-13 18:24:05,745 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain
block blk_-17314133620670477_23702 from any node: java.io.IOException: No live
nodes contain current block
2010-12-13 18:24:21,069 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 3007518277568288672
lease expired
2010-12-13 18:24:21,069 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.1.57:60020.leaseChecker closing leases
2010-12-13 18:24:21,069 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.1.57:60020.leaseChecker closed leases
2010-12-13 18:24:29,940 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: Server not running
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1883)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-12-13 18:24:29,940 WARN org.apache.zookeeper.ClientCnxn: Exception closing
session 0x12cc3d60a5b000f to sun.nio.ch.selectionkeyi...@7f0e917a
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2010-12-13 18:24:29,940 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.io.IOException: Server not running
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1883)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-12-13 18:24:29,940 WARN org.apache.zookeeper.ClientCnxn: Exception closing
session 0x12cc3d60a5b0002 to sun.nio.ch.selectionkeyi...@15e232b5
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2010-12-13 18:24:33,361 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher: Got
ZooKeeper event, state: Disconnected, type: None, path: null
2010-12-13 18:24:33,450 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state:
Disconnected, type: None, path: null
2010-12-13 18:24:33,571 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Ignoring ZooKeeper event
while shutting down
purge private log....
2010-12-13 18:24:34,131 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
PhotoIndex_XY,,1285833897239
2010-12-13 18:24:34,131 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No
more row locks outstanding on region PhotoIndex_XY,,1285833897239
2010-12-13 18:24:34,135 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call openScanner([...@318cc625,
startRow=8_64_db5bcbc2-474c-4205-8f82-093408c2ced7,
stopRow=8_93_4890510e-6502-49d2-b4c9-703e035a05bc, maxVersions=1, caching=-1,
cacheBlocks=true, timeRange=[0,9223372036854775807), families={(family=gps,
columns={}), (family=metadata, columns={}}) from 192.168.1.193:38148: output
error
2010-12-13 18:24:34,418 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 10 on 60020 caught: java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1125)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:615)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:679)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:943)
2010-12-13 18:24:34,418 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 10 on 60020: exiting
2010-12-13 18:24:34,447 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server clusterPC1/192.168.1.224:2181
2010-12-13 18:24:34,570 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected
local=/192.168.1.57:42967 remote=clusterPC1/192.168.1.224:2181]
2010-12-13 18:24:34,946 WARN org.apache.zookeeper.ClientCnxn: Exception closing
session 0x12cc3d60a5b0002 to sun.nio.ch.selectionkeyi...@3f588025
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2010-12-13 18:24:34,946 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state:
Expired, type: None, path: null
2010-12-13 18:24:35,372 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Ignoring ZooKeeper event
while shutting down
purge private log....
2010-12-13 18:24:35,433 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher: Got
ZooKeeper event, state: Expired, type: None, path: null
2010-12-13 18:24:35,433 WARN org.apache.zookeeper.ClientCnxn: Exception closing
session 0x12cc3d60a5b000f to sun.nio.ch.selectionkeyi...@4b451da0
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2010-12-13 18:24:36,135 INFO org.apache.zookeeper.ZooKeeper: Closing session:
0x12cc3d60a5b000f
2010-12-13 18:24:36,986 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x12cc3d60a5b000f
purge private log....
2010-12-13 18:24:49,501 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x12cc3d60a5b000f
2010-12-13 18:24:49,538 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
closing hlog writer in
hdfs://clusterPC1:54001/hbase20_6/.logs/clusterPC3,60020,1291775815759
2010-12-13 18:24:49,784 INFO org.apache.zookeeper.ZooKeeper: Session:
0x12cc3d60a5b000f closed
2010-12-13 18:24:49,784 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Closed connection with
ZooKeeper
2010-12-13 18:24:49,784 INFO org.apache.zookeeper.ClientCnxn: EventThread shut
down
2010-12-13 18:25:33,346 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292234961106 File
does not exist. Holder DFSClient_-724890119 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1332)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1323)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1251)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2937)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2819)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
2010-12-13 18:25:33,415 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
for block null bad datanode[0] nodes == null
2010-12-13 18:25:33,416 WARN org.apache.hadoop.hdfs.DFSClient: Could not get
block locations. Source file
"/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292234961106" -
Aborting...
2010-12-13 18:25:33,462 ERROR org.apache.hadoop.hdfs.DFSClient: Exception
closing file
/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292234961106 :
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292234961106 File
does not exist. Holder DFSClient_-724890119 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1332)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1323)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1251)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292234961106 File
does not exist. Holder DFSClient_-724890119 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1332)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1323)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1251)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2937)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2819)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
2010-12-13 18:25:33,818 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Close and delete failed
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase20_6/.logs/clusterPC3,60020,1291775815759/hlog.dat.1292234961106 File
does not exist. Holder DFSClient_-724890119 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1332)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1323)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1251)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:627)
at java.lang.Thread.run(Thread.java:619)
2010-12-13 18:25:33,818 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: telling master that region
server is shutting down at: 192.168.1.57:60020
2010-12-13 18:25:33,907 INFO org.apache.hadoop.fs.FileSystem: Could not cancel
cleanup thread, though no FileSystems are open
2010-12-13 18:25:33,962 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutting down
HRegionServer: file system not available
java.io.IOException: File system is not available
at
org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:126)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkFileSystem(HRegionServer.java:902)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:850)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:833)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1921)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.io.IOException: Call to clusterPC1/192.168.1.224:54001 failed
on local exception: java.nio.channels.ClosedByInterruptException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.getFileInfo(Unknown Source)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:619)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:648)
at
org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:115)
... 9 more
Caused by: java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341)
at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:480)
at org.apache.hadoop.ipc.Client.call(Client.java:721)
... 21 more
2010-12-13 18:25:33,962 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutting down
HRegionServer: file system not available
java.io.IOException: File system is not available
at
org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:126)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkFileSystem(HRegionServer.java:902)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:850)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.cleanup(HRegionServer.java:833)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1921)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.io.IOException: Call to clusterPC1/192.168.1.224:54001 failed
on local exception: java.nio.channels.ClosedByInterruptException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.getFileInfo(Unknown Source)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:619)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:648)
at
org.apache.hadoop.hbase.util.FSUtils.checkFileSystemAvailable(FSUtils.java:115)
... 9 more
Caused by: java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341)
at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:480)
at org.apache.hadoop.ipc.Client.call(Client.java:721)
... 21 more
2010-12-13 18:25:34,305 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=41, stores=148, storefiles=145, storefileIndexSize=9,
memstoreSize=0, compactionQueueSize=0, usedHeap=715, maxHeap=3111,
blockCacheSize=0, blockCacheFree=0, blockCacheCount=0, blockCacheHitRatio=0,
fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
2010-12-13 18:25:34,398 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020, call next(6407567603869720194, 1) from 192.168.1.205:40676:
error: java.io.IOException: Server not running
java.io.IOException: Server not running
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1883)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-12-13 18:25:34,398 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call next(6407567603869720194, 1) from 192.168.1.205:40676: output
error
2010-12-13 18:25:34,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020 caught: java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1125)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:615)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:679)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:943)
2010-12-13 18:25:34,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2010-12-13 18:25:36,544 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=41, stores=148, storefiles=145, storefileIndexSize=9,
memstoreSize=0, compactionQueueSize=0, usedHeap=715, maxHeap=3111,
blockCacheSize=0, blockCacheFree=0, blockCacheCount=0, blockCacheHitRatio=0,
fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
2010-12-13 18:25:36,586 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server at:
192.168.1.57:60020
2010-12-13 18:25:36,586 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020, call next(-5223221819126724420, 1) from
192.168.1.205:40680: error: java.io.IOException: Server not running
java.io.IOException: Server not running
at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1883)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-12-13 18:25:36,586 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call next(-5223221819126724420, 1) from 192.168.1.205:40680: output
error
2010-12-13 18:25:36,695 INFO org.apache.zookeeper.ZooKeeper: Closing session:
0x12cc3d60a5b0002
2010-12-13 18:25:36,695 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x12cc3d60a5b0002
2010-12-13 18:25:36,695 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x12cc3d60a5b0002
2010-12-13 18:25:36,695 INFO org.apache.zookeeper.ZooKeeper: Session:
0x12cc3d60a5b0002 closed
2010-12-13 18:25:36,695 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Closed connection with
ZooKeeper
2010-12-13 18:25:36,695 INFO org.apache.zookeeper.ClientCnxn: EventThread shut
down
2010-12-13 18:25:36,695 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020 caught: java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1125)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:615)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:679)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:943)
2010-12-13 18:25:36,695 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2010-12-13 18:25:37,430 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer:
regionserver/192.168.1.57:60020 exiting
2010-12-13 18:25:47,447 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown thread.
2010-12-13 18:25:47,447 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
