Hello,
we have a small architecture of 4 servers with 1
Namenode/Jobtracker/HbaseMaster, 2 Datanode/Tasktracker, 1 server
Failover Namenode/Jobtracker.
We often Hbase crashes with this error:
DEBUG org.apache.hadoop.ipc.HBaseServer: got #385
2012-03-26 15:15:08,690 DEBUG org.apache.hadoop.ipc.HBaseServer:
IPC Server handler 3 on 48895: has #385 from 172.16.0.1:49493
2012-03-26 15:15:08,690 DEBUG org.apache.hadoop.ipc.HBaseServer:
Served: regionServerReport queueTime= 0 procesingTime= 0
2012-03-26 15:15:08,691 DEBUG org.apache.hadoop.ipc.HBaseServer:
IPC Server Responder: responding to #385 from 172.16.0.1:49493
2012-03-26 15:15:08,691 DEBUG org.apache.hadoop.ipc.HBaseServer:
IPC Server Responder: responding to #385 from 172.16.0.1:49493
Wrote 8 bytes.
2012-03-26 15:15:08,691 DEBUG org.apache.hadoop.ipc.HBaseClient:
IPC Client (47) connection to nm1.pus2011.com/172.16.0.1:48895
from an unknown user got value #385
2012-03-26 15:15:08,691 DEBUG org.apache.hadoop.ipc.HbaseRPC:
Call: regionServerReport 2
2012-03-26 15:15:08,941 DEBUG org.apache.hadoop.ipc.Client: IPC
Client (47) connection to nm.pus2011.com/172.16.0.3:9000 from
hbase: closed
2012-03-26 15:15:08,941 DEBUG org.apache.hadoop.ipc.Client: IPC
Client (47) connection to nm.pus2011.com/172.16.0.3:9000 from
hbase: stopped, remaining connections 0
2012-03-26 15:15:11,043 DEBUG org.apache.hadoop.ipc.HBaseServer:
Served: next queueTime= 41371 procesingTime= 43542
2012-03-26 15:15:11,043 DEBUG org.apache.hadoop.ipc.HBaseServer:
IPC Server Responder: responding to #7 from 172.16.0.5:38633
2012-03-26 15:15:11,043 WARN org.apache.hadoop.ipc.HBaseServer:
IPC Server Responder, call next(-8649074184087149864, 1) from
172.16.0.5:38633: output error
2012-03-26 15:15:11,043 WARN org.apache.hadoop.ipc.HBaseServer:
IPC Server handler 7 on 58237 caught:
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:249)
at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:440)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1341)
at org.apache.hadoop.hbase.ipc.HBaseServer
$Responder.processResponse(HBaseServer.java:727)
at org.apache.hadoop.hbase.ipc.HBaseServer
$Responder.doRespond(HBaseServer.java:792)
at org.apache.hadoop.hbase.ipc.HBaseServer
$Handler.run(HBaseServer.java:1083)
Can you help us ?
Best regards
Simon Gilliot
