Hi Friends
I am using a mini distributed cluster of 3 nodes. Regionservers are running on
all of 3 nodes. This cluster is operational from 2 weeks now without any issue.
Today morning I saw one of the region servers are down.
I restarted the regionserver dynamically and it started fine and master also
distributed regions to this region server.
I looked at the log but not much understood what was the problem. Logs state
that one of my datanodes are bad, but when I checked, all datanodes are working
fine.
Please suggest
Regionserver Log :
2011-10-13 14:35:47,974 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog:
Could not append. Requesting close of hlog
java.io.IOException: Reflection
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986)
at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:937)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1477)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1386)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2549)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
... 10 more
Caused by: java.io.IOException: DFSOutputStream is closed
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3238)
at
org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
at
org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
... 14 more
2011-10-13 14:35:47,999 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs --
HDFS-200
2011-10-13 14:35:48,001 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
serverName=master,60020,1316669478524, load=(requests=0, regions=4,
usedHeap=31, maxHeap=993): IOE in log roller
java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2680)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371)
2011-10-13 14:35:48,002 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0,
memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31,
maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440,
blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9,
blockCacheEvictedCount=0, blockCacheHitRatio=99, blockCacheHitCachingRatio=99
2011-10-13 14:35:48,002 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log roller
2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
::DISCLAIMER::
-----------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended
for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates.
Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the
opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of
this message without the prior written consent of the author of this e-mail is
strictly prohibited. If you have
received this email in error please delete it and notify the sender
immediately. Before opening any mail and
attachments please check them for viruses and defect.
-----------------------------------------------------------------------------------------------------------------------
