Hi Lars, I am using Hbase 0.90.3 and there was no inserting done on the nodes at that time.
-----Original Message----- From: lars hofhansl [mailto:[email protected]] Sent: Friday, October 14, 2011 10:46 AM To: [email protected] Subject: Re: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append Did you do some heavy inserting during that time? I assume you use 0.90.x(?) Might be HBASE-4387. -- Lars ----- Original Message ----- From: Stuti Awasthi <[email protected]> To: "[email protected]" <[email protected]> Cc: Sent: Thursday, October 13, 2011 9:44 PM Subject: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append Hi Friends I am using a mini distributed cluster of 3 nodes. Regionservers are running on all of 3 nodes. This cluster is operational from 2 weeks now without any issue. Today morning I saw one of the region servers are down. I restarted the regionserver dynamically and it started fine and master also distributed regions to this region server. I looked at the log but not much understood what was the problem. Logs state that one of my datanodes are bad, but when I checked, all datanodes are working fine. Please suggest Regionserver Log : 2011-10-13 14:35:47,974 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting close of hlog java.io.IOException: Reflection at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147) at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986) at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:937) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1477) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1386) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2549) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145) ... 10 more Caused by: java.io.IOException: DFSOutputStream is closed at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3238) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97) at org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944) ... 14 more 2011-10-13 14:35:47,999 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200 2011-10-13 14:35:48,001 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=master,60020,1316669478524, load=(requests=0, regions=4, usedHeap=31, maxHeap=993): IOE in log roller java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2680) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371) 2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0, memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31, maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440, blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9, blockCacheEvictedCount=0, blockCacheHitRatio=99, blockCacheHitCachingRatio=99 2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log roller 2011-10-13 14:35:48,002 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting. ::DISCLAIMER:: ----------------------------------------------------------------------------------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any mail and attachments please check them for viruses and defect. -----------------------------------------------------------------------------------------------------------------------
