Also make sure this isn't an exception that came after a bunch of others, look for the first FATAL.
J-D On Fri, Oct 14, 2011 at 2:25 AM, Ramkrishna S Vasudevan < [email protected]> wrote: > Hi > > Are you sure there was no insert. The stack trace shows a put operation > was > going on. You can check your datanode logs at the corresponding time when > this exception occured in the RS to see if any write request has reached > the > DN and that you can find using the HLog file name for that RS. > > Regards > Ram > > -----Original Message----- > From: Stuti Awasthi [mailto:[email protected]] > Sent: Friday, October 14, 2011 2:47 PM > To: [email protected] > Subject: RE: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not > append > > Hi Lars, > I am using Hbase 0.90.3 and there was no inserting done on the nodes at > that > time. > > -----Original Message----- > From: lars hofhansl [mailto:[email protected]] > Sent: Friday, October 14, 2011 10:46 AM > To: [email protected] > Subject: Re: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not > append > > Did you do some heavy inserting during that time? I assume you use > 0.90.x(?) > > Might be HBASE-4387. > > -- Lars > > > > ----- Original Message ----- > From: Stuti Awasthi <[email protected]> > To: "[email protected]" <[email protected]> > Cc: > Sent: Thursday, October 13, 2011 9:44 PM > Subject: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not > append > > Hi Friends > I am using a mini distributed cluster of 3 nodes. Regionservers are running > on all of 3 nodes. This cluster is operational from 2 weeks now without any > issue. Today morning I saw one of the region servers are down. > I restarted the regionserver dynamically and it started fine and master > also > distributed regions to this region server. > I looked at the log but not much understood what was the problem. Logs > state > that one of my datanodes are bad, but when I checked, all datanodes are > working fine. > > Please suggest > > Regionserver Log : > > 2011-10-13 14:35:47,974 FATAL > org.apache.hadoop.hbase.regionserver.wal.HLog: > Could not append. Requesting close of hlog > java.io.IOException: Reflection > at > > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(Sequence > FileLogWriter.java:147) > at > org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986) > at > org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:937) > at > > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:147 > 7) > at > org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1386) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java: > 2549) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(Sequence > FileLogWriter.java:145) > ... 10 more > Caused by: java.io.IOException: DFSOutputStream is closed > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3238) > at > org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97) > at > org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944) > ... 14 more > 2011-10-13 14:35:47,999 INFO > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using > syncFs > -- HDFS-200 > 2011-10-13 14:35:48,001 FATAL > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server > serverName=master,60020,1316669478524, load=(requests=0, regions=4, > usedHeap=31, maxHeap=993): IOE in log roller > java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting... > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSCli > ent.java:2680) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java: > 2172) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient. > java:2371) > 2011-10-13 14:35:48,002 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: > requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0, > memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31, > maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440, > blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9, > blockCacheEvictedCount=0, blockCacheHitRatio=99, > blockCacheHitCachingRatio=99 > 2011-10-13 14:35:48,002 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log > roller > 2011-10-13 14:35:48,002 INFO > org.apache.hadoop.hbase.regionserver.LogRoller: > LogRoller exiting. > > > > ::DISCLAIMER:: > > ---------------------------------------------------------------------------- > ------------------------------------------- > > The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. > It shall not attach any liability on the originator or HCL or its > affiliates. Any views or opinions presented in this email are solely those > of the author and may not necessarily reflect the opinions of HCL or its > affiliates. > Any form of reproduction, dissemination, copying, disclosure, modification, > distribution and / or publication of this message without the prior written > consent of the author of this e-mail is strictly prohibited. If you have > received this email in error please delete it and notify the sender > immediately. Before opening any mail and attachments please check them for > viruses and defect. > > > ---------------------------------------------------------------------------- > ------------------------------------------- > > >
