Hi, Thanks! The patch applies without change to hadoop-0.18.0, and should be included in a 0.18.1.
However, I'm still seeing: in hadoop.log: 2008-09-05 11:13:54,805 WARN dfs.DFSClient - Exception while reading from blk_3428404120239503595_2664 of /user/trank/segments/20080905102650/crawl_generate/part-00010 from somehost:50010: java.io.IOException: Premeture EOF from in putStream in datanode.log: 2008-09-05 11:15:09,554 WARN dfs.DataNode - DatanodeRegistration(somehost:50010, storageID=DS-751763840-somehost-50010-1219931304453, infoPort=50075, ipcPort=50020):Got exception while serving blk_-4682098638573619471_2662 to /somehost: java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/somehost:50010 remote=/somehost:45244] These entries in datanode.log happens a few minutes apart repeatedly. I've reduced # map-tasks so load on this node is below 1.0 with 5GB of free memory (so it's not resource starvation). Espen On Thu, Sep 4, 2008 at 3:33 PM, Devaraj Das <[EMAIL PROTECTED]> wrote: >> I started a profile of the reduce-task. I've attached the profiling output. >> It seems from the samples that ramManager.waitForDataToMerge() doesn't >> actually wait. >> Has anybody seen this behavior. > > This has been fixed in HADOOP-3940 > > > On 9/4/08 6:36 PM, "Espen Amble Kolstad" <[EMAIL PROTECTED]> wrote: > >> I have the same problem on our cluster. >> >> It seems the reducer-tasks are using all cpu, long before there's anything to >> shuffle. >> >> I started a profile of the reduce-task. I've attached the profiling output. >> It seems from the samples that ramManager.waitForDataToMerge() doesn't >> actually wait. >> Has anybody seen this behavior. >> >> Espen >> >> On Thursday 28 August 2008 06:11:42 wangxu wrote: >>> Hi,all >>> I am using hadoop-0.18.0-core.jar and nutch-2008-08-18_04-01-55.jar, >>> and running hadoop on one namenode and 4 slaves. >>> attached is my hadoop-site.xml, and I didn't change the file >>> hadoop-default.xml >>> >>> when data in segments are large,this kind of errors occure: >>> >>> java.io.IOException: Could not obtain block: blk_-2634319951074439134_1129 >>> file=/user/root/crawl_debug/segments/20080825053518/content/part-00002/data >>> at >>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.jav >>> a:1462) at >>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1 >>> 312) at >>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417) at >>> java.io.DataInputStream.readFully(DataInputStream.java:178) >>> at >>> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:64 >>> ) at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:102) >>> at >>> org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1646) >>> at >>> org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.ja >>> va:1712) at >>> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java: >>> 1787) at >>> org.apache.hadoop.mapred.SequenceFileRecordReader.getCurrentValue(SequenceF >>> ileRecordReader.java:104) at >>> org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordRe >>> ader.java:79) at >>> org.apache.hadoop.mapred.join.WrappedRecordReader.next(WrappedRecordReader. >>> java:112) at >>> org.apache.hadoop.mapred.join.WrappedRecordReader.accept(WrappedRecordReade >>> r.java:130) at >>> org.apache.hadoop.mapred.join.CompositeRecordReader.fillJoinCollector(Compo >>> siteRecordReader.java:398) at >>> org.apache.hadoop.mapred.join.JoinRecordReader.next(JoinRecordReader.java:5 >>> 6) at >>> org.apache.hadoop.mapred.join.JoinRecordReader.next(JoinRecordReader.java:3 >>> 3) at >>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:165) >>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45) >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) >>> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209) >>> >>> >>> how can I correct this? >>> thanks. >>> Xu >> > > >
