[jira] [Commented] (MAPREDUCE-6869) org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers :
[ https://issues.apache.org/jira/browse/MAPREDUCE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944493#comment-15944493 ] 翟玉勇 commented on MAPREDUCE-6869: thx > org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers : > -- > > Key: MAPREDUCE-6869 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6869 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, yarn >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0-cdh5.8.2 >Reporter: 翟玉勇 >Priority: Minor > > nodemanager log > 2017-03-25 21:07:03,071 ERROR org.apache.hadoop.mapred.ShuffleHandler: > Shuffle error in populating headers : > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > usercache/master/appcache/application_1489067586592_930490/output/attempt_1489067586592_930490_m_002811_0/file.out.index > in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:488) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:165) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.getMapOutputInfo(ShuffleHandler.java:1000) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.populateHeaders(ShuffleHandler.java:1022) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:908) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) > at > org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) > at > org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at > org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) > at > org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > reduce log > 2017-03-25 05:40:33,919 WARN [fetcher#2] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: Invalid map id > java.lang.IllegalArgumentException: TaskAttemptId string : TTP/1.1 500 > Internal Server Error > Content-Type: text/plain;
[jira] [Commented] (MAPREDUCE-6869) org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers :
[ https://issues.apache.org/jira/browse/MAPREDUCE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943317#comment-15943317 ] Jason Lowe commented on MAPREDUCE-6869: --- This error is simply stating that a map output file could not be found. That can happen if the disk goes bad, something like tmpwatch is coming along and cleaning things up when it shouldn't, etc. It is not in itself indicative of a problem in Hadoop unless you have supporting evidence for that. Is this reproducible or are there other things that indicate there is something Hadoop itself can do to prevent this? If not then this is more like a user support request which is better served by the [Hadoop mailing lists|http://hadoop.apache.org/mailing_lists.html]. JIRA is for tracking known problems in Hadoop itself and not for general support. > org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers : > -- > > Key: MAPREDUCE-6869 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6869 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, yarn >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0-cdh5.8.2 >Reporter: 翟玉勇 >Priority: Minor > > nodemanager log > 2017-03-25 21:07:03,071 ERROR org.apache.hadoop.mapred.ShuffleHandler: > Shuffle error in populating headers : > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > usercache/master/appcache/application_1489067586592_930490/output/attempt_1489067586592_930490_m_002811_0/file.out.index > in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:488) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:165) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.getMapOutputInfo(ShuffleHandler.java:1000) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.populateHeaders(ShuffleHandler.java:1022) > at > org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:908) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) > at > org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) > at > org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at > org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) > at > org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) > at > org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at >
