[
https://issues.apache.org/jira/browse/YARN-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith resolved YARN-1432.
--------------------------
Resolution: Not A Problem
Hi [~ramgopalnaali] For the different users to run in the Kerberos set up,
LinuxContainerExecutor should be used. For more detailed security set up ,
kindly refer [Hadoop In Security Mode |
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SecureMode.html]
> Reduce phase is failing with shuffle error in kerberos enabled cluster
> ----------------------------------------------------------------------
>
> Key: YARN-1432
> URL: https://issues.apache.org/jira/browse/YARN-1432
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.1.0-beta
> Reporter: Ramgopal N
> Labels: security
>
> {code}
> OS user: user3
> kerberos user: hdfs
> Reducer is trying to read the map intermediate output using kerberos
> user(hdfs),but the owner of this file is OS user(user3)
> 2013-11-21 20:35:48,169 ERROR org.apache.hadoop.mapred.ShuffleHandler:
> Shuffle error :
> java.io.IOException: Error Reading IndexFile
> at
> org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:123)
> at
> org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:68)
> at
> org.apache.hadoop.mapred.ShuffleHandler$Shuffle.sendMapOutput(ShuffleHandler.java:595)
> at
> org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:506)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
> at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:144)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
> at
> org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:99)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
> at
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:523)
> at
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:507)
> at
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:444)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Owner 'user3' for path
> /home/user3/NodeAgentTmpDir/data/mapred/nm-local-dir/usercache/hdfs/appcache/application_1385040658134_0011/output/attempt_1385040658134_0011_m_000000_0/file.out.index
> did not match expected owner 'hdfs'
> at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:285)
> at
> org.apache.hadoop.io.SecureIOUtils.forceSecureOpenFSDataInputStream(SecureIOUtils.java:174)
> at
> org.apache.hadoop.io.SecureIOUtils.openFSDataInputStream(SecureIOUtils.java:158)
> at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:70)
> at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:62)
> at
> org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:119)
> ... 30 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
