[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617303#comment-14617303
]
zhihai xu commented on MAPREDUCE-6426:
--------------------------------------
{{testGetMapOutputInfo}} can only work with "simple" mode, If I enable security
by adding the following code to {{testGetMapOutputInfo}}
{code}
conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
"kerberos");
UserGroupInformation.setConfiguration(conf);
{code}
I will get the same error:
{code}
2015-07-07 12:44:44,246 ERROR [New I/O worker #188] mapred.ShuffleHandler
(ShuffleHandler.java:messageReceived(829)) - Shuffle error :
java.io.IOException: Error Reading IndexFile
at
org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:123)
at
org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:68)
at
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.getMapOutputInfo(ShuffleHandler.java:865)
at
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:818)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
I also find out how security is enabled. Although {{testGetMapOutputInfo}}
didn't enable security, {{testMapFileAccess}} enabled security "kerberos" by
{{UserGroupInformation.setConfiguration(conf);}} which is a static member
function, if {{testMapFileAccess}} is running before {{testGetMapOutputInfo}},
the configuration used by UserGroupInformation is not changed by
{{testGetMapOutputInfo}}, So UserGroupInformation is still using the
configuration from {{testMapFileAccess}} for {{testGetMapOutputInfo}}.
I attached a patch MAPREDUCE-6426.000.patch, which manually disable security in
{testGetMapOutputInfo}}. I believe it should fix this issue.
> TestShuffleHandler fails in trunk
> ---------------------------------
>
> Key: MAPREDUCE-6426
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Affects Versions: 2.8.0
> Reporter: Devaraj K
> Assignee: zhihai xu
> Attachments: MAPREDUCE-6426.000.patch
>
>
> {code:xml}
> expected:<1> but was:<0>
> Stacktrace
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
> {code}
> https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
> https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
