[
https://issues.apache.org/jira/browse/PIG-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14137877#comment-14137877
]
Rohini Palaniswamy commented on PIG-3507:
-----------------------------------------
Sorry. My bad. Forgot that core-site.xml is also loaded as part of defaults. We
use a perl script instead of the bash bin/pig and what we do there is not add
HADOOP_CONF_DIR to classpath if it is local mode.
if [ "$HADOOP_CONF_DIR" != "" ]; then
CLASSPATH=${CLASSPATH}:${HADOOP_CONF_DIR}
fi
Either you can do that or even this sounds ok. But you can simplify the code
as below as you don't have to initialize UserGroupInformation in case of non
local mode.
{code}
if (opts.getValStr().toLowerCase().contains("local") {
UserGroupInformation.setConfiguration(new
Configuration(false));
}
{code}
Above check also ensures kerberos is not enabled for tez_local mode.
> It fails to run pig in local mode on a Kerberos enabled Hadoop cluster
> ----------------------------------------------------------------------
>
> Key: PIG-3507
> URL: https://issues.apache.org/jira/browse/PIG-3507
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0, 0.11
> Reporter: chiyang
> Assignee: liyunzhang_intel
> Fix For: 0.14.0
>
> Attachments: PIG-3507.patch, PIG_3507_1.patch
>
>
> It fails to run pig in local mode on a Kerberos enabled Hadoop cluster
> *Command*
> pig -x local <pig script>
>
> *Pig script*
> A = load '/etc/passwd';
> dump A;
>
> *Root cause*
> When running pig in local mode, jobConf in HExecutionEngine is initiated with
> core-default.xml (hadoop.security.authentication = simple),
> mapred-default.xml, and yarn-default.xml. However, the settings are not
> passed to UserGroupInformation. That's why obtainTokensForNamenodesInternal()
> is called from obtainTokensForNamenodes(), and causes the exception to happen.
> {noformat}
> public static void obtainTokensForNamenodes(Credentials credentials, Path[]
> ps, Configuration conf) throws IOException {
> if (!UserGroupInformation.isSecurityEnabled()) {
> return;
> }
> obtainTokensForNamenodesInternal(credentials, ps, conf);
> }
> {noformat}
> *Error*
> Pig Stack Trace
> ---------------
> ERROR 6000: Output Location Validation Failed for:
> 'file:/tmp/temp-308998488/tmp-2025176494 More info to follow:
> Can't get JT Kerberos principal for use as renewer
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> open iterator for alias A
> at org.apache.pig.PigServer.openIterator(PigServer.java:841)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:604)
> at org.apache.pig.Main.main(Main.java:157)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
> at org.apache.pig.PigServer.storeEx(PigServer.java:940)
> at org.apache.pig.PigServer.store(PigServer.java:903)
> at org.apache.pig.PigServer.openIterator(PigServer.java:816)
> ... 12 more
> Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: Output
> Location Validation Failed for: 'file:/tmp/temp-308998488/tmp-2025176494 More
> info to follow:
> Can't get JT Kerberos principal for use as renewer
> at
> org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95)
> at
> org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
> at
> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
> at
> org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
> at
> org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
> at
> org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:288)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1327)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1252)
> at org.apache.pig.PigServer.storeEx(PigServer.java:936)
> ... 14 more
> Caused by: java.io.IOException: Can't get JT Kerberos principal for use as
> renewer
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:129)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:111)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:85)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:127)
> at
> org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
> ... 24 more
> ================================================================================
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
