[
https://issues.apache.org/jira/browse/HIVE-10098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-10098:
--------------------------------
Component/s: Security
> HS2 local task for map join fails in KMS encrypted cluster
> ----------------------------------------------------------
>
> Key: HIVE-10098
> URL: https://issues.apache.org/jira/browse/HIVE-10098
> Project: Hive
> Issue Type: Bug
> Components: Security
> Reporter: Yongzhi Chen
> Assignee: Yongzhi Chen
> Fix For: 1.2.0
>
> Attachments: HIVE-10098.1.patch, HIVE-10098.2.patch
>
>
> Env: KMS was enabled after cluster was kerberos secured.
> Problem: PROBLEM: Any Hive query via beeline that performs a MapJoin fails
> with a java.lang.reflect.UndeclaredThrowableException from
> KMSClientProvider.addDelegationTokens.
> {code}
> 2015-03-18 08:49:17,948 INFO [main]: Configuration.deprecation
> (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.input.dir is
> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
> 2015-03-18 08:49:19,048 WARN [main]: security.UserGroupInformation
> (UserGroupInformation.java:doAs(1645)) - PriviledgedActionException as:hive
> (auth:KERBEROS)
> cause:org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> 2015-03-18 08:49:19,050 ERROR [main]: mr.MapredLocalTask
> (MapredLocalTask.java:executeFromChildJVM(314)) - Hive Runtime Error: Map
> local work failed
> java.io.IOException: java.io.IOException:
> java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:634)
>
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:363)
>
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:337)
>
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:303)
> at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:735)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.io.IOException:
> java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:826)
>
> at
> org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
>
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2017)
>
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:121)
>
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
>
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
>
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:205)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:413)
>
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:559)
>
> ... 9 more
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655)
>
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:808)
>
> ... 18 more
> Caused by:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
>
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
>
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
> {code}
> To make sure map join happen, test need a small table join with a large one,
> for example:
> {code}
> CREATE TABLE if not exists jsmall (code string, des string, t int, s int) ROW
> FORMAT DELIMITED FIELDS TERMINATED BY '\t';
> CREATE TABLE if not exists jbig1 (code string, des string, t int, s int) ROW
> FORMAT DELIMITED FIELDS TERMINATED BY '\t';
> load data local inpath '/tmp/jdata' into table jsmall;
> load data local inpath '/tmp/jdata' into table jbig1;
> load data local inpath '/tmp/jdata' into table jbig1;
> load data local inpath '/tmp/jdata' into table jbig1;
> load data local inpath '/tmp/jdata' into table jbig1;
> load data local inpath '/tmp/jdata' into table jbig1;
> load data local inpath '/tmp/jdata' into table jbig1;
> load data local inpath '/tmp/jdata' into table jbig1;
> select count(*) from jsmall small join jbig1 big on (small.code = big.code);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
