[
https://issues.apache.org/jira/browse/HIVE-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429881#comment-16429881
]
Changshu Liu edited comment on HIVE-13532 at 4/9/18 2:44 AM:
-------------------------------------------------------------
We encountered this issue using Hive 1.2 branch, the problem is that the
MapredLocalTask dose not respect the Hive (Hive CLI and HiveServer2) security
settings.
We just need to pass the correct Hadoop permission information down to local
task via env variable.
here are the full call stack if this issue happened:
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator
(Operator.java:initialize(373)) - Initialization Done 1 HASHTABLESINK done is
reset.
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator
(Operator.java:initializeChildren(430)) - Operator 1 HASHTABLESINK initialized
2018-03-06 19:48:00,344 INFO [main]: mr.MapredLocalTask
(MapredLocalTask.java:initializeOperators(465)) - fetchoperator for $INTNAME
initialized
2018-03-06 19:48:02,281 ERROR [main]: mr.MapredLocalTask
(MapredLocalTask.java:executeInProcess(361)) - Hive Runtime Error: Map local
work failed
java.io.IOException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User: abc is not allowed to impersonate abc
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:409)
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:384)
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:350)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:746)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User: monarch is not allowed to impersonate monarch
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
was (Author: csliu):
We encountered this issue using Hive 1.2 branch, the problem is that the
MapredLocalTask dose not respect the Hive (Hive CLI and HiveServer2) security
settings, here are the full call stack:
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator
(Operator.java:initialize(373)) - Initialization Done 1 HASHTABLESINK done is
reset.
2018-03-06 19:48:00,343 INFO [main]: exec.HashTableSinkOperator
(Operator.java:initializeChildren(430)) - Operator 1 HASHTABLESINK initialized
2018-03-06 19:48:00,344 INFO [main]: mr.MapredLocalTask
(MapredLocalTask.java:initializeOperators(465)) - fetchoperator for $INTNAME
initialized
2018-03-06 19:48:02,281 ERROR [main]: mr.MapredLocalTask
(MapredLocalTask.java:executeInProcess(361)) - Hive Runtime Error: Map local
work failed
java.io.IOException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User: abc is not allowed to impersonate abc
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:409)
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:384)
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:350)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:746)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User: monarch is not allowed to impersonate monarch
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> MapredLocalTask should use the same security settings as remote task
> --------------------------------------------------------------------
>
> Key: HIVE-13532
> URL: https://issues.apache.org/jira/browse/HIVE-13532
> Project: Hive
> Issue Type: Bug
> Affects Versions: 1.1.0
> Environment: HADOOP_PROXY_USER is set.
> Reporter: Zhiwen Sun
> Assignee: Changshu Liu
> Priority: Major
> Attachments: HIVE-13532.1.patch
>
>
> Map join set HADOOP_USER_NAME should be realuser's username.
> Current, hive set HADOOP_USER_NAME env for mapjoin local process according:
> {quote}
> String endUserName = Utils.getUGI().getShortUserName();
> {quote}
> suppose set HADOOP_PROXY_USER=abc in shell.
> map join local job will have following env:
> {quote}
> HADOOP_USER_NAME=abc
> HADOOP_PROXY_NAME=abc
> {quote}
> this will cause such exception:
> {quote}
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User: abc is not allowed to impersonate
> {quote}
> I think we should set HADOOP_USER_NAME to realuser:
> {quote}
> String endUserName = Utils.getUGI().getRealUser().getShortUserName();
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
