[
https://issues.apache.org/jira/browse/IMPALA-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570724#comment-16570724
]
ASF subversion and git services commented on IMPALA-6789:
---------------------------------------------------------
Commit 8bd9f175b79f117c425def065dbeaa92098bdecd in impala's branch
refs/heads/master from stiga-huang
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=8bd9f17 ]
IMPALA-6789: disable impersonation in hive in minicluster
Due to bug of HADOOP-7050, users with dots in their username can't
launch HiveServer2 in minicluster. To work arround this, we can set
hive.server2.enable.doAs to false to disable impersonation in hive.
Impala's authorization only depends on Sentry, so we can disable this
without breaking any tests.
This patch also quotes the group name in AuthorizationStmtTest#testShow
to avoid syntax errors when group name contains dots.
Test:
* Build succeed with username quanlong.huang locally
Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a
Reviewed-on: http://gerrit.cloudera.org:8080/11119
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Failed to launch HiveServer2 in minicluster after switching to Hadoop3
> ----------------------------------------------------------------------
>
> Key: IMPALA-6789
> URL: https://issues.apache.org/jira/browse/IMPALA-6789
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
>
> After switching to Hadoop3 by IMPALA-4277 (0812f87), the HiveServer2 in my
> dev env failed to launch and kept warning that "User: quanlong.huang is not
> allowed to impersonate foo".
> {code:java}
> java.lang.RuntimeException: java.lang.RuntimeException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User: quanlong.huang is not allowed to impersonate foo
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at com.sun.proxy.$Proxy37.open(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:411)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:363)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:189)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:423)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:312)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User: quanlong.huang is not allowed to impersonate foo
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:596)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_121]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> ... 21 more
> Caused by: org.apache.hadoop.ipc.RemoteException: User: quanlong.huang is not
> allowed to impersonate foo
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at com.sun.proxy.$Proxy31.getFileInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_121]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at com.sun.proxy.$Proxy32.getFileInfo(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1494)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1491)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1506)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:701)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:640)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:572)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_121]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> ... 21 more
> {code}
> [~philip] helps me to find that the root cause is HADOOP-7050. Hadoop
> impersonation does not support username with dots so far. My username
> contains a dot so it failed.
> It's hard for me to change my username or use another username due to some
> company-specific security issues. Finally, I found a workaround for this.
> The source of the invocation is in ServiceCLIService#getSessionHandle
> {code:java}
> if
> (cliService.getHiveConf().getBoolVar(ConfVars.HIVE_SERVER2_ENABLE_DOAS) &&
> (userName != null)) {
> String delegationTokenStr = getDelegationToken(userName);
> sessionHandle = cliService.openSessionWithImpersonation(protocol,
> userName,
> req.getPassword(), ipAddress, req.getConfiguration(),
> delegationTokenStr);
> } else {
> sessionHandle = cliService.openSession(protocol, userName,
> req.getPassword(),
> ipAddress, req.getConfiguration());
> }
> {code}
> So add this to the hive-site.xml works around the issue:
> {code:xml}
> <property>
> <name>hive.server2.enable.doAs</name>
> <value>false</value>
> </property>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
