It's hard for me to change username due to some company-specific security
issues. I finally found a workaround for my problem.
The source of the impersonation invocation is in
ThriftCLIService#getSessionHandle
if (cliService.getHiveConf().getBoolVar(ConfVars.HIVE_SERVER2_ENABLE_DOAS)
&&
(userName != null)) {
...
sessionHandle = cliService.openSessionWithImpersonation(...);
} else {
sessionHandle = cliService.openSession(...);
}
So setting hive.server2.enable.doAs to false in hive-site.xml can avoid this
invocatino. I modified my hive-site.xml templates in fe/src/test/resources for
this and HiveServer2 launched successfully.
Just filed a JIRA for this workaround in case someone has an odd username like
me: https://issues.apache.org/jira/browse/IMPALA-6789
Thanks,
Quanlong
在 2018-04-01 22:51:53，"Quanlong Huang" <[email protected]> 写道：
>Thanks, Philip!
>Since it's still unfixed, let me try to use another username.
>
>
>在 2018-04-01 12:35:42，"Philip Zeyliger" <[email protected]> 写道：
>
>Hi Quanlong,
>
>
>You're running into https://issues.apache.org/jira/browse/HADOOP-7050. I found
>that JIRA via https://kb.informatica.com/solution/23/Pages/61/510035.aspx (and
>Google).
>
>
>What's surprising to me is that this code has been around in Hadoop for a long
>time. I think HiveServer2 must have changed in that it now invokes the
>impersonation code, whereas it must have not before.
>
>
>Here are some pointers:
>
>
>
>testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl:
><name>hadoop.proxyuser.${USER}.hosts</name>
>
>
>
>https://github.com/apache/hadoop/blob/dc8e3432013153ac11d31d6b462aa96f8ca2c443/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/DefaultImpersonationProvider.java#L78
>
>
>
> String usersGroupsRegEx = prefixRegEx + "[^.]*(" +
> Pattern.quote(CONF_USERS) + "|" + Pattern.quote(CONF_GROUPS) + ")";
>
>
>To my eye, it looks like that regular expression is wrong and is what's
>disallowing usernames with periods/dots in them. (Once it's loosened, any code
>that's parsing these may also need to be fixed.)
>
>
>
>
>-- Philip
>
>
>
>
>
>On Fri, Mar 30, 2018 at 7:34 PM, Quanlong Huang <[email protected]> wrote:
>
>I failed to start the minicluster too but encountered another errors.
>HiveServer2 failed to launch and kept warning:
>
>
>2018-03-30T18:54:05,526 WARN [HiveServer2-Handler-Pool: Thread-49]
>thrift.ThriftCLIService: Error opening session:
>java.lang.RuntimeException: java.lang.RuntimeException:
>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User: quanlong.huang is not allowed to impersonate foo
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at com.sun.proxy.$Proxy37.open(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:411)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:363)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:189)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:423)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:312)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
>Caused by: java.lang.RuntimeException:
>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User: quanlong.huang is not allowed to impersonate foo
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:596)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_121]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> ... 21 more
>Caused by: org.apache.hadoop.ipc.RemoteException: User: quanlong.huang is not
>allowed to impersonate foo
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at com.sun.proxy.$Proxy31.getFileInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_121]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at com.sun.proxy.$Proxy32.getFileInfo(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1494)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1491)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1506)
> ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
> ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?]
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:701)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:640)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:572)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_121]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> ... 21 more
>
>
>I just rebase the codes and run ./buildall.sh -format
>Any thoughts?
>
>
>Thanks,
>Quanlong
>
>At 2018-03-30 13:08:33, "Dimitris Tsirogiannis" <[email protected]>
>wrote:
>>I enabled full logging in my postgres that hosts the sentry and metastore
>>db and I don't see the table being created. If anyone has gone through the
>>process, can you: a) verify that relation SENTRY_ROLE exists in your
>>sentry_policy db, and b) tell me how many relations are in your policy_db.
>>
>>Thanks
>>Dimitris
>>
>>On Thu, Mar 29, 2018 at 9:32 PM, Dimitris Tsirogiannis <
>>[email protected]> wrote:
>>
>>> Good point. I used -format that in theory handles both the metastore and
>>> the sentry policy dB. The sentry_policy db is created and has some tables
>>> but not the SENTRY_ROLE.
>>>
>>> Dimitris
>>>
>>> On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <[email protected]> wrote:
>>>
>>>> I think I might have once fixed that using
>>>>
>>>> ./buildall.sh -notests -format_metastore -format_sentry_policy_db
>>>>
>>>>
>>>> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
>>>> [email protected]> wrote:
>>>>
>>>> > I tried rebuilding my minicluster but Sentry refuses to start. I get
>>>> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does
>>>> that
>>>> > ring any bells?
>>>> >
>>>> > Thanks
>>>> > Dimitris
>>>> >
>>>> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <[email protected]>
>>>> > wrote:
>>>> >
>>>> > > Hi folks,
>>>> > >
>>>> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
>>>> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which
>>>> changes
>>>> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry
>>>> 2.0,
>>>> > > and so on. This change *will not* be back-ported to 2.x.
>>>> > >
>>>> > > When you pull that change in, you'll need to re-build your minicluster
>>>> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull
>>>> in
>>>> > the
>>>> > > new dependencies, format your cluster, and load up all the data. As
>>>> you
>>>> > > know, it takes 1-2 hours.
>>>> > >
>>>> > > If you want to hold off, you can elso set export
>>>> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
>>>> > >
>>>> > > Note that this choice between versions happens at build time, and
>>>> CMake
>>>> > > depends on it. So, switching back and forth requires re-running CMake.
>>>> > >
>>>> > > Please let me know if you run into any trouble. This is a big enough
>>>> that
>>>> > > there may be some bumps on the road.
>>>> > >
>>>> > > -- Philip
>>>> > >
>>>> >
>>>>
>>>
>
>
