This worked for me finally after Vineet's suggestion via HCC. https://community.hortonworks.com/questions/90662/hawq-issues-with-ranger-kms.html
Thank you all. Regards, Gagan Brahmi On Thu, Mar 23, 2017 at 11:15 AM, Gagan Brahmi <[email protected]> wrote: > Hi All, > > Is there any known issue around running HAWQ in a cluster with Ranger KMS > over YARN? > > Seems that HAWQ is not able to obtain containers when requesting it to > YARN ResourceManager. > > The following is what I am seeing in the YARN RM logs: > > --------------------- > > 2017-03-23 10:56:30,816 INFO hdfs.DFSClient > (DFSClient.java:getDelegationToken(1043)) > - Created HDFS_DELEGATION_TOKEN token 20049 for postgres on > 192.168.59.104:8020 > 2017-03-23 10:56:30,889 WARN security.DelegationTokenRenewer > (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(895)) - Unable > to add the application to the delegation token renewer. > java.io.IOException: java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.crypto.key.kms.KMSClientProvider. > addDelegationTokens(KMSClientProvider.java:1032) > at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExte > nsion.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110) > at org.apache.hadoop.hdfs.DistributedFileSystem. > addDelegationTokens(DistributedFileSystem.java:2298) > at org.apache.hadoop.yarn.server.resourcemanager.security. > DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:685) > at org.apache.hadoop.yarn.server.resourcemanager.security. > DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:680) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1724) > at org.apache.hadoop.yarn.server.resourcemanager.security. > DelegationTokenRenewer.obtainSystemTokensForUser( > DelegationTokenRenewer.java:679) > at org.apache.hadoop.yarn.server.resourcemanager.security. > DelegationTokenRenewer.requestNewHdfsDelegationToken( > DelegationTokenRenewer.java:643) > at org.apache.hadoop.yarn.server.resourcemanager.security. > DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java: > 488) > at org.apache.hadoop.yarn.server.resourcemanager.security. > DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:77) > at org.apache.hadoop.yarn.server.resourcemanager.security. > DelegationTokenRenewer$DelegationTokenRenewerRunnable. > handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:891) > at org.apache.hadoop.yarn.server.resourcemanager.security. > DelegationTokenRenewer$DelegationTokenRenewerRunnable > .run(DelegationTokenRenewer.java:868) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1742) > at org.apache.hadoop.crypto.key.kms.KMSClientProvider. > addDelegationTokens(KMSClientProvider.java:1014) > ... 16 more > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > Authentication failed, URL: http://hdp-hdb-200.gagan.com:9292/kms/v1/?op= > GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com% > 40gagan.com&doAs=postgres&user.name=yarn, status: 403, message: Forbidden > at org.apache.hadoop.security.authentication.client. > AuthenticatedURL.extractToken(AuthenticatedURL.java:278) > at org.apache.hadoop.security.authentication.client. > PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77) > at org.apache.hadoop.security.token.delegation.web. > DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator. > java:132) > at org.apache.hadoop.security.authentication.client. > KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212) > at org.apache.hadoop.security.token.delegation.web. > DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator. > java:132) > at org.apache.hadoop.security.authentication.client. > AuthenticatedURL.openConnection(AuthenticatedURL.java:216) > at org.apache.hadoop.security.token.delegation.web. > DelegationTokenAuthenticator.doDelegationTokenOperation( > DelegationTokenAuthenticator.java:298) > at org.apache.hadoop.security.token.delegation.web. > DelegationTokenAuthenticator.getDelegationToken( > DelegationTokenAuthenticator.java:170) > at org.apache.hadoop.security.token.delegation.web. > DelegationTokenAuthenticatedURL.getDelegationToken( > DelegationTokenAuthenticatedURL.java:371) > at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run( > KMSClientProvider.java:1019) > at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run( > KMSClientProvider.java:1014) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1724) > ... 17 more > > --------------------- > > The following is what I see in Ranger KMS log (kms.lo) > > --------------------- > > 2017-03-23 11:02:00,734 DEBUG LimitLatch - Counting > up[http-bio-9292-Acceptor-0] latch=7 > 2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [uriBC] has > value [/kms/v1/] > 2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [semicolon] has > value [-1] > 2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [enc] has value > [ISO-8859-1] > 2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Security checking > request OPTIONS /kms/v1/ > 2017-03-23 11:02:00,738 DEBUG RealmBase - No applicable constraints > defined > 2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Not subject to any > constraint > 2017-03-23 11:02:00,738 TRACE StandardWrapper - Returning non-STM > instance > 2017-03-23 11:02:00,739 DEBUG Http11Protocol - Socket: [ > org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/ > 192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State > out: [OPEN] > 2017-03-23 11:02:00,758 DEBUG Http11Processor - Error parsing HTTP request > header > java.io.EOFException: Unexpected EOF read on the socket > at org.apache.coyote.http11.Http11Processor. > setRequestLineReadTimeout(Http11Processor.java:169) > at org.apache.coyote.http11.AbstractHttp11Processor.process( > AbstractHttp11Processor.java:990) > at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler. > process(AbstractProtocol.java:625) > at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor. > run(JIoEndpoint.java:318) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run( > TaskThread.java:61) > at java.lang.Thread.run(Thread.java:745) > 2017-03-23 11:02:00,758 DEBUG Http11Protocol - Socket: [ > org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/ > 192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State > out: [CLOSE > D] > 2017-03-23 11:02:00,758 TRACE JIoEndpoint - Closing > socket:org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/ > 192.168.59.104,port=58547,localport=9292] > > --------------------- > > The following is from the Ranger KMS access log: > > --------------------- > > 2017-03-23 11:02:00,738 UNAUTHENTICATED RemoteHost:192.168.59.104 > Method:OPTIONS URL:http://hdp-hdb-200.gagan.com:9292/kms/v1/?op= > GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos > tgres ErrorMsg:'Authentication required' > 2017-03-23 11:02:00,786 UNAUTHENTICATED RemoteHost:192.168.59.104 > Method:OPTIONS URL:http://hdp-hdb-200.gagan.com:9292/kms/v1/?op= > GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos > tgres&user.name=yarn ErrorMsg:'Authentication required' > > --------------------- > > The following is from the Ranger KMS audit log (kms-audit.log) > > --------------------- > > 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op= > GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres > HTTP/1.1" 401 997 > 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op= > GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres > HTTP/1.1" 403 258 > 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op= > GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com > &doAs=postgres&user.name=yarn HTTP/1.1" 401 997 > 192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op= > GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com > &doAs=postgres&user.name=yarn HTTP/1.1" 403 258 > > --------------------- > > > I have added the following proxyuser configuration in Ranger KMS as well: > > hadoop.kms.proxyuser.postgres.users=* > hadoop.kms.proxyuser.postgres.hosts=* > hadoop.kms.proxyuser.yarn.users=* > hadoop.kms.proxyuser.yarn.hosts=* > > The core-site.xml has the required proxyuser configuration as well: > > hadoop.proxyuser.postgres.groups=* > hadoop.proxyuser.postgres.hosts=* > hadoop.proxyuser.yarn.groups=* > hadoop.proxyuser.yarn.hosts=* > > But nothing seem to be working in this case here. > > I would appreciate some inputs on this one. > > > > Regards, > Gagan Brahmi >
