HI Team,
Could you please help me to fix the below ? Not sure where the
blocker is, on SASL or Kerberos Auth_to_local Rules? Thanks
*PROBLEM STATEMENT : Connections established from dataproc client Node to
MSS(metastore service) Instance are not Successful.*
# We were able to build the MSS (metastore service) in a separate VM, now
we tried pointing the mss instance fqdn as metastore uri (thrift url) at
the client node, so the client uses the remote metastore service instead of
local.
# MSS will use Central KDC and DPaaS Client are with local KDC. For this
two to talk each other we have enabled cross-relam trust and tested
manually it is working fine.
# One portion of cross relam trust it to setup the auth_to_local rules at
MSS end, which actually translates the principle and maps to the local
user, In our case “hive/clienthostname@CLIENTL-REALM” from client need to
mapped to “hive” user at MSS. But as per the below logs, exactly at that
stage it is throwing no rules applied error.
# One another event noticed from the logs, is that SASL related error,
regarding there keys.
*ERROR OBSERVED AT MSS SERVER LOGS:*
2023-08-09T17:15:52,187 INFO [pool-6-thread-200] metastore.HiveMetaStore:
200: Done cleaning up thread local RawStore
2023-08-09T17:15:52,187 INFO [pool-6-thread-200] HiveMetaStore.audit:
ugi=hive/clienthostname@CLIENTL-REALM ip=10.xx.xx.190 cmd=Done cleaning up
thread local RawStore
2023-08-09T17:15:52,643 DEBUG [HikariPool-2 housekeeper] pool.HikariPool:
HikariPool-2 - Pool stats (total=10, active=0, idle=10, waiting=0)
2023-08-09T17:15:52,643 DEBUG [HikariPool-1 housekeeper] pool.HikariPool:
HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
2023-08-09T17:15:57,244 DEBUG [pool-6-thread-200]
security.UserGroupInformation: PrivilegedAction [as: hive/
clienthostname@CLIENTL-REALM (auth:KERBEROS)][action:
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1@7b8480aa
]
java.lang.Exception: null
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1852)
[hadoop-client-api-3.3.1.jar:?]
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691)
[hive-exec-3.1.3.jar:3.1.3]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
[hive-exec-3.1.3.jar:3.1.3]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_381]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_381]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_381]
2023-08-09T17:15:57,244 DEBUG [pool-6-thread-200]
transport.TSaslServerTransport: transport map does not contain key *========
HERE IT SAY NO KEY =========*
2023-08-09T17:15:57,244 DEBUG [pool-6-thread-200] transport.TSaslTransport:
opening transport org.apache.thrift.transport.TSaslServerTransport@57f1caac
2023-08-09T17:15:57,245 DEBUG [pool-6-thread-200] transport.TSaslTransport:
SERVER: Received message with status START and payload length 6
2023-08-09T17:15:57,245 DEBUG [pool-6-thread-200]
transport.TSaslServerTransport: Received start message with status START
2023-08-09T17:15:57,245 DEBUG [pool-6-thread-200]
transport.TSaslServerTransport: Received mechanism name 'GSSAPI'
2023-08-09T17:15:57,245 DEBUG [pool-6-thread-200] transport.TSaslTransport:
SERVER: Start message handled
2023-08-09T17:15:57,245 DEBUG [pool-6-thread-200] transport.TSaslTransport:
SERVER: Received message with status OK and payload length 783
2023-08-09T17:15:57,246 DEBUG [pool-6-thread-200] transport.TSaslTransport:
SERVER: Writing message with status OK and payload length 108
2023-08-09T17:15:57,386 DEBUG [pool-6-thread-200] transport.TSaslTransport:
SERVER: Received message with status OK and payload length 0
2023-08-09T17:15:57,387 DEBUG [pool-6-thread-200] transport.TSaslTransport:
SERVER: Writing message with status OK and payload length 32
2023-08-09T17:15:57,411 DEBUG [pool-6-thread-200] transport.TSaslTransport:
SERVER: Received message with status COMPLETE and payload length 32
2023-08-09T17:15:57,411 DEBUG [pool-6-thread-200] security.SaslRpcServer:
SASL server GSSAPI callback: setting canonicalized client ID:
hive/mss-dpaas20-client-m.c.wmt-bfdms-bfddxvenu1.inter...@c.wmt-BFDMS-BFDDXVENU1.INTERNAL
2023-08-09T17:15:57,411 DEBUG [pool-6-thread-200] transport.TSaslTransport:
SERVER: Writing message with status COMPLETE and payload length 0
2023-08-09T17:15:57,411 DEBUG [pool-6-thread-200] transport.TSaslTransport:
SERVER: Main negotiation loop complete *======== HOWEVER SASL NEGOTIATION
COMPLETED HERE =========*
2023-08-09T17:15:57,411 DEBUG [pool-6-thread-200]
security.UserGroupInformation: PrivilegedAction [as: hive/
clienthostname@CLIENTL-REALM (auth:KERBEROS)][action:
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1@1608e328
]
java.lang.Exception: null
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1852)
[hadoop-client-api-3.3.1.jar:?]
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691)
[hive-exec-3.1.3.jar:3.1.3]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:270)
[hive-exec-3.1.3.jar:3.1.3]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_381]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_381]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_381]
2023-08-09T17:15:57,412 DEBUG [pool-6-thread-200]
transport.TSaslServerTransport: transport map does contain key
org.apache.thrift.transport.TSocket@33078b74 *======== AGAIN HERE IT SAY NO
KEY ========= OUR SASL **CLIENT/SERVER CONFIGURATION ARE WITHIN THE NODES,
NO KEYS EXCHANGED BETWEEN MSS NODE AND CLIENT, DOES THIS ERROR SAYS, IT
NEEDS THE TRUSTSTORE KEYS TO EXCHANGED BETWEEN MSS AND CLIENT ?*
2023-08-09T17:15:57,412 DEBUG [pool-6-thread-200]
security.HadoopThriftAuthBridge: AUTH ID ======>hive/
clienthostname@CLIENTL-REALM
2023-08-09T17:15:57,412 ERROR [pool-6-thread-200] server.TThreadPoolServer:
Error occurred during processing of message. *======= EXACTLY FAILS WITH
AUTHBRIDGE PHASE ========= ARE THESE CONNECTION FROM CLIENT, ABLE TO REACH
THE CORE-SITE.XML, DOES IT REQUIRE ANY CONFIGURATION TO DO SO ?*
java.lang.IllegalArgumentException: Illegal principal name hive/
clienthostname@CLIENTL-REALM:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule:
No rules applied to hive/clienthostname@CLIENTL-REALM
at org.apache.hadoop.security.User.<init>(User.java:51)
~[hadoop-client-api-3.3.1.jar:?]
at
org.apache.hadoop.security.UserGroupInformation.createProxyUser(UserGroupInformation.java:1494)
~[hadoop-client-api-3.3.1.jar:?]
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:627)
~[hive-exec-3.1.3.jar:3.1.3]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
[hive-exec-3.1.3.jar:3.1.3]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_381]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_381]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_381]
Caused by:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule:
No rules applied to hive/
<hive/mss-dpaas20-client-m.c.wmt-bfdms-bfddxvenu1.inter...@c.wmt-BFDMS-BFDDXVENU1.INTERNAL>
clienthostname@CLIENTL-REALM *======= ERRORS OUT NO MATCHING RULES
=========*
at
org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:429)
~[hadoop-client-api-3.3.1.jar:?]
at org.apache.hadoop.security.User.<init>(User.java:48)
~[hadoop-client-api-3.3.1.jar:?]
*Rule tried with "Core-Site.xml" at MSS Instance :*
RULE:[1:$1@$0](.*@.*)s/@.*/hive/g
RULE:[2:$1@$0](.*@.*)s/@.*/hive/g
RULE:[2:$1@$0](.*)s/(.*)/hive/g
RULE:[2:$1/$2@$0](.*)s/(.*)/hive/g
RULE:[1:$1](.*)s/(.*)/$1/g
RULE:[2:$1](.*)s/(.*)/$1/g
RULE:[2:$1@$0](.*@\Qclient-hostaname\E$)s/@\Qclient-realm\E$/hive/g
RULE:[1:$1@$0](hive/client-hostname@CLIENT-REALM)s/.*/hive/
RULE:[1:$1@$0](.*@CLIENT-REALM)s/@.*/hive/
RULE:[2:$1@$0](.*@CLIENT-REALM)s/@.*/hive/
RULE:[1:$1@$0](hive/*@CLIENT-REALM)s/.*/hive/
RULE:[2:$1/$2@$0](hive/hostname@CLIENT-REALM)s/(.*)@CLIENT-REALM/hive/
RULE:[1:$1@$0](hive/hostname@CLIENT-REALM)s/.*/hive/
RULE:[2:$1/$2@$0](hive.*@CLIENT-REALM)s/(.*)@CLIENT-REALM/hive/
RULE:[3:$1@$0](hive/client-hostname@CLIENT-REALM)s/.*/hive/
Regards
Sathish Kumar Palani
