[
https://issues.apache.org/jira/browse/IMPALA-12458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812554#comment-17812554
]
Quanlong Huang commented on IMPALA-12458:
-----------------------------------------
Sorry to be late on this. Is this a concurrent issue that only occurs when
catalogd and impalad launch in parallel on the same node? Or does it also
happen when launching impalad on a node that already runs catalogd?
> When impalad and catalogd are started concurrently on a Kerberos-enabled
> node, catalogd will report authentication failure
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-12458
> URL: https://issues.apache.org/jira/browse/IMPALA-12458
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 4.0.0
> Environment: RHEL7
> Reporter: Davy Xu
> Priority: Major
> Attachments: Catalog.log, Impala.log
>
>
> [~tlipcon], would you please take a look at this problem?
> After enabling impala kerberos, start impalad and catalogd concurrently on a
> node. The catalogd log periodically reports authentication failure, with the
> following log:
> 2023-09-23 10:55:08,821 INFO catalog: Invalidating all metadata. Version: 0
> 2023-09-23 10:55:08,983 WARN catalog: Exception encountered while connecting
> to the server : org.apache.hadoop.security.AccessControlException: Client
> cannot authenticate via:[TOKEN, KERBEROS]
> 2023-09-23 10:55:08,995 WARN catalog: Exception encountered while connecting
> to the server : org.apache.hadoop.security.AccessControlException: Client
> cannot authenticate via:[TOKEN, KERBEROS]
> 2023-09-23 10:55:09,052 ERROR catalog: Error loading cache pools:
> Java exception follows:
> java.io.IOException: DestHost:destPort host-10-235-65-168:9000 ,
> LocalHost:localPort host-10-235-65-90/191.188.1.223:0. Failed on local
> exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:840)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:815)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566)
> at org.apache.hadoop.ipc.Client.call(Client.java:1508)
> at org.apache.hadoop.ipc.Client.call(Client.java:1405)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy9.listCachePools(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.listCachePools(ClientNamenodeProtocolTranslatorPB.java:1502)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
> at com.sun.proxy.$Proxy10.listCachePools(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocol.CachePoolIterator.makeRequest(CachePoolIterator.java:51)
> at
> org.apache.hadoop.hdfs.protocol.CachePoolIterator.makeRequest(CachePoolIterator.java:33)
> at
> org.apache.hadoop.fs.BatchedRemoteIterator.makeRequest(BatchedRemoteIterator.java:77)
> at
> org.apache.hadoop.fs.BatchedRemoteIterator.makeRequestIfNeeded(BatchedRemoteIterator.java:85)
> at
> org.apache.hadoop.fs.BatchedRemoteIterator.hasNext(BatchedRemoteIterator.java:99)
> at
> org.apache.impala.catalog.CatalogServiceCatalog$CachePoolReader.run(CatalogServiceCatalog.java:544)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:1904)
> at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:138)
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:778)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:741)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:835)
> at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
> at org.apache.hadoop.ipc.Client.call(Client.java:1452)
> ... 23 more
> Caused by: org.apache.hadoop.security.AccessControlException: Client cannot
> authenticate via:[TOKEN, KERBEROS]
> at
> org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
> at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
> ... 26 more
> After logging in with impala-shell, an error is reported when executing the
> table creation statement, with the following log:
> ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
> Metastore:
> CAUSED BY: MetaException: Got exception: java.io.IOException
> DestHost:destPort host-10-235-65-168:9000 , LocalHost:localPort
> host-10-235-65-90/191.188.1.223:0. Failed on local exception:
> java.io.IOException: org.apache.hadoop.security.AccessControlException:
> Client cannot authenticate via:[TOKEN, KERBEROS]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
