Davy Xu created IMPALA-12458:
--------------------------------
Summary: When impalad and catalogd are started concurrently on a
Kerberos-enabled node, catalogd will report authentication failure
Key: IMPALA-12458
URL: https://issues.apache.org/jira/browse/IMPALA-12458
Project: IMPALA
Issue Type: Bug
Components: Catalog
Affects Versions: Impala 4.0.0
Environment: RHEL7
Reporter: Davy Xu
After enabling impala kerberos, start impalad and catalogd concurrently on a
node. The catalogd log periodically reports authentication failure, with the
following log:
2023-09-23 10:55:08,821 INFO catalog: Invalidating all metadata. Version: 0
2023-09-23 10:55:08,983 WARN catalog: Exception encountered while connecting to
the server : org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
2023-09-23 10:55:08,995 WARN catalog: Exception encountered while connecting to
the server : org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
2023-09-23 10:55:09,052 ERROR catalog: Error loading cache pools:
Java exception follows:
java.io.IOException: DestHost:destPort host-10-235-65-168:9000 ,
LocalHost:localPort host-10-235-65-90/191.188.1.223:0. Failed on local
exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:840)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:815)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566)
at org.apache.hadoop.ipc.Client.call(Client.java:1508)
at org.apache.hadoop.ipc.Client.call(Client.java:1405)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.listCachePools(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.listCachePools(ClientNamenodeProtocolTranslatorPB.java:1502)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
at com.sun.proxy.$Proxy10.listCachePools(Unknown Source)
at
org.apache.hadoop.hdfs.protocol.CachePoolIterator.makeRequest(CachePoolIterator.java:51)
at
org.apache.hadoop.hdfs.protocol.CachePoolIterator.makeRequest(CachePoolIterator.java:33)
at
org.apache.hadoop.fs.BatchedRemoteIterator.makeRequest(BatchedRemoteIterator.java:77)
at
org.apache.hadoop.fs.BatchedRemoteIterator.makeRequestIfNeeded(BatchedRemoteIterator.java:85)
at
org.apache.hadoop.fs.BatchedRemoteIterator.hasNext(BatchedRemoteIterator.java:99)
at
org.apache.impala.catalog.CatalogServiceCatalog$CachePoolReader.run(CatalogServiceCatalog.java:544)
at
org.apache.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:1904)
at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:138)
Caused by: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:778)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:741)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:835)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
at org.apache.hadoop.ipc.Client.call(Client.java:1452)
... 23 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
at
org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
... 26 more
After logging in with impala-shell, an error is reported when executing the
table creation statement, with the following log:
ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
Metastore:
CAUSED BY: MetaException: Got exception: java.io.IOException DestHost:destPort
host-10-235-65-168:9000 , LocalHost:localPort
host-10-235-65-90/191.188.1.223:0. Failed on local exception:
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
