Sammi Chen created HADOOP-20082:
-----------------------------------
Summary: Resolved FQDN by dnsjava from short name has tailing "."
Key: HADOOP-20082
URL: https://issues.apache.org/jira/browse/HADOOP-20082
Project: Hadoop Common
Issue Type: Bug
Reporter: Sammi Chen
Assignee: Sammi Chen
A failure of Spark 3 jobs accessing HDFS-backed Hive data.
{code:java}
26/10/27 10:00:11 ERROR executor.Executor: [Executor task launch worker for
task 2.0 in stage 0.0 (TID 1)]: Exception in task 2.0 in stage 0.0 (TID 1)
java.io.IOException: DestHost:destPort NN1.example.com.:8020 ,
LocalHost:localPort *.example.com/10.30.102.11:0. Failed on local exception:
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:892)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:867)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1620)
at org.apache.hadoop.ipc.Client.call(Client.java:1562)
at org.apache.hadoop.ipc.Client.call(Client.java:1459)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy38.getBlockLocations(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:330)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
at com.sun.proxy.$Proxy39.getBlockLocations(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:880)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:869)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:858)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1026)
at
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:344)
at
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:340)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:357)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:957)
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:546)
at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:369)
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:61)
at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:112)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1972)
at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:290)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:289)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:247)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:62)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:62)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:62)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:62)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:95)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:808)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1910)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:771)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:866)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:430)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1681)
at org.apache.hadoop.ipc.Client.call(Client.java:1506)
... 58 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
at
org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:174)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:391)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:639)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:430)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:853)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:849)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1910)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:849)
... 61 more{code}
It appeared that executors fail to authenticate due to a trailing dot appended
to the NameNode hostname (e.g., `nn1.example.com.` vs `nn1.example.com`).
The problem is that the delegation token acquired by spark AM is using the fqdn
nn1.example.com, so the delegation service is "nn1.example.com:8020". And the
hive table file location use simple name "nn1", so when the executor try to
access HDFS, short name "nn1" is resolved to "nn1.example.com." by dnsjava, so
it doesn't match the "nn1.example.com:8080" in the delegation token.
If both side use the same fdqn or shortname, then resolved service name will
match.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
