What is the JIRA ticket for this problem? SPARK-25355 <https://issues.apache.org/jira/browse/SPARK-25355> was marked as resolved for 2 years ago. But now there are a lot of new comments on whether things work and not.
SPARK-39033 Support --proxy-user for Spark on K8s not working <https://issues.apache.org/jira/projects/SPARK/issues/SPARK-39033> is this yours problem? By having a good JIRA where the problem is discussed and the same with the PR description helps when others try to change the code in the future. tir. 3. mai 2022 kl. 19:20 skrev Pralabh Kumar <pralabhku...@gmail.com>: > Hi Steve > > Thx for the input . Actually I have wrongly put the error stack > trace(complete error trace and other details are available in the Jira , > apologies for the same. ) . The error is related to authenticatication > and it's happening only when the proxy user option is being used , > otherwise things are working fine. > > Also my intention wasn't to point out any mistakes or escalate the problem > to the community. I was just looking for help and direction to solve the > issue. > > 22/04/26 08:54:40 DEBUG Client: closing ipc connection to <server>/<ip>:8020: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS] > java.io.IOException: org.apache.hadoop.security.AccessControlException: > Client cannot authenticate via:[TOKEN, KERBEROS] > at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757) > at java.base/java.security.AccessController.doPrivileged(Native Method) > at java.base/javax.security.auth.Subject.doAs(Unknown Source) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) > at > org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720) > at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813) > at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558) > at org.apache.hadoop.ipc.Client.call(Client.java:1389) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown > Source) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > at java.base/java.lang.reflect.Method.invoke(Unknown Source) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) > at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654) > at > org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579) > at > org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65) > at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270) > at org.apache.hadoop.fs.Globber.glob(Globber.java:149) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2067) > at > org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:318) > at > org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273) > at > org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271) > at > scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293) > at > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > at > scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38) > at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293) > at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108) > at > org.apache.spark.util.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:271) > at > org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:364) > at scala.Option.map(Option.scala:230) > at > org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:364) > at > org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898) > at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165) > at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163) > at java.base/java.security.AccessController.doPrivileged(Native Method) > at java.base/javax.security.auth.Subject.doAs(Unknown Source) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) > at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) > > > > Again Thx for the input and advice regarding documentation and > apologies for putting the wrong error stack earlier. > > Regards > Pralabh Kumar > On Tue, May 3, 2022 at 7:39 PM Steve Loughran <ste...@cloudera.com> wrote: > >> >> Prablah, did you follow the URL provided in the exception message? i put >> a lot of effort in to improving the diagnostics, where the wiki articles >> are part of the troubleshooing process >> https://issues.apache.org/jira/browse/HADOOP-7469 >> >> it's really disappointing when people escalate the problem to open source >> developers before trying to fix the problem themselves, in this case, read >> the error message. >> >> now, if there is some k8s related issue which makes this more common, you >> are encouraged to update the wiki entry with a new cause. documentation is >> an important contribution to open source projects, and if you have >> discovered a new way to recreate the failure, it would be welcome. which >> reminds me, i have to add something to connection reset and docker which >> comes down to "turn off http keepalive in maven builds" >> >> -Steve >> >> >> >> >> >> On Sat, 30 Apr 2022 at 10:45, Gabor Somogyi <gabor.g.somo...@gmail.com> >> wrote: >> >>> Hi, >>> >>> Please be aware that ConnectionRefused exception is has nothing to do w/ >>> authentication. See the description from Hadoop wiki: >>> "You get a ConnectionRefused >>> <https://cwiki.apache.org/confluence/display/HADOOP2/ConnectionRefused> >>> Exception >>> when there is a machine at the address specified, but there is no program >>> listening on the specific TCP port the client is using -and there is no >>> firewall in the way silently dropping TCP connection requests. If you do >>> not know what a TCP connection request is, please consult the >>> specification <http://www.ietf.org/rfc/rfc793.txt>." >>> >>> This means the namenode on host:port is not reachable in the TCP layer. >>> Maybe there are multiple issues but I'm pretty sure that something is wrong >>> in the K8S net config. >>> >>> BR, >>> G >>> >>> >>> On Fri, Apr 29, 2022 at 6:23 PM Pralabh Kumar <pralabhku...@gmail.com> >>> wrote: >>> >>>> Hi dev Team >>>> >>>> Spark-25355 added the functionality of the proxy user on K8s . However >>>> proxy user on K8s with Kerberized HDFS is not working . It is throwing >>>> exception and >>>> >>>> 22/04/21 17:50:30 WARN Client: Exception encountered while connecting >>>> to the server : org.apache.hadoop.security.AccessControlException: Client >>>> cannot authenticate via:[TOKEN, KERBEROS] >>>> >>>> >>>> Exception in thread "main" java.net.ConnectException: Call From >>>> <driverpod> to <namenode> failed on connection exception: >>>> java.net.ConnectException: Connection refused; For more details see: http: >>>> //wiki.apache.org/hadoop/ConnectionRefused >>>> >>>> at >>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>> Method) >>>> >>>> at >>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown >>>> Source) >>>> >>>> at >>>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown >>>> Source) >>>> >>>> at java.base/java.lang.reflect.Constructor.newInstance(Unknown >>>> Source) >>>> >>>> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) >>>> >>>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755) >>>> >>>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) >>>> >>>> at org.apache.hadoop.ipc.Client.call(Client.java:1443) >>>> >>>> at org.apache.hadoop.ipc.Client.call(Client.java:1353) >>>> >>>> at >>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) >>>> >>>> at >>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) >>>> >>>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source) >>>> >>>> at >>>> >>>> >>>> >>>> On debugging deep , we found the proxy user doesn't have access to >>>> delegation tokens in case of K8s .SparkSubmit.submit explicitly creating >>>> the proxy user and this user doesn't have delegation token. >>>> >>>> >>>> Please help me with the same. >>>> >>>> >>>> Regards >>>> >>>> Pralabh Kumar >>>> >>>> >>>> >>>> -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
