[ https://issues.apache.org/jira/browse/HDFS-13419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432402#comment-16432402 ]
Kihwal Lee commented on HDFS-13419: ----------------------------------- Yes, a token is checked on connection establishment. If the connection stays up, token expiration won't affect on-going requests. Do you have a use case that require connection to be closed by server on token expiration? > client can communicate to server even if hdfs delegation token expired > ---------------------------------------------------------------------- > > Key: HDFS-13419 > URL: https://issues.apache.org/jira/browse/HDFS-13419 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: wangqiang.shen > Priority: Minor > > i was testing hdfs delegation token expired problem use spark streaming, if i > set my batch interval small than 10 sec, my spark streaming program will not > dead, but if batch interval was setted bigger than 10 sec, the spark > streaming program will dead because of hdfs delegation token expire problem, > and the exception as follows > {code:java} > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > token (HDFS_DELEGATION_TOKEN token 14042 for test) is expired > at org.apache.hadoop.ipc.Client.call(Client.java:1468) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) > at com.sun.proxy.$Proxy11.getListing(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554) > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy12.getListing(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969) > at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105) > at > org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755) > at > org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751) > at com.envisioncn.arch.App$2$1.call(App.java:120) > at com.envisioncn.arch.App$2$1.call(App.java:91) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:218) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:218) > at > org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902) > at > org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {code} > the spark streaming program only call FileSystem.listStatus function in every > batch > {code:java} > FileSystem fs = FileSystem.get(new Configuration()); > FileStatus[] status = fs.listStatus(new Path("/")); > for(FileStatus status1 : status){ > System.out.println(status1.getPath()); > } > {code} > and i found when hadoop client send rpc request to server, it will first get > a connection object and set up the connection if the connection dose not > exists.And it will get a SaslRpcClient to connect to the server side in the > connection setup stage.Also server will authenticate the client at the > connection setup stage. But if the connection exists, client will use the > existed connection, so the authentication stage will not happen. > The connection between client and server will be closed if it's idle time > exceeds ipc.client.connection.maxidletime, and > ipc.client.connection.maxidletime default value is 10sec. Therefore, if i > continue send request to server at fixed interval as long as the interval > small than 10sec, the connection will not be closed, so delegation token > expire problem will not happen. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org