Sure Pramod. Please respond on this mail chain when you get to know.. Thanks very much.
Regards, Raja. From: Pramod Immaneni <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, June 20, 2016 at 4:54 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Raja, I believe it would. I will check and get back to you but the easiest way for you to check is that the file should appear in HDFS under /user/<username>/datatorrent with the same filename as it is in your local filesystem. Thanks On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <[email protected]<mailto:[email protected]>> wrote: Thanks for the response Pramod. My quick question is, I see we should mention these properties in dt-site.xml !! I am not sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass some configuration to application. Can I set these in properties.xml file and it will still work ? Regards, Raja. From: Pramod Immaneni <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, June 20, 2016 at 4:32 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Hi Raja, Yes the keytab would be copied over to HDFS and reused for getting a new token before the old one expires. By default it is 7 days. If it is different in your cluster please set the properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying the property dt.authentication.store.keytab.All this is described in the document that Thomas sent over. Thanks On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <[email protected]<mailto:[email protected]>> wrote: Hi Thomas, To ensure auto renewal of delegation tokens life time, Can I use the the below properties in properties.xml file ? <property> <name>dt.authentication.principal</name> <value>kerberos-principal-of-user</value> </property> <property> <name>dt.authentication.keytab</name> <value>absolute-path-to-keytab-file</value> </property> FYI, I am launching application from Apex CLI! And till this time I haven’t used the above properties when launching apex applications in our secure hadoop environment, still they worked fine without any issues, but failing after 7days!! If I set the above properties in properties.xml, will that do auto-renewal and run successfully without any issues of failing again due to delegation token lifetime expiry ?? Please advise. Thanks a lot in advance. Regards, Raja. From: "Raja.Aravapalli" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Sunday, June 19, 2016 at 3:30 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Thanks a lot Thomas. Will take this as reference and test our application. Great! Regards, Raja. From: Thomas Weise <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Sunday, June 19, 2016 at 2:01 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Token expiration working as expected! Please have a look on how to extend or refresh it: https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh Thanks, Thomas On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <[email protected]<mailto:[email protected]>> wrote: Hi, My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!! Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ? Exception details below: ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired at org.apache.hadoop.ipc.Client.call(Client.java:1427) at org.apache.hadoop.ipc.Client.call(Client.java:1358) Thanks a lot in advance. Regards, Raja.
