Hi Pramod, As suggested in step2, I added the properties you provided below, and ran a new job, to see the debug lines "number of tokens: " and "updated token: "
And, as you said, I can see those lines in the application log now.. Does this mean, the renewal is working as expected ? So, will my application run continously even after 7days now ? We have only changed refersh times here… how does this ensure the application runs indefinite time… Regards, Raja. From: "Raja.Aravapalli" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Saturday, July 2, 2016 at 9:52 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Thanks Pramod, I will these suggestions and let you know. Thanks a lot. Regards, Raja. From: Pramod Immaneni <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Friday, July 1, 2016 at 8:36 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Hi Raja, Some questions for you and also some options for you to verify without waiting for a long time. 1. Do you see a warning message like ""No keytab specified for refreshing tokens, application may not be able to run indefinitely" when application is being launched from command line. 2. For testing, set the parameters below in your dt-site.xml and launch your application. This will make the application think that the tokens are only valid for 5 minutes and within 5 * 0.7 = 3.5 minutes (token refresh factor is 0.7) the application should try to get new tokens. It should print the debug lines "number of tokens: " and "updated token: " in the application master logs. The application master logs are in the log file of the first container of the application. Let me know if you see those log lines. <property> <name>dt.resourcemanager.delegation.token.max-lifetime</name> <value>300000</value> </property> <property> <name>dt.namenode.delegation.token.max-lifetime</name> <value>300000</value> </property> <property> <name>dt.attr.DEBUG</name> <value>true</value> </property> For more information about application auto-fetching new tokens read here https://github.com/apache/apex-core/blob/master/docs/security.md Thanks On Fri, Jul 1, 2016 at 1:08 PM, Raja.Aravapalli <[email protected]<mailto:[email protected]>> wrote: Thanks a lot Pramod. Will wait for your response. Regards, Raja. From: Pramod Immaneni <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Friday, July 1, 2016 at 10:56 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Hi Raja, Let me look at this and get back to you. Thanks On Thu, Jun 30, 2016 at 11:20 PM, Raja.Aravapalli <[email protected]<mailto:[email protected]>> wrote: Can someone pls help me, how can I ensure, my apex application doesn’t fail after 7days… Thanks a lot. Regards, Raja. From: "Raja.Aravapalli" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, June 30, 2016 at 6:06 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Hi, I triggered my application by specifying properties, “dt.authentication.principal” & “dt.authentication.keytab” , BUT, did not specify the property “dt.authentication.store.keytab”. I also observed the keytab is copied to hdfs location “/user/<user>/datatorrent”. But, still my apex application failed after 7days!!! I am setting these properties in “properties.xml” file! How can I ensure my settings are working correct. Having waiting for 7days to learn its failure is a very tough thing. Hope there should be some other alternatives. Can someone pls help me fix this …. Thanks a lot !! Regards, Raja. From: "Raja.Aravapalli" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, June 20, 2016 at 5:43 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Sure Pramod. Please respond on this mail chain when you get to know.. Thanks very much. Regards, Raja. From: Pramod Immaneni <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, June 20, 2016 at 4:54 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Raja, I believe it would. I will check and get back to you but the easiest way for you to check is that the file should appear in HDFS under /user/<username>/datatorrent with the same filename as it is in your local filesystem. Thanks On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <[email protected]<mailto:[email protected]>> wrote: Thanks for the response Pramod. My quick question is, I see we should mention these properties in dt-site.xml !! I am not sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass some configuration to application. Can I set these in properties.xml file and it will still work ? Regards, Raja. From: Pramod Immaneni <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, June 20, 2016 at 4:32 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Hi Raja, Yes the keytab would be copied over to HDFS and reused for getting a new token before the old one expires. By default it is 7 days. If it is different in your cluster please set the properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying the property dt.authentication.store.keytab.All this is described in the document that Thomas sent over. Thanks On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <[email protected]<mailto:[email protected]>> wrote: Hi Thomas, To ensure auto renewal of delegation tokens life time, Can I use the the below properties in properties.xml file ? <property> <name>dt.authentication.principal</name> <value>kerberos-principal-of-user</value> </property> <property> <name>dt.authentication.keytab</name> <value>absolute-path-to-keytab-file</value> </property> FYI, I am launching application from Apex CLI! And till this time I haven’t used the above properties when launching apex applications in our secure hadoop environment, still they worked fine without any issues, but failing after 7days!! If I set the above properties in properties.xml, will that do auto-renewal and run successfully without any issues of failing again due to delegation token lifetime expiry ?? Please advise. Thanks a lot in advance. Regards, Raja. From: "Raja.Aravapalli" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Sunday, June 19, 2016 at 3:30 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Thanks a lot Thomas. Will take this as reference and test our application. Great! Regards, Raja. From: Thomas Weise <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Sunday, June 19, 2016 at 2:01 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? Token expiration working as expected! Please have a look on how to extend or refresh it: https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh Thanks, Thomas On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <[email protected]<mailto:[email protected]>> wrote: Hi, My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!! Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ? Exception details below: ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired at org.apache.hadoop.ipc.Client.call(Client.java:1427) at org.apache.hadoop.ipc.Client.call(Client.java:1358) Thanks a lot in advance. Regards, Raja.
