Hi Raja, Do you have admin access to the cluster? If so look for the properties yarn.resourcemanager.delegation.token.max-lifetime and dfs.namenode.delegation.token.max-lifetime in your Hadoop configuration. They should currently be set to 7 days. This is what is making the tokens expire after 7days. For testing you can change it to half hour for example and restart hadoop services. You can now see without waiting for a long time if the application is getting new tokens and running with them.
Thanks On Tue, Jul 5, 2016 at 2:23 PM, Raja.Aravapalli <[email protected]> wrote: > > Hi Pramod, > > As suggested in step2, I added the properties you provided below, and ran > a new job, to see the debug lines "number of tokens: " and "updated > token: " > > And, as you said, I can see those lines in the application log now.. > > Does this mean, the renewal is working as expected ? So, will my > application run continously even after 7days now ? We have only changed > refersh times here… how does this ensure the application runs indefinite > time… > > > > > Regards, > Raja. > > From: "Raja.Aravapalli" <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Saturday, July 2, 2016 at 9:52 AM > > To: "[email protected]" <[email protected]> > Subject: Re: how to increase lifetime of hdfs delegation tokens ? > > > Thanks Pramod, I will these suggestions and let you know. Thanks a lot. > > > Regards, > Raja. > > From: Pramod Immaneni <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Friday, July 1, 2016 at 8:36 PM > To: "[email protected]" <[email protected]> > Subject: Re: how to increase lifetime of hdfs delegation tokens ? > > Hi Raja, > > Some questions for you and also some options for you to verify without > waiting for a long time. > > 1. Do you see a warning message like ""No keytab specified for refreshing > tokens, application may not be able to run indefinitely" when application > is being launched from command line. > > 2. For testing, set the parameters below in your dt-site.xml and launch > your application. This will make the application think that the tokens are > only valid for 5 minutes and within 5 * 0.7 = 3.5 minutes (token refresh > factor is 0.7) the application should try to get new tokens. It should > print the debug lines "number of tokens: " and "updated token: " in the > application master logs. The application master logs are in the log file of > the first container of the application. Let me know if you see those log > lines. > > <property> > <name>dt.resourcemanager.delegation.token.max-lifetime</name> > <value>300000</value> > </property> > > <property> > <name>dt.namenode.delegation.token.max-lifetime</name> > <value>300000</value> > </property> > > <property> > <name>dt.attr.DEBUG</name> > <value>true</value> > </property> > > For more information about application auto-fetching new tokens read here > > https://github.com/apache/apex-core/blob/master/docs/security.md > > Thanks > > On Fri, Jul 1, 2016 at 1:08 PM, Raja.Aravapalli < > [email protected]> wrote: > >> >> Thanks a lot Pramod. Will wait for your response. >> >> >> Regards, >> Raja. >> >> From: Pramod Immaneni <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Friday, July 1, 2016 at 10:56 AM >> >> To: "[email protected]" <[email protected]> >> Subject: Re: how to increase lifetime of hdfs delegation tokens ? >> >> Hi Raja, >> >> Let me look at this and get back to you. >> >> Thanks >> >> On Thu, Jun 30, 2016 at 11:20 PM, Raja.Aravapalli < >> [email protected]> wrote: >> >>> >>> Can someone pls help me, how can I ensure, my apex application doesn’t >>> fail after 7days… >>> >>> Thanks a lot. >>> >>> >>> Regards, >>> Raja. >>> >>> From: "Raja.Aravapalli" <[email protected]> >>> Reply-To: "[email protected]" <[email protected]> >>> Date: Thursday, June 30, 2016 at 6:06 AM >>> >>> To: "[email protected]" <[email protected]> >>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? >>> >>> >>> Hi, >>> >>> I triggered my application by specifying properties, “ >>> dt.authentication.principal” & “dt.authentication.keytab” , BUT, did >>> not specify the property “dt.authentication.store.keytab”. >>> >>> I also observed the keytab is copied to hdfs location >>> “/user/<user>/datatorrent”. But, still my apex application failed after >>> 7days!!! >>> >>> I am setting these properties in “properties.xml” file! >>> >>> How can I ensure my settings are working correct. Having waiting for >>> 7days to learn its failure is a very tough thing. Hope there should be some >>> other alternatives. >>> >>> Can someone pls help me fix this …. Thanks a lot !! >>> >>> >>> Regards, >>> Raja. >>> >>> From: "Raja.Aravapalli" <[email protected]> >>> Reply-To: "[email protected]" <[email protected]> >>> Date: Monday, June 20, 2016 at 5:43 PM >>> To: "[email protected]" <[email protected]> >>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? >>> >>> >>> Sure Pramod. Please respond on this mail chain when you get to know.. >>> >>> Thanks very much. >>> >>> >>> Regards, >>> Raja. >>> >>> From: Pramod Immaneni <[email protected]> >>> Reply-To: "[email protected]" <[email protected]> >>> Date: Monday, June 20, 2016 at 4:54 PM >>> To: "[email protected]" <[email protected]> >>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? >>> >>> Raja, >>> >>> I believe it would. I will check and get back to you but the easiest way >>> for you to check is that the file should appear in HDFS under >>> /user/<username>/datatorrent with the same filename as it is in your local >>> filesystem. >>> >>> Thanks >>> >>> On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli < >>> [email protected]> wrote: >>> >>>> >>>> Thanks for the response Pramod. >>>> >>>> My quick question is, I see we should mention these properties in >>>> dt-site.xml !! I am not sure about dt-site.xml, all I am using is only >>>> properites.xml file, which I am using to pass some configuration to >>>> application. >>>> Can I set these in properties.xml file and it will still work ? >>>> >>>> >>>> Regards, >>>> Raja. >>>> >>>> From: Pramod Immaneni <[email protected]> >>>> Reply-To: "[email protected]" <[email protected]> >>>> Date: Monday, June 20, 2016 at 4:32 PM >>>> >>>> To: "[email protected]" <[email protected]> >>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? >>>> >>>> Hi Raja, >>>> >>>> Yes the keytab would be copied over to HDFS and reused for getting a >>>> new token before the old one expires. By default it is 7 days. If it is >>>> different in your cluster please set the >>>> properties dt.resourcemanager.delegation.token.max-lifetime and >>>> dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't >>>> the default keytab to be copied over into HDFS and reused you can specify >>>> your own keytab file for fetching a new token by putting it in HDFS and >>>> specifying the property dt.authentication.store.keytab.All this is >>>> described in the document that Thomas sent over. >>>> >>>> Thanks >>>> >>>> On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli < >>>> [email protected]> wrote: >>>> >>>>> >>>>> Hi Thomas, >>>>> >>>>> To ensure auto renewal of delegation tokens life time, Can I use the >>>>> the below properties in properties.xml file ? >>>>> >>>>> <property> >>>>> <name>dt.authentication.principal</name> >>>>> <value>kerberos-principal-of-user</value> >>>>> </property> >>>>> <property> >>>>> <name>dt.authentication.keytab</name> >>>>> <value>absolute-path-to-keytab-file</value> >>>>> </property> >>>>> >>>>> >>>>> FYI, >>>>> I am launching application from Apex CLI! And till this time I haven’t >>>>> used the above properties when launching apex applications in our secure >>>>> hadoop environment, still they worked fine without any issues, but failing >>>>> after 7days!! >>>>> >>>>> If I set the above properties in properties.xml, will that do >>>>> auto-renewal and run successfully without any issues of failing again due >>>>> to delegation token lifetime expiry ?? >>>>> >>>>> Please advise. >>>>> >>>>> >>>>> Thanks a lot in advance. >>>>> >>>>> >>>>> Regards, >>>>> Raja. >>>>> >>>>> From: "Raja.Aravapalli" <[email protected]> >>>>> Reply-To: "[email protected]" <[email protected]> >>>>> Date: Sunday, June 19, 2016 at 3:30 PM >>>>> >>>>> To: "[email protected]" <[email protected]> >>>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? >>>>> >>>>> >>>>> Thanks a lot Thomas. >>>>> >>>>> Will take this as reference and test our application. Great! >>>>> >>>>> >>>>> Regards, >>>>> Raja. >>>>> >>>>> From: Thomas Weise <[email protected]> >>>>> Reply-To: "[email protected]" <[email protected]> >>>>> Date: Sunday, June 19, 2016 at 2:01 PM >>>>> To: "[email protected]" <[email protected]> >>>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ? >>>>> >>>>> Token expiration working as expected! >>>>> >>>>> Please have a look on how to extend or refresh it: >>>>> >>>>> >>>>> https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh >>>>> >>>>> Thanks, >>>>> Thomas >>>>> >>>>> >>>>> On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> My Apex application failed exactly after running 7days in our >>>>>> distributed hadoop environment, with delegation token expiry!! >>>>>> >>>>>> Can someone pls help me with details, on how I can increase the >>>>>> delegation token time to lifetime or any other process running in >>>>>> parallel >>>>>> to renew the tokens ? >>>>>> >>>>>> *Exception details below:* >>>>>> >>>>>> ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - >>>>>> Failed to close inode 11111111 >>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >>>>>> token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1427) >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1358) >>>>>> >>>>>> >>>>>> >>>>>> Thanks a lot in advance. >>>>>> >>>>>> >>>>>> Regards, >>>>>> Raja. >>>>>> >>>>>> >>>>> >>>> >>> >> >
