Raja, I believe it would. I will check and get back to you but the easiest way for you to check is that the file should appear in HDFS under /user/<username>/datatorrent with the same filename as it is in your local filesystem.
Thanks On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <[email protected] > wrote: > > Thanks for the response Pramod. > > My quick question is, I see we should mention these properties in > dt-site.xml !! I am not sure about dt-site.xml, all I am using is only > properites.xml file, which I am using to pass some configuration to > application. > Can I set these in properties.xml file and it will still work ? > > > Regards, > Raja. > > From: Pramod Immaneni <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Monday, June 20, 2016 at 4:32 PM > > To: "[email protected]" <[email protected]> > Subject: Re: how to increase lifetime of hdfs delegation tokens ? > > Hi Raja, > > Yes the keytab would be copied over to HDFS and reused for getting a new > token before the old one expires. By default it is 7 days. If it is > different in your cluster please set the > properties dt.resourcemanager.delegation.token.max-lifetime and > dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't > the default keytab to be copied over into HDFS and reused you can specify > your own keytab file for fetching a new token by putting it in HDFS and > specifying the property dt.authentication.store.keytab.All this is > described in the document that Thomas sent over. > > Thanks > > On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli < > [email protected]> wrote: > >> >> Hi Thomas, >> >> To ensure auto renewal of delegation tokens life time, Can I use the the >> below properties in properties.xml file ? >> >> <property> >> <name>dt.authentication.principal</name> >> <value>kerberos-principal-of-user</value> >> </property> >> <property> >> <name>dt.authentication.keytab</name> >> <value>absolute-path-to-keytab-file</value> >> </property> >> >> >> FYI, >> I am launching application from Apex CLI! And till this time I haven’t >> used the above properties when launching apex applications in our secure >> hadoop environment, still they worked fine without any issues, but failing >> after 7days!! >> >> If I set the above properties in properties.xml, will that do >> auto-renewal and run successfully without any issues of failing again due >> to delegation token lifetime expiry ?? >> >> Please advise. >> >> >> Thanks a lot in advance. >> >> >> Regards, >> Raja. >> >> From: "Raja.Aravapalli" <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Sunday, June 19, 2016 at 3:30 PM >> >> To: "[email protected]" <[email protected]> >> Subject: Re: how to increase lifetime of hdfs delegation tokens ? >> >> >> Thanks a lot Thomas. >> >> Will take this as reference and test our application. Great! >> >> >> Regards, >> Raja. >> >> From: Thomas Weise <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Sunday, June 19, 2016 at 2:01 PM >> To: "[email protected]" <[email protected]> >> Subject: Re: how to increase lifetime of hdfs delegation tokens ? >> >> Token expiration working as expected! >> >> Please have a look on how to extend or refresh it: >> >> >> https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh >> >> Thanks, >> Thomas >> >> >> On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli < >> [email protected]> wrote: >> >>> >>> Hi, >>> >>> My Apex application failed exactly after running 7days in our >>> distributed hadoop environment, with delegation token expiry!! >>> >>> Can someone pls help me with details, on how I can increase the >>> delegation token time to lifetime or any other process running in parallel >>> to renew the tokens ? >>> >>> *Exception details below:* >>> >>> ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - >>> Failed to close inode 11111111 >>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >>> token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired >>> at org.apache.hadoop.ipc.Client.call(Client.java:1427) >>> at org.apache.hadoop.ipc.Client.call(Client.java:1358) >>> >>> >>> >>> Thanks a lot in advance. >>> >>> >>> Regards, >>> Raja. >>> >>> >> >
