Re: Long running Spark job on YARN throws "No AMRMToken"

2016-02-13 Thread Steve Loughran
On 11 Feb 2016, at 15:24, Prabhu Joseph > wrote: Steve, When ResourceManager is submitted with an application, AMLauncher creates the token YARN_AM_RM_TOKEN (token used between RM and AM). When ApplicationMaster is

Re: Long running Spark job on YARN throws "No AMRMToken"

2016-02-11 Thread Prabhu Joseph
Steve, When ResourceManager is submitted with an application, AMLauncher creates the token YARN_AM_RM_TOKEN (token used between RM and AM). When ApplicationMaster is launched, it tries to contact RM for registering request, allocate request to receive containers, finish request. In all the

Re: Long running Spark job on YARN throws "No AMRMToken"

2016-02-09 Thread Steve Loughran
On 9 Feb 2016, at 05:55, Prabhu Joseph > wrote: + Spark-Dev On Tue, Feb 9, 2016 at 10:04 AM, Prabhu Joseph > wrote: Hi All, A long running Spark job on YARN throws

Re: Long running Spark job on YARN throws "No AMRMToken"

2016-02-09 Thread Steve Loughran
On 9 Feb 2016, at 11:26, Steve Loughran > wrote: On 9 Feb 2016, at 05:55, Prabhu Joseph > wrote: + Spark-Dev On Tue, Feb 9, 2016 at 10:04 AM, Prabhu Joseph

Re: Long running Spark job on YARN throws "No AMRMToken"

2016-02-09 Thread Hari Shreedharan
The credentials file approach (using keytab for spark apps) will only update HDFS tokens. YARN's AMRM tokens should be taken care of by YARN internally. Steve - correct me if I am wrong here: If the AMRM tokens are disappearing it might be a YARN bug (does the AMRM token have a 7 day limit as

Re: Long running Spark job on YARN throws "No AMRMToken"

2016-02-08 Thread Prabhu Joseph
+ Spark-Dev On Tue, Feb 9, 2016 at 10:04 AM, Prabhu Joseph wrote: > Hi All, > > A long running Spark job on YARN throws below exception after running > for few days. > > yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. >