GitHub user harishreedharan opened a pull request:
https://github.com/apache/spark/pull/7394
[SPARK-8851][YARN] In Client mode, make sure the client logs in and updates
tokens
In client side, the flow is SparkSubmit -> SparkContext -> yarn/Client.
Since the yarn client only gets a cloned config and the staging dir is set
here, it is not really possible to do re-logins in the SparkContext. So, do the
initial logins in Spark Submit and do re-logins as we do now in the AM, but the
Client behaves like an executor in this specific context and reads the
credentials file to update the tokens. This way, even if the streaming context
is started up from checkpoint - it is fine since we have logged in from
SparkSubmit itself itself.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/harishreedharan/spark yarn-client-login
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7394.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7394
----
commit 58b1969ae4efc0c995ffa817f54e370721f4058f
Author: Hari Shreedharan <[email protected]>
Date: 2015-07-10T23:19:51Z
Simple attempt 1.
commit 26f8bfa20f15b2a1be62d23b9b1d09a7b4b874d1
Author: Hari Shreedharan <[email protected]>
Date: 2015-07-14T07:47:42Z
[SPARK-8851][YARN] In Client mode, make sure the client logs in and updates
tokens.
In client side, the flow is SparkSubmit -> SparkContext -> yarn/Client.
Since the yarn client only gets a cloned config and the staging dir is set
here, it is not really possible to do re-logins in the SparkContext. So, do the
initial logins in Spark Submit and do re-logins as we do now in the AM, but the
Client behaves like an executor in this specific context and reads the
credentials file to update the tokens. This way, even if the streaming context
is started up from checkpoint - it is fine since we have logged in from
SparkSubmit itself itself.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
