GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/11721
[SPARK-13885][YARN] Fix attempt id regression for Spark running on Yarn
## What changes were proposed in this pull request?
This regression is introduced in #9182, previously attempt id is simply as
counter "1" or "2". With the change of #9182, it is changed to full name as
"appattemtp-xxx-00001", this will affect all the parts which uses this attempt
id, like event log file name, history server app url link. So here change it
back to the counter to keep consistent with previous code.
Also revert back this patch #11518, this patch fix the url link of history
log according to the new way of attempt id, since here we change back to the
previous way, so this patch is not necessary, here to revert it.
Also clean "spark.yarn.app.id" and "spark.yarn.app.attemptId", since it is
useless now.
## How was this patch tested?
Test it with unit test and manually test different scenario:
1. application running in yarn-client mode.
2. application running in yarn-cluster mode.
3. application running in yarn-cluster mode with multiple attempts.
Checked both the event log file name and url link.
CC @vanzin @tgravescs , please help to review, thanks a lot.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-13885
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11721.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11721
----
commit b777533ffd39e91cab6b6a5c9a28364e81aca50e
Author: jerryshao <[email protected]>
Date: 2016-03-15T04:26:26Z
Revert "[SPARK-13675][UI] Fix wrong historyserver url link for application
running in yarn cluster mode"
This reverts commit 9e86e6efd136182bb00fa925c3818c9baccbd1fc.
commit 6339a01574c619cc4a36f95966b8ba65a640f7fe
Author: jerryshao <[email protected]>
Date: 2016-03-15T05:43:02Z
Change to the correct attempt id
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
