[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-50025677 +1 thanks @rahulsinghaliitd ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1094 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49874400 I've only used the sbt/sbt scalastyle target. I'm not sure if there is a reason it wasn't implemented in maven. you could file a jira for it and get comments from the folks who did it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49874426 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49874824 QA tests have started for PR 1094. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17034/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49889792 QA results for PR 1094:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17034/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49733427 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49733823 QA tests have started for PR 1094. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16962/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49733906 QA results for PR 1094:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16962/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49736022 Jenkins, this is okay to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49736019 @rahulsinghaliitd sorry for this dragging on I forgot to kick jenkins. Looks like its complaining about scalastyle. Can you please update. /home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala message=File line length exceeds 100 characters line=172 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user rahulsinghaliitd commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49828457 @tgravescs No problem at all. Updated patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user rahulsinghaliitd commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49830504 @tgravescs There does not seem to be a maven goal to check scalastyle, isn't this something we want? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/1094#discussion_r15177387 --- Diff: yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala --- @@ -289,7 +289,7 @@ class ExecutorLauncher(args: ApplicationMasterArguments, conf: Configuration, sp .asInstanceOf[FinishApplicationMasterRequest] finishReq.setAppAttemptId(appAttemptId) finishReq.setFinishApplicationStatus(status) - finishReq.setTrackingUrl(sparkConf.get(spark.yarn.historyServer.address, )) + finishReq.setTrackingUrl(sparkConf.get(spark.driver.appUIHistoryAddress, )) --- End diff -- typo, need the other in the default case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user rahulsinghaliitd commented on a diff in the pull request: https://github.com/apache/spark/pull/1094#discussion_r15206722 --- Diff: yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala --- @@ -289,7 +289,7 @@ class ExecutorLauncher(args: ApplicationMasterArguments, conf: Configuration, sp .asInstanceOf[FinishApplicationMasterRequest] finishReq.setAppAttemptId(appAttemptId) finishReq.setFinishApplicationStatus(status) - finishReq.setTrackingUrl(sparkConf.get(spark.yarn.historyServer.address, )) + finishReq.setTrackingUrl(sparkConf.get(spark.driver.appUIHistoryAddress, )) --- End diff -- woah, thanks for catching that! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user rahulsinghaliitd commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49572410 @tgravescs updated according to your comments and rebased to current HEAD of master branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/1094#discussion_r15112301 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala --- @@ -114,7 +114,7 @@ class HistoryServer( attachHandler(createStaticHandler(SparkUI.STATIC_RESOURCE_DIR, /static)) val contextHandler = new ServletContextHandler -contextHandler.setContextPath(/history) + contextHandler.setContextPath(HistoryServer.UI_PATH_PREFIX.stripSuffix(/)) --- End diff -- I think it makes more sense to do this the other way. have the UI_PATH_PREFIX /history and add in the / wherever its needed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user rahulsinghaliitd commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49133859 @tgravescs updated according to your comments and rebased to current HEAD of master branch. Thanks for following up on this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/1094#discussion_r14936355
--- Diff:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
---
@@ -132,4 +135,17 @@ object YarnSparkHadoopUtil {
}
}
+ def getUIHistoryAddress(sc: SparkContext, conf: SparkConf) : String = {
+val eventLogDir = sc.eventLogger match {
+ case Some(logger) = logger.logDir.split(/).last
--- End diff --
I think it would be better to add a routine to the eventLogger to just give
us the name of the directory rather then us splitting it and it possibly
breaking in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49039069 This PR conflicts with pr1112. I would like to put that one in first and then upmerge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/1094#discussion_r14936811
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala ---
@@ -172,6 +172,8 @@ class HistoryServer(
object HistoryServer {
private val conf = new SparkConf
+ val UI_PATH_PREFIX = /history/
--- End diff --
If we are adding this we should also use it in this file to set the path.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49069859 OK, thank you for your reply. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-49083255 @rahulsinghaliitd can you please upmerge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-48990919 Is someone working on this? or facing some problem? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-47250080 Generally we ask for the PR submitter to resolve conflicts. You just have to rebase your branch on top of master, resolve conflicts, and force push it into your repo, the PR will be updated automatically. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user rahulsinghaliitd commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-47064230 @vanzin I was only referring to how the UI URL is passed around. I have used the longer way of passing it around using command line arguments whereas the other change uses spark conf by simply setting it as another property. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-47125879 @rahulsinghaliitd ah, good point. Passing as a SparkConf property should work now that I fixed some things in the yarn-cluster backend. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-47126414 Latest patch LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-47126454 (Aside from rebasing to fix the merge conflicts.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user rahulsinghaliitd commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-47184407 Am I supposed to resolve the conflicts or will they be resolved by the admin who merges this change? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-46869006 @rahulsinghaliitd that bug is about linking to the running web ui of the driver in yarn-client mode. Slightly different than this one, which is about setting the link after the application is finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user rahulsinghaliitd commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-46804664 @sryza thanks for the thumbs up. Although I wonder if the approach in https://github.com/apache/spark/pull/1112 is better for passing the UI address (certainly is much cleaner). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-46732050 This change looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-46201382 Hi @rahulsinghaliitd, Thanks for filing this; this is something I had on my radar but hadn't gotten down to it yet. While I also want the feature, I think we need a better approach here. For one, I'm not a big fan of using the log directory name as the application's URL in the history server; it's not very discoverable, since it's generated internally by the event logger. Also, slightly unrelated, but I have upcoming changes that make those names even more ugly, so they're not a very user-friendly thing to use as a public-facing URL. I've actually been playing with this and have some changes that use the actual application ID, when it's available (such as in the case of Yarn), so server the application from the history server. I think that's a better approach. But if some committer wants to check this in while I work on that, it would cause (many) conflicts with my change. While we're here, you'd need to make a similar change in the yarn/alpha project, and also in yarn-client mode (ExecutorLauncher.scala in both projects). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user rahulsinghaliitd commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-46213984 Hi @vanzin, I don't have any preference as to how the URL is formatted and I think that changing the URL can be a separate activity. I am hoping this can be committed so that we atleast have the functionality. Thanks for pointing out the changes, I had also realized the same and I am currently in the process of testing them. They should be available soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
GitHub user rahulsinghaliitd opened a pull request: https://github.com/apache/spark/pull/1094 SPARK-2150: Provide direct link to finished application UI in yarn resou... ...rce manager UI Use the event logger directory to provide a direct link to finished application UI in yarn resourcemanager UI. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Guavus/spark SPARK-2150 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1094.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1094 commit 38ea5c7625fc11889f1bd51b1ee15e99644f0867 Author: Rahul Singhal rahul.sing...@guavus.com Date: 2014-06-16T05:04:52Z SPARK-2150: Provide direct link to finished application UI in yarn resource manager UI Use the event logger directory to provide a direct link to finished application UI in yarn resourcemanager UI. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2150: Provide direct link to finished ap...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1094#issuecomment-46141864 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
