[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-69037810 Alright LGTM I'm merging this into master thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3467#discussion_r22591785 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -329,6 +319,17 @@ private[history] class FsHistoryProvider(conf: SparkConf) extends ApplicationHis /** Returns the system's mononotically increasing time. */ private def getMonotonicTimeMs(): Long = System.nanoTime() / (1000 * 1000) + /** + * Return true when the application already completed. --- End diff -- ``` Return true when the application has completed. ``` Don't worry I'll fix the wording myself when I merge this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3467#discussion_r22591679
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -180,14 +176,15 @@ private[history] class FsHistoryProvider(conf:
SparkConf) extends ApplicationHis
appListener.startTime.getOrElse(-1L),
appListener.endTime.getOrElse(-1L),
getModificationTime(dir),
- appListener.sparkUser.getOrElse(NOT_STARTED)))
+ appListener.sparkUser.getOrElse(NOT_STARTED),
+ !fs.isFile(new Path(dir.getPath(),
EventLoggingListener.APPLICATION_COMPLETE
} catch {
case e: Exception =
logInfo(sFailed to load application log data from $dir., e)
None
}
}
-.sortBy { info = -info.endTime }
+.sortBy { info = (-info.endTime, -info.startTime) }
--- End diff --
I see
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3467 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-68837548 resolved conflicts! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-68096064 [Test build #24818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24818/consoleFull) for PR 3467 at commit [`f9ef854`](https://github.com/apache/spark/commit/f9ef8547228d4c56afb9fc6f43431a458a2325ca). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on a diff in the pull request:
https://github.com/apache/spark/pull/3467#discussion_r22270614
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -180,14 +176,15 @@ private[history] class FsHistoryProvider(conf:
SparkConf) extends ApplicationHis
appListener.startTime.getOrElse(-1L),
appListener.endTime.getOrElse(-1L),
getModificationTime(dir),
- appListener.sparkUser.getOrElse(NOT_STARTED)))
+ appListener.sparkUser.getOrElse(NOT_STARTED),
+ !fs.isFile(new Path(dir.getPath(),
EventLoggingListener.APPLICATION_COMPLETE
} catch {
case e: Exception =
logInfo(sFailed to load application log data from $dir., e)
None
}
}
-.sortBy { info = -info.endTime }
+.sortBy { info = (-info.endTime, -info.startTime) }
--- End diff --
About completed applications, they are sorted by endTime because they have
proper endTime (almost unique).
About incomplete applications, they are sorted by startTime because they
have the same invalid endTime(-1). As the first order is not effective, the
second order is effective.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-68098394 [Test build #24818 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24818/consoleFull) for PR 3467 at commit [`f9ef854`](https://github.com/apache/spark/commit/f9ef8547228d4c56afb9fc6f43431a458a2325ca). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-68098398 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24818/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-68122296 [Test build #24830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24830/consoleFull) for PR 3467 at commit [`76205d2`](https://github.com/apache/spark/commit/76205d2b66a3c591931bbd0cc7cf358200563d21). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-68124725 [Test build #24830 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24830/consoleFull) for PR 3467 at commit [`76205d2`](https://github.com/apache/spark/commit/76205d2b66a3c591931bbd0cc7cf358200563d21). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-68124728 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24830/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-68043299 Thank you for your comments! I'm going to do it in a few days! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3467#discussion_r22139289
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -180,14 +176,15 @@ private[history] class FsHistoryProvider(conf:
SparkConf) extends ApplicationHis
appListener.startTime.getOrElse(-1L),
appListener.endTime.getOrElse(-1L),
getModificationTime(dir),
- appListener.sparkUser.getOrElse(NOT_STARTED)))
+ appListener.sparkUser.getOrElse(NOT_STARTED),
+ !fs.isFile(new Path(dir.getPath(),
EventLoggingListener.APPLICATION_COMPLETE
} catch {
case e: Exception =
logInfo(sFailed to load application log data from $dir., e)
None
}
}
-.sortBy { info = -info.endTime }
+.sortBy { info = (-info.endTime, -info.startTime) }
--- End diff --
what does this mean? Which column does it sort by?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3467#discussion_r22139290
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala ---
@@ -117,8 +129,9 @@ private[spark] class HistoryPage(parent: HistoryServer)
extends WebUIPage() {
private def appRow(info: ApplicationHistoryInfo): Seq[Node] = {
val uiAddress = HistoryServer.UI_PATH_PREFIX + s/${info.id}
val startTime = UIUtils.formatDate(info.startTime)
-val endTime = UIUtils.formatDate(info.endTime)
-val duration = UIUtils.formatDuration(info.endTime - info.startTime)
+val endTime = if(info.endTime != -1) UIUtils.formatDate(info.endTime)
else -
--- End diff --
space before if
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3467#discussion_r22139294
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala ---
@@ -117,8 +129,9 @@ private[spark] class HistoryPage(parent: HistoryServer)
extends WebUIPage() {
private def appRow(info: ApplicationHistoryInfo): Seq[Node] = {
val uiAddress = HistoryServer.UI_PATH_PREFIX + s/${info.id}
val startTime = UIUtils.formatDate(info.startTime)
-val endTime = UIUtils.formatDate(info.endTime)
-val duration = UIUtils.formatDuration(info.endTime - info.startTime)
+val endTime = if(info.endTime != -1) UIUtils.formatDate(info.endTime)
else -
--- End diff --
also might be good to check for `info.endTime 0` instead, here and other
places
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3467#discussion_r22139306 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/ApplicationHistoryProvider.scala --- @@ -25,7 +25,8 @@ private[spark] case class ApplicationHistoryInfo( startTime: Long, endTime: Long, lastUpdated: Long, -sparkUser: String) +sparkUser: String, +incomplete: Boolean = false) --- End diff -- the double negative is a little confusing. Why not just make this `complete: Boolean`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3467#discussion_r22139319
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala ---
@@ -65,11 +67,12 @@ private[spark] class HistoryPage(parent: HistoryServer)
extends WebUIPage() {
h4
Showing {actualFirst + 1}-{last + 1} of {allApps.size}
+{if (requestedIncomplete) (incomplete applications)}
--- End diff --
should capitalize `Incomplete`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3467#discussion_r22139329 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -180,14 +176,15 @@ private[history] class FsHistoryProvider(conf: SparkConf) extends ApplicationHis appListener.startTime.getOrElse(-1L), appListener.endTime.getOrElse(-1L), getModificationTime(dir), - appListener.sparkUser.getOrElse(NOT_STARTED))) + appListener.sparkUser.getOrElse(NOT_STARTED), + !fs.isFile(new Path(dir.getPath(), EventLoggingListener.APPLICATION_COMPLETE --- End diff -- when you resolve the merge conflicts you should factor this out into its own method: ``` def isApplicationComplete(...): Boolean = ``` or something --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-67722347 Hey @tsudukim this patch is pretty straightforward and it looks good. Unfortunately you will need to resolve a non-trivial merge conflict with another patch that I just merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-67545486 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-67546098 [Test build #24596 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24596/consoleFull) for PR 3467 at commit [`5454869`](https://github.com/apache/spark/commit/5454869cf8087edb7e22bab5b6b6828d2acd31d7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-67557713 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24596/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-67557700 [Test build #24596 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24596/consoleFull) for PR 3467 at commit [`5454869`](https://github.com/apache/spark/commit/5454869cf8087edb7e22bab5b6b6828d2acd31d7). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class RandomModuleHook(object):` * `class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Boolean)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-67559755 By the way I'm going to block this on #1222, since that one has been open for a while and I would like to get it merged first before @vanzin has to resolve more conflicts again. After that one goes in this PR will likely have to resolve non-trivial merge conflicts. I might hold back reviewing this PR until then. Just a heads up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-66809813 [Test build #24413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24413/consoleFull) for PR 3467 at commit [`5454869`](https://github.com/apache/spark/commit/5454869cf8087edb7e22bab5b6b6828d2acd31d7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-66810500 @andrewor14 Thank you for following this ticket. I finished rebasing to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-66820869 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24413/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-66820853 [Test build #24413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24413/consoleFull) for PR 3467 at commit [`5454869`](https://github.com/apache/spark/commit/5454869cf8087edb7e22bab5b6b6828d2acd31d7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-66519222 Hey @tsudukim you will need to rebase to master because there have been changes to the HS that went in recently. Once you do that I will take a detailed pass through the PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user ryan-williams commented on a diff in the pull request:
https://github.com/apache/spark/pull/3467#discussion_r21033452
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala ---
@@ -61,6 +72,15 @@ private[spark] class HistoryPage(parent: HistoryServer)
extends WebUIPage() {
h4No Completed Applications Found/h4
}
}
+ a href={makePageLink(actualPage, !requestedIncomplete)}
+{
+ if (requestedIncomplete) {
+Back to completed applications
+ } else {
+Show incompleted applications
--- End diff --
nit: s/incompleted/incomplete/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64922238 @ryan-williams Thank you for your review! I fixed them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64922504 [Test build #23941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23941/consoleFull) for PR 3467 at commit [`d6dd750`](https://github.com/apache/spark/commit/d6dd750ae6afb33b75862bc55d8e143f56c9881b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64926902 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23941/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64601739 [Test build #23892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23892/consoleFull) for PR 3467 at commit [`e8cdf7b`](https://github.com/apache/spark/commit/e8cdf7b7ab72f8b441496156dfb0dce6acdcc5a8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64653027 [Test build #23892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23892/consoleFull) for PR 3467 at commit [`e8cdf7b`](https://github.com/apache/spark/commit/e8cdf7b7ab72f8b441496156dfb0dce6acdcc5a8). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64653042 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23892/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
GitHub user tsudukim opened a pull request: https://github.com/apache/spark/pull/3467 [SPARK-2458] Make failed application log visible on History Server Enabled HistoryServer to show incomplete applications. We can see the log for incomplete applications by clicking the bottom link. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tsudukim/spark feature/SPARK-2458-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3467.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3467 commit 66fc15b75a16a3dd0345c0fca0babc56c2529af9 Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp Date: 2014-11-25T08:00:03Z [SPARK-2458] Make failed application log visible on History Server Enabled HistoryServer to show incomplete applications. We can see the log for incomplete applications by clicking the bottom link. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64509912 In this PR, the sort order is (-endTime, -startTime) which means that the sorting is still end time for completed apps but is start time for incomplete apps because incomplete apps don't have the end time generally. Please see also the discuss in this previous PR about the need of this feature. https://github.com/apache/spark/pull/1558 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64510056 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64510079 And these are the screenshots of new UI. A new link for the page of incomplete applications is added at bottom. (Show incomplete applications)  And this is the new separated page for incomplete applications. Incomplete apps donsn't have the end time generally so end time are all shown as -.  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-64510185 @andrewor14 I created a new PR (#3467) as your comment. Please check it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64513458 Looks great. I will take a detailed pass once the 1.2 release is out of the window. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64513464 add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64513684 [Test build #23875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23875/consoleFull) for PR 3467 at commit [`66fc15b`](https://github.com/apache/spark/commit/66fc15b75a16a3dd0345c0fca0babc56c2529af9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64513767 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23875/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3467#issuecomment-64513766 [Test build #23875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23875/consoleFull) for PR 3467 at commit [`66fc15b`](https://github.com/apache/spark/commit/66fc15b75a16a3dd0345c0fca0babc56c2529af9). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-57430441 Thank you @andrewor14 I've researched this problem these days with our environment and it turned out to be a very rare case as @vanzin suggested first. (like jvm lost and failed to call SparkContext::stop(), failed to write to HDFS for some reason, etc) And my PR is not the smart way to solve the rare case. so I drop this PR. Thank you for your comments again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim closed the pull request at: https://github.com/apache/spark/pull/1558 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-57500764 @tsudukim Actually the high-level fix here is not a bad idea. Right now if the logs don't show up here the user has to manually figure out whether the APPLICATION_COMPLETE file is present. It would be good to show some feedback to the user so they don't have to guess if their paths are set properly or their application terminated properly etc. Let me know if you're interested in submitting a new PR that addresses the comments raised in this one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-55333925 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-55334577 Hi @tsudukim, how does the user see the incomplete applications? As @vanzin suggested, the semantics of a history server is that it displays completed applications only. That said, since we can't distinguish running and failed applications, we might want a way to expose the potentially failed applications. I think the UI should have a subtle Show incomplete applications link that only expands if the user clicks on it. These should be in a separate table by themselves so we don't mix them with the ones we know are complete. As for sorting, I agree with @vanzin that end time is more natural than start time. For incomplete applications, actually, won't the end time always be infinity or some special value? Maybe we can use that to detect whether an application has finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-55334702 QA tests have started for PR 1558. This patch DID NOT merge cleanly! brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20169/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-55334841 QA results for PR 1558:br- This patch FAILED unit tests.brbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20169/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-55334845 Also, looks like this has merge conflicts. It would be great if you could rebase to master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-55164137 QA tests have started for PR 1558. This patch DID NOT merge cleanly! brView progress: https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/14/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-55164303 QA results for PR 1558:br- This patch FAILED unit tests.brbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/14/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-54694566 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-50052724 Disclaimer: haven't looked at the code yet. I'm a little conflicted about exposing running apps in the history server, especially this way. First, history sort of implies things that happened in the past. Second, misbehaving apps can cause log files to never go into a finished state (e.g. by failing to call `SparkContext::stop()`) - although you can make the argument that anyone can write anything to the root log dir anyway. Third, the user experience from your screenshots is very weird. When just looking at finished apps, things are sorted one way, but when including unfinished ones, they're sorted another way. That's super confusing, especially when you have paging. If listing running apps in the HS is really wanted, I'd suggest an approach where running apps are shown separately from finished ones. Either in a separate table, or a separate tab in the UI. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-50060159 Thank you for following this PR. Let me explain a little. I'm sorry I made you misunderstand my purpose with the improper word uncompleted. The purpose of this PR is to show failed apps in the HS, but not the running apps. But it is true that we can't recognize if the app already failed or still running from the log in this way, so as a result they both show up in the HS. First point, the purpose is to show failed apps in the past, so this PR still matches for the concept of HS. Second point, the target of this PR is apps that never go into finished state. And third point, sorting ways are the same in the both mode. But your suggestion makes sense. Separate table or tab might be better. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1558#issuecomment-50060945
Hmm. A properly-written app that fails should still show up as finished:
val sc = new SparkContext(blah)
try {
doStuff()
} finally {
sc.stop()
}
Of course that's not guaranteed to work 100% of the time (for that we'd
need an external entity monitoring the app, since we can't trust the app itself
to do the right thing), but should cover most cases.
re: sorting, I see what you mean. Still, I think sorting by end time is
more natural for someone checking app history. Perhaps at some point we should
let the user pick how to sort / filter the list, but that's a separate
discussion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
GitHub user tsudukim opened a pull request: https://github.com/apache/spark/pull/1558 [SPARK-2458] Make failed application log visible on History Server Modified to show uncompleted applications in History Server ui. Modified apps sort rule to startTime base (originally it was endTime base) because uncompleted apps doesn't have proper endTime. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tsudukim/spark feature/SPARK-2458 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1558.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1558 commit 503d8abb9ae24eb6b211481a60f4b348d125a69a Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp Date: 2014-07-16T00:12:42Z [SPARK-2458] Make failed application log visible on History Server Modified to show completed applications in History Server ui. Modified apps sort rule to startTime base (originally it was endTime base) because failed apps doesn't have proper endTime. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-49956854 We get the same ui as now by default.  When clicked the link above the table, we can also get the list that also include the apps which doesn't finished successfully.  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2458] Make failed application log visib...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1558#issuecomment-49956952 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
