[GitHub] spark pull request #21809: [SPARK-24851] : Map a Stage ID to it's Associated...
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/21809#discussion_r204634512
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -182,6 +184,12 @@ private[ui] class StagePage(parent: StagesTab, store:
AppStatusStore) extends We
{Utils.bytesToString(stageData.diskBytesSpilled)}
}}
+ {if (!stageJobIds.isEmpty) {
+
+ Associated Job Ids:
+ {stageJobIds}
--- End diff --
My suggestion is to map each job id as a href link.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21809: [SPARK-24851] : Map a Stage ID to it's Associated...
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21809#discussion_r204634172 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -115,6 +115,8 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We return UIUtils.headerSparkPage(request, stageHeader, content, parent) } +val stageJobIds = parent.store.getJobIdsAssociatedWithStage(stageId, stageAttemptId) --- End diff -- E.g. we can add a function to return the whole StageDataWrapper in AppStatusStore --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21809: [SPARK-24851] : Map a Stage ID to it's Associated...
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/21809#discussion_r204633878 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -115,6 +115,8 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We return UIUtils.headerSparkPage(request, stageHeader, content, parent) } +val stageJobIds = parent.store.getJobIdsAssociatedWithStage(stageId, stageAttemptId) --- End diff -- In https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala#L109 there is a query for the stage data already. We can reduce the query to the store here. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21809: [SPARK-24851] : Map a Stage ID to it's Associated...
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21809#discussion_r204514740
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
---
@@ -94,6 +94,13 @@ private[spark] class AppStatusStore(
}.toSeq
}
+ def getJobIdsAssociatedWithStage(stageId: Int): Seq[Set[Int]] = {
+
store.view(classOf[StageDataWrapper]).index("stageId").first(stageId).last(stageId)
--- End diff --
I really did not get the question, perhaps you are suggesting to try an
alternative way but is there any other alternative way to do this? Let me know
your thoughts. Thank you.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21809: [SPARK-24851] : Map a Stage ID to it's Associated...
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21809#discussion_r204514110
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -182,6 +184,12 @@ private[ui] class StagePage(parent: StagesTab, store:
AppStatusStore) extends We
{Utils.bytesToString(stageData.diskBytesSpilled)}
}}
+ {if (!stageJobIds.isEmpty) {
+
+ Associated Job Ids:
+ {stageJobIds}
--- End diff --
The problem here is that the stage could also have multiple job ids, in
that case, we get a bunch of them. Do you want a generic link instead that will
take you to the jobs page? Let me know what you think.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21809: [SPARK-24851] : Map a Stage ID to it's Associated...
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21809#discussion_r204513701
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
---
@@ -94,6 +94,13 @@ private[spark] class AppStatusStore(
}.toSeq
}
+ def getJobIdsAssociatedWithStage(stageId: Int): Seq[Set[Int]] = {
--- End diff --
Good point! Have taken care of it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21809: [SPARK-24851] : Map a Stage ID to it's Associated...
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/21809#discussion_r204103981
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -182,6 +184,12 @@ private[ui] class StagePage(parent: StagesTab, store:
AppStatusStore) extends We
{Utils.bytesToString(stageData.diskBytesSpilled)}
}}
+ {if (!stageJobIds.isEmpty) {
+
+ Associated Job Ids:
+ {stageJobIds}
--- End diff --
make it href link?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21809: [SPARK-24851] : Map a Stage ID to it's Associated...
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/21809#discussion_r204105954
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
---
@@ -94,6 +94,13 @@ private[spark] class AppStatusStore(
}.toSeq
}
+ def getJobIdsAssociatedWithStage(stageId: Int): Seq[Set[Int]] = {
+
store.view(classOf[StageDataWrapper]).index("stageId").first(stageId).last(stageId)
--- End diff --
can we avoid the store look up here?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21809: [SPARK-24851] : Map a Stage ID to it's Associated...
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21809#discussion_r203936074
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
---
@@ -94,6 +94,13 @@ private[spark] class AppStatusStore(
}.toSeq
}
+ def getJobIdsAssociatedWithStage(stageId: Int): Seq[Set[Int]] = {
--- End diff --
We don't need to fetch all the stage attempts, just the first of it is
enough to get all the jobIds.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21809: [SPARK-24851] : Map a Stage ID to it's Associated...
GitHub user pgandhi999 opened a pull request: https://github.com/apache/spark/pull/21809 [SPARK-24851] : Map a Stage ID to it's Associated Job ID in UI It would be nice to have a field in Stage Page UI which would show mapping of the current stage id to the job id's to which that stage belongs to. ## What changes were proposed in this pull request? Added a field in Stage UI to display the corresponding job id for that particular stage. ## How was this patch tested? https://user-images.githubusercontent.com/8190/42900762-47d1a6d2-8a8f-11e8-9dc5-45713eaac296.png;> You can merge this pull request into a Git repository by running: $ git pull https://github.com/pgandhi999/spark SPARK-24851 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21809.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21809 commit 7be05201239c9fec7133e8f09455224786e91b59 Author: pgandhi Date: 2018-07-18T18:31:20Z [SPARK-24851] : Map a Stage ID to it's Associated Job ID in UI Added a field in Stage UI to display the corresponding job id for that particular stage. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
