Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-54101534
I think this was ultimately fixed by #1545 so we can close this issue. But
feel free to open another PR if that one did not fix this.
---
If your project is set up for
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1384
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-50149066
@tsudukim yes,SPARK-2298 is that i want to. but i think a simple way is on
this PR add a jobid column to stage table.it is very easy to achieve it.
---
If your
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49850442
It turned out much trickier than I thought to add attempt id. I submitted a
PR here #1545
That PR already modifies the UI, since that's the only way I could test.
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49860984
i think you can add jobid to stageTable. because jobid is very useful when
a application has many jobs.that can distinguish every job's stages.
---
If your project
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49944747
@rxin Surely we can also fix them all in one patch. But it can be a little
bit hard work to modify them compatibly in one patch so I just have thought to
separate into
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49944917
@lianhuiwang It appears to be a different problem to SPARK-2298.
Is your aim same as this ticket?
https://issues.apache.org/jira/browse/SPARK-1362
If so, how
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49495727
Modified PR as your comments. thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49207731
@pwendell I agree that there are many room for improvement about handling
of stageId and attemptId. It might be better to break this problems into some
sub-tasks. I
Github user tsudukim commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r15019018
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49209319
@rxin OK. After that, I think I can make this patch better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49208410
Let's hold off merging this one until we merge #1262. Then it will be
easier to index the information based on stage + attempt.
---
If your project is set up for it, you
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49209857
@rxin in #1262, can I expect the key of the stagedata in
JobProgressListener become stageId + attemptId instead of stageId only?
---
If your project is set up for it,
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r14975903
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def
Github user tsudukim commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r14976665
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-4990
@tsudukim @kayousterhout so I think in general here, our handling of stage
re-submissions is broken in the UI. For instance, I looked in the
`JobProgressListener` and
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49111342
@rxin is this something you've thought about in your various schedule
refactoring things?
---
If your project is set up for it, you can reply to this email and
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-49111726
@tsudukim I created a JIRA to deal with the broader issue. If you want to
take that on as well, let me know:
https://issues.apache.org/jira/browse/SPARK-2501 it might
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r14977192
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48963364
I'm wondering how to show it. I gave it a shot. Is it smart?
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48803681
@andrewor14 Thank you for your comment.
I think it is more weird if the display style of ID/attempt changes by
conditions.
Surely most stages will only have 1
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48803729
@tsudukim The concept of TaskSet should be internal to Spark. Users
shouldn't have to aware of task set. Users should only care about stage +
attempt.
---
If your project
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48804094
@xrin OK, thanks. Then attempt id is still required in the web ui for users
to know stage + attempt. Have I got that right?
---
If your project is set up for it, you
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48804157
Yup - but let's avoid exposing the concept of TaskSet to users in the UI.
That's only for internal engineering.
---
If your project is set up for it, you can reply to
GitHub user tsudukim opened a pull request:
https://github.com/apache/spark/pull/1384
SPARK-2298: Show stage attempt in UI
Added attempt ID column into stage page of webUI.
Added attemptId handling code into StageInfo, JsonProtocol.
Modified DAGScheduler to identify stages
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48800648
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48800698
Attempt Id shows up in web ui. Submitted and Duration became individual
value to stage attempts.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48800948
To make this a bit more concise, what about having one column on the left
whose header is `ID: Attempt` and separating them with a colon. Current the
word Stage is
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48801078
/cc @rxin who is interested in this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user tsudukim commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48802747
@pwendell Thank you for your response. You mean like this?
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1384#issuecomment-48802862
Hm the latest screenshot looks a little funky to me.
Most stages will only have 1 attempt, so I think it makes sense to only
show the attempt if this is not
31 matches
Mail list logo
