Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18351
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18351
Yes, I'm with the current solution, since there's no better solution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18351
Actually, re-reading, @jerryshao are you OK with this approach?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18351
**[Test build #3814 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3814/testReport)**
for PR 18351 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18351
**[Test build #3814 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3814/testReport)**
for PR 18351 at commit
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18351
This isn't my area, but I see no objection to merging it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user fjh100456 commented on the issue:
https://github.com/apache/spark/pull/18351
Hi, @srowen , what's your opinion about this issue? There's been no
progress for days.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user 10075197 commented on the issue:
https://github.com/apache/spark/pull/18351
Since it's hard to compute the accurate duration, I think it's better to
hide this field.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user fjh100456 commented on the issue:
https://github.com/apache/spark/pull/18351
It seems to be a big change to solve this problem completely, it may not be
worth making such a big change for such a small problem, so I think we should
hide it.
cc @srowen
---
If your
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/18351
I think hiding it is the correct choice personally. Users are currently
used to it always set to 0 so this wouldn't be a change in functionality. Also
it seems from discussion that any attempt to
Github user fjh100456 commented on the issue:
https://github.com/apache/spark/pull/18351
According to Jerryshao, I should change to use "Currenttime - StartTime",
although in some scenarios the value is wrong.
But I tend to hide the duration for incomplete applications,
Github user eatoncys commented on the issue:
https://github.com/apache/spark/pull/18351
I think it is better to hide it. @fjh100456
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user eatoncys commented on the issue:
https://github.com/apache/spark/pull/18351
I think it is better to hide it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user eatoncys commented on the issue:
https://github.com/apache/spark/pull/18351
I think it is better to hide it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user fjh100456 commented on the issue:
https://github.com/apache/spark/pull/18351
Oops, I didn't notice the comment. But it seems that if we don't change the
"lastUpdated", it just be a "Currenttime".
Even soï¼I'd like to ask for other contributor's opinions, hide or
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18351
Using modification time may have some issues, please see the comment.
>// Use loading time as lastUpdated since some filesystems don't update
modifiedTime
// each time file is
Github user fjh100456 commented on the issue:
https://github.com/apache/spark/pull/18351
Sorry, I forgot to explain that what I mean by âLastupdatedâ refers to
the value of "Filestatus.Getmodificationtime".Maybe the value of lastUpdated
field needs a change too. @jerryshao
---
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18351
Looks from the
[code](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L458),
`lastUpdated` will still be increased even
Github user fjh100456 commented on the issue:
https://github.com/apache/spark/pull/18351
@jerryshao For example, you use shell commands to kill the application's
submit process from outside, this application is always treated as a
"incomplete application" in history server.In this
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18351
@fjh100456 IMO, abrupt abortion of application without application end time
is not the normal case. For the most of the incompleted applications
"currTimeInMs - startTime" as **Duration** is
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/18351
@jerryshao Yes, I have just seen your PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18351
@guoxiaolongzte this column is already hidden in the master code.
@fjh100456 would you please explain more about this?
>the application of the exception abort will always be
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/18351
I think the field needs to be hidden.
---
If your project is set up
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/18351
@fjh100456 Try to follow jerryshao's advice. I think this is very
necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user fjh100456 commented on the issue:
https://github.com/apache/spark/pull/18351
@jerryshao I thought about doing like this, but the application of the
exception abort will always be treated as âincompleted applicationâ, if we
use âcurrTimeInMs - startTimeâ, the
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18351
I'm not familiar with JS, I'm wondering if we could use `currTimeInMs -
startTime` as the **Duration** instead of "0". Not sure if it is easy to do in
JS.
---
If your project is set up for it,
Github user fjh100456 commented on the issue:
https://github.com/apache/spark/pull/18351
@guoxiaolongzte I have considered this problem, but there is no
âEndTimeâ value for applications in processing, including the application
of the exception abort I mentioned. So in the Scala
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/18351
I think it should show the specific time. It need to calculate the
duration, rather than hide the title.
@srowen @ajbozarth @jerryshao
---
If your project is set up for it, you can
Github user fjh100456 commented on the issue:
https://github.com/apache/spark/pull/18351
Yes,it should be. @ajbozarth
The screenshotï¼@zhuoliu
Github user zhuoliu commented on the issue:
https://github.com/apache/spark/pull/18351
@fjh100456 this change looks good to me. Could you please add a screenshot
for your test?
Regarding your question of " the application of an exception abort (such as
the application of a
Github user fjh100456 commented on the issue:
https://github.com/apache/spark/pull/18351
I have not found a similar problem on other pages? The CompleteTime of this
page has been hidden, indicating that it should have been considered before.
Have you considered the other question
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18351
CC @zhuoliu
Wouldn't there be more places to apply logic like this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18351
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
33 matches
Mail list logo
