[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18351 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18351 Yes, I'm with the current solution, since there's no better solution. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18351 Actually, re-reading, @jerryshao are you OK with this approach? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18351 **[Test build #3814 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3814/testReport)** for PR 18351 at commit [`f91ed07`](https://github.com/apache/spark/commit/f91ed07718d4462e487237353bf82c80c5e148f7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18351 **[Test build #3814 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3814/testReport)** for PR 18351 at commit [`f91ed07`](https://github.com/apache/spark/commit/f91ed07718d4462e487237353bf82c80c5e148f7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18351 This isn't my area, but I see no objection to merging it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/18351 Hi, @srowen , what's your opinion about this issue? There's been no progress for days. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user 10075197 commented on the issue: https://github.com/apache/spark/pull/18351 Since it's hard to compute the accurate duration, I think it's better to hide this field. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/18351 It seems to be a big change to solve this problem completely, it may not be worth making such a big change for such a small problem, so I think we should hide it. cc @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/18351 I think hiding it is the correct choice personally. Users are currently used to it always set to 0 so this wouldn't be a change in functionality. Also it seems from discussion that any attempt to calculate the value would be error prone. If someone could show it's not error prone I might change my mind though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/18351 According to Jerryshao, I should change to use "Currenttime - StartTime", although in some scenarios the value is wrong. But I tend to hide the duration for incomplete applications, because the duration of incomplete applications has been shown on the master page, there is no need to display a value that might not be always correct here. Can you help me decide? @srowen @ajbozarth @zhuoliu --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user eatoncys commented on the issue: https://github.com/apache/spark/pull/18351 I think it is better to hide it. @fjh100456 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user eatoncys commented on the issue: https://github.com/apache/spark/pull/18351 I think it is better to hide it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user eatoncys commented on the issue: https://github.com/apache/spark/pull/18351 I think it is better to hide it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/18351 Oops, I didn't notice the comment. But it seems that if we don't change the "lastUpdated", it just be a "Currenttime". Even soï¼I'd like to ask for other contributor's opinions, hide or change to currentTime-startTime,which is better, after all, abnormal scenes may also occur often, such as driver downtime, or task kill process. I tend to show the right content, not more but possibly incorrect content. And the duration of the in-process applications can be seen on other pages, not unknowable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18351 Using modification time may have some issues, please see the comment. >// Use loading time as lastUpdated since some filesystems don't update modifiedTime // each time file is updated. However use modifiedTime for completed jobs so lastUpdated // won't change whenever HistoryServer restarts and reloads the file. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/18351 Sorry, I forgot to explain that what I mean by âLastupdatedâ refers to the value of "Filestatus.Getmodificationtime".Maybe the value of lastUpdated field needs a change too. @jerryshao --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18351 Looks from the [code](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L458), `lastUpdated` will still be increased even after application is finished unexpectedly. So will `lastUpdated` get correct behavior? Please correct me if I'm wrong. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/18351 @jerryshao For example, you use shell commands to kill the application's submit process from outside, this application is always treated as a "incomplete application" in history server.In this case it would be wrong to use "currenttime - startTime", and the time will increases with each refresh which should actually be discontinued after the kill. If we must keep the âdurationâ, I tend to use "last updated - startTime", although sometimes may not be very accurate, but at least closer to the real value, and more acceptable than display an error value, you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18351 @fjh100456 IMO, abrupt abortion of application without application end time is not the normal case. For the most of the incompleted applications "currTimeInMs - startTime" as **Duration** is meaningful. If we'd like to display this column, I think considering normal case should be enough. (My two cents) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/18351 @jerryshao Yes, I have just seen your PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18351 @guoxiaolongzte this column is already hidden in the master code. @fjh100456 would you please explain more about this? >the application of the exception abort will always be treated as âincompleted applicationâ, if we use âcurrTimeInMs - startTimeâ, the duration of this kind of application will be wrong and look very strange. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/18351  I think the field needs to be hidden. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/18351 @fjh100456 Try to follow jerryshao's advice. I think this is very necessary. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/18351 @jerryshao I thought about doing like this, but the application of the exception abort will always be treated as âincompleted applicationâ, if we use âcurrTimeInMs - startTimeâ, the duration of this kind of application will be wrong and look very strange. I have also considered using the lastUpdated - Starttime, but in some cases this is not very accurate. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18351 I'm not familiar with JS, I'm wondering if we could use `currTimeInMs - startTime` as the **Duration** instead of "0". Not sure if it is easy to do in JS. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/18351 @guoxiaolongzte I have considered this problem, but there is no âEndTimeâ value for applications in processing, including the application of the exception abort I mentioned. So in the Scala code this value is not counted but directly set to 0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/18351 I think it should show the specific time. It need to calculate the duration, rather than hide the title. @srowen @ajbozarth @jerryshao --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/18351 Yes,it should be. @ajbozarth The screenshotï¼@zhuoliu  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user zhuoliu commented on the issue: https://github.com/apache/spark/pull/18351 @fjh100456 this change looks good to me. Could you please add a screenshot for your test? Regarding your question of " the application of an exception abort (such as the application of a background kill or driver outage) will always be treated as an Incompleted application, and I'm not sure if this is a problem", locally such exceptionally aborted application would rather be treated as Completed or at least STOPPED. I am not sure whether some other logic or some clients are using this field to decide whether they should retry or not. In that sense, we would have more to consider. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/18351 I have not found a similar problem on other pages? The CompleteTime of this page has been hidden, indicating that it should have been considered before. Have you considered the other question I mentioned in Jira about the abort of the anomaly? @zhouliu @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18351 CC @zhuoliu Wouldn't there be more places to apply logic like this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18351: [SPARK-21135][WEB UI] On history server page，duration ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18351 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
