[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19132 @jerryshao Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Cannot cleanly merge to 2.2, so this will only land to master branch. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 LGTM, merging to master, if possible to 2.2. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81768/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19132 **[Test build #81768 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81768/testReport)** for PR 19132 at commit [`06035f7`](https://github.com/apache/spark/commit/06035f7cd637511a01d3e9dc04c1d6f78afc27cc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19132 **[Test build #81768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81768/testReport)** for PR 19132 at commit [`06035f7`](https://github.com/apache/spark/commit/06035f7cd637511a01d3e9dc04c1d6f78afc27cc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81764/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19132 **[Test build #81764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81764/testReport)** for PR 19132 at commit [`8ec4319`](https://github.com/apache/spark/commit/8ec4319dd81cf4e19917fe4a64a016049f3a2e83). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19132 **[Test build #81764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81764/testReport)** for PR 19132 at commit [`8ec4319`](https://github.com/apache/spark/commit/8ec4319dd81cf4e19917fe4a64a016049f3a2e83). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81710/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19132 **[Test build #81710 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81710/testReport)** for PR 19132 at commit [`f0e1ded`](https://github.com/apache/spark/commit/f0e1dedf1a2f167f9a4be66050cb8948e9ccbea2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19132 **[Test build #81710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81710/testReport)** for PR 19132 at commit [`f0e1ded`](https://github.com/apache/spark/commit/f0e1dedf1a2f167f9a4be66050cb8948e9ccbea2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81709/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19132 **[Test build #81709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81709/testReport)** for PR 19132 at commit [`560d442`](https://github.com/apache/spark/commit/560d442a8d25e37f9d831699663b1c8413ddd6a9). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19132 **[Test build #81709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81709/testReport)** for PR 19132 at commit [`560d442`](https://github.com/apache/spark/commit/560d442a8d25e37f9d831699663b1c8413ddd6a9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19132 I've been following, still LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Overall LGTM, @ajbozarth can you please review again? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19132 @jerryshao @HyukjinKwon Jenkins done.Thanks too much. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81678/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19132 **[Test build #81678 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81678/testReport)** for PR 19132 at commit [`25fe22c`](https://github.com/apache/spark/commit/25fe22cddde276f846fd4808de1b575a87b1c059). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Thanks @HyukjinKwon , I will ping Josh about this thing ð . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19132 @jerryshao, for triggering tests on Jenkins, I think this should be added by its admin manually as well if I understood correctly. In my case, I asked this to Josh Rosen before via email privately. I am quite sure you are facing the same issue I (and Holden, Felix and Takuya) met before if I understood correctly. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19132 **[Test build #81678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81678/testReport)** for PR 19132 at commit [`25fe22c`](https://github.com/apache/spark/commit/25fe22cddde276f846fd4808de1b575a87b1c059). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19132 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19132 Yes i have confirmed.Below is a testing job: @jerryshao    --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Looks like I don't have the Jenkins permission to trigger UT ð . Let me ping @srowen to trigger the test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 ok to test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19132 @jerryshao Thanks for your time. IIUC, event log is completed since driver has not dropped any event of executor which has problem described above.See below,driver only drop two events after shutting down: `2017-09-11,17:08:57,605 ERROR org.apache.spark.scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(300,WrappedArray()) 2017-09-11,17:08:57,623 INFO org.apache.spark.deploy.yarn.YarnAllocator: Driver requested a total number of 0 executor(s). 2017-09-11,17:08:57,624 INFO org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Shutting down all executors 2017-09-11,17:08:57,624 INFO org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Asking each executor to shut down 2017-09-11,17:08:57,629 ERROR org.apache.spark.scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(258,WrappedArray())` And below showing an executor has this problem:   Case here is a job running today of our cluster. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Still I have a question about history server, is you event log an incomplete event log or completed when you met such issue? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19132 Done @jerryshao --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 So your PR description is quite confusing, would you please elaborate your problem in detail and describe how to reproduce the issue. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19132 I just appear on history ui since Live ui will use heartbeat metric result @jerryshao(From my understanding) .It appears when Driver exit unnormally,then status of taskinfo can not be updated correctlly. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Is this a problem only in History UI, or it also has issues in Live UI? From my understanding you only pass a last update time for history UI, so is it intended? Also you mentioned "When executor failed and task metrics have not send to driver,the status will always be 'RUNNING'", is this a bug in scheduler? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19132 @vanzin @zsxwing could you help reivew this?Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19132 Ping @kiszk @jerryshao can you help reivew this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19132 @ajbozarth I have updated the implementation which only access FS in FSHistoryServerProvider --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/19132 Thanks for your recommendation @ajbozarth .Could you put a link for your pr?For the problem you mentioned,i have thought about them. 1ãFsHistoryServer will always use FS to get event log 2ãFor spark ui, my implementation will not access FS. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/19132 Though this may fix the problem it is 100% the wrong way to do it for multiple reasons: 1. The UI should not touch the FS, FS access it abstracted out so the UI is can work with multiple FS impls 2. This fix assumes it's running the History Server in a class used by both the Web UI and SHS Without a deeper look at the issue I would instead look at why the problem is occurring and fixing that rather than putting a bandaid on the UI like this. (Note: I've done this before and got the same response, nothing personally :) I would recommend closing this and opening a new PR once you figure out a new solution. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19132 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
