[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-23 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 thanks for the reviews @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18162 thanks, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78468/ Test PASSed. ---

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #78468 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78468/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78460/ Test PASSed. ---

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #78460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78460/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #78468 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78468/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78467/ Test FAILed. ---

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #78467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78467/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #78467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78467/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 failure is from previous push of code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78466/ Test FAILed. ---

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #78466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78466/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #78466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78466/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 upmerged to master and updated default and removed unneeded changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 will upmerge shortly, since there are conflicts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #78460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78460/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-21 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 sorry missed that you had commented, yes we can change that --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18162 @tgravescs can you address https://github.com/apache/spark/pull/18162#discussion_r122363701 ? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78300/ Test FAILed. ---

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #78300 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78300/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #78300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78300/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-20 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-14 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 sorry been out on vacation and probably won't have time this week to respond much but will update early next week. Thanks @rdblue . I am running this in our production as well and can clearly

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-08 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/18162 @tgravescs, I deployed this to our production environment (based on 2.0.0) a few days ago and haven't hit any problems with it. I think this is good to go, unless something has been added recently

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77687/ Test FAILed. ---

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #77687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77687/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77684/ Test FAILed. ---

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #77684 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77684/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #77687 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77687/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 I'm not sure what you mean by its not doable? what places are you seeing update the block statuses that I haven't covered here? most of it was done by the BlockManager. Maybe I'm missing

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18162 I'm wondering if it's doable to stop tracking the `updatedBlockStatus` according to a config... there are many places that update `updatedBlockStatus` right? --- If your project is set up for

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #77684 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77684/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #77682 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77682/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77682/ Test FAILed. ---

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 turned on by default for backwards compatibility but don't really agree with it. We should make it more stable/usable for people by turning it off. I'm assuming anyone that is using this would

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #77682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77682/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-02 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 ok, I'll update the default. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-01 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18162 Yea I think we should merge this to 2.2, but we need to change the default value of the new config to `true`, to not surprise users. --- If your project is set up for it, you can reply to this

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-01 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 @JoshRosen what do you think should we add the deprecated? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-01 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 It would be nice to get this into spark 2.2 if we can --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-01 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18162 Yeah I was figuring I would file another jira to remove it later. I can add the deprecated flag here if you guys agree. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18162 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77625/ Test PASSed. ---

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #77625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77625/testReport)** for PR 18162 at commit

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-06-01 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18162 I do love the clean up you did by removing `updatedBlockStatuses` entirely... Since `SparkListenerTaskEnd` is marked as a developer API, is it acceptable to make

[GitHub] spark issue #18162: [SPARK-20923] turn tracking of TaskMetrics._updatedBlock...

2017-05-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18162 **[Test build #77625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77625/testReport)** for PR 18162 at commit