[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-14 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/17533 I think the failed unit test can be fixed in https://github.com/apache/spark/pull/17634 and https://github.com/apache/spark/pull/17603 --- If your project is set up for it, you can reply to

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75802/ Test FAILed. ---

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75802/testReport)** for PR 17533 at commit

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75800/testReport)** for PR 17533 at commit

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75800/ Test FAILed. ---

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-14 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/17533 @squito Thank you so much for reviewing thus far and sorry for the complexity I bring in. I tried to simplify the code according to your comment and please take another look when tests

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75802/testReport)** for PR 17533 at commit

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75800/testReport)** for PR 17533 at commit

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75697/ Test PASSed. ---

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75697 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75697/testReport)** for PR 17533 at commit

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75697/testReport)** for PR 17533 at commit

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75695/ Test FAILed. ---

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75695/testReport)** for PR 17533 at commit

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75542/testReport)** for PR 17533 at commit

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-05 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/17533 Yes, I did the test in my cluster. In highly-skew stage, the time cost can be reduced significantly. Tasks are scheduled with locality preference. But in current code, input size of tasks are not

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-05 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17533 Tasks are scheduled by locality (which includes shuffle tasks too to some extent). This is making a lot of state mutable within TSM - is there any tests done which show improvements due to

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75538/testReport)** for PR 17533 at commit

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75529/ Test FAILed. ---

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75532/ Test FAILed. ---

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75531/ Test FAILed. ---

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17533 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75532/testReport)** for PR 17533 at commit

[GitHub] spark issue #17533: [WIP][SPARK-20219] Schedule tasks based on size of input...

2017-04-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17533 **[Test build #75531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75531/testReport)** for PR 17533 at commit