[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18031 **[Test build #77072 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77072/testReport)** for PR 18031 at commit

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/18031 @HyukjinKwon Thank you so much ! Really helpful 👍 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18031 **[Test build #77070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77070/testReport)** for PR 18031 at commit

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18031 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77070/ Test FAILed. ---

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18031 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18031 **[Test build #77070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77070/testReport)** for PR 18031 at commit

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18031 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18031 **[Test build #77069 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77069/testReport)** for PR 18031 at commit

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18031 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77069/ Test FAILed. ---

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18031 **[Test build #77069 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77069/testReport)** for PR 18031 at commit

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18031 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77058/ Test FAILed. ---

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18031 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18031 **[Test build #77058 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77058/testReport)** for PR 18031 at commit

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/18031 Gentle ping to @JoshRosen @cloud-fan @mridulm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18031 **[Test build #77058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77058/testReport)** for PR 18031 at commit

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/18031 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18031 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18031 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77056/ Test FAILed. ---

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18031 **[Test build #77056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77056/testReport)** for PR 18031 at commit

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/18031 I try to give user a way to control the memory strictly and no blocks are underestimated(setting spark.shuffle.accurateBlockThreshold=0 and spark.shuffle.accurateBlockThresholdByTimesAverage=1).

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/18031 To resolve the comments in https://github.com/apache/spark/pull/16989 : >minimum size before we consider something a large block : if average is 10kb, and some blocks are > 20kb, spilling them

[GitHub] spark issue #18031: Record accurate size of blocks in MapStatus when it's ab...

2017-05-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18031 **[Test build #77056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77056/testReport)** for PR 18031 at commit