[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-17 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15915 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15915 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 @srowen All tests passed, take a look please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70248/ Test PASSed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70248 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70248/testReport)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 There are some unrelated failures, retest it please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #3503 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3503/testReport)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #3503 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3503/testReport)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70248 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70248/testReport)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70245/ Test FAILed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70245 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70245/testReport)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70245/testReport)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 OK, we reach a consensus, I will give a update as soon as possible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15915 There's still one comment outstanding, but other than that, yes I think it's fine to make this change without another additional test, because it's hard to test and because the change is such a

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15915 Because the actual change is far from complicated, if we can't have proper test case for it, can we skip the added test? @srowen @JoshRosen --- If your project is set up for it, you can reply to

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-16 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 @srowen Looks like it is not very convenient to test this case, with `java.lang.OutOfMemoryError: Requested array size exceeds VM limit` in jenkins. Do you think it is necessary to test

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #3500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3500/testReport)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #3500 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3500/testReport)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70191/ Test FAILed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70191/testReport)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70191/testReport)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-15 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 OK, I will give an update as soon as possible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-15 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15915 You could also cap it at 16MB here. Or 128MB or whatever. That is unrelated to introducing a new config. It seems funny that setting to allow Int.MaxValue but map anything else to

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-14 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 @srowen The truth is that `chunk size` is passed from different code path with different value. There is not a reasonable value when `chunk size` as a invalid value. Experiencedly, `chunk size`

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-14 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15915 It seems to do the job. It seems a little funny that if you specify a huge chunk size like 1GB, that's accepted, but if you specify ~2GB, it becomes 4MB. Is the logic that this probably happens when

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70118/ Test PASSed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70118 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70118/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15915 Current change looks good to me. cc @JoshRosen @srowen to check it again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70115/ Test PASSed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70115 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70115/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 ``` Discovery starting. Discovery completed in 37 seconds, 984 milliseconds. Run starting. Expected test count is: 10 MemoryStoreSuite: - reserve/release unroll memory -

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70118/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15915 Can you also add a test into MemoryStoreSuite? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70115 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70115/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70083/ Test PASSed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70083 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70083/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70083 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70083/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70063/ Test FAILed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70065/ Test FAILed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70065/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70063 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70063/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-12 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15915 I'd like option 1 that to add some checks and throw error message for users. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-12 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 Generally, there are three ways to fix this issue: 1. Check if `chunkSize` exceed `Int.MaxValue` to narrow this issue. 2. Provide a new config to set `chunkSize`. 3. Reuse existing

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70007/ Test FAILed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70008/ Test FAILed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70008/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70007 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70007/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70001/ Test FAILed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70001/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #70001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70001/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-11 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 @srowen Sorry for the delay, I will update it as soon as possible on the basis of comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-11 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15915 @uncleGen can you follow up on this so we can finish this out? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-07 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/15915 I'd use `SparkEnv.memoryManager.pageSizeBytes` (which is calculated at

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-07 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 @srowen +1 to your comment. > I think the suggestion was to base it off of some existing property, like page size? Yes, introducing a new config property is a practice of lazy.

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-04 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15915 This introduces a whole new config property. I think the suggestion was to base it off of some existing property, like page size? @JoshRosen would be best positioned to comment on that. We can more

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69460/ Test PASSed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #69460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69460/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-12-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69461/ Test FAILed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #69461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69461/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #69460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69460/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-30 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 Is there any update? @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69382/ Test PASSed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #69382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69382/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #69382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69382/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69098/ Test PASSed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #69098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69098/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/15915 Haven't forgotten about this; it's on my list to review post-Thanksgiving. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #69098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69098/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 @SparkQA retest again --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #3435 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3435/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #3435 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3435/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69068/ Test FAILed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #69068 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69068/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 @JoshRosen The main cause for this issue is the mix-up of "chunkSize" and other "size" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #69068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69068/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-23 Thread uncleGen
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/15915 @JoshRosen I find some other usage of "chunkSize" in other code path, and we set it with **1024 * 1024 * 4**. Is there any reason? Now, I use it as default value for "chunkSize" --- If your

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-21 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15915 @JoshRosen sounds like you don't disagree with this change because anything more than `Integer.MAX_VALUE` is going to cause an outright failure, but mean it could impose a significantly smaller

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-18 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/15915 To clarify, does this overflow only occur when `spark.storage.unrollMemoryThreshold` has been manually set to a huge value which doesn't fit into an int? Setting the chunk size to

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #68776 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68776/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-17 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15915 **[Test build #68776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68776/consoleFull)** for PR 15915 at commit

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68759/ Test PASSed. ---

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

2016-11-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15915 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

  1   2   >