[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19266 @srowen Thanks! @liufengdb Could you submit a separate PR to fix the issues and also please include the test cases? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19266 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19266 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19266 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81984/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19266 **[Test build #81984 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81984/testReport)** for PR 19266 at commit [`9bb9e77`](https://github.com/apache/spark/commit/9bb9e77850484672dabefcd719996daa9a4cad17). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19266 **[Test build #81984 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81984/testReport)** for PR 19266 at commit [`9bb9e77`](https://github.com/apache/spark/commit/9bb9e77850484672dabefcd719996daa9a4cad17). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19266 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19266 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81921/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19266 **[Test build #81921 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81921/testReport)** for PR 19266 at commit [`4594b55`](https://github.com/apache/spark/commit/4594b55b4d4fb26968e50c73558376df8f895dda). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19266 **[Test build #81921 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81921/testReport)** for PR 19266 at commit [`4594b55`](https://github.com/apache/spark/commit/4594b55b4d4fb26968e50c73558376df8f895dda). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19266 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81882/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19266 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19266 **[Test build #81882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81882/testReport)** for PR 19266 at commit [`8dbfa30`](https://github.com/apache/spark/commit/8dbfa30848b49a9fe4038c327222ddc9fd9a0ec0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19266 Yeah, agree, it could be some global constant. I don't think it should be configurable. Ideally it's determined from the JVM, but don't know a way to do that. In many cases, assuming Int.MaxValue is the max array size when it's Int.MaxValue-8 doesn't matter much. For example, arguably I should leave the ML changes alone here, because, in the very rare case that a matrix size is somewhere between Int.MaxValue-8 and Int.MaxValue, it will fail anyway, and it's not avoidable given the user input. It's also, maybe, more conservative to not always assume anything beyond Int.MaxValue-8 is going to fail, and not "proactively" fail at this cutoff. However I think there are a smallish number of identifiable cases where Spark can very much avoid the failure (like BufferHolder), and they're the instances where an array size keeps doubling. Maybe we can stick to those clear cases? especially any one that seems to have triggered the original error? Those cases are few enough and related enough that I'm sure they're just one issue, not several. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/19266 I though, if this limit highly depends on JVM implementations, better to put the limit as a global variable somewhere (e.g., `ARRAY_INT_MAX` in `spark.util.Utils` or other places)? As another option, how about adding internal option in `SparkConf` for that? Also, how about making a parent jira tcket to track the similar issue cuz it seems difficult to cover all the possible place in this pr. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19266 CC @maropu --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19266: [SPARK-22033][CORE] BufferHolder, other size checks shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19266 **[Test build #81882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81882/testReport)** for PR 19266 at commit [`8dbfa30`](https://github.com/apache/spark/commit/8dbfa30848b49a9fe4038c327222ddc9fd9a0ec0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
