[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[tedyu]

Github user tedyu closed the pull request at:

https://github.com/apache/spark/pull/12814


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-216457498
  
This is really trivial, and will become unnecessary once we have new unsafe 
format for array. Can you close it for now and wait for the new format?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[tedyu]

Github user tedyu commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-215974354
  
```
sbt.ForkMain$ForkError: java.lang.AssertionError: 
expected:<0.9986422261219262> but was:<0.9986422261219272>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.spark.mllib.stat.JavaStatisticsSuite.testCorr(JavaStatisticsSuite.java:75)
```
The above assertion failure is not related to the change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-215973310
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-215973311
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57448/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-215973298
  
**[Test build #57448 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57448/consoleFull)**
 for PR 12814 at commit 
[`bce9d12`](https://github.com/apache/spark/commit/bce9d12e8b78fcb6f228cb2f103cf2b50ce351ef).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/12814#discussion_r61670779
  
--- Diff: 
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java
 ---
@@ -338,9 +338,10 @@ public UnsafeArrayData copy() {
   }
 
   public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
-if (arr.length > (Integer.MAX_VALUE - 4) / 8) {
+int limit = (Integer.MAX_VALUE - 4) / 8;
+if (arr.length > limit) {
   throw new UnsupportedOperationException("Cannot convert this array 
to unsafe format as " +
-"it's too big.");
+"its length (" + arr.length + ") exceeds " + limit + ".");
--- End diff --

Oh, duh sorry about that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/12814#discussion_r61670775
  
--- Diff: 
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java
 ---
@@ -338,9 +338,10 @@ public UnsafeArrayData copy() {
   }
 
   public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
-if (arr.length > (Integer.MAX_VALUE - 4) / 8) {
+int limit = (Integer.MAX_VALUE - 4) / 8;
+if (arr.length > limit) {
   throw new UnsupportedOperationException("Cannot convert this array 
to unsafe format as " +
-"it's too big.");
+"its length (" + arr.length + ") exceeds " + limit + ".");
--- End diff --

This is Java.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-215972055
  
Ted this is pretty trivial, and tacked on to another JIRA. It's not really 
a logically necessary part of the other changes. I don't think it needs a JIRA 
anyway; it's barely worth doing unless you're going to survey lots of similar 
messages and try to uniformly improve them. I'd rather not have to review a 
stream of tiny ad-hoc changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/12814#discussion_r61670705
  
--- Diff: 
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java
 ---
@@ -338,9 +338,10 @@ public UnsafeArrayData copy() {
   }
 
   public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
-if (arr.length > (Integer.MAX_VALUE - 4) / 8) {
+int limit = (Integer.MAX_VALUE - 4) / 8;
+if (arr.length > limit) {
   throw new UnsupportedOperationException("Cannot convert this array 
to unsafe format as " +
-"it's too big.");
+"its length (" + arr.length + ") exceeds " + limit + ".");
--- End diff --

String interpolation?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-215971578
  
**[Test build #57448 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57448/consoleFull)**
 for PR 12814 at commit 
[`bce9d12`](https://github.com/apache/spark/commit/bce9d12e8b78fcb6f228cb2f103cf2b50ce351ef).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-215971224
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-215971221
  
**[Test build #57447 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57447/consoleFull)**
 for PR 12814 at commit 
[`ed47ef9`](https://github.com/apache/spark/commit/ed47ef9994bbdd23aa14584da19e20483a35b1f2).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-215971225
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57447/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12814#issuecomment-215971076
  
**[Test build #57447 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57447/consoleFull)**
 for PR 12814 at commit 
[`ed47ef9`](https://github.com/apache/spark/commit/ed47ef9994bbdd23aa14584da19e20483a35b1f2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14850] Show limit for array size when a...

[tedyu]

GitHub user tedyu opened a pull request:

https://github.com/apache/spark/pull/12814

[SPARK-14850] Show limit for array size when array is too big

## What changes were proposed in this pull request?

This PR shows the size of array and the limit when array is too big.

## How was this patch tested?

Existing tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tedyu/spark master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12814.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12814


commit ed47ef9994bbdd23aa14584da19e20483a35b1f2
Author: tedyu 
Date:   2016-04-30T14:42:55Z

[SPARK-14850] Show limit for array size when array is too big




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org