[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai closed the pull request at:

https://github.com/apache/spark/pull/10619


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10619#issuecomment-170220630
  
@QiangCai you may want to close this one, and then make a new one that 
incorporates the original change and also 
https://github.com/apache/spark/pull/10670 once it's merged


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10619#issuecomment-169700777
  
@sarutak I will do it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[sarutak]

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169274323
  
@QiangCai We have many conflicts against `branch-1.6` so I'd merge this 
into only `master` for now.
If you want merge this into `branch-1.6` please feel free open another PR.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[sarutak]

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169272584
  
Merging this into `master` and `branch-1.6`. Thanks @QiangCai !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169270903
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10562


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169369194
  
OK. I have created another PR https://github.com/apache/spark/pull/10619.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10619#issuecomment-169369211
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

GitHub user QiangCai opened a pull request:

https://github.com/apache/spark/pull/10619

[SPARK-12340][SQL]fix Int overflow in the SparkPlan.executeTake, RDD.take 
and AsyncRDDActions.takeAsync for branch-1.6

 I create this PR to merge this code into branch-1.6. And I have merged 
this code into master in PR https://github.com/apache/spark/pull/10562.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/spark bugfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10619.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10619


commit b35e9b922e893ab825bfd43370cc91852f6cc50a
Author: QiangCai 
Date:   2016-01-06T15:51:17Z

avoid Int overflow




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[sarutak]

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/10619#issuecomment-169370129
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10619#issuecomment-169369849
  
**[Test build #2337 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2337/consoleFull)**
 for PR 10619 at commit 
[`b35e9b9`](https://github.com/apache/spark/commit/b35e9b922e893ab825bfd43370cc91852f6cc50a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10619#issuecomment-169400213
  
**[Test build #2337 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2337/consoleFull)**
 for PR 10619 at commit 
[`b35e9b9`](https://github.com/apache/spark/commit/b35e9b922e893ab825bfd43370cc91852f6cc50a).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48989147
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1291,11 +1291,11 @@ abstract class RDD[T: ClassTag](
 } else {
   val buf = new ArrayBuffer[T]
   val totalParts = this.partitions.length
-  var partsScanned = 0
+  var partsScanned = 0L
--- End diff --

why is this change necessary? When can partsScanned go above 2B?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[sarutak]

Github user sarutak commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48993704
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1291,11 +1291,11 @@ abstract class RDD[T: ClassTag](
 } else {
   val buf = new ArrayBuffer[T]
   val totalParts = this.partitions.length
-  var partsScanned = 0
+  var partsScanned = 0L
--- End diff --

Ah, you're right. `partScanned` cannot exceed the value of `totalParts`.
I'll return it to `Int`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48993960
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1291,11 +1291,11 @@ abstract class RDD[T: ClassTag](
 } else {
   val buf = new ArrayBuffer[T]
   val totalParts = this.partitions.length
-  var partsScanned = 0
+  var partsScanned = 0L
--- End diff --

I think there is a legit problem here. Imagine `totalParts` is close to 
`Int.MaxValue`, and imagine `partsScanned` is close to `totalParts`. Adding 
`p.size` to it below could cause it to roll over. I think this change is needed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48995991
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1291,11 +1291,11 @@ abstract class RDD[T: ClassTag](
 } else {
   val buf = new ArrayBuffer[T]
   val totalParts = this.partitions.length
-  var partsScanned = 0
+  var partsScanned = 0L
--- End diff --

That's never possible -- if we have anywhere near 2B partitions, the 
scheduler won't be fast enough to schedule them. As a matter of fact, if we 
have anywhere larger than a few millions, the scheduler will likely crash.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48996433
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1291,11 +1291,11 @@ abstract class RDD[T: ClassTag](
 } else {
   val buf = new ArrayBuffer[T]
   val totalParts = this.partitions.length
-  var partsScanned = 0
+  var partsScanned = 0L
--- End diff --

Fair point, in practice this all but certainly won't happen. Note that this 
patch was already committed to `master` making this a `Long`. It doesn't hurt 
and is very very theoretically more correct locally. I suppose I don't think 
it's worth updating again, but I do not feel strongly about it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48997252
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1291,11 +1291,11 @@ abstract class RDD[T: ClassTag](
 } else {
   val buf = new ArrayBuffer[T]
   val totalParts = this.partitions.length
-  var partsScanned = 0
+  var partsScanned = 0L
--- End diff --

I'd prefer to change it back since it is so little work, in case this 
starts a trend to change all ints to longs for no reason, and also raise 
questions about why this can be greater than int.max when we read this code in 
the future.

Also @srowen even if totalParts is close to int.max, I don't think 
partsScanned can be greater than int.max because we never scan more parts than 
the number of parts available.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48997385
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2067,4 +2067,16 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   )
 }
   }
+
+  test("SPARK-12340: overstep the bounds of Int in SparkPlan.executeTake") 
{
+val rdd = sqlContext.sparkContext.parallelize(1 to 3 , 3 )
--- End diff --

also remove the extra space before comma here.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48997514
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2067,4 +2067,16 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   )
 }
   }
+
+  test("SPARK-12340: overstep the bounds of Int in SparkPlan.executeTake") 
{
+val rdd = sqlContext.sparkContext.parallelize(1 to 3 , 3 )
+rdd.toDF("key").registerTempTable("spark12340")
+checkAnswer(
+  sql("select key from spark12340 limit 2147483638"),
+  Row(1) :: Row(2) :: Row(3) :: Nil
+)
+assert(rdd.take(2147483638).size === 3)
--- End diff --

we should have a unit test in RDDSuite for the rdd tests, not in 
SQLQuerySuite.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48997784
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1291,11 +1291,11 @@ abstract class RDD[T: ClassTag](
 } else {
   val buf = new ArrayBuffer[T]
   val totalParts = this.partitions.length
-  var partsScanned = 0
+  var partsScanned = 0L
--- End diff --

Ah ok you were referring to partsScanned + numPartsToTry - we should just 
cast that to long to minimize the impact.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48997992
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2067,4 +2067,16 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   )
 }
   }
+
+  test("SPARK-12340: overstep the bounds of Int in SparkPlan.executeTake") 
{
+val rdd = sqlContext.sparkContext.parallelize(1 to 3 , 3 )
--- End diff --

for the sql part i'd just move this into the existing limit test case, and 
add a line of comment explaining this.

also you should explain in the comment why 2147483638 is chosen as a value.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169430744
  
@QiangCai it would be great if you can submit a new pull request to address 
the comments. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10619#issuecomment-169502987
  
spark-12340 has passed this test, but just other error has happend.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[sarutak]

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/10619#issuecomment-169547090
  
@QiangCai Do you have time to follow @rxin 's comment in #10562 ?
Also if you'd like to merge this into `branch-1.6`, could you fix this 
similarly? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168967549
  
@QiangCai I think the test failures are unrelated. However before we can 
retest you'll have to rebase as there is a merge conflict now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[sarutak]

Github user sarutak commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48867966
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2067,4 +2067,16 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   )
 }
   }
+
+  test("SPARK-12340: overstep the bounds of Int in SparkPlan.executeTake"){
--- End diff --

If you have a chance to modify again, please insert a white space between 
`)` and `{`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169063131
  
@sarutak  Maybe we have found another bug. I will try to fix it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169070917
  
I have remove the initial size of the ArrayBuffer instance. And the default 
size is 16.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169076579
  
**[Test build #2326 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2326/consoleFull)**
 for PR 10562 at commit 
[`e7577ee`](https://github.com/apache/spark/commit/e7577ee98630d1e53782e6f7dbc7979c1bc558a8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[sarutak]

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169061245
  
Why the instance of `ArrayBuffer` in `AsyncRDDActions#takeAsync` is created 
with initial size?
On the other hand, the instance of `ArrayBuffer` in `RDD#take` is created 
without initial size.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169056589
  
@srowen I find error message in test bulid log, a OutOfMemoryError 
exception has happened. The code in 71 line of  the file AsyncRDDActions.scala 
is  "val results = new ArrayBuffer[T](num) ", because num(2147483638) is too 
large, JVM can't allocate enough memory space. I think I should modify unit 
test to smaller num in 2079 line of SQLQuerySuite.scala.

error message:
[info] Exception encountered when attempting to run a suite with class 
name: org.apache.spark.sql.SQLQuerySuite *** ABORTED *** (31 seconds, 447 
milliseconds)
[info]   java.lang.OutOfMemoryError: Java heap space
[info]   at 
scala.collection.mutable.ResizableArray$class.$init$(ResizableArray.scala:32)
[info]   at 
scala.collection.mutable.ArrayBuffer.(ArrayBuffer.scala:47)
[info]   at 
org.apache.spark.rdd.AsyncRDDActions$$anonfun$takeAsync$1.apply(AsyncRDDActions.scala:71)
[info]   at 
org.apache.spark.rdd.AsyncRDDActions$$anonfun$takeAsync$1.apply(AsyncRDDActions.scala:66)
[info]   at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
[info]   at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
[info]   at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
[info]   at 
org.apache.spark.rdd.AsyncRDDActions.takeAsync(AsyncRDDActions.scala:66)
[info]   at 
org.apache.spark.sql.SQLQuerySuite$$anonfun$132.apply$mcV$sp(SQLQuerySuite.scala:2079)
[info]   at 
org.apache.spark.sql.SQLQuerySuite$$anonfun$132.apply(SQLQuerySuite.scala:2071)
[info]   at 
org.apache.spark.sql.SQLQuerySuite$$anonfun$132.apply(SQLQuerySuite.scala:2071)
..


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48868638
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2067,4 +2067,16 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   )
 }
   }
+
+  test("SPARK-12340: overstep the bounds of Int in SparkPlan.executeTake"){
--- End diff --

I will do it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169033103
  
**[Test build #2325 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2325/consoleFull)**
 for PR 10562 at commit 
[`3d340f7`](https://github.com/apache/spark/commit/3d340f730309f9a2930051caea0e516ef52b1d06).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48844685
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2028,6 +2028,7 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   Row(false) :: Row(true) :: Nil)
   }
 
+<<< 1cdc42d2b99edfec01066699a7620cca02b61f0e
--- End diff --

I have remove it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169002316
  
**[Test build #2325 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2325/consoleFull)**
 for PR 10562 at commit 
[`3d340f7`](https://github.com/apache/spark/commit/3d340f730309f9a2930051caea0e516ef52b1d06).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169106952
  
**[Test build #2326 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2326/consoleFull)**
 for PR 10562 at commit 
[`e7577ee`](https://github.com/apache/spark/commit/e7577ee98630d1e53782e6f7dbc7979c1bc558a8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168997795
  
@srowen I have rebased from master and resolved all conflicts.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48844423
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2028,6 +2028,7 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   Row(false) :: Row(true) :: Nil)
   }
 
+<<< 1cdc42d2b99edfec01066699a7620cca02b61f0e
--- End diff --

This shouldn't be here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-169191091
  
I think I have resolved this problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168639329
  
@QiangCai the problem isn't blank lines but whitespace at the end of your 
lines.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168863650
  
@srowen  I have no idea how to resolve this error. Would you help me?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai closed the pull request at:

https://github.com/apache/spark/pull/10562


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168674310
  
@srowen I have removed some whitespaces.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

GitHub user QiangCai reopened a pull request:

https://github.com/apache/spark/pull/10562

[SPARK-12340][SQL]fix Int overflow in the SparkPlan.executeTake, RDD.take 
and AsyncRDDActions.takeAsync

I have closed pull request https://github.com/apache/spark/pull/10487. And 
I create this pull request to resolve the problem.

spark jira
https://issues.apache.org/jira/browse/SPARK-12340

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/spark bugfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10562.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10562


commit 4974f05d93852f4ee302715d811eb585affddd2d
Author: QiangCai 
Date:   2016-01-03T13:47:58Z

[SPARK-12340][SQL]fix Int overflow in the SparkPlan.executeTake, RDD.take 
and AsyncRDDActions.takeAsync

commit 9c102bc81bdf704ef1e1c916727f2ec78702be5b
Author: cai qiang 
Date:   2016-01-03T23:42:24Z

delete some blank lines

commit 639cfb26f33f2d0f31530b57a147f42904e635ef
Author: QiangCai 
Date:   2016-01-04T13:00:40Z

remove whitespaces




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168676335
  
**[Test build #2310 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2310/consoleFull)**
 for PR 10562 at commit 
[`639cfb2`](https://github.com/apache/spark/commit/639cfb26f33f2d0f31530b57a147f42904e635ef).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168700395
  
**[Test build #2310 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2310/consoleFull)**
 for PR 10562 at commit 
[`639cfb2`](https://github.com/apache/spark/commit/639cfb26f33f2d0f31530b57a147f42904e635ef).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168564772
  
I have removed some blank lines.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[sarutak]

Github user sarutak commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48698010
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2028,4 +2028,18 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   Row(false) :: Row(true) :: Nil)
   }
 
+  test("SPARK-12340: overstep the bounds of Int in SparkPlan.executeTake"){
+val rdd = sqlContext.sparkContext.parallelize(1 to 3 , 3 )
+
+rdd.toDF("key").registerTempTable("spark12340")
+checkAnswer(
+  sql("select key from spark12340 limit 2147483638"),
+  Row(1) :: Row(2) :: Row(3) :: Nil
+)
+
+assert(rdd.take(2147483638).size === 3)
+
+assert(rdd.takeAsync(2147483638).get.size === 3)
--- End diff --

Does this test case pass the compilation? `2147483638` is not within the 
range of `Int`.
Should we say `Int.MaxValue` ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[sarutak]

Github user sarutak commented on a diff in the pull request:

https://github.com/apache/spark/pull/10562#discussion_r48697977
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 
---
@@ -2028,4 +2028,18 @@ class SQLQuerySuite extends QueryTest with 
SharedSQLContext {
   Row(false) :: Row(true) :: Nil)
   }
 
+  test("SPARK-12340: overstep the bounds of Int in SparkPlan.executeTake"){
+val rdd = sqlContext.sparkContext.parallelize(1 to 3 , 3 )
+
+rdd.toDF("key").registerTempTable("spark12340")
+checkAnswer(
+  sql("select key from spark12340 limit 2147483638"),
+  Row(1) :: Row(2) :: Row(3) :: Nil
+)
+
+assert(rdd.take(2147483638).size === 3)
+
--- End diff --

I don't think a blank line is not needed here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-168478981
  
@QiangCai the merge commit is wrong as you can see. You can revert it, 
commit, rebase from master, fix conflicts, and force-push a merge to fix. Or, 
if you're not sure how to do it, close this PR and open a new one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-168500336
  
I have created another pull request 
https://github.com/apache/spark/pull/10562.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168500320
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai closed the pull request at:

https://github.com/apache/spark/pull/10487


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

GitHub user QiangCai opened a pull request:

https://github.com/apache/spark/pull/10562

[SPARK-12340][SQL]fix Int overflow in the SparkPlan.executeTake, RDD.take 
and AsyncRDDActions.takeAsync

I have close pull request https://github.com/apache/spark/pull/10487. And I 
create this pull request to resolve the problem.

spark jira
https://issues.apache.org/jira/browse/SPARK-12340

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/spark bugfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10562.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10562


commit 4974f05d93852f4ee302715d811eb585affddd2d
Author: QiangCai 
Date:   2016-01-03T13:47:58Z

[SPARK-12340][SQL]fix Int overflow in the SparkPlan.executeTake, RDD.take 
and AsyncRDDActions.takeAsync




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168534382
  
**[Test build #2307 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2307/consoleFull)**
 for PR 10562 at commit 
[`4974f05`](https://github.com/apache/spark/commit/4974f05d93852f4ee302715d811eb585affddd2d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10562#issuecomment-168534430
  
**[Test build #2307 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2307/consoleFull)**
 for PR 10562 at commit 
[`4974f05`](https://github.com/apache/spark/commit/4974f05d93852f4ee302715d811eb585affddd2d).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-168467402
  
@sarutak When  I am rebasing  from master,  I get many conflicts. I don't 
known how to resolve it. I have  just pushed  the commit "merge".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-167540446
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[sarutak]

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-167653173
  
@QiangCai LGTM but could you add test cases?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-167709123
  
@sarutak I will try to add test cases.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[QiangCai]

GitHub user QiangCai opened a pull request:

https://github.com/apache/spark/pull/10487

[SPARK-12340][SQL]fix Int overflow in the SparkPlan.executeTake, RDD.take 
and AsyncRDDActions.takeAsync

@srowen  I new this pull request to to resolve the problem.

another pull request 
https://github.com/apache/spark/pull/10310

jira
https://issues.apache.org/jira/browse/SPARK-12340

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/spark bugfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10487.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10487


commit 18f1b81ba1fa47202ab069ce12f6f9141e981ed3
Author: QiangCai 
Date:   2015-12-27T08:20:44Z

fix Int overflow




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-167393041
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-167436105
  
**[Test build #2256 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2256/consoleFull)**
 for PR 10487 at commit 
[`18f1b81`](https://github.com/apache/spark/commit/18f1b81ba1fa47202ab069ce12f6f9141e981ed3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-167426345
  
**[Test build #2256 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2256/consoleFull)**
 for PR 10487 at commit 
[`18f1b81`](https://github.com/apache/spark/commit/18f1b81ba1fa47202ab069ce12f6f9141e981ed3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org