date:20151227

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167399270
  
**[Test build #48351 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48351/consoleFull)**
 for PR 10489 at commit 
[`6c1d8b5`](https://github.com/apache/spark/commit/6c1d8b540d2cc5e871cd06c1c8add710ae84b506).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167399420
  
**[Test build #48352 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48352/consoleFull)**
 for PR 10435 at commit 
[`a629e75`](https://github.com/apache/spark/commit/a629e754be3c087466de9f3b0aa8634e8d640ea0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12530][Build] Fix build break at Spark-...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10488#issuecomment-167401548
  
**[Test build #48348 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48348/consoleFull)**
 for PR 10488 at commit 
[`9658c3b`](https://github.com/apache/spark/commit/9658c3b6b66034a515a3e5d3cfd3d6cc9c4a3305).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12530][Build] Fix build break at Spark-...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10488#issuecomment-167401573
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48348/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12530][Build] Fix build break at Spark-...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10488#issuecomment-167401572
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167403074
  
**[Test build #48352 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48352/consoleFull)**
 for PR 10435 at commit 
[`a629e75`](https://github.com/apache/spark/commit/a629e754be3c087466de9f3b0aa8634e8d640ea0).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`case class Hash(children: Seq[Expression]) extends Expression `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167403081
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48351/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167403091
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48352/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167403063
  
**[Test build #48351 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48351/consoleFull)**
 for PR 10489 at commit 
[`6c1d8b5`](https://github.com/apache/spark/commit/6c1d8b540d2cc5e871cd06c1c8add710ae84b506).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167403080
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167403090
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL] Fix overstep the bounds of ...

2015-12-27 Thread QiangCai

Github user QiangCai commented on the pull request:

https://github.com/apache/spark/pull/10310#issuecomment-167393002
  
@srowen I have new another pull request. This pull request will be closed.
another pull request https://github.com/apache/spark/pull/10487


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12530][Build] Fix build break at Spark-...

2015-12-27 Thread kiszk

GitHub user kiszk opened a pull request:

https://github.com/apache/spark/pull/10488

[SPARK-12530][Build] Fix build break at Spark-Master-Maven-Snapshots from 
#1293

Compilation error caused due to string concatenations that are not a 
constant
Use raw string literal to avoid string concatenations


https://amplab.cs.berkeley.edu/jenkins/view/Spark-Packaging/job/Spark-Master-Maven-Snapshots/1293/

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kiszk/spark SPARK-12530

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10488


commit 9658c3b6b66034a515a3e5d3cfd3d6cc9c4a3305
Author: Kazuaki Ishizaki 
Date:   2015-12-27T08:24:38Z

fix compilation error

use raw string literal to avoid string concatenations




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

2015-12-27 Thread QiangCai

GitHub user QiangCai opened a pull request:

https://github.com/apache/spark/pull/10487

[SPARK-12340][SQL]fix Int overflow in the SparkPlan.executeTake, RDD.take 
and AsyncRDDActions.takeAsync

@srowen  I new this pull request to to resolve the problem.

another pull request 
https://github.com/apache/spark/pull/10310

jira
https://issues.apache.org/jira/browse/SPARK-12340

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/spark bugfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10487.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10487


commit 18f1b81ba1fa47202ab069ce12f6f9141e981ed3
Author: QiangCai 
Date:   2015-12-27T08:20:44Z

fix Int overflow




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-167393041
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL] Fix overstep the bounds of ...

2015-12-27 Thread QiangCai

Github user QiangCai closed the pull request at:

https://github.com/apache/spark/pull/10310


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread cloud-fan

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167395236
  
retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167396600
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167396598
  
**[Test build #48349 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48349/consoleFull)**
 for PR 10435 at commit 
[`c130097`](https://github.com/apache/spark/commit/c130097e3f12a0d45eed2e2d5eb6682d65f11c9a).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`case class Hash(children: Seq[Expression]) extends Expression `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12530][Build] Fix build break at Spark-...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10488#issuecomment-167396615
  
**[Test build #48348 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48348/consoleFull)**
 for PR 10488 at commit 
[`9658c3b`](https://github.com/apache/spark/commit/9658c3b6b66034a515a3e5d3cfd3d6cc9c4a3305).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167396601
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48349/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167396554
  
**[Test build #48349 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48349/consoleFull)**
 for PR 10435 at commit 
[`c130097`](https://github.com/apache/spark/commit/c130097e3f12a0d45eed2e2d5eb6682d65f11c9a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167396643
  
**[Test build #48350 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48350/consoleFull)**
 for PR 10435 at commit 
[`c130097`](https://github.com/apache/spark/commit/c130097e3f12a0d45eed2e2d5eb6682d65f11c9a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167396684
  
**[Test build #48350 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48350/consoleFull)**
 for PR 10435 at commit 
[`c130097`](https://github.com/apache/spark/commit/c130097e3f12a0d45eed2e2d5eb6682d65f11c9a).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`case class Hash(children: Seq[Expression]) extends Expression `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167396686
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48350/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167396685
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread vectorijk

GitHub user vectorijk opened a pull request:

https://github.com/apache/spark/pull/10489

[SPARK-12461] [SQL] Add ExpressionDescription to math functions

cc @yhuai 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vectorijk/spark spark-12461

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10489.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10489


commit 6c1d8b540d2cc5e871cd06c1c8add710ae84b506
Author: Kai Jiang 
Date:   2015-12-27T07:33:33Z

Add ExpressionDescription to math functions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-167436105
  
**[Test build #2256 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2256/consoleFull)**
 for PR 10487 at commit 
[`18f1b81`](https://github.com/apache/spark/commit/18f1b81ba1fa47202ab069ce12f6f9141e981ed3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12415] Do not use closure serializer to...

2015-12-27 Thread srowen

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/10368#discussion_r48458317
  
--- Diff: core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala 
---
@@ -220,7 +222,7 @@ class HeartbeatReceiverSuite
 } else {
   assert(!response.reregisterBlockManager)
   // Additionally verify that the scheduler callback is called with 
the correct parameters
-  verify(scheduler).executorHeartbeatReceived(
+  verify(scheduler, 
VerificationModeFactory.atLeast(1)).executorHeartbeatReceived(
--- End diff --

@tedyu this still needs attention. Is the rest OK to commit @andrewor14 or 
are you interested in refining the tests?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12530][Build] Fix build break at Spark-...

2015-12-27 Thread kiszk

Github user kiszk commented on the pull request:

https://github.com/apache/spark/pull/10488#issuecomment-167418794
  
Thanks for letting me know them. I will check them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

2015-12-27 Thread saucam

Github user saucam commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-167422237
  
Hey @marmbrus sorry for the delay in this update, I have added the same 
thing to the planner. Also rebased to latest master. How does it look now ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-167423222
  
**[Test build #48353 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48353/consoleFull)**
 for PR 9858 at commit 
[`1e1422d`](https://github.com/apache/spark/commit/1e1422d3039279cd62b988cc13e2ceba3c43b309).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10490#issuecomment-167436825
  
**[Test build #48355 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48355/consoleFull)**
 for PR 10490 at commit 
[`918ea2c`](https://github.com/apache/spark/commit/918ea2ce80d2b0f749349ff58fe4967760388814).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-167432879
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48354/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-167432877
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-167426083
  
**[Test build #48354 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48354/consoleFull)**
 for PR 9858 at commit 
[`5e5994d`](https://github.com/apache/spark/commit/5e5994d0caca2b8d3ef862ac7a76fd6dd5b51225).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Branch 1.2

2015-12-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10485#issuecomment-167431807
  
@ErlangZ please close this PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12515][SQL][DOC] minor doc update for r...

2015-12-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10465#issuecomment-167437583
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12530][Build] Fix build break at Spark-...

2015-12-27 Thread kiszk

Github user kiszk commented on the pull request:

https://github.com/apache/spark/pull/10488#issuecomment-167424111
  
@hvanhovell, a set of following two conditions causes this compilation 
failure:
1. Use more than one string concatenations
2. Use scala 2.11 compiler (open question is that [this 
failure](https://amplab.cs.berkeley.edu/jenkins/view/Spark-Packaging/job/Spark-Master-Maven-Snapshots/1293/consoleFull)
 occurs by using both scala 2.10 and 2.11 compilers on Jenkins 

I can reproduce this compilation error by using the following command:

$ java -version
openjdk version "1.8.0_65"
OpenJDK Runtime Environment (build 1.8.0_65-b17)
OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
$ build/mvn -Dscala-2.11 -Pyarn -Phadoop-2.4 -DskipTests clean package 
install -pl sql/catalyst



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12263][Docs]: IllegalStateException: Me...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10483#issuecomment-167431974
  
**[Test build #2257 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2257/consoleFull)**
 for PR 10483 at commit 
[`b55fe70`](https://github.com/apache/spark/commit/b55fe70811528f16c7dd63b689dfdb96b6adfd14).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...

2015-12-27 Thread gatorsmile

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/10490

[SPARK-12532] [SQL] Join-key Pushdown via Predicate Transitivity

More predicates in join conditions/filters can be pushed down via predicate 
transitivity. More predicate pushdown could greatly improve the join 
performance. 

For example, we can infer the extra predicate `upperCaseData.N = 3` in the 
following query:
```
"SELECT * FROM upperCaseData JOIN lowerCaseData where lowerCaseData.n = 
upperCaseData.N and lowerCaseData.n = 3"
```

Before the improvement, the optimized logical plan is
```
== Optimized Logical Plan ==
Project [N#16,L#17,n#18,l#19]
+- Join Inner, Some((n#18 = N#16))
   :- LogicalRDD [N#16,L#17], MapPartitionsRDD[17] at beforeAll at 
BeforeAndAfterAll.scala:187
   +- Filter (n#18 = 3)
  +- LogicalRDD [n#18,l#19], MapPartitionsRDD[19] at beforeAll at 
BeforeAndAfterAll.scala:187
```
After the improvement, the optimized logical plan should be like
```
== Optimized Logical Plan ==
Project [N#16,L#17,n#18,l#19]
+- Join Inner, Some((n#18 = N#16))
   :- Filter (N#16 = 3)
   :  +- LogicalRDD [N#16,L#17], MapPartitionsRDD[17] at beforeAll at 
BeforeAndAfterAll.scala:187
   +- Filter (n#18 = 3)
  +- LogicalRDD [n#18,l#19], MapPartitionsRDD[19] at beforeAll at 
BeforeAndAfterAll.scala:187
```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark conditionInfer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10490


commit fb84dbac240361e086fdb0d4a18760006c0d4e76
Author: gatorsmile 
Date:   2015-12-27T18:42:04Z

Infer and push down join/filter conditions

commit 918ea2ce80d2b0f749349ff58fe4967760388814
Author: gatorsmile 
Date:   2015-12-27T18:42:29Z

Merge remote-tracking branch 'upstream/master' into conditionInfer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-167423751
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-167423743
  
**[Test build #48353 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48353/consoleFull)**
 for PR 9858 at commit 
[`1e1422d`](https://github.com/apache/spark/commit/1e1422d3039279cd62b988cc13e2ceba3c43b309).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-167423754
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48353/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12263][Docs]: IllegalStateException: Me...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10483#issuecomment-167431912
  
**[Test build #2257 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2257/consoleFull)**
 for PR 10483 at commit 
[`b55fe70`](https://github.com/apache/spark/commit/b55fe70811528f16c7dd63b689dfdb96b6adfd14).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12263][Docs]: IllegalStateException: Me...

2015-12-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10483#issuecomment-167431844
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10490#issuecomment-167440556
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48355/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10490#issuecomment-167440555
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12532] [SQL] Join-key Pushdown via Pred...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10490#issuecomment-167440524
  
**[Test build #48355 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48355/consoleFull)**
 for PR 10490 at commit 
[`918ea2c`](https://github.com/apache/spark/commit/918ea2ce80d2b0f749349ff58fe4967760388814).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10487#issuecomment-167426345
  
**[Test build #2256 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2256/consoleFull)**
 for PR 10487 at commit 
[`18f1b81`](https://github.com/apache/spark/commit/18f1b81ba1fa47202ab069ce12f6f9141e981ed3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11878][SQL]: Eliminate distribute by in...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9858#issuecomment-167432837
  
**[Test build #48354 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48354/consoleFull)**
 for PR 9858 at commit 
[`5e5994d`](https://github.com/apache/spark/commit/5e5994d0caca2b8d3ef862ac7a76fd6dd5b51225).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12530][Build] Fix build break at Spark-...

2015-12-27 Thread hvanhovell

Github user hvanhovell commented on the pull request:

https://github.com/apache/spark/pull/10488#issuecomment-167418258
  
There a quite a few builds passing which are using string concatenation in 
combination with ```ExpressionDescription```:
https://github.com/apache/spark/pull/10489
https://github.com/apache/spark/pull/10418
https://github.com/apache/spark/pull/10402

Maybe something else is causing this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-10625] [SQL] Spark SQL JDBC read/write ...

2015-12-27 Thread tribbloid

Github user tribbloid commented on a diff in the pull request:

https://github.com/apache/spark/pull/8785#discussion_r48460773
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/jdbc/UnserializableDriverHelper.scala
 ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc
+
+import java.sql.{DriverManager, Connection}
+import java.util.Properties
+import java.util.logging.Logger
+
+object UnserializableDriverHelper {
+
+  def replaceDriverDuring[T](f: => T): T = {
+import scala.collection.JavaConverters._
+
+object UnserializableH2Driver extends org.h2.Driver {
+
+  override def connect(url: String, info: Properties): Connection = {
+
+val result = super.connect(url, info)
+info.put("unserializableDriver", this)
+result
+  }
+
+  override def getParentLogger: Logger = null
+}
+
+val oldDrivers = 
DriverManager.getDrivers.asScala.filter(_.acceptsURL("jdbc:h2:"))
+oldDrivers.foreach{ DriverManager.deregisterDriver }
--- End diff --

Yes, both foreach are made one line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-10625] [SQL] Spark SQL JDBC read/write ...

2015-12-27 Thread tribbloid

Github user tribbloid commented on a diff in the pull request:

https://github.com/apache/spark/pull/8785#discussion_r48460762
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/jdbc/UnserializableDriverHelper.scala
 ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc
+
+import java.sql.{DriverManager, Connection}
+import java.util.Properties
+import java.util.logging.Logger
+
+object UnserializableDriverHelper {
+
+  def replaceDriverDuring[T](f: => T): T = {
+import scala.collection.JavaConverters._
--- End diff --

Yes, promote to under object


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12168][SPARKR] Add automated tests for ...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10171#issuecomment-167463955
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48357/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12168][SPARKR] Add automated tests for ...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10171#issuecomment-167463702
  
**[Test build #48357 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48357/consoleFull)**
 for PR 10171 at commit 
[`fa4869d`](https://github.com/apache/spark/commit/fa4869d27b106a209ce515202c52d611922f63ab).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-27 Thread felixcheung

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/10491

[SPARK-12534][DOC] update documentation to list command line equivalent to 
properties

Several Spark properties equivalent to Spark submit command line options 
are missing.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark sparksubmitdoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10491.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10491


commit 0fb3a40d8bc6d9186328348881a0bd0a28895124
Author: felixcheung 
Date:   2015-12-28T03:21:42Z

update doc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167469968
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167469970
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48359/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12513] [Streaming] SocketReceiver hang ...

2015-12-27 Thread jerryshao

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/10464#issuecomment-167469967
  
No, you don't. It's OK to have 2 commits. There's a duplication 
[here](https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/SocketInputDStream.scala#L91),
 I think you could simplify it.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167469966
  
**[Test build #48359 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48359/consoleFull)**
 for PR 10489 at commit 
[`e31723d`](https://github.com/apache/spark/commit/e31723de94997b54731a9f48214c843f6b766dd4).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167473135
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48358/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167473131
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12453][Streaming] Remove explicit depen...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10492#issuecomment-167480578
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167480572
  
**[Test build #48360 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48360/consoleFull)**
 for PR 10489 at commit 
[`12d129d`](https://github.com/apache/spark/commit/12d129dc64a68551f3a18b9fe8f2f08aca810fde).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-10625] [SQL] Spark SQL JDBC read/write ...

2015-12-27 Thread tribbloid

Github user tribbloid closed the pull request at:

https://github.com/apache/spark/pull/8785


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-10625] [SQL] Spark SQL JDBC read/write ...

2015-12-27 Thread tribbloid

Github user tribbloid commented on the pull request:

https://github.com/apache/spark/pull/8785#issuecomment-167461046
  
All issues should be fixed by now, submitting final pull request


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-10625] [SQL] Spark SQL JDBC read/write ...

2015-12-27 Thread tribbloid

GitHub user tribbloid reopened a pull request:

https://github.com/apache/spark/pull/8785

[Spark-10625] [SQL] Spark SQL JDBC read/write is unable to handle JDBC 
Drivers that adds unserializable objects into connection properties

Connection properties are now deep copied before they are used by JDBC 
Drivers, this solvs all problems in unit tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Schedule1/spark SPARK-10625

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8785.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8785


commit 267afca7caa3e06a0fdac123f8a50033a726c564
Author: tribbloid 
Date:   2015-09-15T23:02:57Z

test case demonstrating SPARK-10625:
Spark SQL JDBC read/write is unable to handle JDBC Drivers that adds 
unserializable objects into connection properties

add one more unit test
fix JDBCRelation & DataFrameWriter to pass all tests

revise scala style

put driver replacement code into a shared function

fix styling

upgrade to master and resolve all related issues

commit cd641c01260eeabe910d819fffd9193461b37946
Author: tribbloid 
Date:   2015-12-10T01:15:18Z

remove the useless second deep copy of properties
rename test names to be more explicit

commit d33dc9c773e01695b3098f5a69733eacc2a88456
Author: tribbloid 
Date:   2015-12-16T02:39:37Z

minor refactoring based on Sean's suggestion

commit 1dcede3d98575152dda1c46b263fc4e2361f0766
Author: tribbloid 
Date:   2015-12-16T15:37:04Z

move JavaConverters import to under object

commit d2723c8a59301419af6132e06cacbb2860968c71
Author: tribbloid 
Date:   2015-12-21T19:07:05Z

remove the redundant toSeq and pull up lines in brackets

commit 6af8fd8b824f5f343a01868560b74a1f55acd02f
Author: tribbloid 
Date:   2015-12-28T02:05:00Z

improve styling in UnserializableDriverHelper and JDBCRelation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Add more exceptions to Guava relocation

2015-12-27 Thread microhello

Github user microhello commented on a diff in the pull request:

https://github.com/apache/spark/pull/10442#discussion_r48463834
  
--- Diff: pom.xml ---
@@ -99,14 +99,14 @@
 sql/hive
 unsafe
 assembly
-external/twitter
-external/flume
-external/flume-sink
-external/flume-assembly
-external/mqtt
-external/mqtt-assembly
-external/zeromq
-examples
+
--- End diff --

These changes only used by myself, don't need  merge. But I hope the 
following changes be merged:
Add `com.google.common.base.AbstractIterator` exceptions to Guava 
relocation , So far, all guava 14.0.1 `Optional` Class dependencies have added;


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12363] [SQL] Inline Hive parser into sp...

2015-12-27 Thread hvanhovell

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/10420#discussion_r48458564
  
--- Diff: 
sql/hive/src/main/java/org/apache/spark/sql/parser/ParseDriver.java ---
@@ -0,0 +1,260 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.parser;
+
+import java.util.ArrayList;
+import org.antlr.runtime.ANTLRStringStream;
+import org.antlr.runtime.CharStream;
+import org.antlr.runtime.NoViableAltException;
+import org.antlr.runtime.RecognitionException;
+import org.antlr.runtime.Token;
+import org.antlr.runtime.TokenRewriteStream;
+import org.antlr.runtime.TokenStream;
+import org.antlr.runtime.tree.CommonTree;
+import org.antlr.runtime.tree.CommonTreeAdaptor;
+import org.antlr.runtime.tree.TreeAdaptor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hive.ql.Context;
+
+/**
+ * ParseDriver.
+ *
+ */
+public class ParseDriver {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger("hive.ql.parse.ParseDriver");
+
+  /**
+   * ANTLRNoCaseStringStream.
+   *
+   */
+  //This class provides and implementation for a case insensitive token 
checker
+  //for the lexical analysis part of antlr. By converting the token stream 
into
+  //upper case at the time when lexical rules are checked, this class 
ensures that the
+  //lexical rules need to just match the token with upper case letters as 
opposed to
+  //combination of upper case and lower case characteres. This is purely 
used for matching lexical
+  //rules. The actual token text is stored in the same way as the user 
input without
+  //actually converting it into an upper case. The token values are 
generated by the consume()
+  //function of the super class ANTLRStringStream. The LA() function is 
the lookahead funtion
+  //and is purely used for matching lexical rules. This also means that 
the grammar will only
+  //accept capitalized tokens in case it is run from other tools like 
antlrworks which
+  //do not have the ANTLRNoCaseStringStream implementation.
+  public class ANTLRNoCaseStringStream extends ANTLRStringStream {
+
+public ANTLRNoCaseStringStream(String input) {
+  super(input);
+}
+
+@Override
+public int LA(int i) {
+
+  int returnChar = super.LA(i);
+  if (returnChar == CharStream.EOF) {
+return returnChar;
+  } else if (returnChar == 0) {
+return returnChar;
+  }
+
+  return Character.toUpperCase((char) returnChar);
+}
+  }
+
+  /**
+   * HiveLexerX.
+   *
+   */
+  public class HiveLexerX extends SparkSqlLexer {
+
+private final ArrayList errors;
+
+public HiveLexerX() {
+  super();
+  errors = new ArrayList();
+}
+
+public HiveLexerX(CharStream input) {
+  super(input);
+  errors = new ArrayList();
+}
+
+@Override
+public void displayRecognitionError(String[] tokenNames,
+RecognitionException e) {
+
+  errors.add(new ParseError(this, e, tokenNames));
+}
+
+@Override
+public String getErrorMessage(RecognitionException e, String[] 
tokenNames) {
+  String msg = null;
+
+  if (e instanceof NoViableAltException) {
+@SuppressWarnings("unused")
+NoViableAltException nvae = (NoViableAltException) e;
+// for development, can add
+// "decision=<<"+nvae.grammarDecisionDescription+">>"
+// and "(decision="+nvae.decisionNumber+") and
+// "state "+nvae.stateNumber
+msg = "character " + getCharErrorDisplay(e.c) + " not supported 
here";
+  } else {
+msg = super.getErrorMessage(e, tokenNames);
+  }
+
+  return msg;
+}
+
+public ArrayList getErrors() {
+

[GitHub] spark pull request: [Spark-10625] [SQL] Spark SQL JDBC read/write ...

2015-12-27 Thread tribbloid

Github user tribbloid commented on a diff in the pull request:

https://github.com/apache/spark/pull/8785#discussion_r48460893
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
 ---
@@ -75,6 +78,16 @@ private[sql] object JDBCRelation {
 }
 ans.toArray
   }
+
+  def getEffectiveProperties(
+  connectionProperties: Properties,
+  extraOptions: scala.collection.Map[String, String] = Map()): 
Properties = {
--- End diff --

Qualifier removed, but if without wrap the line exceeds 100 characters.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12453] [Streaming] Spark Streaming Kine...

2015-12-27 Thread BrianLondon

Github user BrianLondon commented on the pull request:

https://github.com/apache/spark/pull/10416#issuecomment-167480524
  
@srowen I can't close this because I didn't open it, but there's an 
explicit dependency removal PR at #10492 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12363] [SQL] Inline Hive parser into sp...

2015-12-27 Thread hvanhovell

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/10420#discussion_r48458578
  
--- Diff: 
sql/hive/src/main/java/org/apache/spark/sql/parser/ParseDriver.java ---
@@ -0,0 +1,260 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.parser;
+
+import java.util.ArrayList;
+import org.antlr.runtime.ANTLRStringStream;
+import org.antlr.runtime.CharStream;
+import org.antlr.runtime.NoViableAltException;
+import org.antlr.runtime.RecognitionException;
+import org.antlr.runtime.Token;
+import org.antlr.runtime.TokenRewriteStream;
+import org.antlr.runtime.TokenStream;
+import org.antlr.runtime.tree.CommonTree;
+import org.antlr.runtime.tree.CommonTreeAdaptor;
+import org.antlr.runtime.tree.TreeAdaptor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hive.ql.Context;
+
+/**
+ * ParseDriver.
+ *
+ */
+public class ParseDriver {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger("hive.ql.parse.ParseDriver");
+
+  /**
+   * ANTLRNoCaseStringStream.
+   *
+   */
+  //This class provides and implementation for a case insensitive token 
checker
+  //for the lexical analysis part of antlr. By converting the token stream 
into
+  //upper case at the time when lexical rules are checked, this class 
ensures that the
+  //lexical rules need to just match the token with upper case letters as 
opposed to
+  //combination of upper case and lower case characteres. This is purely 
used for matching lexical
+  //rules. The actual token text is stored in the same way as the user 
input without
+  //actually converting it into an upper case. The token values are 
generated by the consume()
+  //function of the super class ANTLRStringStream. The LA() function is 
the lookahead funtion
+  //and is purely used for matching lexical rules. This also means that 
the grammar will only
+  //accept capitalized tokens in case it is run from other tools like 
antlrworks which
+  //do not have the ANTLRNoCaseStringStream implementation.
+  public class ANTLRNoCaseStringStream extends ANTLRStringStream {
+
+public ANTLRNoCaseStringStream(String input) {
+  super(input);
+}
+
+@Override
+public int LA(int i) {
+
+  int returnChar = super.LA(i);
+  if (returnChar == CharStream.EOF) {
+return returnChar;
+  } else if (returnChar == 0) {
+return returnChar;
+  }
+
+  return Character.toUpperCase((char) returnChar);
+}
+  }
+
+  /**
+   * HiveLexerX.
+   *
+   */
+  public class HiveLexerX extends SparkSqlLexer {
+
+private final ArrayList errors;
+
+public HiveLexerX() {
+  super();
+  errors = new ArrayList();
+}
+
+public HiveLexerX(CharStream input) {
+  super(input);
+  errors = new ArrayList();
+}
+
+@Override
+public void displayRecognitionError(String[] tokenNames,
+RecognitionException e) {
+
+  errors.add(new ParseError(this, e, tokenNames));
+}
+
+@Override
+public String getErrorMessage(RecognitionException e, String[] 
tokenNames) {
+  String msg = null;
+
+  if (e instanceof NoViableAltException) {
+@SuppressWarnings("unused")
+NoViableAltException nvae = (NoViableAltException) e;
+// for development, can add
+// "decision=<<"+nvae.grammarDecisionDescription+">>"
+// and "(decision="+nvae.decisionNumber+") and
+// "state "+nvae.stateNumber
+msg = "character " + getCharErrorDisplay(e.c) + " not supported 
here";
+  } else {
+msg = super.getErrorMessage(e, tokenNames);
+  }
+
+  return msg;
+}
+
+public ArrayList getErrors() {
+

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167447980
  
**[Test build #48356 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48356/consoleFull)**
 for PR 10435 at commit 
[`655800c`](https://github.com/apache/spark/commit/655800cd72e5fabc256cd64a82f6ccd60b491f92).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12363] [SQL] Inline Hive parser into sp...

2015-12-27 Thread hvanhovell

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/10420#discussion_r48458597
  
--- Diff: 
sql/hive/src/main/java/org/apache/spark/sql/parser/ParseDriver.java ---
@@ -0,0 +1,260 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.parser;
+
+import java.util.ArrayList;
+import org.antlr.runtime.ANTLRStringStream;
+import org.antlr.runtime.CharStream;
+import org.antlr.runtime.NoViableAltException;
+import org.antlr.runtime.RecognitionException;
+import org.antlr.runtime.Token;
+import org.antlr.runtime.TokenRewriteStream;
+import org.antlr.runtime.TokenStream;
+import org.antlr.runtime.tree.CommonTree;
+import org.antlr.runtime.tree.CommonTreeAdaptor;
+import org.antlr.runtime.tree.TreeAdaptor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hive.ql.Context;
+
+/**
+ * ParseDriver.
+ *
+ */
+public class ParseDriver {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger("hive.ql.parse.ParseDriver");
+
+  /**
+   * ANTLRNoCaseStringStream.
+   *
+   */
+  //This class provides and implementation for a case insensitive token 
checker
+  //for the lexical analysis part of antlr. By converting the token stream 
into
+  //upper case at the time when lexical rules are checked, this class 
ensures that the
+  //lexical rules need to just match the token with upper case letters as 
opposed to
+  //combination of upper case and lower case characteres. This is purely 
used for matching lexical
+  //rules. The actual token text is stored in the same way as the user 
input without
+  //actually converting it into an upper case. The token values are 
generated by the consume()
+  //function of the super class ANTLRStringStream. The LA() function is 
the lookahead funtion
+  //and is purely used for matching lexical rules. This also means that 
the grammar will only
+  //accept capitalized tokens in case it is run from other tools like 
antlrworks which
+  //do not have the ANTLRNoCaseStringStream implementation.
+  public class ANTLRNoCaseStringStream extends ANTLRStringStream {
+
+public ANTLRNoCaseStringStream(String input) {
+  super(input);
+}
+
+@Override
+public int LA(int i) {
+
+  int returnChar = super.LA(i);
+  if (returnChar == CharStream.EOF) {
+return returnChar;
+  } else if (returnChar == 0) {
+return returnChar;
+  }
+
+  return Character.toUpperCase((char) returnChar);
+}
+  }
+
+  /**
+   * HiveLexerX.
+   *
+   */
+  public class HiveLexerX extends SparkSqlLexer {
+
+private final ArrayList errors;
+
+public HiveLexerX() {
+  super();
+  errors = new ArrayList();
+}
+
+public HiveLexerX(CharStream input) {
+  super(input);
+  errors = new ArrayList();
+}
+
+@Override
+public void displayRecognitionError(String[] tokenNames,
+RecognitionException e) {
+
+  errors.add(new ParseError(this, e, tokenNames));
+}
+
+@Override
+public String getErrorMessage(RecognitionException e, String[] 
tokenNames) {
+  String msg = null;
+
+  if (e instanceof NoViableAltException) {
+@SuppressWarnings("unused")
+NoViableAltException nvae = (NoViableAltException) e;
+// for development, can add
+// "decision=<<"+nvae.grammarDecisionDescription+">>"
+// and "(decision="+nvae.decisionNumber+") and
+// "state "+nvae.stateNumber
+msg = "character " + getCharErrorDisplay(e.c) + " not supported 
here";
+  } else {
+msg = super.getErrorMessage(e, tokenNames);
+  }
+
+  return msg;
+}
+
+public ArrayList getErrors() {
+

[GitHub] spark pull request: [Spark-10625] [SQL] Spark SQL JDBC read/write ...

2015-12-27 Thread tribbloid

Github user tribbloid commented on a diff in the pull request:

https://github.com/apache/spark/pull/8785#discussion_r48460878
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/jdbc/UnserializableDriverHelper.scala
 ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc
+
+import java.sql.{DriverManager, Connection}
+import java.util.Properties
+import java.util.logging.Logger
+
+object UnserializableDriverHelper {
+
+  def replaceDriverDuring[T](f: => T): T = {
+import scala.collection.JavaConverters._
+
+object UnserializableH2Driver extends org.h2.Driver {
+
+  override def connect(url: String, info: Properties): Connection = {
+
+val result = super.connect(url, info)
+info.put("unserializableDriver", this)
+result
+  }
+
+  override def getParentLogger: Logger = null
+}
+
+val oldDrivers = 
DriverManager.getDrivers.asScala.filter(_.acceptsURL("jdbc:h2:"))
+oldDrivers.foreach{ DriverManager.deregisterDriver }
+DriverManager.registerDriver(UnserializableH2Driver)
+
+val result = try { f }
+finally {
--- End diff --

Yes, fixing try-finally formatting


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167472857
  
**[Test build #48358 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48358/consoleFull)**
 for PR 10491 at commit 
[`0fb3a40`](https://github.com/apache/spark/commit/0fb3a40d8bc6d9186328348881a0bd0a28895124).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167452745
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48356/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167452744
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10435#issuecomment-167452730
  
**[Test build #48356 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48356/consoleFull)**
 for PR 10435 at commit 
[`655800c`](https://github.com/apache/spark/commit/655800cd72e5fabc256cd64a82f6ccd60b491f92).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`case class Hash(children: Seq[Expression]) extends Expression `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12168][SPARKR] Add automated tests for ...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10171#issuecomment-167462209
  
**[Test build #48357 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48357/consoleFull)**
 for PR 10171 at commit 
[`fa4869d`](https://github.com/apache/spark/commit/fa4869d27b106a209ce515202c52d611922f63ab).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12168][SPARKR] Add automated tests for ...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10171#issuecomment-167463954
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167469462
  
**[Test build #48359 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48359/consoleFull)**
 for PR 10489 at commit 
[`e31723d`](https://github.com/apache/spark/commit/e31723de94997b54731a9f48214c843f6b766dd4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167469480
  
**[Test build #48358 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48358/consoleFull)**
 for PR 10491 at commit 
[`0fb3a40`](https://github.com/apache/spark/commit/0fb3a40d8bc6d9186328348881a0bd0a28895124).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12453][Streaming] Remove explicit depen...

2015-12-27 Thread BrianLondon

GitHub user BrianLondon opened a pull request:

https://github.com/apache/spark/pull/10492

[SPARK-12453][Streaming] Remove explicit dependency on aws-java-sdk

Successfully ran kinesis demo on a live, aws hosted kinesis stream against 
master and 1.6 branches.  For reasons I don't entirely understand it required a 
manual merge to 1.5 which I did as shown here: 
https://github.com/BrianLondon/spark/commit/075c22e89bc99d5e99be21f40e0d72154a1e23a2

The demo ran successfully on the 1.5 branch as well.

According to `mvn dependency:tree` it is still pulling a fairly old version 
of the aws-java-sdk (1.9.37), but this appears to have fixed the kinesis 
regression in 1.5.2.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/BrianLondon/spark remove-only

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10492.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10492


commit 8e80f5ebf694d7b00f6558ce08bed3d14f82760c
Author: BrianLondon 
Date:   2015-12-10T20:49:07Z

Remove explicit dependency on aws-java-sdk




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12453][Streaming] Remove explicit depen...

2015-12-27 Thread JoshRosen

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/10492#issuecomment-167486919
  
Instead of just omitting the dependency, what about marking it as provided?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167492190
  
**[Test build #48360 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48360/consoleFull)**
 for PR 10489 at commit 
[`12d129d`](https://github.com/apache/spark/commit/12d129dc64a68551f3a18b9fe8f2f08aca810fde).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167492234
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12461] [SQL] Add ExpressionDescription ...

2015-12-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10489#issuecomment-167492235
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48360/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12054] [SQL] Consider nullability of ex...

2015-12-27 Thread davies

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/10333#discussion_r48464673
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ---
@@ -87,18 +87,21 @@ object Cast {
   private def resolvableNullability(from: Boolean, to: Boolean) = !from || 
to
 
   private def forceNullable(from: DataType, to: DataType) = (from, to) 
match {
-case (StringType, _: NumericType) => true
-case (StringType, TimestampType) => true
-case (DoubleType, TimestampType) => true
-case (FloatType, TimestampType) => true
-case (StringType, DateType) => true
-case (_: NumericType, DateType) => true
-case (BooleanType, DateType) => true
-case (DateType, _: NumericType) => true
-case (DateType, BooleanType) => true
-case (DoubleType, _: DecimalType) => true
-case (FloatType, _: DecimalType) => true
-case (_, DecimalType.Fixed(_, _)) => true // TODO: not all upcasts 
here can really give null
+case (NullType, _) => true
+case (_, _) if from == to => false
+
+case (StringType, BinaryType) => false
+case (StringType, _) => true
+
+case (FloatType | DoubleType, TimestampType) => true
+case (TimestampType, DateType) => false
+case (_, DateType) => true
+case (DateType, TimestampType) => false
+case (DateType, _) => true
--- End diff --

It's covered by (_, StringType)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12520] [PySpark] Correct Descriptions a...

2015-12-27 Thread davies

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/10477#issuecomment-167497951
  
LGTM, merging into master and 1.6 (for doc). I think it's fine to not fix 
1.5 branch, until some one hit it, because we may not have a new release for 
1.5 branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Add json property(ALLOW_BACKSLASH_ESCAPING_ANY...

2015-12-27 Thread Cazen

Github user Cazen closed the pull request at:

https://github.com/apache/spark/pull/10493


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Add json property(ALLOW_BACKSLASH_ESCAPING_ANY...

2015-12-27 Thread Cazen

GitHub user Cazen opened a pull request:

https://github.com/apache/spark/pull/10493

Add json property(ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Cazen/spark testCazen

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10493.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10493


commit 93a52b5916f12772f758efae368f87ff3730312e
Author: cazen.lee 
Date:   2015-12-28T07:42:50Z

Add json property(ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Add more exceptions to Guava relocation

2015-12-27 Thread microhello

Github user microhello commented on the pull request:

https://github.com/apache/spark/pull/10442#issuecomment-167493888
  
@srowen 
When I use  [facebook/swift](https://github.com/facebook/swift)  with spark 
1.5.2, throw following exception, so I need add 
`com.google.common.base.AbstractIterator` exceptions to Guava relocation . 
```java
Exception in thread "main" java.lang.IllegalAccessError: class 
com.google.common.base.Optional$1$1 cannot access its superclass 
org.spark-project.guava.base.AbstractIterator
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.google.common.base.Optional$1.iterator(Optional.java:223)
at com.google.common.collect.ImmutableSet.copyOf(ImmutableSet.java:300)
at 
com.facebook.swift.codec.metadata.AbstractThriftMetadataBuilder.inferThriftFieldIds(AbstractThriftMetadataBuilder.java:566)
at 
com.facebook.swift.codec.metadata.AbstractThriftMetadataBuilder.inferThriftFieldIds(AbstractThriftMetadataBuilder.java:542)
at 
com.facebook.swift.codec.metadata.AbstractThriftMetadataBuilder.normalizeThriftFields(AbstractThriftMetadataBuilder.java:483)
at 
com.facebook.swift.codec.metadata.ThriftStructMetadataBuilder.(ThriftStructMetadataBuilder.java:48)
at 
com.facebook.swift.codec.metadata.ThriftCatalog.extractThriftStructMetadata(ThriftCatalog.java:542)
at 
com.facebook.swift.codec.metadata.ThriftCatalog.getThriftStructMetadata(ThriftCatalog.java:408)
at 
com.facebook.swift.codec.metadata.ThriftCatalog.getThriftTypeUncached(ThriftCatalog.java:283)
at 
com.facebook.swift.codec.metadata.ThriftCatalog.getThriftType(ThriftCatalog.java:215)
at 
com.facebook.swift.service.metadata.ThriftMethodMetadata.(ThriftMethodMetadata.java:90)
at 
com.facebook.swift.service.metadata.ThriftServiceMetadata.(ThriftServiceMetadata.java:73)
at 
com.facebook.swift.service.ThriftServiceProcessor.(ThriftServiceProcessor.java:76)
at 
com.facebook.swift.service.ThriftServiceProcessor.(ThriftServiceProcessor.java:65)
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2759][CORE] Generic Binary File Support...

2015-12-27 Thread saeedehA

Github user saeedehA commented on the pull request:

https://github.com/apache/spark/pull/1658#issuecomment-167496362
  
Hello
I'm sorry that I'm beginner in pyspark, but need to get .bin file as byte 
array and I don't understand anything about what I must do, is it possible that 
say me what I must do, step by step(from what I must import to function that 
must call)!! Thanks alot
( Sorry, I can't find any thing that help me, except here)  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12520] [PySpark] Correct Descriptions a...

2015-12-27 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10477


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12520] [PySpark] [1.5] Ensure the join ...

2015-12-27 Thread gatorsmile

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/10484#issuecomment-167506415
  
Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12520] [PySpark] [1.5] Ensure the join ...

2015-12-27 Thread gatorsmile

Github user gatorsmile closed the pull request at:

https://github.com/apache/spark/pull/10484


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12520] [PySpark] Correct Descriptions a...

2015-12-27 Thread gatorsmile

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/10477#issuecomment-167506376
  
Thank you! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12520] [PySpark] [1.5] Ensure the join ...

2015-12-27 Thread davies

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/10484#issuecomment-167498565
  
LGTM, merging into 1.5 branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

1 2 >

1 - 100 of 104 matches

Mail list logo