[GitHub] spark issue #13775: [SPARK-16060][SQL] Vectorized Orc reader

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13775
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scal...

[yanboliang]

GitHub user yanboliang opened a pull request:

https://github.com/apache/spark/pull/16009

[SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs, docs

## What changes were proposed in this pull request?
API review for 2.1, except ```LSH``` related classes which are still under 
development.

## How was this patch tested?
Only doc changes, no new tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yanboliang/spark spark-18318

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16009.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16009


commit 8cfc859aebed82d5249011559ac864fd673a7dda
Author: Yanbo Liang 
Date:   2016-11-24T13:37:44Z

QA New Scala APIs, docs for ml/feature

commit ac72c0514e6fd39d3f0623cc12013474f8f47a11
Author: Yanbo Liang 
Date:   2016-11-25T06:03:07Z

QA New Scala APIs, docs for ml/classification

commit 8c369405e25e1889901da5848383ad0977922e0c
Author: Yanbo Liang 
Date:   2016-11-25T06:20:16Z

QA New Scala APIs, docs for ml/optim




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13775: [SPARK-16060][SQL] Vectorized Orc reader

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13775
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69147/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13775: [SPARK-16060][SQL] Vectorized Orc reader

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13775
  
**[Test build #69147 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69147/consoleFull)**
 for PR 13775 at commit 
[`160e924`](https://github.com/apache/spark/commit/160e92470136282ae3e94dc82ed41571a601017f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16008: [SPARK-18585][SQL] Use `ev.isNull = "false"` if possible...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16008
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16008: [SPARK-18585][SQL] Use `ev.isNull = "false"` if possible...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16008
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69146/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16008: [SPARK-18585][SQL] Use `ev.isNull = "false"` if possible...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16008
  
**[Test build #69146 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69146/consoleFull)**
 for PR 16008 at commit 
[`6992bd0`](https://github.com/apache/spark/commit/6992bd08c410f5560f68cf59c3a9b139cc4464e8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15977
  
**[Test build #69151 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69151/consoleFull)**
 for PR 15977 at commit 
[`99d4623`](https://github.com/apache/spark/commit/99d46234853f4b59f6930e69f3adc06fd88035c8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15977
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69150/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15977
  
**[Test build #69150 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69150/consoleFull)**
 for PR 15977 at commit 
[`05b5016`](https://github.com/apache/spark/commit/05b501652bcc1dac1d432d6a116767eeddc25af9).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15977
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15977
  
**[Test build #69150 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69150/consoleFull)**
 for PR 15977 at commit 
[`05b5016`](https://github.com/apache/spark/commit/05b501652bcc1dac1d432d6a116767eeddc25af9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13775: [SPARK-16060][SQL] Vectorized Orc reader

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13775
  
**[Test build #69149 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69149/consoleFull)**
 for PR 13775 at commit 
[`bd15842`](https://github.com/apache/spark/commit/bd15842e7b146cf292d9c29b896362412b22b8c1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13775: [SPARK-16060][SQL] Vectorized Orc reader

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13775
  
**[Test build #69148 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69148/consoleFull)**
 for PR 13775 at commit 
[`3014834`](https://github.com/apache/spark/commit/3014834391906264797b38d2e1b2e50b8e6d5327).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16003: [SPARK-18482][SQL] make sure Spark can access the table ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16003
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69145/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16003: [SPARK-18482][SQL] make sure Spark can access the table ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16003
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16003: [SPARK-18482][SQL] make sure Spark can access the table ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16003
  
**[Test build #69145 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69145/consoleFull)**
 for PR 16003 at commit 
[`6f785e3`](https://github.com/apache/spark/commit/6f785e3685ea80784ed7c38dbd29f5976d257f20).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class HiveExternalCatalogBackwardCompatibilitySuite extends QueryTest 
with TestHiveSingleton `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15992: [SPARK-18560][CORE][STREAMING] Receiver data can not be ...

[uncleGen]

Github user uncleGen commented on the issue:

https://github.com/apache/spark/pull/15992
  
@JoshRosen and @zsxwing waiting for your feedback.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14136
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69144/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14136
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14136
  
**[Test build #69144 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69144/consoleFull)**
 for PR 14136 at commit 
[`e01d0b2`](https://github.com/apache/spark/commit/e01d0b2ca5c7a0f7d693d48cb2dbefeed569fec0).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  class CountingsSerializer `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16001: [SPARK-18575][WEB] Keep same style: adjust the position ...

[uncleGen]

Github user uncleGen commented on the issue:

https://github.com/apache/spark/pull/16001
  
@srowen waiting for your feedback :D 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15955: [SPARK-18434][ML][FOLLOWUP] Add checking for setS...

[zhengruifeng]

Github user zhengruifeng closed the pull request at:

https://github.com/apache/spark/pull/15955


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15955: [SPARK-18434][ML][FOLLOWUP] Add checking for setSolver i...

[yanboliang]

Github user yanboliang commented on the issue:

https://github.com/apache/spark/pull/15955
  
After offline discussion with @zhengruifeng , we think the validation in 
the ```Param``` itself should be better, so other ways (which not calling 
setter method) of setting params can also benefit from the validation. We open 
[SPARK-18518](https://issues.apache.org/jira/browse/SPARK-18518) to a more 
perfect solution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15966
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15966
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69143/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15966
  
**[Test build #69143 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69143/consoleFull)**
 for PR 15966 at commit 
[`4a957e4`](https://github.com/apache/spark/commit/4a957e420a873e8dd976fc0aa2d59564c05d667b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13775: [SPARK-16060][SQL] Vectorized Orc reader

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13775
  
**[Test build #69147 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69147/consoleFull)**
 for PR 13775 at commit 
[`160e924`](https://github.com/apache/spark/commit/160e92470136282ae3e94dc82ed41571a601017f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16008: [SPARK-18585][SQL] Use `ev.isNull = "false"` if possible...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16008
  
**[Test build #69146 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69146/consoleFull)**
 for PR 16008 at commit 
[`6992bd0`](https://github.com/apache/spark/commit/6992bd08c410f5560f68cf59c3a9b139cc4464e8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16008: [SPARK-18585][SQL] Use `ev.isNull = "false"` if p...

[ueshin]

GitHub user ueshin opened a pull request:

https://github.com/apache/spark/pull/16008

[SPARK-18585][SQL] Use `ev.isNull = "false"` if possible for Janino to have 
a chance to optimize.

## What changes were proposed in this pull request?

Janino can optimize `true ? a : b` into `a` or `false ? a : b` into `b`, or 
if/else with literal condition, so we should use literal as `ev.isNull` if 
possible.

## How was this patch tested?

Existing tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ueshin/apache-spark issues/SPARK-18585

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16008.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16008


commit 6992bd08c410f5560f68cf59c3a9b139cc4464e8
Author: Takuya UESHIN 
Date:   2016-11-22T09:00:09Z

Use `isNull = "false"` if possible.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16007: [SPARK-18583][SQL] Fix nullability of InputFileName.

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16007
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16003: [SPARK-18482][SQL] make sure Spark can access the table ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16003
  
**[Test build #69145 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69145/consoleFull)**
 for PR 16003 at commit 
[`6f785e3`](https://github.com/apache/spark/commit/6f785e3685ea80784ed7c38dbd29f5976d257f20).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16007: [SPARK-18583][SQL] Fix nullability of InputFileName.

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16007
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69142/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16007: [SPARK-18583][SQL] Fix nullability of InputFileName.

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16007
  
**[Test build #69142 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69142/consoleFull)**
 for PR 16007 at commit 
[`4592c09`](https://github.com/apache/spark/commit/4592c09a6e3712474761db28471fb6e1a3a5148f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

[ChorPangChan]

Github user ChorPangChan commented on a diff in the pull request:

https://github.com/apache/spark/pull/16000#discussion_r89563670
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/status/api/v1/JacksonMessageWriter.scala
 ---
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.status.api.v1
+
+import java.io.OutputStream
+import java.lang.annotation.Annotation
+import java.lang.reflect.Type
+import java.nio.charset.StandardCharsets
+import java.text.SimpleDateFormat
+import java.util.{Calendar, Locale, SimpleTimeZone}
+import javax.ws.rs.Produces
+import javax.ws.rs.core.{MediaType, MultivaluedMap}
+import javax.ws.rs.ext.{MessageBodyWriter, Provider}
+
+import com.fasterxml.jackson.annotation.JsonInclude
+import com.fasterxml.jackson.databind.{ObjectMapper, SerializationFeature}
+
+/**
+ * This class converts the POJO metric responses into json, using jackson.
+ *
+ * This doesn't follow the standard jersey-jackson plugin options, because 
we want to stick
+ * with an old version of jersey (since we have it from yarn anyway) and 
don't want to pull in lots
+ * of dependencies from a new plugin.
+ *
+ * Note that jersey automatically discovers this class based on its 
package and its annotations.
+ */
+@Provider
+@Produces(Array(MediaType.APPLICATION_JSON))
+private[v1] class JacksonMessageWriter extends MessageBodyWriter[Object]{
+
--- End diff --

I agree in terms of functionality
but how about the design/structure

JacksonMessageWriter in spark-core is private to api.v1
Althouth we can use it though holder.setInitParameter
shoud we?
is there any sample of reusing the JacksonMessageWriter in other component?

I imaging someday in the future, sommeone will create a v2 api
and inside that v2 api, there will be another JacksonMessageWriter as well
because even they have the same function, I dont think they have the same 
meaning

in order to share the class of JacksonMessageWriter
we may need to move it outside v1 first
and i think that will be too much to consider for our current goal.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/15977
  
Thank you for comment@rxin@srowen@gatorsmile! I'll update that later today!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14136
  
**[Test build #69144 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69144/consoleFull)**
 for PR 14136 at commit 
[`e01d0b2`](https://github.com/apache/spark/commit/e01d0b2ca5c7a0f7d693d48cb2dbefeed569fec0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15966
  
**[Test build #69143 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69143/consoleFull)**
 for PR 15966 at commit 
[`4a957e4`](https://github.com/apache/spark/commit/4a957e420a873e8dd976fc0aa2d59564c05d667b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions`...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/15966#discussion_r89561319
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala
 ---
@@ -35,11 +35,12 @@ class JdbcRelationProvider extends 
CreatableRelationProvider
 val upperBound = jdbcOptions.upperBound
 val numPartitions = jdbcOptions.numPartitions
 
-val partitionInfo = if (partitionColumn == null) {
+val partitionInfo = if (partitionColumn.isEmpty || lowerBound.isEmpty 
|| upperBound.isEmpty ||
--- End diff --

I'll use the following.
```scala
val partitionInfo = if (partitionColumn.isEmpty) {
  assert(lowerBound.isEmpty && upperBound.isEmpty)
  null
} else {
  assert(lowerBound.nonEmpty && upperBound.nonEmpty && 
numPartitions.nonEmpty)
  JDBCPartitioningInfo(
partitionColumn.get, lowerBound.get, upperBound.get, 
numPartitions.get)
}
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions`...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/15966#discussion_r89560765
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala
 ---
@@ -35,11 +35,12 @@ class JdbcRelationProvider extends 
CreatableRelationProvider
 val upperBound = jdbcOptions.upperBound
 val numPartitions = jdbcOptions.numPartitions
 
-val partitionInfo = if (partitionColumn == null) {
+val partitionInfo = if (partitionColumn.isEmpty || lowerBound.isEmpty 
|| upperBound.isEmpty ||
--- End diff --

Ur, @rxin .
We are using `numPartition` for both writing and reading, and 
`numPartition` can be used alone.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions`...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/15966#discussion_r89560594
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala
 ---
@@ -35,11 +35,12 @@ class JdbcRelationProvider extends 
CreatableRelationProvider
 val upperBound = jdbcOptions.upperBound
 val numPartitions = jdbcOptions.numPartitions
 
-val partitionInfo = if (partitionColumn == null) {
+val partitionInfo = if (partitionColumn.isEmpty || lowerBound.isEmpty 
|| upperBound.isEmpty ||
--- End diff --

I assumed the following.
```
if (partitionColumn.isEmpty) {
  assert(lowerBound.isEmpty && upperBound.isEmpty && numPartitions.isEmpty)
  null
} else {
  ...
}
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions`...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/15966#discussion_r89560507
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala
 ---
@@ -35,11 +35,12 @@ class JdbcRelationProvider extends 
CreatableRelationProvider
 val upperBound = jdbcOptions.upperBound
 val numPartitions = jdbcOptions.numPartitions
 
-val partitionInfo = if (partitionColumn == null) {
+val partitionInfo = if (partitionColumn.isEmpty || lowerBound.isEmpty 
|| upperBound.isEmpty ||
--- End diff --

Thanks. I'll update soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16007: [SPARK-18583][SQL] Fix nullability of InputFileName.

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16007
  
**[Test build #69142 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69142/consoleFull)**
 for PR 16007 at commit 
[`4592c09`](https://github.com/apache/spark/commit/4592c09a6e3712474761db28471fb6e1a3a5148f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16007: [SPARK-18583][SQL] Fix nullability of InputFileNa...

[ueshin]

GitHub user ueshin opened a pull request:

https://github.com/apache/spark/pull/16007

[SPARK-18583][SQL] Fix nullability of InputFileName.

## What changes were proposed in this pull request?

The nullability of `InputFileName` should be `false`.

## How was this patch tested?

Existing tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ueshin/apache-spark issues/SPARK-18583

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16007.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16007


commit 4592c09a6e3712474761db28471fb6e1a3a5148f
Author: Takuya UESHIN 
Date:   2016-11-24T07:58:24Z

Fix nullability of `InputFileName`.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15955: [SPARK-18434][ML][FOLLOWUP] Add checking for setSolver i...

[zhengruifeng]

Github user zhengruifeng commented on the issue:

https://github.com/apache/spark/pull/15955
  
After offline discussion with @yanboliang , I will update this PR to make 
solvers more private and include MLPC in this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16004: [STS][SQL]multiple STS running in the same machin...

[tanxinz]

Github user tanxinz closed the pull request at:

https://github.com/apache/spark/pull/16004


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15955: [SPARK-18434][ML][FOLLOWUP] Add checking for setSolver i...

[zhengruifeng]

Github user zhengruifeng commented on the issue:

https://github.com/apache/spark/pull/15955
  
@yanboliang I add the `private[regression]` limitation refering to 
`MultilayerPerceptronClassifier` implementation: 
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala#L272


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16000: [SPARK-18537][Web UI]Add a REST api to spark stre...

[ChorPangChan]

Github user ChorPangChan commented on a diff in the pull request:

https://github.com/apache/spark/pull/16000#discussion_r89558292
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/status/api/v1/JacksonMessageWriter.scala
 ---
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.streaming.status.api.v1
+
+import java.io.OutputStream
+import java.lang.annotation.Annotation
+import java.lang.reflect.Type
+import java.nio.charset.StandardCharsets
+import java.text.SimpleDateFormat
+import java.util.{Calendar, Locale, SimpleTimeZone}
+import javax.ws.rs.Produces
+import javax.ws.rs.core.{MediaType, MultivaluedMap}
+import javax.ws.rs.ext.{MessageBodyWriter, Provider}
+
+import com.fasterxml.jackson.annotation.JsonInclude
+import com.fasterxml.jackson.databind.{ObjectMapper, SerializationFeature}
+
+/**
+ * This class converts the POJO metric responses into json, using jackson.
+ *
+ * This doesn't follow the standard jersey-jackson plugin options, because 
we want to stick
+ * with an old version of jersey (since we have it from yarn anyway) and 
don't want to pull in lots
+ * of dependencies from a new plugin.
+ *
+ * Note that jersey automatically discovers this class based on its 
package and its annotations.
+ */
+@Provider
+@Produces(Array(MediaType.APPLICATION_JSON))
+private[v1] class JacksonMessageWriter extends MessageBodyWriter[Object]{
+
--- End diff --

actually im not sure about this.
can you explain a little bit more?

there is a posibility that someone can change JacksonMessageWriter in 
spark-core
without even notice that streaming is also using that class

in the view of providing an api, im not sure if that would be a good idea.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits

[witgo]

Github user witgo commented on the issue:

https://github.com/apache/spark/pull/14995
  
Thanks, I am glad to hear this, and I want to solve the issues of reading, 
storing and transmitting data as much as possible.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15916: [SPARK-18487][SQL] Add completion listener to HashAggreg...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/15916
  
Hello @rxin @hvanhovell Can you let me know if this is needed or not? So I 
can close it. Thank you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15970: [SPARK-18134][SQL] Comparable MapTypes [POC]

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15970
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15970: [SPARK-18134][SQL] Comparable MapTypes [POC]

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15970
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69141/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15970: [SPARK-18134][SQL] Comparable MapTypes [POC]

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15970
  
**[Test build #69141 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69141/consoleFull)**
 for PR 15970 at commit 
[`7d847b8`](https://github.com/apache/spark/commit/7d847b810a14f7783a42cc501dfb01a57769687f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15961: [SPARK-18523][PySpark]Make SparkContext.stop more reliab...

[kxepal]

Github user kxepal commented on the issue:

https://github.com/apache/spark/pull/15961
  
@holdenk @srowen 
Added a warning message, please take a second look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions`...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/15966#discussion_r89554371
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala
 ---
@@ -35,11 +35,12 @@ class JdbcRelationProvider extends 
CreatableRelationProvider
 val upperBound = jdbcOptions.upperBound
 val numPartitions = jdbcOptions.numPartitions
 
-val partitionInfo = if (partitionColumn == null) {
+val partitionInfo = if (partitionColumn.isEmpty || lowerBound.isEmpty 
|| upperBound.isEmpty ||
--- End diff --

I'd change this to 

```
if (partitionColumn == null) {
  assert(lowerBound.isEmpty && upperBound.isEmpty && numPartitions.isEmpty)
  null
} else {
  ...
}
```

to be future proof.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15966: [SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instea...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15966
  
LGTM other than that one change.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15970: [SPARK-18134][SQL] Comparable MapTypes [POC]

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15970
  
**[Test build #69141 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69141/consoleFull)**
 for PR 15970 at commit 
[`7d847b8`](https://github.com/apache/spark/commit/7d847b810a14f7783a42cc501dfb01a57769687f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15462: [SPARK-17680] [SQL] [TEST] Added test cases for InMemory...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/15462
  
@andrewor14 thank you for your review. Could you please see it again?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE an...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/15358#discussion_r89547862
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ddl.scala ---
@@ -56,6 +56,14 @@ case class CreateTempViewUsing(
   s"Temporary view '$tableIdent' should not have specified a database")
   }
 
+  override def argString: String = {
+s"[tableIdent:$tableIdent " +
+  userSpecifiedSchema.getOrElse("") +
--- End diff --

Does this need a space?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE an...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/15358#discussion_r89547695
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -766,4 +766,5 @@ object DDLUtils {
   }
 }
   }
+
--- End diff --

Nit: ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE an...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/15358#discussion_r89547692
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
 ---
@@ -52,12 +52,10 @@ case class CatalogStorageFormat(
 properties: Map[String, String]) {
 
   override def toString: String = {
-val serdePropsToString =
-  if (properties.nonEmpty) {
-s"Properties: " + properties.map(p => p._1 + "=" + 
p._2).mkString("[", ", ", "]")
-  } else {
-""
-  }
+val serdePropsToString = CatalogUtils.maskCredentials(properties) 
match {
+  case props if props.isEmpty => ""
+  case props => s"Properties: " + props.map(p => p._1 + "=" + 
p._2).mkString("[", ", ", "]")
--- End diff --

NIT: string interpolation not needed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE and DESC ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15358
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE and DESC ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15358
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69140/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE and DESC ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15358
  
**[Test build #69140 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69140/consoleFull)**
 for PR 15358 at commit 
[`bc9a508`](https://github.com/apache/spark/commit/bc9a5082cf4576cfa7a6d4911db74bc0476c4360).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16004: [STS][SQL]multiple STS running in the same machine throw...

[hvanhovell]

Github user hvanhovell commented on the issue:

https://github.com/apache/spark/pull/16004
  
Hmmm. Seems that you have opened against the wrong branch. Could you close, 
open against the correct branch, and add the JIRA ticket and the Spark 
component to the title (look at every other PR).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16001: [SPARK-18575][WEB] Keep same style: adjust the position ...

[ajbozarth]

Github user ajbozarth commented on the issue:

https://github.com/apache/spark/pull/16001
  
Not sure this is worth a whole commit, but it is probably good to fix. LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching Using ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15975
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching Using ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15975
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69139/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching Using ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15975
  
**[Test build #69139 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69139/consoleFull)**
 for PR 15975 at commit 
[`5c5b3ca`](https://github.com/apache/spark/commit/5c5b3cab4ec77484467e960319abbddb49313952).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15462: [SPARK-17680] [SQL] [TEST] Added test cases for InMemory...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15462
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69138/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15462: [SPARK-17680] [SQL] [TEST] Added test cases for InMemory...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15462
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15462: [SPARK-17680] [SQL] [TEST] Added test cases for InMemory...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15462
  
**[Test build #69138 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69138/consoleFull)**
 for PR 15462 at commit 
[`42eade8`](https://github.com/apache/spark/commit/42eade8b13a4a8eef10f424a8f58d7096fea89c4).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16005: [SPARK-18578][SQL] Full outer join in correlated ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16005


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16005: [SPARK-18578][SQL] Full outer join in correlated subquer...

[hvanhovell]

Github user hvanhovell commented on the issue:

https://github.com/apache/spark/pull/16005
  
Hmmm. I cannot merge to 2.0 :(... Can you open a backport against 2.0?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16005: [SPARK-18578][SQL] Full outer join in correlated subquer...

[hvanhovell]

Github user hvanhovell commented on the issue:

https://github.com/apache/spark/pull/16005
  
LGTM. Merging to master/2.1/2.0. Thanks! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15977
  
Also use NonFoldableLiteral to do the test. Make sure you create a null 
nonfoldable literal to verify the result is null.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15977
  
How about adding the following two lines into the test case in 
[`PredicateSuite.scala`](https://github.com/apache/spark/blob/84284e8c82542d80dad94e458a0c0210bf803db3/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/PredicateSuite.scala#L122)?
 
```Scala
checkEvaluation(In(Literal.create(null, IntegerType), Seq.empty), null)
checkEvaluation(In(Literal(1), Seq.empty), false)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15977: [SPARK-18436][SQL] isin causing SQL syntax error ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/15977#discussion_r89540400
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
 ---
@@ -120,7 +120,9 @@ object JDBCRDD extends Logging {
   case StringStartsWith(attr, value) => s"${attr} LIKE '${value}%'"
   case StringEndsWith(attr, value) => s"${attr} LIKE '%${value}'"
   case StringContains(attr, value) => s"${attr} LIKE '%${value}%'"
-  case In(attr, value) => s"$attr IN (${compileValue(value)})"
+  case In(null, value) => "NULL"
--- End diff --

Yeah. `NULL` in predicates means `UNKNOWN`. 

FYI, 
- our source code for the `IN` operator: 
https://github.com/apache/spark/blob/84284e8c82542d80dad94e458a0c0210bf803db3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala#L141-L142

Oracle Docs of `NULL`: 
https://docs.oracle.com/cd/B19306_01/server.102/b14200/sql_elements005.htm


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods f...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/15913
  
One more thing: This breaks several public APIs, although in mostly minor 
ways and with good reasons.  Could you please update the JIRA to list the 
public API changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated me...

[jkbradley]

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/15913#discussion_r89537340
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/param/params.scala ---
@@ -547,21 +547,6 @@ trait Params extends Identifiable with Serializable {
   }
 
   /**
-   * Validates parameter values stored internally.
-   * Raise an exception if any parameter value is invalid.
-   *
-   * This only needs to check for interactions between parameters.
--- End diff --

Could you please add the relevant text from here to transformSchema?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated me...

[jkbradley]

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/15913#discussion_r89537785
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala ---
@@ -443,12 +429,11 @@ private[ml] trait GBTParams extends 
TreeEnsembleParams with HasMaxIter with HasS
* (default = 0.1)
* @group setParam
*/
-  def setStepSize(value: Double): this.type = set(stepSize, value)
-
-  override def validateParams(): Unit = {
+  def setStepSize(value: Double): this.type = {
 require(ParamValidators.inRange(0, 1, lowerInclusive = false, 
upperInclusive = true)(
-  getStepSize), "GBT parameter stepSize should be in interval (0, 1], 
" +
-  s"but it given invalid value $getStepSize.")
+  value), "GBT parameter stepSize should be in interval (0, 1], " +
+  s"but it given invalid value $value.")
--- End diff --

"it given invalid value" -> "it was given an invalid value"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated me...

[jkbradley]

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/15913#discussion_r89537864
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala ---
@@ -317,8 +317,32 @@ private[ml] trait TreeEnsembleParams extends 
DecisionTreeParams {
   }
 }
 
-/** Used for [[RandomForestParams]] */
-private[ml] trait HasFeatureSubsetStrategy extends Params {
+/**
+ * Parameters for Random Forest algorithms.
+ */
+private[ml] trait RandomForestParams extends TreeEnsembleParams {
+
+  /**
+   * Number of trees to train (>= 1).
+   * If 1, then no bootstrapping is used.  If > 1, then bootstrapping is 
done.
+   * TODO: Change to always do bootstrapping (simpler).  SPARK-7130
+   * (default = 20)
+   *
+   * Note: The reason that we cannot add this to both GBT and RF (i.e. in 
TreeEnsembleParams)
+   * is the param `maxIter` controls how many trees a GBT has. The 
semantics in the algorithms
+   * are a bit different.
+   * @group param
+   */
+  final val numTrees: IntParam = new IntParam(this, "numTrees", "Number of 
trees to train (>= 1)",
+ParamValidators.gtEq(1))
+
+  setDefault(numTrees -> 20)
+
+  /** @group setParam */
+  def setNumTrees(value: Int): this.type = set(numTrees, value)
--- End diff --

Are these setter methods in traits Java-compatible?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15913: [SPARK-18481][ML] ML 2.1 QA: Remove deprecated me...

[jkbradley]

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/15913#discussion_r89537771
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala ---
@@ -443,12 +425,11 @@ private[ml] trait GBTParams extends 
TreeEnsembleParams with HasMaxIter with HasS
* (default = 0.1)
* @group setParam
*/
-  def setStepSize(value: Double): this.type = set(stepSize, value)
-
-  override def validateParams(): Unit = {
+  def setStepSize(value: Double): this.type = {
 require(ParamValidators.inRange(0, 1, lowerInclusive = false, 
upperInclusive = true)(
--- End diff --

This validation should really be in the Param itself.  It's not since it's 
a sharedParam, but perhaps we should just copy the shared Param code here.  The 
problem with putting validation in the setter is that Params can get set in 
other ways too (such as ```gbt.set(stepSize, value)```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16005: [SPARK-18578][SQL] Full outer join in correlated subquer...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16005
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16005: [SPARK-18578][SQL] Full outer join in correlated subquer...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16005
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69137/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16005: [SPARK-18578][SQL] Full outer join in correlated subquer...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16005
  
**[Test build #69137 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69137/consoleFull)**
 for PR 16005 at commit 
[`5d6cc8e`](https://github.com/apache/spark/commit/5d6cc8ea670e3f35508c8be5d84f5a221ef45c8b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15977: [SPARK-18436][SQL] isin causing SQL syntax error with JD...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15977
  
I believe the correct replacement for "x in ()" is "if (isnull(x)) null 
else false"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE and DESC ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15358
  
**[Test build #69140 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69140/consoleFull)**
 for PR 15358 at commit 
[`bc9a508`](https://github.com/apache/spark/commit/bc9a5082cf4576cfa7a6d4911db74bc0476c4360).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching Using ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15975
  
**[Test build #69139 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69139/consoleFull)**
 for PR 15975 at commit 
[`5c5b3ca`](https://github.com/apache/spark/commit/5c5b3cab4ec77484467e960319abbddb49313952).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching Using ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15975
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15717: [SPARK-17910][SQL] Allow users to update the comment of ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15717
  
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15358: [SPARK-17783] [SQL] Hide Credentials in CREATE and DESC ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15358
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16005: [SPARK-18578][SQL] Full outer join in correlated subquer...

[nsyca]

Github user nsyca commented on the issue:

https://github.com/apache/spark/pull/16005
  
@hvanhovell I will work on the whitelist in a new JIRA under SPARK-18455. 
It will be my top priority task and I hope we can merge it in the next minor 
release of 2.0.x. Let's have this PR scoped for the FOJ and Window cases. Shall 
we?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits

[opme]

Github user opme commented on the issue:

https://github.com/apache/spark/pull/14995
  
@witgo  I have a Pyspark application that was failing in 3 different places 
but is able to run without errors now.  I'm glad for this patch as I am not 
sure how I would have explained to my professors why the big data application I 
chose to do my analysis has 32 bit limitations.  This is my final project for a 
Georgia Tech Big data class and I will write about the these limitations of 
Spark in my paper.   My app is called the Surgeon Scorecard and it computes 
surgical complication rate for surgeons on the Medicare synthetic cms dataset 
which is about 1.6 billion records.  https://github.com/opme/SurgeonScorecard.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16005: [SPARK-18578][SQL] Full outer join in correlated subquer...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16005
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69135/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16005: [SPARK-18578][SQL] Full outer join in correlated subquer...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16005
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16005: [SPARK-18578][SQL] Full outer join in correlated subquer...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16005
  
**[Test build #69135 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69135/consoleFull)**
 for PR 16005 at commit 
[`099f75d`](https://github.com/apache/spark/commit/099f75d6b8f21434717e38442519ff67663e52e2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15462: [SPARK-17680] [SQL] [TEST] Added test cases for InMemory...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15462
  
**[Test build #69138 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69138/consoleFull)**
 for PR 15462 at commit 
[`42eade8`](https://github.com/apache/spark/commit/42eade8b13a4a8eef10f424a8f58d7096fea89c4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15462: [SPARK-17680] [SQL] [TEST] Added test cases for I...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/15462#discussion_r89535856
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/InMemoryColumnarQuerySuite.scala
 ---
@@ -20,18 +20,83 @@ package org.apache.spark.sql.execution.columnar
 import java.nio.charset.StandardCharsets
 import java.sql.{Date, Timestamp}
 
-import org.apache.spark.sql.{QueryTest, Row}
+import org.apache.spark.sql.{DataFrame, QueryTest, Row}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSQLContext
 import org.apache.spark.sql.test.SQLTestData._
 import org.apache.spark.sql.types._
-import org.apache.spark.storage.StorageLevel.MEMORY_ONLY
+import org.apache.spark.storage.StorageLevel._
 
 class InMemoryColumnarQuerySuite extends QueryTest with SharedSQLContext {
   import testImplicits._
 
   setupTestData()
 
+  def cachePrimitiveTest(data: DataFrame, dataType: String) {
+data.createOrReplaceTempView(s"testData$dataType")
+val storageLevel = MEMORY_ONLY
+val plan = spark.sessionState.executePlan(data.logicalPlan).sparkPlan
+val inMemoryRelation = InMemoryRelation(useCompression = true, 5, 
storageLevel, plan, None)
+
+assert(inMemoryRelation.cachedColumnBuffers.getStorageLevel == 
storageLevel)
+inMemoryRelation.cachedColumnBuffers.collect().head match {
+  case _: CachedBatch => assert(true)
+  case other => fail(s"Unexpected cached batch type: 
${other.getClass.getName}")
+}
+checkAnswer(inMemoryRelation, data.collect().toSeq)
+  }
+
+  test("all data type w && w/o nullability") {
+// all primitives
+Seq(true, false).map { nullability =>
--- End diff --

OK. I created smaller multiple tests


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15462: [SPARK-17680] [SQL] [TEST] Added test cases for I...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/15462#discussion_r89535786
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/InMemoryColumnarQuerySuite.scala
 ---
@@ -20,18 +20,83 @@ package org.apache.spark.sql.execution.columnar
 import java.nio.charset.StandardCharsets
 import java.sql.{Date, Timestamp}
 
-import org.apache.spark.sql.{QueryTest, Row}
+import org.apache.spark.sql.{DataFrame, QueryTest, Row}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSQLContext
 import org.apache.spark.sql.test.SQLTestData._
 import org.apache.spark.sql.types._
-import org.apache.spark.storage.StorageLevel.MEMORY_ONLY
+import org.apache.spark.storage.StorageLevel._
 
 class InMemoryColumnarQuerySuite extends QueryTest with SharedSQLContext {
   import testImplicits._
 
   setupTestData()
 
+  def cachePrimitiveTest(data: DataFrame, dataType: String) {
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org