[GitHub] spark issue #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partiti...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18769
  
**[Test build #80050 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80050/testReport)**
 for PR 18769 at commit 
[`cee6838`](https://github.com/apache/spark/commit/cee6838e6f188c1d751ef6d3bc2a191057f45e34).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18775: [SPARK-8577][SparkR] Eliminate needless synchronization ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18775
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18775: [SPARK-8577][SparkR] Eliminate needless synchroni...

[SereneAnt]

GitHub user SereneAnt opened a pull request:

https://github.com/apache/spark/pull/18775

[SPARK-8577][SparkR] Eliminate needless synchronization in java-R 
serialization

## What changes were proposed in this pull request?
Remove surplus synchronized blocks.

## How was this patch tested?
Unit tests run OK.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SereneAnt/spark 
eliminate_unnecessary_synchronization_in_java-R_serialization

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18775.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18775


commit eafad7d0cdf73c9fab9a03f2898039de7b127bc7
Author: iurii.ant 
Date:   2017-07-30T05:04:06Z

[SPARK-8577][SparkR] Eliminate needless synchronization in java-R 
serialization




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18772
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80049/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18772
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18772
  
**[Test build #80049 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80049/testReport)**
 for PR 18772 at commit 
[`6c93f85`](https://github.com/apache/spark/commit/6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18745: [SPARK-21544][DEPLOY] Tests jar of some module should no...

[caneGuy]

Github user caneGuy commented on the issue:

https://github.com/apache/spark/pull/18745
  
cc 
@vanzin @srowen thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18772
  
**[Test build #80049 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80049/testReport)**
 for PR 18772 at commit 
[`6c93f85`](https://github.com/apache/spark/commit/6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/18772
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18774: [MINOR][DOC]Replace numTasks with numPartitions in progr...

[remisharoon]

Github user remisharoon commented on the issue:

https://github.com/apache/spark/pull/18774
  
@polarke : 
please refer to this Pull Request [ 
https://github.com/apache/spark/pull/18770 ] for best practices


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18774: [MINOR][DOC]Replace numTasks with numPartitions in progr...

[remisharoon]

Github user remisharoon commented on the issue:

https://github.com/apache/spark/pull/18774
  
@polarke , can you please provide the details of how you tested this?
please provide screenshots of the locally tested doc site


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17180
  
LGTM Wait for @JoshRosen for final sign off.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17849: [SPARK-10931][ML][PYSPARK] PySpark Models Copy Param Val...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/17849
  
This looks pretty reasonable, sorry for the delay. If you have a chance to 
update this to master would be good to do.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/18761
  
@gatorsmile OK. Usually the constructor expression parameters should also 
be in children. `RuntimeReplaceable` is a special case. I'd check if there are 
other expressions like it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17180
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80048/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17180
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17180
  
**[Test build #80048 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80048/testReport)**
 for PR 17180 at commit 
[`2403c30`](https://github.com/apache/spark/commit/2403c30fc904bb6f4d1a1f8c79a9c513fc21e9f8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18772
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18772
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80047/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18772
  
**[Test build #80047 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80047/testReport)**
 for PR 18772 at commit 
[`6c93f85`](https://github.com/apache/spark/commit/6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18774: [MINOR][DOC]Replace numTasks with numPartitions in progr...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18774
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18774: [MINOR][DOC]Replace numTasks with numPartitions i...

[polarke]

GitHub user polarke opened a pull request:

https://github.com/apache/spark/pull/18774

[MINOR][DOC]Replace numTasks with numPartitions in programming guide

In programming guide, `numTasks` is used in several places as arguments of 
Transformations. However, in code, `numPartitions` is used. In this fix, I 
replace `numTasks` with `numPartitions` in programming guide for consistency.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/polarke/spark 
replace-numtasks-with-numpartitions-in-doc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18774.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18774


commit ec11300f4a2c38fdb7aca3774096a46076b58caf
Author: Cheng Wang 
Date:   2017-07-29T20:13:48Z

Replace numTasks with numPartitions in programming guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18773: [MINOR][DOC] Replace numTasks with numPartitions ...

[polarke]

Github user polarke closed the pull request at:

https://github.com/apache/spark/pull/18773


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18771: [SPARK-21478][SQL] Avoid unpersisting related Datasets

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18771
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18771: [SPARK-21478][SQL] Avoid unpersisting related Datasets

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18771
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80046/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18771: [SPARK-21478][SQL] Avoid unpersisting related Datasets

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18771
  
**[Test build #80046 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80046/testReport)**
 for PR 18771 at commit 
[`857b3dd`](https://github.com/apache/spark/commit/857b3dd5804355331d3bef4eb8136e6604232758).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18773: [MINOR][DOC] Replace numTasks with numPartitions ...

[polarke]

GitHub user polarke opened a pull request:

https://github.com/apache/spark/pull/18773

[MINOR][DOC] Replace numTasks with numPartitions in programming guide

In programming guide, `numTasks` is used in several places as arguments of 
Transformations. However, in code, `numPartitions` is used. In this fix, I 
replace `numTasks` with `numPartitions` in programming guide for consistency.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/polarke/spark 
rename-numtasks-to-numpartitions-in-doc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18773.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18773


commit 8a9f3da9aa20a5b3b61d22f74962510d65db9ce0
Author: Cheng Wang 
Date:   2017-07-29T19:43:58Z

Rename numTasks to numPartitions in programming guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18773: [MINOR][DOC] Replace numTasks with numPartitions in prog...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18773
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17180
  
**[Test build #80048 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80048/testReport)**
 for PR 17180 at commit 
[`2403c30`](https://github.com/apache/spark/commit/2403c30fc904bb6f4d1a1f8c79a9c513fc21e9f8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17180
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/18772
  
In `visitTableName` in `AstBuilder`, we already have a different analyzer 
path to resolve column aliases 
(https://github.com/maropu/spark/blob/6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala#L685).
 We could also use `UnresolvedSubqueryColumnAliases` along with 
`visitAliasedQuery` and `visitAliasedRelation` though, I'm not sure it is worth 
fixing so. So, could you give me insight on this, too? Thanks.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18771: [SPARK-21478][SQL] Avoid unpersisting related Dat...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18771#discussion_r130226083
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala ---
@@ -114,13 +114,13 @@ class CacheManager extends Logging {
   }
 
   /**
-   * Un-cache all the cache entries that refer to the given plan.
+   * Un-cache the cache entry that refers to the given plan.
*/
   def uncacheQuery(spark: SparkSession, plan: LogicalPlan, blocking: 
Boolean): Unit = writeLock {
 val it = cachedData.iterator()
 while (it.hasNext) {
   val cd = it.next()
-  if (cd.plan.find(_.sameResult(plan)).isDefined) {
+  if (plan.sameResult(cd.plan)) {
--- End diff --

This is by design. This is for avoiding to get the incorrect results. See 
the original PR: https://github.com/apache/spark/pull/17097


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18772
  
**[Test build #80047 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80047/testReport)**
 for PR 18772 at commit 
[`6c93f85`](https://github.com/apache/spark/commit/6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18749
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18749
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80044/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18749
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80043/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18749
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18749
  
**[Test build #80044 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80044/testReport)**
 for PR 18749 at commit 
[`f51ee74`](https://github.com/apache/spark/commit/f51ee74c77a476113cec164d3cc6bce9997e3519).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18749
  
**[Test build #80043 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80043/testReport)**
 for PR 18749 at commit 
[`bf48875`](https://github.com/apache/spark/commit/bf48875057ca5978b88cacab455292b64daf1557).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18772: [SPARK-20963][SQL] Support column aliases for joi...

[maropu]

GitHub user maropu opened a pull request:

https://github.com/apache/spark/pull/18772

[SPARK-20963][SQL] Support column aliases for join relations in FROM clause

## What changes were proposed in this pull request?
This pr added parsing rules to support column aliases for join relations in 
FROM clause.
This pr is a sub-task of #18079.

## How was this patch tested?
Added tests in `AnalysisSuite`, `PlanParserSuite,` and `SQLQueryTestSuite`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maropu/spark SPARK-20963-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18772.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18772


commit 6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84
Author: Takeshi Yamamuro 
Date:   2017-07-24T09:08:11Z

Support aliases for join relations




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18768: [SPARK-21533][SQL] Print warning messages when ov...

[maropu]

Github user maropu commented on a diff in the pull request:

https://github.com/apache/spark/pull/18768#discussion_r130225918
  
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala 
---
@@ -111,6 +112,31 @@ private[hive] object HiveShim {
 }
   }
 
+  private def isSubClassOf(t: Type, parent: Class[_]): Boolean = t match {
+case cls: Class[_] => parent.isAssignableFrom(cls)
+case _ => false
+  }
+
+  private def hasInheritanceOf[UDFType: ClassTag](funcName: String, clazz: 
Class[_]): Boolean = {
+val clsTag = classTag[UDFType].runtimeClass
+if (isSubClassOf(clazz, clsTag)) {
+  val funcClass = clazz.getMethod(funcName, classOf[MapredContext])
+  funcClass.getDeclaringClass != clsTag
+} else {
+  false
+}
+  }
+
+  def validateHiveUserDefinedFunction(udfClass: Class[_]): Unit = {
+if (hasInheritanceOf[GenericUDF]("configure", udfClass) ||
--- End diff --

I checked it seemed this API had been implemented in v0.11.0 
(https://github.com/apache/hive/blob/release-0.11.0/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java).
 This is [the 
commit](https://github.com/apache/hive/commit/a4705e51809e0f6c24cdc417c91600627cbe3482#diff-3111a4cd48c72a32f7713451d7b9ca1aR98)
 for this API.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit problem i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18641
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit problem i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18641
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80045/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit problem i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18641
  
**[Test build #80045 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80045/testReport)**
 for PR 18641 at commit 
[`e5431a0`](https://github.com/apache/spark/commit/e5431a02fda88c442146eb2297a04c2e36da65ad).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18771: [SPARK-21478][SQL] Avoid unpersisting related Datasets

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18771
  
**[Test build #80046 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80046/testReport)**
 for PR 18771 at commit 
[`857b3dd`](https://github.com/apache/spark/commit/857b3dd5804355331d3bef4eb8136e6604232758).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18771: [SPARK-21478][SQL] Avoid unpersisting related Dat...

[vinodkc]

GitHub user vinodkc opened a pull request:

https://github.com/apache/spark/pull/18771

[SPARK-21478][SQL]  Avoid unpersisting related Datasets

## What changes were proposed in this pull request?
While unpersisting a dataset, only unpersist and remove that datasets's 
plan from Cachemanager's cachedData.
## How was this patch tested?
 Added unit tests




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vinodkc/spark br_SPARK-21478

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18771.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18771


commit e187aeb67a13493c6f5a9e540779a677d3502b04
Author: vinodkc 
Date:   2017-07-29T15:10:24Z

Fixed unpersisting related DFs

commit 857b3dd5804355331d3bef4eb8136e6604232758
Author: vinodkc 
Date:   2017-07-29T18:01:01Z

Updated test cases and condition for unpersist




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18645: [SPARK-14280][BUILD][WIP] Update change-version.sh and p...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/18645
  
Oh nice, it will help if that's not a required update along with everything 
else here as it requires a Jackson update. The Jackson update may be important 
eventually but nice if it can be separate.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18645: [SPARK-14280][BUILD][WIP] Update change-version.sh and p...

[xuwei-k]

Github user xuwei-k commented on the issue:

https://github.com/apache/spark/pull/18645
  
json4s 3.2.11 for Scala 2.12 released

- 
https://oss.sonatype.org/content/repositories/releases/org/json4s/json4s-jackson_2.12/3.2.11
- https://github.com/json4s/json4s/releases/tag/v3.2.11_2.12


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18185: [SPARK-20962][SQL] Support subquery column aliase...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18185


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18185: [SPARK-20962][SQL] Support subquery column aliases in FR...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18185
  
LGTM.

Thanks! Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16818: [SPARK-19451][SQL][Core] Underlying integer overf...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16818


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18540


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18540
  
Thanks! Merging to master/2.2


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18540
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18540#discussion_r130224334
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ---
@@ -1179,32 +1179,26 @@ class AstBuilder(conf: SQLConf) extends 
SqlBaseBaseVisitor[AnyRef] with Logging
   }
 
   /**
-   * Create or resolve a [[FrameBoundary]]. Simple math expressions are 
allowed for Value
-   * Preceding/Following boundaries. These expressions must be constant 
(foldable) and return an
-   * integer value.
+   * Create or resolve a frame boundary expressions.
*/
-  override def visitFrameBound(ctx: FrameBoundContext): FrameBoundary = 
withOrigin(ctx) {
-// We currently only allow foldable integers.
-def value: Int = {
+  override def visitFrameBound(ctx: FrameBoundContext): Expression = 
withOrigin(ctx) {
+def value: Expression = {
   val e = expression(ctx.expression)
-  validate(e.resolved && e.foldable && e.dataType == IntegerType,
-"Frame bound value must be a constant integer.",
-ctx)
-  e.eval().asInstanceOf[Int]
+  validate(e.resolved && e.foldable, "Frame bound value must be a 
literal.", ctx)
+  e
 }
 
-// Create the FrameBoundary
 ctx.boundType.getType match {
   case SqlBaseParser.PRECEDING if ctx.UNBOUNDED != null =>
 UnboundedPreceding
   case SqlBaseParser.PRECEDING =>
-ValuePreceding(value)
+UnaryMinus(value)
   case SqlBaseParser.CURRENT =>
 CurrentRow
   case SqlBaseParser.FOLLOWING if ctx.UNBOUNDED != null =>
 UnboundedFollowing
   case SqlBaseParser.FOLLOWING =>
-ValueFollowing(value)
+value
--- End diff --

It sounds like we already allowed it in the previous release. Thus, we need 
to follow what we have now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18761: [SPARK-21555][SQL] RuntimeReplaceable should be c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18761


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18761
  
Thanks! Merging to master/2.2/2.1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18761
  
LGTM



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18761
  
It looks much better now. Since `CleanupAliases` has such an issue, could 
you check whether the other expressions are facing the same issue? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...

[icexelloss]

Github user icexelloss commented on the issue:

https://github.com/apache/spark/pull/18664
  
It looks like "SESSION_LOCAL_TIMEZONE" is not respected in most of the 
pyspark functionality. 

I think `df.collect()` and `df.toPandas` can be fixed to respect 
SESSION_LOCAL_TIMEZONE.

`TimestampType().toInternal(dt)` is tricky because it doesn't have a 
reference to the SQLConf object, we should maybe deprecate this method?

I think we can also create separate Jira to address these, this PR can fix 
just the Arrow path. @BryanCutler and @ueshin do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes exam...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/18749#discussion_r130223158
  
--- Diff: 
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionDescription.java
 ---
@@ -24,20 +24,50 @@
 
 /**
  * ::DeveloperApi::
-
+ *
  * A function description type which can be recognized by 
FunctionRegistry, and will be used to
  * show the usage of the function in human language.
  *
  * `usage()` will be used for the function usage in brief way.
- * `extended()` will be used for the function usage in verbose way, suppose
- *  an example will be provided.
  *
- *  And we can refer the function name by `_FUNC_`, in `usage` and 
`extended`, as it's
+ * These below are concatenated and used for the function usage in verbose 
way, suppose arguments,
+ * examples, note and since will be provided.
+ *
+ * `arguments()` describes arguments for the expression. This should 
follow the format as below:
+ *
+ *   Arguments:
+ * * arg0 - ...
+ * 
+ * * arg1 - ...
+ * 
+ *
+ * `examples()` describes examples for the expression. This should follow 
the format as below:
+ *
+ *   Examples:
+ * > SELECT ...;
+ *  ...
+ * > SELECT ...;
+ *  ...
+ *
+ * `note()` contains some notes for the expression optionally.
+ *
+ * `since()` contains version information for the expression. Version is 
specified by,
+ * for example, "2.2.0".
+ *
+ *  We can refer the function name by `_FUNC_`, in `usage`, `arguments` 
and `examples`, as it's
  *  registered in `FunctionRegistry`.
+ *
+ *  Note that, if `extended()` is defined, `arguments()`, `examples()`, 
`note()` and `since()` will
+ *  be ignored and `extended()` will be used for the extended description 
for backward
+ *  compatibility.
  */
 @DeveloperApi
 @Retention(RetentionPolicy.RUNTIME)
 public @interface ExpressionDescription {
-String usage() default "_FUNC_ is undocumented";
-String extended() default "\nNo example/argument for _FUNC_.\n";
+String usage() default "";
+String extended() default "";
--- End diff --

In the last commit 
https://github.com/apache/spark/pull/18749/commits/bf48875057ca5978b88cacab455292b64daf1557,
 I tried to keep the original behaviour and compatibility (although I guess we 
are fine to break this as said in few comments above):

For `ExpressionDescription` and `ExpressionInfo`,


if `extended` is an empty string (the default value), it creates the 
`extended` made by `arguments`, `examples`, `note` and `since` and use this as 
the extended description. 

if `extended` is a non-empty string, it ignores the new `arguments`, 
`examples`, `note` and `since` but use this as the extended description, 
assuming this was set manually. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18749
  
**[Test build #80044 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80044/testReport)**
 for PR 18749 at commit 
[`f51ee74`](https://github.com/apache/spark/commit/f51ee74c77a476113cec164d3cc6bce9997e3519).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit problem i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18641
  
**[Test build #80045 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80045/testReport)**
 for PR 18641 at commit 
[`e5431a0`](https://github.com/apache/spark/commit/e5431a02fda88c442146eb2297a04c2e36da65ad).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit pr...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/18641#discussion_r130223022
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
 ---
@@ -270,37 +271,91 @@ case class CaseWhenCodegen(
 // }
 //   }
 // }
+
+val isNull = ctx.freshName("caseWhenIsNull")
+val value = ctx.freshName("caseWhenValue")
+// Split these expressions only if they are created from a row object
--- End diff --

good catch. done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18749
  
**[Test build #80043 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80043/testReport)**
 for PR 18749 at commit 
[`bf48875`](https://github.com/apache/spark/commit/bf48875057ca5978b88cacab455292b64daf1557).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes exam...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/18749#discussion_r130222959
  
--- Diff: 
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionDescription.java
 ---
@@ -24,20 +24,50 @@
 
 /**
  * ::DeveloperApi::
-
+ *
  * A function description type which can be recognized by 
FunctionRegistry, and will be used to
  * show the usage of the function in human language.
  *
  * `usage()` will be used for the function usage in brief way.
- * `extended()` will be used for the function usage in verbose way, suppose
- *  an example will be provided.
  *
- *  And we can refer the function name by `_FUNC_`, in `usage` and 
`extended`, as it's
+ * These below are concatenated and used for the function usage in verbose 
way, suppose arguments,
+ * examples, note and since will be provided.
+ *
+ * `arguments()` describes arguments for the expression. This should 
follow the format as below:
+ *
+ *   Arguments:
+ * * arg0 - ...
+ * 
+ * * arg1 - ...
+ * 
+ *
+ * `examples()` describes examples for the expression. This should follow 
the format as below:
+ *
+ *   Examples:
+ * > SELECT ...;
+ *  ...
+ * > SELECT ...;
+ *  ...
+ *
+ * `note()` contains some notes for the expression optionally.
+ *
+ * `since()` contains version information for the expression. Version is 
specified by,
+ * for example, "2.2.0".
+ *
+ *  We can refer the function name by `_FUNC_`, in `usage`, `arguments` 
and `examples`, as it's
  *  registered in `FunctionRegistry`.
+ *
+ *  Note that, if `extended()` is defined, `arguments()`, `examples()`, 
`note()` and `since()` will
+ *  be ignored and `extended()` will be used for the extended description 
for backward
+ *  compatibility.
  */
 @DeveloperApi
 @Retention(RetentionPolicy.RUNTIME)
 public @interface ExpressionDescription {
-String usage() default "_FUNC_ is undocumented";
-String extended() default "\nNo example/argument for _FUNC_.\n";
+String usage() default "";
+String extended() default "";
--- End diff --

I manually tested source compatibility and I guess it also won't break 
binary compatibility - 
https://stackoverflow.com/questions/21197255/is-adding-a-method-to-a-java-annotation-safe-for-backward-compatibility


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18750: Skip maven-compiler-plugin main and test compilations in...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18750
  
**[Test build #3861 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3861/testReport)**
 for PR 18750 at commit 
[`d9dfe3a`](https://github.com/apache/spark/commit/d9dfe3a147dd5530315572c523588d55c28120de).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes exam...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/18749#discussion_r13064
  
--- Diff: 
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionDescription.java
 ---
@@ -29,15 +29,40 @@
  * show the usage of the function in human language.
  *
  * `usage()` will be used for the function usage in brief way.
- * `extended()` will be used for the function usage in verbose way, suppose
- *  an example will be provided.
  *
- *  And we can refer the function name by `_FUNC_`, in `usage` and 
`extended`, as it's
+ * These below are concatenated and used for the function usage in verbose 
way, suppose arguments,
+ * examples, note and since will be provided.
+ *
+ * `arguments()` describes arguments for the expression. This should 
follow the format as below:
+ *
+ *   Arguments:
+ * * arg0 - ...
+ * 
+ * * arg1 - ...
+ * 
+ *
+ * `examples()` describes examples for the expression. This should follow 
the format as below:
+ *
+ *   Examples:
+ * > SELECT ...;
+ *  ...
+ * > SELECT ...;
+ *  ...
+ *
+ * `note()` contains some notes for the expression optionally.
+ *
+ * `since()` contains version information for the expression. Version is 
specified by,
+ * for example, "2.2.0".
+ *
+ *  We can refer the function name by `_FUNC_`, in `usage`, `arguments` 
and `examples`, as it's
  *  registered in `FunctionRegistry`.
  */
 @DeveloperApi
--- End diff --

Will give a try.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes exam...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/18749#discussion_r13001
  
--- Diff: 
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java
 ---
@@ -26,6 +26,10 @@
 private String name;
 private String extended;
 private String db;
+private String arguments;
--- End diff --

Yes, I think so for the former and sure will keep the original constructor.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18767: [SPARK-21572][Spark Shell] Add description on how to exi...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/18767
  
I also don't think it is worth being printed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18315: [SPARK-21108] [ML] [WIP] convert LinearSVC to aggregator...

[hhbyyh]

Github user hhbyyh commented on the issue:

https://github.com/apache/spark/pull/18315
  
Thanks for the review. Updated to address the comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18750: Skip maven-compiler-plugin main and test compilations in...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18750
  
**[Test build #3861 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3861/testReport)**
 for PR 18750 at commit 
[`d9dfe3a`](https://github.com/apache/spark/commit/d9dfe3a147dd5530315572c523588d55c28120de).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18750: Skip maven-compiler-plugin main and test compilations in...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/18750
  
This will need a JIRA @gslowikowski , by the way


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partiti...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18769
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partiti...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18769
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80042/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partiti...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18769
  
**[Test build #80042 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80042/testReport)**
 for PR 18769 at commit 
[`4bdc9ad`](https://github.com/apache/spark/commit/4bdc9ad5506a6055b9bb0b82ad4cfb1af09708c0).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18718: [SPARK-21357][DStreams] FileInputDStream not remo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18718


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18770: [SPARK-21508][DOC] Fix example code provided in S...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18770


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18718: [SPARK-21357][DStreams] FileInputDStream not remove out ...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/18718
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18770: [SPARK-21508][DOC] Fix example code provided in Spark St...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/18770
  
Merged to master/2.2


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18770: [SPARK-21508][DOC] Fix example code provided in Spark St...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18770
  
**[Test build #3860 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3860/testReport)**
 for PR 18770 at commit 
[`646699e`](https://github.com/apache/spark/commit/646699eca80969a82e82c6b7c7c100f910458266).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18770: [SPARK-21508][DOC] Fix example code provided in Spark St...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18770
  
**[Test build #3860 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3860/testReport)**
 for PR 18770 at commit 
[`646699e`](https://github.com/apache/spark/commit/646699eca80969a82e82c6b7c7c100f910458266).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18540
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18540
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80041/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18540
  
**[Test build #80041 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80041/testReport)**
 for PR 18540 at commit 
[`38b04df`](https://github.com/apache/spark/commit/38b04df792caf298da63be565d1b46a4eae93d98).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18770: Error in example code provided in Spark Streaming Docume...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18770
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18770: Error in example code provided in Spark Streaming...

[remisharoon]

GitHub user remisharoon opened a pull request:

https://github.com/apache/spark/pull/18770

Error in example code provided in Spark Streaming Documentation

## What changes were proposed in this pull request?

JIRA ticket : SPARK-21508
correcting a mistake in example code provided in Spark Streaming Custom 
Receivers Documentation

## How was this patch tested?
this documentation change is tested manually by jekyll build , running 
below commands
```
jekyll build
jekyll serve --watch
```
screen-shots provided below

![screenshot1](https://user-images.githubusercontent.com/8828470/28744636-a6de1ac6-7482-11e7-843b-ff84b5855ec0.png)

![screenshot2](https://user-images.githubusercontent.com/8828470/28744637-a6def496-7482-11e7-9512-7f4bbe027c6a.png)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/remisharoon/spark master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18770.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18770


commit 646699eca80969a82e82c6b7c7c100f910458266
Author: Remis Haroon 
Date:   2017-07-25T14:34:20Z

correcting an error in example code provided in Spark Streaming Custom 
Receivers Documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18767: [SPARK-21572][Spark Shell] Add description on how to exi...

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/18767
  
I’m not convinced this would be useful, could you close this please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18768: [SPARK-21533][SQL] Print warning messages when override ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18768
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80040/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18768: [SPARK-21533][SQL] Print warning messages when override ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18768
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18768: [SPARK-21533][SQL] Print warning messages when override ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18768
  
**[Test build #80040 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80040/testReport)**
 for PR 18768 at commit 
[`2d586a8`](https://github.com/apache/spark/commit/2d586a80e6f67e3adac17a8c1b5d09fbbf4252d5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partiti...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18769
  
**[Test build #80042 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80042/testReport)**
 for PR 18769 at commit 
[`4bdc9ad`](https://github.com/apache/spark/commit/4bdc9ad5506a6055b9bb0b82ad4cfb1af09708c0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic....

[wangyum]

GitHub user wangyum opened a pull request:

https://github.com/apache/spark/pull/18769

[SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partitions lose effect.

## What changes were proposed in this pull request?

How to reproduce:
```scala
val data = (0 until 1001).map(i => (i, i))
spark.createDataFrame(data).toDF("c1", 
"c2").createOrReplaceTempView("sourceTable")
spark.sql("create table targetTable(c1 int) PARTITIONED BY(c2 int)")
spark.sql("set hive.exec.dynamic.partition.mode=nonstrict")
spark.sql("set hive.exec.max.dynamic.partitions=1001")
spark.sql("insert overwrite table targetTable partition(c2) select * from 
sourceTable")
```

This PR fix this issue.

## How was this patch tested?
 unit tests


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangyum/spark SPARK-21574

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18769.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18769


commit 4bdc9ad5506a6055b9bb0b82ad4cfb1af09708c0
Author: Yuming Wang 
Date:   2017-07-29T10:48:02Z

Fix  set hive.exec.max.dynamic.partitions lose effect.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18761
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18761
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80039/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18761
  
**[Test build #80039 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80039/testReport)**
 for PR 18761 at commit 
[`82d54ed`](https://github.com/apache/spark/commit/82d54ed769e75d2a8b74b8fb87d49904e3d405d9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18768: [SPARK-21533][SQL] Print warning messages when ov...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18768#discussion_r130217731
  
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala 
---
@@ -111,6 +112,31 @@ private[hive] object HiveShim {
 }
   }
 
+  private def isSubClassOf(t: Type, parent: Class[_]): Boolean = t match {
+case cls: Class[_] => parent.isAssignableFrom(cls)
+case _ => false
+  }
+
+  private def hasInheritanceOf[UDFType: ClassTag](funcName: String, clazz: 
Class[_]): Boolean = {
+val clsTag = classTag[UDFType].runtimeClass
+if (isSubClassOf(clazz, clsTag)) {
+  val funcClass = clazz.getMethod(funcName, classOf[MapredContext])
+  funcClass.getDeclaringClass != clsTag
+} else {
+  false
+}
+  }
+
+  def validateHiveUserDefinedFunction(udfClass: Class[_]): Unit = {
+if (hasInheritanceOf[GenericUDF]("configure", udfClass) ||
--- End diff --

When `GenericUDF` API has `configure` method? Seems `GenericUDF` at 0.10.0 
has no such method?


https://hive.apache.org/javadocs/r0.10.0/api/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18540
  
**[Test build #80041 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80041/testReport)**
 for PR 18540 at commit 
[`38b04df`](https://github.com/apache/spark/commit/38b04df792caf298da63be565d1b46a4eae93d98).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

[jiangxb1987]

Github user jiangxb1987 commented on a diff in the pull request:

https://github.com/apache/spark/pull/18540#discussion_r130217250
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/expressions/WindowSpec.scala ---
@@ -174,28 +191,22 @@ class WindowSpec private[sql](
*/
   // Note: when updating the doc for this method, also update 
Window.rangeBetween.
   def rangeBetween(start: Long, end: Long): WindowSpec = {
--- End diff --

We'll get and compute empty frames.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...

[jiangxb1987]

Github user jiangxb1987 commented on a diff in the pull request:

https://github.com/apache/spark/pull/18540#discussion_r130217244
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ---
@@ -1179,32 +1179,26 @@ class AstBuilder(conf: SQLConf) extends 
SqlBaseBaseVisitor[AnyRef] with Logging
   }
 
   /**
-   * Create or resolve a [[FrameBoundary]]. Simple math expressions are 
allowed for Value
-   * Preceding/Following boundaries. These expressions must be constant 
(foldable) and return an
-   * integer value.
+   * Create or resolve a frame boundary expressions.
*/
-  override def visitFrameBound(ctx: FrameBoundContext): FrameBoundary = 
withOrigin(ctx) {
-// We currently only allow foldable integers.
-def value: Int = {
+  override def visitFrameBound(ctx: FrameBoundContext): Expression = 
withOrigin(ctx) {
+def value: Expression = {
   val e = expression(ctx.expression)
-  validate(e.resolved && e.foldable && e.dataType == IntegerType,
-"Frame bound value must be a constant integer.",
-ctx)
-  e.eval().asInstanceOf[Int]
+  validate(e.resolved && e.foldable, "Frame bound value must be a 
literal.", ctx)
+  e
 }
 
-// Create the FrameBoundary
 ctx.boundType.getType match {
   case SqlBaseParser.PRECEDING if ctx.UNBOUNDED != null =>
 UnboundedPreceding
   case SqlBaseParser.PRECEDING =>
-ValuePreceding(value)
+UnaryMinus(value)
   case SqlBaseParser.CURRENT =>
 CurrentRow
   case SqlBaseParser.FOLLOWING if ctx.UNBOUNDED != null =>
 UnboundedFollowing
   case SqlBaseParser.FOLLOWING =>
-ValueFollowing(value)
+value
--- End diff --

May I ask how should we parse it into an `unsigned-integer`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org