date:20170508

[GitHub] spark issue #17912: [SPARK-20670] [ML] Simplify FPGrowth transform

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17912
  
**[Test build #76635 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76635/testReport)**
 for PR 17912 at commit 
[`b9e3e47`](https://github.com/apache/spark/commit/b9e3e47706af2b9b09fa73101487d31a00779dc3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17879
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76621/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17879
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17879
  
**[Test build #76621 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76621/testReport)**
 for PR 17879 at commit 
[`ff9b1d6`](https://github.com/apache/spark/commit/ff9b1d66873eb8cad1a4a13f323555da2706a849).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17911: [SPARK-20668][SQL] Modify ScalaUDF to handle nullability...

2017-05-08 Thread ueshin

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/17911
  
cc @gatorsmile


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17912: [SPARK-20670] [ML] Simplify FPGrowth transform

2017-05-08 Thread hhbyyh

Github user hhbyyh commented on the issue:

https://github.com/apache/spark/pull/17912
  
cc @srowen @jkbradley @felixcheung 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17858: [SPARK-20594][SQL]The staging directory should be a chil...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17858
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76617/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17858: [SPARK-20594][SQL]The staging directory should be a chil...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17858
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17858: [SPARK-20594][SQL]The staging directory should be a chil...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17858
  
**[Test build #76617 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76617/testReport)**
 for PR 17858 at commit 
[`6b22d3e`](https://github.com/apache/spark/commit/6b22d3ea694c4133965ddface73c52c3566cd156).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17912: [SPARK-20670] [ML] Simplify FPGrowth transform

2017-05-08 Thread hhbyyh

GitHub user hhbyyh opened a pull request:

https://github.com/apache/spark/pull/17912

[SPARK-20670] [ML] Simplify FPGrowth transform

## What changes were proposed in this pull request?

As suggested by Sean Owen in https://github.com/apache/spark/pull/17130, 
the transform code in FPGrowthModel can be simplified.

As I tested on some public dataset http://fimi.ua.ac.be/data/, the 
performance of the new transform code is even or better than the old 
implementation.

## How was this patch tested?

Existing unit test.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hhbyyh/spark fpgrowthTransform

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17912.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17912






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16985: [SPARK-19122][SQL] Unnecessary shuffle+sort added if joi...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16985
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16985: [SPARK-19122][SQL] Unnecessary shuffle+sort added if joi...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16985
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76614/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17905
  
**[Test build #76634 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76634/testReport)**
 for PR 17905 at commit 
[`b37a760`](https://github.com/apache/spark/commit/b37a760417ea5f9b958a7329dbccd110478821ff).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tabl...

2017-05-08 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17905


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17910: [SPARK-20669][ML] LogisticRegression family should be ca...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17910
  
**[Test build #76633 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76633/testReport)**
 for PR 17910 at commit 
[`33c0f9e`](https://github.com/apache/spark/commit/33c0f9e52c239a6067a535be9c0ce19772d32aef).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16985: [SPARK-19122][SQL] Unnecessary shuffle+sort added if joi...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16985
  
**[Test build #76614 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76614/testReport)**
 for PR 16985 at commit 
[`e202ac1`](https://github.com/apache/spark/commit/e202ac1eda5fd1be3e466eea8975a1b0af54129f).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17911: [SPARK-20668][SQL] Modify ScalaUDF to handle nullability...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17911
  
**[Test build #76632 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76632/testReport)**
 for PR 17911 at commit 
[`120c862`](https://github.com/apache/spark/commit/120c862bada2e8a574f29ea4eb4434a528d59b3b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17911: [SPARK-20668][SQL] Modify ScalaUDF to handle null...

2017-05-08 Thread ueshin

GitHub user ueshin opened a pull request:

https://github.com/apache/spark/pull/17911

[SPARK-20668][SQL] Modify ScalaUDF to handle nullability.

## What changes were proposed in this pull request?

When registering Scala UDF, we can know if the udf will return nullable 
value or not. `ScalaUDF` and related classes should handle the nullability.

## How was this patch tested?

Existing tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ueshin/apache-spark issues/SPARK-20668

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17911.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17911


commit 120c862bada2e8a574f29ea4eb4434a528d59b3b
Author: Takuya UESHIN 
Date:   2017-05-05T04:17:18Z

Modify ScalaUDF to handle nullability.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17905
  
merged to master/2.2


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17910: [SPARK-20669][ML] LogisticRegression family shoul...

2017-05-08 Thread zhengruifeng

GitHub user zhengruifeng opened a pull request:

https://github.com/apache/spark/pull/17910

[SPARK-20669][ML] LogisticRegression family should be case insensitive 

## What changes were proposed in this pull request?
make param `family` case insensitive 

## How was this patch tested?
updated tests


@yanboliang 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhengruifeng/spark lr_family_lowercase

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17910.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17910


commit 33c0f9e52c239a6067a535be9c0ce19772d32aef
Author: Zheng RuiFeng 
Date:   2017-05-09T05:43:13Z

create pr




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17905
  
ok Jenkins passes, I'm going to merge this in since there are a bunch of PR 
failing because of this, even when they say it's up-to-date with master.
I'm going to investigate further though.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15435
  
**[Test build #76631 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76631/testReport)**
 for PR 15435 at commit 
[`449782a`](https://github.com/apache/spark/commit/449782a36ed139919bec6b114938590a383eaf43).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16989
  
**[Test build #76630 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76630/testReport)**
 for PR 16989 at commit 
[`308b7c7`](https://github.com/apache/spark/commit/308b7c72984d66030551f58ba000c5090d308dde).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17909: [SPARK-20661][WIP] try to dump table names

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17909
  
**[Test build #76629 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76629/testReport)**
 for PR 17909 at commit 
[`986dbdd`](https://github.com/apache/spark/commit/986dbdddb27218bf271402eb4a93eaccc763d4d5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17909: [SPARK-20661][WIP] try to dump table names

2017-05-08 Thread felixcheung

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/17909

[SPARK-20661][WIP] try to dump table names

## What changes were proposed in this pull request?

.. to see what tables are leaked.

Do not merge

## How was this patch tested?

Jenkins

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark trylisttable

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17909.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17909


commit 6332e9662f232eb871a795cf004465d0de6d500d
Author: Felix Cheung 
Date:   2017-05-09T05:44:26Z

try to dump table names

commit 986dbdddb27218bf271402eb4a93eaccc763d4d5
Author: Felix Cheung 
Date:   2017-05-09T05:45:50Z

to trigger sql tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17905
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17905
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76612/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17905
  
**[Test build #76612 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76612/testReport)**
 for PR 17905 at commit 
[`1aa17d8`](https://github.com/apache/spark/commit/1aa17d80590d88354065d409e1dd64961823eb2e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17879
  
**[Test build #76628 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76628/testReport)**
 for PR 17879 at commit 
[`53381ea`](https://github.com/apache/spark/commit/53381ea6ba41cc26ed89a6fc42252f7126198d9f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16985: [SPARK-19122][SQL] Unnecessary shuffle+sort added if joi...

2017-05-08 Thread cloud-fan

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/16985
  
shall we introduce a physical optimizer rule which reorders join predicates 
based on `child.outputOrdering` and `outputPartitioning`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...

2017-05-08 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17666
  
LGTM. 

Thank you! @maropu 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17887: [SPARK-20399][SQL] Add a config to fallback string liter...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17887
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17887: [SPARK-20399][SQL] Add a config to fallback string liter...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17887
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76611/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17887: [SPARK-20399][SQL] Add a config to fallback string liter...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17887
  
**[Test build #76611 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76611/testReport)**
 for PR 17887 at commit 
[`04a9fd3`](https://github.com/apache/spark/commit/04a9fd34c7489079da2b02a8f3a5ca84d87b0017).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17865: [SPARK-20456][Docs] Add examples for functions collectio...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17865
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76625/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17865: [SPARK-20456][Docs] Add examples for functions collectio...

2017-05-08 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17865
  
@map222 Unfortunately, our PySpark did not follow what we did in Scala. 
Will review it more carefully in the future. Thanks! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17865: [SPARK-20456][Docs] Add examples for functions collectio...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17865
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17865: [SPARK-20456][Docs] Add examples for functions collectio...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17865
  
**[Test build #76625 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76625/testReport)**
 for PR 17865 at commit 
[`ca8b5f7`](https://github.com/apache/spark/commit/ca8b5f7d666bd13a515ba1358e4f69ff13df9711).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17908: [SPARK-20667] [SQL] [TESTS] Cleanup the cataloged metada...

2017-05-08 Thread cloud-fan

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/17908
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15259: [SPARK-17685][SQL] Make SortMergeJoinExec's currentVars ...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15259
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76610/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15259: [SPARK-17685][SQL] Make SortMergeJoinExec's currentVars ...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15259
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17902: [SPARK-20641][core] Add key-value store abstraction and ...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17902
  
**[Test build #76605 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76605/testReport)**
 for PR 17902 at commit 
[`63e0a58`](https://github.com/apache/spark/commit/63e0a58b01bd622d6a3f2dc8fbe72c819493c152).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15259: [SPARK-17685][SQL] Make SortMergeJoinExec's currentVars ...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15259
  
**[Test build #76610 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76610/testReport)**
 for PR 15259 at commit 
[`2bb54b5`](https://github.com/apache/spark/commit/2bb54b569fcaf3c431bf792f594c485064d3cd37).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17902: [SPARK-20641][core] Add key-value store abstraction and ...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17902
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76605/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17902: [SPARK-20641][core] Add key-value store abstraction and ...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17902
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17879: [SPARK-20619][ML] StringIndexer supports multiple...

2017-05-08 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17879#discussion_r115409190
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala ---
@@ -131,6 +167,12 @@ object StringIndexer extends 
DefaultParamsReadable[StringIndexer] {
   private[feature] val KEEP_INVALID: String = "keep"
   private[feature] val supportedHandleInvalids: Array[String] =
 Array(SKIP_INVALID, ERROR_INVALID, KEEP_INVALID)
+  private[feature] val FREQ_DESC: String = "frequency_desc"
+  private[feature] val FREQ_ASC: String = "frequency_asc"
+  private[feature] val ALPHABET_DESC: String = "alphabet_desc"
+  private[feature] val ALPHABET_ASC: String = "alphabet_asc"
--- End diff --

Normally, we do not use underscore in the names. `lowerCamelCase` is our 
rules for naming. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17770: [SPARK-20392][SQL] Set barrier to prevent re-ente...

2017-05-08 Thread cloud-fan

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17770#discussion_r115408985
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -670,7 +671,9 @@ class Analyzer(
  * Generate a new logical plan for the right child with different 
expression IDs
  * for all conflicting attributes.
  */
-private def dedupRight (left: LogicalPlan, right: LogicalPlan): 
LogicalPlan = {
+private def dedupRight (left: LogicalPlan, oriRight: LogicalPlan): 
LogicalPlan = {
+  // Remove analysis barrier if any.
+  val right = CleanupBarriers(oriRight)
--- End diff --

shall we still keep the `AnalysisBarrier` for the right side?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17711: [SPARK-19951][SQL] Add string concatenate operator || to...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17711
  
**[Test build #76626 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76626/testReport)**
 for PR 17711 at commit 
[`cb4b26e`](https://github.com/apache/spark/commit/cb4b26e5e3bf112afadf69f0eacbd71a464fedaf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16989
  
**[Test build #76627 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76627/testReport)**
 for PR 16989 at commit 
[`ecb0882`](https://github.com/apache/spark/commit/ecb0882415887c47fb3b3de34c278955d2cf9214).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17770: [SPARK-20392][SQL] Set barrier to prevent re-ente...

2017-05-08 Thread cloud-fan

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17770#discussion_r115408504
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -188,6 +188,9 @@ class Dataset[T] private[sql](
 }
   }
 
+  // Wrap analyzed logical plan with an analysis barrier so we won't 
traverse/resolve it again.
+  @transient private val planBarrier: LogicalPlan = 
AnalysisBarrier(logicalPlan)
--- End diff --

`planWithBarrier`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17770: [SPARK-20392][SQL] Set barrier to prevent re-ente...

2017-05-08 Thread cloud-fan

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17770#discussion_r115408432
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
 ---
@@ -47,36 +47,11 @@ abstract class LogicalPlan extends 
QueryPlan[LogicalPlan] with Logging {
   def isStreaming: Boolean = children.exists(_.isStreaming == true)
 
   /**
-   * Returns a copy of this node where `rule` has been recursively applied 
first to all of its
-   * children and then itself (post-order). When `rule` does not apply to 
a given node, it is left
-   * unchanged.  This function is similar to `transformUp`, but skips 
sub-trees that have already
-   * been marked as analyzed.
-   *
-   * @param rule the function use to transform this nodes children
-   */
-  def resolveOperators(rule: PartialFunction[LogicalPlan, LogicalPlan]): 
LogicalPlan = {
-if (!analyzed) {
-  val afterRuleOnChildren = mapChildren(_.resolveOperators(rule))
-  if (this fastEquals afterRuleOnChildren) {
-CurrentOrigin.withOrigin(origin) {
-  rule.applyOrElse(this, identity[LogicalPlan])
-}
-  } else {
-CurrentOrigin.withOrigin(origin) {
-  rule.applyOrElse(afterRuleOnChildren, identity[LogicalPlan])
-}
-  }
-} else {
-  this
-}
-  }
-
-  /**
* Recursively transforms the expressions of a tree, skipping nodes that 
have already
* been analyzed.
*/
   def resolveExpressions(r: PartialFunction[Expression, Expression]): 
LogicalPlan = {
--- End diff --

this should also be removed, we should use `transformExpressions`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17876: [SPARK-20569][SQL] RuntimeReplaceable functions should n...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17876
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17876: [SPARK-20569][SQL] RuntimeReplaceable functions should n...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17876
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76609/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17876: [SPARK-20569][SQL] RuntimeReplaceable functions should n...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17876
  
**[Test build #76609 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76609/testReport)**
 for PR 17876 at commit 
[`0021ec3`](https://github.com/apache/spark/commit/0021ec370904fe01eb671624bef61066121e60ef).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16989
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16989
  
**[Test build #76622 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76622/testReport)**
 for PR 16989 at commit 
[`c58dcf4`](https://github.com/apache/spark/commit/c58dcf448723ea51d38bc07bf83c079a293c8d88).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16989
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76622/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17908: [SPARK-20667] [SQL] [TESTS] Cleanup the cataloged...

2017-05-08 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17908#discussion_r115407806
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -1251,9 +1251,10 @@ class SessionCatalog(
 dropTempFunction(func.funcName, ignoreIfNotExists = false)
   }
 }
-tempTables.clear()
+clearTempTables()
--- End diff --

This is to call the public function. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17908: [SPARK-20667] [SQL] [TESTS] Cleanup the cataloged...

2017-05-08 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17908#discussion_r115407771
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala ---
@@ -488,14 +488,9 @@ private[hive] class TestHiveSparkSession(
 
   sharedState.cacheManager.clearCache()
   loadedTables.clear()
-  sessionState.catalog.clearTempTables()
-  sessionState.catalog.tableRelationCache.invalidateAll()
--- End diff --

This is part of `sessionState.catalog.reset()` after this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17908: [SPARK-20667] [SQL] [TESTS] Cleanup the cataloged...

2017-05-08 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17908#discussion_r115407765
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala ---
@@ -488,14 +488,9 @@ private[hive] class TestHiveSparkSession(
 
   sharedState.cacheManager.clearCache()
   loadedTables.clear()
-  sessionState.catalog.clearTempTables()
-  sessionState.catalog.tableRelationCache.invalidateAll()
-
+  sessionState.catalog.reset()
   metadataHive.reset()
 
-  
FunctionRegistry.getFunctionNames.asScala.filterNot(originalUDFs.contains(_)).
-foreach { udfName => 
FunctionRegistry.unregisterTemporaryUDF(udfName) }
--- End diff --

This is part of `sessionState.catalog.reset()`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17879
  
**[Test build #76624 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76624/testReport)**
 for PR 17879 at commit 
[`07198d9`](https://github.com/apache/spark/commit/07198d9bb45a54d3c257ad37e772cc31154ffcb6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17865: [SPARK-20456][Docs] Add examples for functions collectio...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17865
  
**[Test build #76625 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76625/testReport)**
 for PR 17865 at commit 
[`ca8b5f7`](https://github.com/apache/spark/commit/ca8b5f7d666bd13a515ba1358e4f69ff13df9711).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17908: [SPARK-20667] [SQL] [TESTS] Cleanup the cataloged metada...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17908
  
**[Test build #76623 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76623/testReport)**
 for PR 17908 at commit 
[`4976215`](https://github.com/apache/spark/commit/4976215fa16f88d4c8772cfc67cb1866319f8a1f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17905
  
How about https://github.com/apache/spark/pull/17908? It tries to reset the 
cataloged metadata objects and temporary objects.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17879
  
**[Test build #76621 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76621/testReport)**
 for PR 17879 at commit 
[`ff9b1d6`](https://github.com/apache/spark/commit/ff9b1d66873eb8cad1a4a13f323555da2706a849).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16989
  
**[Test build #76622 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76622/testReport)**
 for PR 16989 at commit 
[`c58dcf4`](https://github.com/apache/spark/commit/c58dcf448723ea51d38bc07bf83c079a293c8d88).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17908: [SPARK-20667] [SQL] [TESTS] Cleanup the cataloged...

2017-05-08 Thread gatorsmile

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/17908

[SPARK-20667] [SQL] [TESTS] Cleanup the cataloged metadata after completing 
the package of sql/core and sql/hive

## What changes were proposed in this pull request?

So far, we do not drop all the cataloged tables after each package. 
Sometimes, we might hit strange test case errors because the previous test 
suite did not drop the tables/functions/database. At least, we can first clean 
up the environment when completing the package of `sql/core` and `sql/hive`.

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Please review http://spark.apache.org/contributing.html before opening a 
pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark reset

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17908.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17908


commit 4976215fa16f88d4c8772cfc67cb1866319f8a1f
Author: Xiao Li 
Date:   2017-05-09T04:49:47Z

fix.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread actuaryzhang

Github user actuaryzhang commented on the issue:

https://github.com/apache/spark/pull/17879
  
Thanks much @felixcheung and @viirya. I have addressed your comments. 
- update from 2.2 to 2.3
- change `freq_desc` to `frequency_desc`. 
- move toLowerCase to the getter method. 
Please let me know if there is anything needed. Thanks!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17904: [SPARK-20630] [Web UI] Fixed column visibility in Execut...

2017-05-08 Thread ajbozarth

Github user ajbozarth commented on the issue:

https://github.com/apache/spark/pull/17904
  
I'm not sure why it's failing those tests, plus my branch is up to date 
with master (minus one unrelated commit)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17904: [SPARK-20630] [Web UI] Fixed column visibility in Execut...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17904
  
**[Test build #76620 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76620/testReport)**
 for PR 17904 at commit 
[`766bfb0`](https://github.com/apache/spark/commit/766bfb0f45366b790710e75579c8207370e56560).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17887: [SPARK-20399][SQL] Add a config to fallback strin...

2017-05-08 Thread cloud-fan

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17887#discussion_r115406775
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
 ---
@@ -413,38 +428,102 @@ class ExpressionParserSuite extends PlanTest {
   }
 
   test("strings") {
--- End diff --

how about something like
```
Seq(true, false).foreach { escape =>
  val conf = new SQLConf()
  conf.setConfString(SQLConf.ESCAPED_STRING_LITERALS.key, "true")
  val parser = new CatalystSqlParser(conf)

  // tests that have same result whatever the conf is
  assertEqual("\"hello\"", "hello")
  ...

  // tests that have different result regarding the conf
  if (escape) {
assert(...) 
...
  } else {
assert(...)
...
  }

}

```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17904: [SPARK-20630] [Web UI] Fixed column visibility in Execut...

2017-05-08 Thread ajbozarth

Github user ajbozarth commented on the issue:

https://github.com/apache/spark/pull/17904
  
Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17879
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17879
  
**[Test build #76619 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76619/testReport)**
 for PR 17879 at commit 
[`ba34043`](https://github.com/apache/spark/commit/ba340437fee99f848dfa88eab2e10d87651eab0a).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17879
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76619/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17901: [SPARK-20639][SQL] Add single argument support fo...

2017-05-08 Thread HyukjinKwon

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/17901#discussion_r115406526
  
--- Diff: R/pkg/R/functions.R ---
@@ -1752,15 +1752,15 @@ setMethod("toRadians",
 
 #' to_date
 #'
-#' Converts the column into a DateType. You may optionally specify a format
+#' Converts the column into a date column. You may optionally specify a 
format
 #' according to the rules in:
 #' 
\url{http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html}.
 #' If the string cannot be parsed according to the specified format (or 
default),
 #' the value of the column will be null.
-#' The default format is '-MM-dd'.
+#' By default, it follows casting rules to a date if the format is omitted.
--- End diff --

Ah, let me give a shot with adding an example - `cast(df$x, "date")`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17879: [SPARK-20619][ML] StringIndexer supports multiple ways t...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17879
  
**[Test build #76619 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76619/testReport)**
 for PR 17879 at commit 
[`ba34043`](https://github.com/apache/spark/commit/ba340437fee99f848dfa88eab2e10d87651eab0a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17887: [SPARK-20399][SQL] Add a config to fallback strin...

2017-05-08 Thread cloud-fan

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17887#discussion_r115406428
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 
---
@@ -1168,6 +1169,18 @@ class DatasetSuite extends QueryTest with 
SharedSQLContext {
 val ds = Seq(WithMapInOption(Some(Map(1 -> 1.toDS()
 checkDataset(ds, WithMapInOption(Some(Map(1 -> 1
   }
+
+  test("do not unescaped regex pattern string") {
--- End diff --

add jira id and when we should not unescape


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17905
  
right. I think it's a good way to decouple R tests from any earlier states 
and also not to mask the error/leak. I'll get that in when Jenkins pass (and 
see if I could figure out what is leaked)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17865: [SPARK-20456][Docs] Add examples for functions collectio...

2017-05-08 Thread map222

Github user map222 commented on the issue:

https://github.com/apache/spark/pull/17865
  
@gatorsmile I checked four functions, `approx_count_distinct`, `coalesce`, 
`covar_samp`, and `countDistinct`, comparing the python and Scala 
documentation. None of them are the same. My guess is that the python docs 
differ for most functions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread yhuai

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/17905
  
i see. I think 
https://github.com/apache/spark/pull/17905/commits/d4c1a9db25ee7386f7b12e4dabb54210a9892510
 is good. How about we get it checked in first (after jenkins passes)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17905
  
hmm, spoke too soon I think - looks to me like all the `withTable` clause 
are in place and complete.
not sure what can be leaking through then..


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17904: [SPARK-20630] [Web UI] Fixed column visibility in Execut...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17904
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76607/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17904: [SPARK-20630] [Web UI] Fixed column visibility in Execut...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17904
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17904: [SPARK-20630] [Web UI] Fixed column visibility in Execut...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17904
  
**[Test build #76607 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76607/testReport)**
 for PR 17904 at commit 
[`766bfb0`](https://github.com/apache/spark/commit/766bfb0f45366b790710e75579c8207370e56560).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

2017-05-08 Thread WeichenXu123

Github user WeichenXu123 commented on the issue:

https://github.com/apache/spark/pull/15435
  
jenkins test please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

2017-05-08 Thread WeichenXu123

Github user WeichenXu123 commented on the issue:

https://github.com/apache/spark/pull/15435
  
@felixcheung allready updated..


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...

2017-05-08 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17869
  
I think that's fine. It should be safe.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17858: [SPARK-20594][SQL]The staging directory should be...

2017-05-08 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17858#discussion_r115404865
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
 ---
@@ -97,12 +97,23 @@ case class InsertIntoHiveTable(
 val inputPathUri: URI = inputPath.toUri
 val inputPathName: String = inputPathUri.getPath
 val fs: FileSystem = inputPath.getFileSystem(hadoopConf)
-val stagingPathName: String =
+var stagingPathName: String =
   if (inputPathName.indexOf(stagingDir) == -1) {
 new Path(inputPathName, stagingDir).toString
   } else {
 inputPathName.substring(0, inputPathName.indexOf(stagingDir) + 
stagingDir.length)
   }
+
+// SPARK-20594: The staging directory should be a child directory 
starts with "." to avoid
+// being deleted if we set hive.exec.stagingdir under the table 
directory.
+if (FileUtils.isSubDir(new Path(stagingPathName), inputPath, fs)
+  && !stagingPathName.stripPrefix(inputPathName).startsWith(".")) {
--- End diff --

This is just to hide the issue and make the test cases passed, right?

We need to drop the created staging directory no matter what is the value 
users set. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread yhuai

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/17905
  
lgtm


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17865: [SPARK-20456][Docs] Add examples for functions co...

2017-05-08 Thread map222

Github user map222 commented on a diff in the pull request:

https://github.com/apache/spark/pull/17865#discussion_r115404673
  
--- Diff: python/pyspark/sql/functions.py ---
@@ -153,7 +173,7 @@ def _():
 # math functions that take two arguments as input
 _binary_mathfunctions = {
 'atan2': 'Returns the angle theta from the conversion of rectangular 
coordinates (x, y) to' +
- 'polar coordinates (r, theta).',
+ 'polar coordinates (r, theta). Units in radians.',
--- End diff --

Most libraries seem to default to radians. However, I checked the R, numpy, 
and MATLAB notes for common trigonometry functions, and they all note the units 
in the function documentation, e.g.:
https://docs.scipy.org/doc/numpy/reference/generated/numpy.sin.html
https://www.mathworks.com/help/matlab/ref/sin.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17666
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76608/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17666
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17666
  
**[Test build #76608 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76608/testReport)**
 for PR 17666 at commit 
[`625dbda`](https://github.com/apache/spark/commit/625dbda3aab90922d6301f044dc90746d2ffb238).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17666
  
Build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...

2017-05-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17666
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76602/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread yhuai

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/17905
  
@falaki's PR did not actually trigger that test. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17905: [SPARK-20661][SPARKR][TEST][FOLLOWUP] SparkR tableNames(...

2017-05-08 Thread yhuai

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/17905
  
@felixcheung  you are right. That is the problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...

2017-05-08 Thread heary-cao

Github user heary-cao commented on the issue:

https://github.com/apache/spark/pull/17869
  
@HyukjinKwon
I suggest to add to the beforeAllã
If the added beforeEach, Most of the unit tests will run the  
Utils.clearLocalRootDirs() twice.
What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17666: [SPARK-20311][SQL] Support aliases for table value funct...

2017-05-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17666
  
**[Test build #76602 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76602/testReport)**
 for PR 17666 at commit 
[`f494e41`](https://github.com/apache/spark/commit/f494e417557539369c2a5c6ee472d9697937a587).
 * This patch passes all tests.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

1 2 3 4 5 6 >

1 - 100 of 573 matches

Mail list logo