date:20170108

[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16387
  
**[Test build #71068 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71068/testReport)**
 for PR 16387 at commit 
[`5645533`](https://github.com/apache/spark/commit/5645533ede8bb37aa316afbc0d003d042bf5d52d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16511: [SPARK-19133][SPARKR][ML] fix glm for Gamma, clarify glm...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16511
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16511: [SPARK-19133][SPARKR][ML] fix glm for Gamma, clarify glm...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16511
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71066/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16511: [SPARK-19133][SPARKR][ML] fix glm for Gamma, clarify glm...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16511
  
**[Test build #71066 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71066/testReport)**
 for PR 16511 at commit 
[`0688aa4`](https://github.com/apache/spark/commit/0688aa4691e630e22a2016ded7e002221a1ec16a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16486: [SPARK-13610][ML] Create a Transformer to disassemble ve...

2017-01-08 Thread leonfl

Github user leonfl commented on the issue:

https://github.com/apache/spark/pull/16486
  
@jkbradley, Could you also help to check this patch cause you are familiar 
with this defect, Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16513: [SPARK-12757][CORE] lower "block locks were not released...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16513
  
**[Test build #71067 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71067/testReport)**
 for PR 16513 at commit 
[`dc81625`](https://github.com/apache/spark/commit/dc816254226baa2ff3cbdbdf421a3ac517f5a2c8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16513: [SPARK-12757][CORE] lower "block locks were not released...

2017-01-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16513
  
@JoshRosen


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16513: [SPARK-12757][CORE] lower "block locks were not r...

2017-01-08 Thread felixcheung

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/16513

[SPARK-12757][CORE] lower "block locks were not released" log to info level

## What changes were proposed in this pull request?

lower "block locks were not released" log to info level


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark blocklockswarn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16513.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16513


commit dc816254226baa2ff3cbdbdf421a3ac517f5a2c8
Author: Felix Cheung 
Date:   2017-01-09T07:23:39Z

lower log to info




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15671
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15671
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71064/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15671
  
**[Test build #71064 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71064/testReport)**
 for PR 15671 at commit 
[`e6b4615`](https://github.com/apache/spark/commit/e6b46153ee82f722ab95021dc82ab81d34dfd4ff).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16511: [SPARK-19133][SPARKR][DOCS] fix glm for Gamma, clarify g...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16511
  
**[Test build #71066 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71066/testReport)**
 for PR 16511 at commit 
[`0688aa4`](https://github.com/apache/spark/commit/0688aa4691e630e22a2016ded7e002221a1ec16a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported

2017-01-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16511
  
turns out Gamma is broken. will be working on adding tests


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16432
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71065/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16432
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16432
  
**[Test build #71065 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71065/testReport)**
 for PR 16432 at commit 
[`2673af2`](https://github.com/apache/spark/commit/2673af288c0c9472af3aa8e68782d6e572d12346).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15539: [SPARK-17994] [SQL] Add back a file status cache for cat...

2017-01-08 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15539
  
It could return incorrect results, but I need to prove it using a use case. 
We always call 
[refresh0](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala#L50)
 in 
[`filterPartitions`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala#L78-L79).
 Thus, it covers most cases. Let me try more in the next few days. 

Tonight, let me first fix the cache inconsistence issue caused by `SET 
LOCATION`. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16510
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71062/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16510
  
**[Test build #71062 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71062/testReport)**
 for PR 16510 at commit 
[`a706c15`](https://github.com/apache/spark/commit/a706c15ea699e41dd720e3c1f4c76e880ea7adc8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16510
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15539: [SPARK-17994] [SQL] Add back a file status cache for cat...

2017-01-08 Thread ericl

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/15539
  
Hm, what use cases are we trying to address? As I understand, the worst 
that can happen if the cache size flag is toggled at runtime is that the old 
settings might still apply. And when the filesource partition management flag 
is changed, it only takes effect after the table entry in 
`cachedDataSourceTables` in HiveMetastoreCatalog is refreshed.

If we document the above, is that enough?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16512: [SPARK-18335][SPARKR] createDataFrame to support numPart...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16512
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71061/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16512: [SPARK-18335][SPARKR] createDataFrame to support numPart...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16512
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16512: [SPARK-18335][SPARKR] createDataFrame to support numPart...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16512
  
**[Test build #71061 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71061/testReport)**
 for PR 16512 at commit 
[`b66a0ac`](https://github.com/apache/spark/commit/b66a0ac4748bbf14dfb992aeff95028122b6d7a9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16497: [SPARK-19118] [SQL] Percentile support for frequency dis...

2017-01-08 Thread jiangxb1987

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/16497
  
@tanejagagan Would you please revert the unnecessary code changes?

BTW: You can test the scala style on local environment by running 
`./dev/lint-scala` from console.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16501: [SPARK-19117][TESTS] Skip the tests using script transfo...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16501
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71058/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16501: [SPARK-19117][TESTS] Skip the tests using script transfo...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16501
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16501: [SPARK-19117][TESTS] Skip the tests using script transfo...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16501
  
**[Test build #71058 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71058/testReport)**
 for PR 16501 at commit 
[`0f654f3`](https://github.com/apache/spark/commit/0f654f347627d18a65d748a2ec9d60023568a531).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16432
  
**[Test build #71065 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71065/testReport)**
 for PR 16432 at commit 
[`2673af2`](https://github.com/apache/spark/commit/2673af288c0c9472af3aa8e68782d6e572d12346).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16511
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16511
  
**[Test build #71060 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71060/testReport)**
 for PR 16511 at commit 
[`3403567`](https://github.com/apache/spark/commit/3403567704946a546ab229bceb8ea28e91e0d759).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16511
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71060/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16474
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71055/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16474
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16474
  
**[Test build #71055 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71055/testReport)**
 for PR 16474 at commit 
[`6b562eb`](https://github.com/apache/spark/commit/6b562eba8b7d2a508cfd6f972bc18a108c3ed044).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16233: [SPARK-18801][SQL] Support resolve a nested view

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16233
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16233: [SPARK-18801][SQL] Support resolve a nested view

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16233
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71057/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15671: [SPARK-18206][ML]Add instrumentation for MLP,NB,LDA,AFT,...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15671
  
**[Test build #71064 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71064/testReport)**
 for PR 15671 at commit 
[`e6b4615`](https://github.com/apache/spark/commit/e6b46153ee82f722ab95021dc82ab81d34dfd4ff).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16474
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16233: [SPARK-18801][SQL] Support resolve a nested view

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16233
  
**[Test build #71057 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71057/testReport)**
 for PR 16233 at commit 
[`ad8957c`](https://github.com/apache/spark/commit/ad8957cbdb408e3c18f65f1bff5d14df447bd39c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16474
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71053/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16474
  
**[Test build #71053 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71053/testReport)**
 for PR 16474 at commit 
[`d6878e1`](https://github.com/apache/spark/commit/d6878e1087eb9d7c32c1084e908a967326c75087).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...

2017-01-08 Thread wzhfy

Github user wzhfy commented on the issue:

https://github.com/apache/spark/pull/16401
  
OK I'll submit a pr to fix this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...

2017-01-08 Thread rxin

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16401
  
hm concrete suggestion:

1. pass the conf and cache the computed statistics the first time

2. have a simple invalidateStatsCache method that can be called manually to 
invalidate.


Basically what I suggested in 
https://github.com/apache/spark/pull/16401#issuecomment-270841604


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...

2017-01-08 Thread wzhfy

Github user wzhfy commented on the issue:

https://github.com/apache/spark/pull/16401
  
So you mean we still pass the conf and do caching, and also add the 
invalidation method, although the cache invalidation logic has some problem in 
current stage?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16411: [SPARK-17984][YARN][Mesos][Deploy][WIP] add executor lau...

2017-01-08 Thread xiaochang-wu

Github user xiaochang-wu commented on the issue:

https://github.com/apache/spark/pull/16411
  
@tgravescs Thanks for your comments. There are two things we have tried.

1) To add a prefix command on executor launch
I agree this opens a door for user to do anything for launching the 
executor. This patch is intended for profiling and debugging. May not fit for 
production. I am not sure it's the best form to implement, it fixed our problem 
quickly. 

2) NUMA
The script attached about NUMA is only an example to show how to use this 
patch. User can customize it to fulfill their specific needs. Automatic NUMA 
balancing is by default enabled on our system. As mentioned in the original 
Redhat slides, It can only deal with certain cases and still can not beat 
manual pinning. From our experiments, not all cases have big NUMA penalties. We 
should use some platform tools such as Intel VTune to identify if there is a 
NUMA problem and tune case by case.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-08 Thread wzhfy

Github user wzhfy commented on the issue:

https://github.com/apache/spark/pull/16395
  
@ron8hu Can you update the test cases based on the latest master? We have a 
new test infrastructure now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12414: [SPARK-14657] [ML] RFormula w/o intercept should output ...

2017-01-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/12414
  
hi - where are we on this?
@yanboliang could you add `[SPARKR]` to the title


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16510
  
**[Test build #71062 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71062/testReport)**
 for PR 16510 at commit 
[`a706c15`](https://github.com/apache/spark/commit/a706c15ea699e41dd720e3c1f4c76e880ea7adc8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15480: [SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` gro...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15480
  
**[Test build #71063 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71063/testReport)**
 for PR 15480 at commit 
[`3d31cb3`](https://github.com/apache/spark/commit/3d31cb3e6309950bb723a1b83a004caab493c5e7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16512: [SPARK-18335][SPARKR] createDataFrame to support numPart...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16512
  
**[Test build #71061 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71061/testReport)**
 for PR 16512 at commit 
[`b66a0ac`](https://github.com/apache/spark/commit/b66a0ac4748bbf14dfb992aeff95028122b6d7a9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...

2017-01-08 Thread rxin

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16401
  
But you can't do a lazy val unless you have the conf passed in, since you 
will be adding knobs to the CBO stats soon wouldn't you?

Also there is really nothing basic vs CBO here. Both are returning the same 
set of statistics, just using a different way to estimate.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

2017-01-08 Thread jerryshao

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/16432#discussion_r95104135
  
--- Diff: 
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala
 ---
@@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends 
ServiceCredentialProvider
 }
 
 // Get the time of next renewal.
-tokenRenewalInterval.map { interval =>
-  creds.getAllTokens.asScala
-.filter(_.getKind == 
DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
+tokenRenewalInterval.flatMap { interval =>
+  val nextRenewalDates = creds.getAllTokens.asScala
+
.filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier])
--- End diff --

Yeah, that's the issue, let me think about how to handle it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16512: [SPARK-18335][SPARKR] createDataFrame to support ...

2017-01-08 Thread felixcheung

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/16512

[SPARK-18335][SPARKR] createDataFrame to support numPartitions parameter

## What changes were proposed in this pull request?

To allow specifying number of partitions when the DataFrame is created

## How was this patch tested?

manual, unit tests


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rnumpart

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16512


commit b66a0ac4748bbf14dfb992aeff95028122b6d7a9
Author: Felix Cheung 
Date:   2017-01-09T05:16:39Z

add numPartitions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

2017-01-08 Thread jerryshao

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/16432#discussion_r95104079
  
--- Diff: 
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala
 ---
@@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends 
ServiceCredentialProvider
 }
 
 // Get the time of next renewal.
-tokenRenewalInterval.map { interval =>
-  creds.getAllTokens.asScala
-.filter(_.getKind == 
DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
+tokenRenewalInterval.flatMap { interval =>
+  val nextRenewalDates = creds.getAllTokens.asScala
+
.filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier])
 .map { t =>
-  val identifier = new DelegationTokenIdentifier()
-  identifier.readFields(new DataInputStream(new 
ByteArrayInputStream(t.getIdentifier)))
+  val identifier = 
t.decodeIdentifier().asInstanceOf[AbstractDelegationTokenIdentifier]
   identifier.getIssueDate + interval
-  }.foldLeft(0L)(math.max)
+}
+  if (nextRenewalDates.isEmpty) None else Some(nextRenewalDates.min)
--- End diff --

I think the logic was incorrect previously, we should get the minimum 
renewal date, that's why I changed here. But the code was probably OK if only 
we only have one default HDFS configured. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16511
  
**[Test build #71060 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71060/testReport)**
 for PR 16511 at commit 
[`3403567`](https://github.com/apache/spark/commit/3403567704946a546ab229bceb8ea28e91e0d759).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16511: [SPARKR][DOCS] clarify glm family supported

2017-01-08 Thread felixcheung

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/16511#discussion_r95103899
  
--- Diff: R/pkg/R/mllib_regression.R ---
@@ -52,6 +52,8 @@ setClass("IsotonicRegressionModel", representation(jobj = 
"jobj"))
 #'   This can be a character string naming a family function, 
a family function or
 #'   the result of a call to a family function. Refer R family 
at
 #'   
\url{https://stat.ethz.ch/R-manual/R-devel/library/stats/html/family.html}.
+#'   Currently these families are supported: \code{binomial}, 
\code{gaussian},
+#'   \code{Gamma}, and \code{poisson}.
--- End diff --

in R, `Gamma` family is capital `R`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16511: [SPARKR][DOCS] clarify glm family supported

2017-01-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16511
  
@yanboliang 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16511: [SPARKR][DOCS] clarify glm family supported

2017-01-08 Thread felixcheung

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/16511

[SPARKR][DOCS] clarify glm family supported

## What changes were proposed in this pull request?

R family is a longer list than what Spark supports.

## How was this patch tested?

manual

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rdocglmfamily

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16511.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16511


commit 3403567704946a546ab229bceb8ea28e91e0d759
Author: Felix Cheung 
Date:   2017-01-09T05:31:06Z

doc family supported




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

2017-01-08 Thread viirya

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/16432#discussion_r95103785
  
--- Diff: 
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala
 ---
@@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends 
ServiceCredentialProvider
 }
 
 // Get the time of next renewal.
-tokenRenewalInterval.map { interval =>
-  creds.getAllTokens.asScala
-.filter(_.getKind == 
DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
+tokenRenewalInterval.flatMap { interval =>
+  val nextRenewalDates = creds.getAllTokens.asScala
+
.filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier])
--- End diff --

Can we make sure all instances of `AbstractDelegationTokenIdentifier` are 
tokens for Hadoop file systems? Isn't Hive's `DelegationTokenIdentifier` also 
an `AbstractDelegationTokenIdentifier`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16431: [SPARK-19020] [SQL] Cardinality estimation of aggregate ...

2017-01-08 Thread wzhfy

Github user wzhfy commented on the issue:

https://github.com/apache/spark/pull/16431
  
OK, I'll update this pr today.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...

2017-01-08 Thread wzhfy

Github user wzhfy commented on the issue:

https://github.com/apache/spark/pull/16401
  
If our goal is to fix two sets of stats issue first, is it ok to just use 
two `lazy val`s now, and leave the configuration and cache invalidation issues 
in the future? In current implementation, we don't need any configurations 
except the cbo switch conf.
```
def statistics(conf: CatalystConf): Statistics = {
  if (conf.cboEnabled) cboStats else basicStats
}
lazy val cboStats...
lazy val basicStats...
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16432: [SPARK-19021][YARN] Generailize HDFSCredentialPro...

2017-01-08 Thread viirya

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/16432#discussion_r95103643
  
--- Diff: 
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala
 ---
@@ -56,14 +57,14 @@ private[security] class HDFSCredentialProvider extends 
ServiceCredentialProvider
 }
 
 // Get the time of next renewal.
-tokenRenewalInterval.map { interval =>
-  creds.getAllTokens.asScala
-.filter(_.getKind == 
DelegationTokenIdentifier.HDFS_DELEGATION_KIND)
+tokenRenewalInterval.flatMap { interval =>
+  val nextRenewalDates = creds.getAllTokens.asScala
+
.filter(_.decodeIdentifier().isInstanceOf[AbstractDelegationTokenIdentifier])
 .map { t =>
-  val identifier = new DelegationTokenIdentifier()
-  identifier.readFields(new DataInputStream(new 
ByteArrayInputStream(t.getIdentifier)))
+  val identifier = 
t.decodeIdentifier().asInstanceOf[AbstractDelegationTokenIdentifier]
   identifier.getIssueDate + interval
-  }.foldLeft(0L)(math.max)
+}
+  if (nextRenewalDates.isEmpty) None else Some(nextRenewalDates.min)
--- End diff --

Why previously we use math.max and now changed to minimum value?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16431: [SPARK-19020] [SQL] Cardinality estimation of agg...

2017-01-08 Thread rxin

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/16431#discussion_r95103316
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/AggregateEstimation.scala
 ---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical.estimation
+
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Statistics}
+
+
+object AggregateEstimation {
+  import EstimationUtils._
+
+  def estimate(agg: Aggregate): Option[Statistics] = {
+val childStats = agg.child.statistics
+// Check if we have column stats for all group-by columns.
+val colStatsExist = agg.groupingExpressions.forall { e =>
+  e.isInstanceOf[Attribute] && 
childStats.attributeStats.contains(e.asInstanceOf[Attribute])
+}
+if (rowCountsExist(agg.child) && colStatsExist) {
+  // Initial value for agg without group expressions
+  var outputRows: BigInt = 1
+  agg.groupingExpressions.map(_.asInstanceOf[Attribute]).foreach { 
attr =>
+val colStat = childStats.attributeStats(attr)
+// Multiply distinct counts of group by columns. This is an upper 
bound, which assumes
+// the data contains all combinations of distinct values of group 
by columns.
+outputRows *= colStat.distinctCount
+  }
+
+  // The number of output rows must not be larger than child's number 
of rows.
+  // Note that this also covers the case of uniqueness of column. If 
one of the group-by columns
--- End diff --

OK I don't think you need this explanation here -- it simply makes it more 
confusing. You are just putting an upper bound on cardinality, and that 
explains everything.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16431: [SPARK-19020] [SQL] Cardinality estimation of aggregate ...

2017-01-08 Thread rxin

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16431
  
Can you update the pull request and the test cases to use the new test 
infra?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-08 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16430


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16510
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16510
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71059/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16510
  
**[Test build #71059 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71059/testReport)**
 for PR 16510 at commit 
[`f90a841`](https://github.com/apache/spark/commit/f90a8414bfc71f69b5785e689e719b6d45c45024).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16430: [SPARK-17077] [SQL] Cardinality estimation for project o...

2017-01-08 Thread rxin

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16430
  
Alright I'm going to merge this since this patch introduces test 
infrastructure that can be used by other tests. Please submit a follow-up PR to 
add more test cases.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field

2017-01-08 Thread gczsjdy

Github user gczsjdy commented on a diff in the pull request:

https://github.com/apache/spark/pull/16476#discussion_r95102912
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
 ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
 CaseWhen(cases, elseValue)
   }
 }
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 
if not found.
+ * It takes at least 2 parameters, and all parameters' types should be 
subtypes of AtomicType.
+ */
+@ExpressionDescription(
+  usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the 
str1,str2,... or 0 if not found.",
+  extended = """
+Examples:
+  > SELECT _FUNC_(10, 9, 3, 10, 4);
+   3
+  """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+  override def nullable: Boolean = false
+  override def foldable: Boolean = children.forall(_.foldable)
+
+  private lazy val ordering = 
TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+  override def checkInputDataTypes(): TypeCheckResult = {
+if (children.length <= 1) {
+  TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 
arguments")
+} else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+  TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to 
be of AtomicType")
+} else
+  TypeCheckResult.TypeCheckSuccess
+  }
+
+  override def dataType: DataType = IntegerType
+
+  override def eval(input: InternalRow): Any = {
+val target = children.head.eval(input)
+val targetDataType = children.head.dataType
+def findEqual(target: Any, params: Seq[Expression], index: Int): Int = 
{
+  params.toList match {
--- End diff --

@chenghao-intel I have added that annotation and remove the `toList`.
Still working on the rule for folding not match datatypes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field

2017-01-08 Thread gczsjdy

Github user gczsjdy commented on a diff in the pull request:

https://github.com/apache/spark/pull/16476#discussion_r95102842
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
 ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
 CaseWhen(cases, elseValue)
   }
 }
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 
if not found.
+ * It takes at least 2 parameters, and all parameters' types should be 
subtypes of AtomicType.
+ */
+@ExpressionDescription(
+  usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the 
str1,str2,... or 0 if not found.",
+  extended = """
+Examples:
+  > SELECT _FUNC_(10, 9, 3, 10, 4);
+   3
+  """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+  override def nullable: Boolean = false
+  override def foldable: Boolean = children.forall(_.foldable)
+
+  private lazy val ordering = 
TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+  override def checkInputDataTypes(): TypeCheckResult = {
+if (children.length <= 1) {
+  TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 
arguments")
+} else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+  TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to 
be of AtomicType")
+} else
+  TypeCheckResult.TypeCheckSuccess
+  }
+
+  override def dataType: DataType = IntegerType
+
+  override def eval(input: InternalRow): Any = {
+val target = children.head.eval(input)
+val targetDataType = children.head.dataType
+def findEqual(target: Any, params: Seq[Expression], index: Int): Int = 
{
+  params.toList match {
+case Nil => 0
+case head::tail if targetDataType == head.dataType
+  && head.eval(input) != null && ordering.equiv(target, 
head.eval(input)) => index
+case _ => findEqual(target, params.tail, index + 1)
+  }
+}
+if(target == null)
+  0
+else
+  findEqual(target, children.tail, 1)
+  }
+
+  protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
--- End diff --

Yes, because in `checkEvaluation` function there is : 
`checkEvaluationWithGeneratedMutableProjection`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16507: [SPARK-18903][SPARKR][BACKPORT-2.1] Add API to ge...

2017-01-08 Thread felixcheung

Github user felixcheung closed the pull request at:

https://github.com/apache/spark/pull/16507


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16507: [SPARK-18903][SPARKR][BACKPORT-2.1] Add API to get Spark...

2017-01-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16507
  
merged to branch-2.1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16504: [SPARK-19126][Docs] Update Join Documentation Acr...

2017-01-08 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16504


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16504: [SPARK-19126][Docs] Update Join Documentation Across Lan...

2017-01-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16504
  
merged to master & branch-2.1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16504: [SPARK-19126][Docs] Update Join Documentation Across Lan...

2017-01-08 Thread felixcheung

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16504
  
@anabranch the failure is not from Jenkins but from (R only) AppVeyor 
tests. It looks like it is nothing related - it failed at access github.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16510: [SPARK-19130][SPARKR] Support setting literal value as c...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16510
  
**[Test build #71059 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71059/testReport)**
 for PR 16510 at commit 
[`f90a841`](https://github.com/apache/spark/commit/f90a8414bfc71f69b5785e689e719b6d45c45024).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16510: [SPARK-19130][SPARKR] Support setting literal val...

2017-01-08 Thread felixcheung

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/16510

[SPARK-19130][SPARKR] Support setting literal value as column implicitly

## What changes were proposed in this pull request?

```
df$foo <- 1
```

instead of
```
df$foo <- lit(1)
```

## How was this patch tested?

unit tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rlitcol

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16510.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16510


commit f90a8414bfc71f69b5785e689e719b6d45c45024
Author: Felix Cheung 
Date:   2017-01-09T04:22:15Z

setting literal value




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16432
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71056/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16432
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16432
  
**[Test build #71056 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71056/testReport)**
 for PR 16432 at commit 
[`3b4c4b0`](https://github.com/apache/spark/commit/3b4c4b0bed43a04fbbb1f3de7e45e34f1007a222).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16432
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71054/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16432
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16501: [WIP][SPARK-19117][TESTS] Skip the tests using script tr...

2017-01-08 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16501
  
Build started: [TESTS] `org.apache.spark.sql.hive.execution.HiveQuerySuite` 
[![PR-16501](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=072A48DF-9BFF-488E-9510-4FE37B211F68=true)](https://ci.appveyor.com/project/spark-test/spark/branch/072A48DF-9BFF-488E-9510-4FE37B211F68)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16432
  
**[Test build #71054 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71054/testReport)**
 for PR 16432 at commit 
[`86c76a6`](https://github.com/apache/spark/commit/86c76af0bdb4fda15e420d1103dffcb375e3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16501: [WIP][SPARK-19117][TESTS] Skip the tests using script tr...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16501
  
**[Test build #71058 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71058/testReport)**
 for PR 16501 at commit 
[`0f654f3`](https://github.com/apache/spark/commit/0f654f347627d18a65d748a2ec9d60023568a531).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15539: [SPARK-17994] [SQL] Add back a file status cache for cat...

2017-01-08 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15539
  
Yeah, I think we should document the behavior issues when different 
sessions are using different conf values. Will do it. I think we also need to 
evict all the cache that are associated with the specific client when users 
make a switch. Do you think this makes sense?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16443: [SPARK-19042] spark executor can't download the jars whe...

2017-01-08 Thread hustfxj

Github user hustfxj commented on the issue:

https://github.com/apache/spark/pull/16443
  
@srowen I can't  fix the description at this PR, so I close the PR and 
recreate the new PR for this bug. You can review it at 
https://github.com/apache/spark/pull/16509.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16443: [SPARK-19042] spark executor can't download the j...

2017-01-08 Thread hustfxj

Github user hustfxj closed the pull request at:

https://github.com/apache/spark/pull/16443


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15539: [SPARK-17994] [SQL] Add back a file status cache for cat...

2017-01-08 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15539
  
`(ClientId, Path), Array[FileStatus]` uh... `FileStatusCache` does not 
share any entries with any other client, but does share memory resources for 
the purpose of cache eviction.

Sorry, I misunderstood the code. I think we need to evict all the cache 
that are associated with the specific client when users make a switch. Do you 
think this makes sense?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16509: [SPARK-19042] spark executor can't download the jars whe...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16509
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16509: [SPARK-19042] spark executor can't download the j...

2017-01-08 Thread hustfxj

GitHub user hustfxj opened a pull request:

https://github.com/apache/spark/pull/16509

[SPARK-19042] spark executor can't download the jars when uber jar's http 
url contaâ¦



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hustfxj/spark bug

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16509.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16509


commit 1a0bd73fa891f8449e2c86f756522323426c37d9
Author: xiaojian.fxj 
Date:   2017-01-09T03:45:59Z

spark executor can't download the jars when uber jar's http url contains 
query string




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16233: [SPARK-18801][SQL] Support resolve a nested view

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16233
  
**[Test build #71057 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71057/testReport)**
 for PR 16233 at commit 
[`ad8957c`](https://github.com/apache/spark/commit/ad8957cbdb408e3c18f65f1bff5d14df447bd39c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16474
  
**[Test build #71055 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71055/testReport)**
 for PR 16474 at commit 
[`6b562eb`](https://github.com/apache/spark/commit/6b562eba8b7d2a508cfd6f972bc18a108c3ed044).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16432
  
**[Test build #71056 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71056/testReport)**
 for PR 16432 at commit 
[`3b4c4b0`](https://github.com/apache/spark/commit/3b4c4b0bed43a04fbbb1f3de7e45e34f1007a222).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16508: [spark-18806] [core] the processors DriverWrapper and Co...

2017-01-08 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16508
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16508: [spark-18806] [core] the processors DriverWrapper...

2017-01-08 Thread liujianhuiouc

GitHub user liujianhuiouc opened a pull request:

https://github.com/apache/spark/pull/16508

[spark-18806] [core] the processors DriverWrapper  and 
CoarseGrainedExecutorBackend should be exit when worker exit

## What changes were proposed in this pull request?

modify the method `isWorker` by adding hostname equals 

## How was this patch tested?

test by manual

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/liujianhuiouc/spark-1 spark-18806

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16508.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16508


commit 665921411606fcbad40de1b7f571aa323c5a0fa7
Author: liujianhui 
Date:   2017-01-09T03:20:14Z

[spark-18806] whether the remote address is worker according to  the ip




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16432: [SPARK-19021][YARN] Generailize HDFSCredentialProvider t...

2017-01-08 Thread jerryshao

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/16432
  
@tgravescs thanks for your comments, I also tested with webhdfs with 
configuration like:

```
--conf spark.yarn.access.namenodes=webhdfs://c6402.ambari.apache.org:50070/
```

And the related logs are:

```
17/01/09 03:15:49 INFO HadoopFSCredentialProvider: getting token for: 
webhdfs://c6402.ambari.apache.org:50070/
17/01/09 03:15:50 INFO HadoopFSCredentialProvider: getting token for: 
hdfs://c6402.ambari.apache.org:8020/user/spark
17/01/09 03:15:50 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 138 
for spark on 192.168.64.102:8020
17/01/09 03:15:50 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 140 
for spark on 192.168.64.102:8020
17/01/09 03:15:50 INFO HadoopFSCredentialProvider: Renewal interval is 
86400112 for token WEBHDFS delegation
17/01/09 03:15:50 INFO HadoopFSCredentialProvider: Renewal interval is 
86400146 for token HDFS_DELEGATION_TOKEN
```

As you could see also get tokens and token renewal interval from webhdfs 
token.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16474: [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parq...

2017-01-08 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16474
  
**[Test build #71053 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71053/testReport)**
 for PR 16474 at commit 
[`d6878e1`](https://github.com/apache/spark/commit/d6878e1087eb9d7c32c1084e908a967326c75087).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

1 2 3 >

1 - 100 of 277 matches

Mail list logo