[GitHub] spark issue #15131: [SPARK-17577][SparkR][Core] SparkR support add files to ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15131
  
**[Test build #65705 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65705/consoleFull)**
 for PR 15131 at commit 
[`9ed3c68`](https://github.com/apache/spark/commit/9ed3c68d752914100a86d04a420ad839a7ed992c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15131: [SPARK-17577][SparkR][Core] SparkR support add files to ...

[yanboliang]

Github user yanboliang commented on the issue:

https://github.com/apache/spark/pull/15131
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15131: [SPARK-17577][SparkR][Core] SparkR support add files to ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15131
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65698/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15131: [SPARK-17577][SparkR][Core] SparkR support add files to ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15131
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15131: [SPARK-17577][SparkR][Core] SparkR support add files to ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15131
  
**[Test build #65698 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65698/consoleFull)**
 for PR 15131 at commit 
[`9ed3c68`](https://github.com/apache/spark/commit/9ed3c68d752914100a86d04a420ad839a7ed992c).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15175: [MINOR][BUILD] Fix CheckStyle Error

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15175
  
**[Test build #65704 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65704/consoleFull)**
 for PR 15175 at commit 
[`ecefe36`](https://github.com/apache/spark/commit/ecefe36645432313e1dc9ca734b38383ce0d8e52).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15173: [SPARK-15698][SQL][Streaming][Follw-up]Fix FileSt...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15173


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15173: [SPARK-15698][SQL][Streaming][Follw-up]Fix FileStream so...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/15173
  
LGTM. Merging to master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15170: [MINOR][BUILD] Fix CheckStyle Error

[Sherry302]

Github user Sherry302 commented on the issue:

https://github.com/apache/spark/pull/15170
  
@lresende @rxin Thank for the review. I have created a 
[PR](https://github.com/apache/spark/pull/15175) to branch-2.0. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15175: [MINOR][BUILD] Fix CheckStyle Error

[Sherry302]

GitHub user Sherry302 opened a pull request:

https://github.com/apache/spark/pull/15175

[MINOR][BUILD] Fix CheckStyle Error

## What changes were proposed in this pull request?
This PR is to fix the code style errors.

## How was this patch tested?
Manual.

Before:
```
./dev/lint-java
Using `mvn` from path: /usr/local/bin/mvn
Checkstyle checks failed at following occurrences:
[ERROR] 
src/main/java/org/apache/spark/network/client/TransportClient.java:[153] 
(sizes) LineLength: Line is longer than 100 characters (found 107).
[ERROR] 
src/main/java/org/apache/spark/network/client/TransportClient.java:[196] 
(sizes) LineLength: Line is longer than 100 characters (found 108).
[ERROR] 
src/main/java/org/apache/spark/network/client/TransportClient.java:[239] 
(sizes) LineLength: Line is longer than 100 characters (found 115).
[ERROR] 
src/main/java/org/apache/spark/network/server/TransportRequestHandler.java:[119]
 (sizes) LineLength: Line is longer than 100 characters (found 107).
[ERROR] 
src/main/java/org/apache/spark/network/server/TransportRequestHandler.java:[129]
 (sizes) LineLength: Line is longer than 100 characters (found 104).
[ERROR] 
src/main/java/org/apache/spark/network/util/LevelDBProvider.java:[124,11] 
(modifier) ModifierOrder: 'static' modifier out of order with the JLS 
suggestions.
[ERROR] 
src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:[184] 
(regexp) RegexpSingleline: No trailing whitespace allowed.
[ERROR] 
src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:[304] 
(regexp) RegexpSingleline: No trailing whitespace allowed.
 ```
After:
```
./dev/lint-java
Using `mvn` from path: /usr/local/bin/mvn
Checkstyle checks passed.
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Sherry302/spark javastylefix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/15175.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15175


commit ecefe36645432313e1dc9ca734b38383ce0d8e52
Author: Weiqing Yang 
Date:   2016-09-21T05:28:13Z

[MINOR][BUILD] Fix CheckStyle Error




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14971: [SPARK-17410] [SPARK-17284] Move Hive-generated Stats In...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14971
  
**[Test build #65703 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65703/consoleFull)**
 for PR 14971 at commit 
[`7ad08fe`](https://github.com/apache/spark/commit/7ad08fe2a488fa759b4abf4e99a7206e031379d9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics

[jisookim0513]

Github user jisookim0513 commented on the issue:

https://github.com/apache/spark/pull/10212
  
@vanzin could you merge this? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15170: [MINOR][BUILD] Fix CheckStyle Error

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15170


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15170: [MINOR][BUILD] Fix CheckStyle Error

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15170
  
I have merged this in master branch.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15170: [MINOR][BUILD] Fix CheckStyle Error

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15170
  
So this pr actually doesn't merge cleanly into branch-2.0. Can you submit a 
new pr for branch-2.0?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15170: [MINOR][BUILD] Fix CheckStyle Error

[lresende]

Github user lresende commented on the issue:

https://github.com/apache/spark/pull/15170
  
Mostly style related changes
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15174: [SPARK-17502] [SQL] [Backport] [2.0] Fix Multiple Bugs i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15174
  
**[Test build #65702 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65702/consoleFull)**
 for PR 15174 at commit 
[`9efbe6b`](https://github.com/apache/spark/commit/9efbe6b37b85637fdb6d511f2def8d62cd8a749a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14959: [SPARK-17387][PYSPARK] Creating SparkContext() from pyth...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/spark/pull/14959
  
I just fix the unti test, @vanzin @davies  @holdenk  Mind to help review it 
? Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15173: [SPARK-15698][SQL][Streaming][Follw-up]Fix FileStream so...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15173
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15173: [SPARK-15698][SQL][Streaming][Follw-up]Fix FileStream so...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15173
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65697/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15173: [SPARK-15698][SQL][Streaming][Follw-up]Fix FileStream so...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15173
  
**[Test build #65697 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65697/consoleFull)**
 for PR 15173 at commit 
[`793ef77`](https://github.com/apache/spark/commit/793ef77c9625dff32ee50a8e3ded15266c10d37f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14995
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65695/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15174: [SPARK-17502] [SQL] [Backport] [2.0] Fix Multiple...

[gatorsmile]

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/15174

[SPARK-17502] [SQL] [Backport] [2.0] Fix Multiple Bugs in DDL Statements on 
Temporary Views [WIP]

### What changes were proposed in this pull request?
This PR is to backport https://github.com/apache/spark/pull/15054 to Spark 
2.0.

- When the permanent tables/views do not exist but the temporary view 
exists, the expected error should be `NoSuchTableException` for 
partition-related ALTER TABLE commands. However, it always reports a confusing 
error message. For example, 
```
Partition spec is invalid. The spec (a, b) must match the partition spec () 
defined in table '`testview`';
```
- When the permanent tables/views do not exist but the temporary view 
exists, the expected error should be `NoSuchTableException` for `ALTER TABLE 
... UNSET TBLPROPERTIES`. However, it reports a missing table property. For 
example, 
```
Attempted to unset non-existent property 'p' in table '`testView`';
```
- When `ANALYZE TABLE` is called on a view or a temporary view, we should 
issue an error message. However, it reports a strange error:
```
ANALYZE TABLE is not supported for Project
```

- When inserting into a temporary view that is generated from `Range`, we 
will get the following error message:
```
assertion failed: No plan for 'InsertIntoTable Range (0, 10, step=1, 
splits=Some(1)), false, false
+- Project [1 AS 1#20]
   +- OneRowRelation$
```

This PR is to fix the above four issues.

### How was this patch tested?
Added multiple test cases

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark PR15054Backport

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/15174.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15174


commit 9efbe6b37b85637fdb6d511f2def8d62cd8a749a
Author: gatorsmile 
Date:   2016-09-21T04:25:08Z

fix.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14995
  
Build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14995
  
**[Test build #65695 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65695/consoleFull)**
 for PR 14995 at commit 
[`5634dea`](https://github.com/apache/spark/commit/5634dea29c062415794e7bba983d7856f4719ee9).
 * This patch passes all tests.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14995
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14995
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65696/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14995
  
**[Test build #65696 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65696/consoleFull)**
 for PR 14995 at commit 
[`58d4b7c`](https://github.com/apache/spark/commit/58d4b7c164d72db820f2819faf020013a92617ee).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14959: [SPARK-17387][PYSPARK] Creating SparkContext() from pyth...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14959
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14959: [SPARK-17387][PYSPARK] Creating SparkContext() from pyth...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14959
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65699/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14959: [SPARK-17387][PYSPARK] Creating SparkContext() from pyth...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14959
  
**[Test build #65699 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65699/consoleFull)**
 for PR 14959 at commit 
[`ad47e3f`](https://github.com/apache/spark/commit/ad47e3f6013eb0d5fa365417327de659fc3ce547).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15171: [SPARK-17617][SQL] Remainder(%) expression.eval returns ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15171
  
**[Test build #65701 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65701/consoleFull)**
 for PR 15171 at commit 
[`955f49a`](https://github.com/apache/spark/commit/955f49a5bf8e99612ddce95c3a1c10ba2c1400f0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15171: [SPARK-17617][SQL] Remainder(%) expression.eval returns ...

[clockfly]

Github user clockfly commented on the issue:

https://github.com/apache/spark/pull/15171
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14917: [SPARK-17142][SQL] Complex query triggers binding error ...

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/14917
  
@hvanhovell In `CheckAnalysis` we have ensured all aggregateExpressions can 
be derived from groupingExpressions (refer to function 
`checkValidAggregateExpression`), but in optimize stage the rule 
`ReorderAssociativeOperator` broke this. I found the `Aggregate.resolved` is 
only checked in analyze stage so we failed to detect this case. So, maybe 
incorporate this rule into `Aggregate.resolved` brings little help to avoid 
similar problem. Should we check analysis again after we have performed all 
optimize rules? Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15131: [SPARK-17577][SparkR][Core] SparkR support add fi...

[yanboliang]

Github user yanboliang commented on a diff in the pull request:

https://github.com/apache/spark/pull/15131#discussion_r79755939
  
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1426,7 +1426,7 @@ class SparkContext(config: SparkConf) extends Logging 
with ExecutorAllocationCli
* supported for Hadoop-supported filesystems.
*/
   def addFile(path: String, recursive: Boolean): Unit = {
-val uri = new URI(path)
+val uri = new Path(path).toUri
--- End diff --

I agree with @HyukjinKwon . Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15131: [SPARK-17577][SparkR][Core] SparkR support add files to ...

[yanboliang]

Github user yanboliang commented on the issue:

https://github.com/apache/spark/pull/15131
  
@felixcheung I totally understand your concern about the naming, but I 
found we can not use  ```spark.getFileDir``` and ```spark.getFiles```. Since 
the ```SparkFiles``` is a term which used to describe the files added by 
```addFile``` and can be shared between driver/executors specially. Spark has 
other kinds of files or directories that can be get by users, so I don't think 
we can simplify the naming. You can refer the definition of 
[```SparkFiles```](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkFiles.scala).
 Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14644
  
**[Test build #65700 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65700/consoleFull)**
 for PR 14644 at commit 
[`3031915`](https://github.com/apache/spark/commit/30319156a9936df4da0249a1e38011ff90c2a3a6).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14644
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65700/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14644
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15059: [SPARK-17506][SQL] Improve the check double value...

[jiangxb1987]

Github user jiangxb1987 commented on a diff in the pull request:

https://github.com/apache/spark/pull/15059#discussion_r79755313
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
 ---
@@ -170,11 +170,9 @@ class ArithmeticExpressionSuite extends SparkFunSuite 
with ExpressionEvalHelper
 checkEvaluation(Remainder(positiveLongLit, positiveLongLit), 0L)
 checkEvaluation(Remainder(negativeLongLit, negativeLongLit), 0L)
 
-// TODO: the following lines would fail the test due to inconsistency 
result of interpret
--- End diff --

The result of interpret and codegen for remainder between giant values are 
equal within relative tolerance, so maybe this no longer requires to be 
resolved. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14959: [SPARK-17387][PYSPARK] Creating SparkContext() from pyth...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14959
  
**[Test build #65699 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65699/consoleFull)**
 for PR 14959 at commit 
[`ad47e3f`](https://github.com/apache/spark/commit/ad47e3f6013eb0d5fa365417327de659fc3ce547).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14644
  
**[Test build #65700 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65700/consoleFull)**
 for PR 14644 at commit 
[`3031915`](https://github.com/apache/spark/commit/30319156a9936df4da0249a1e38011ff90c2a3a6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15131: [SPARK-17577][SparkR][Core] SparkR support add files to ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15131
  
**[Test build #65698 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65698/consoleFull)**
 for PR 15131 at commit 
[`9ed3c68`](https://github.com/apache/spark/commit/9ed3c68d752914100a86d04a420ad839a7ed992c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15041: [SPARK-17488][CORE] TakeAndOrder will OOM when th...

[cenyuhai]

Github user cenyuhai commented on a diff in the pull request:

https://github.com/apache/spark/pull/15041#discussion_r79754097
  
--- Diff: core/src/main/scala/org/apache/spark/util/collection/Utils.scala 
---
@@ -30,10 +34,22 @@ private[spark] object Utils {
* Returns the first K elements from the input as defined by the 
specified implicit Ordering[T]
* and maintains the ordering.
*/
-  def takeOrdered[T](input: Iterator[T], num: Int)(implicit ord: 
Ordering[T]): Iterator[T] = {
-val ordering = new GuavaOrdering[T] {
-  override def compare(l: T, r: T): Int = ord.compare(l, r)
+  def takeOrdered[T](input: Iterator[T], num: Int,
+  ser: Serializer = SparkEnv.get.serializer)(implicit ord: 
Ordering[T]): Iterator[T] = {
+val context = TaskContext.get()
+if (context == null) {
+  val ordering = new GuavaOrdering[T] {
+override def compare(l: T, r: T): Int = ord.compare(l, r)
+  }
+  ordering.leastOf(input.asJava, num).iterator.asScala
+} else {
+  val sorter =
+new ExternalSorter[T, Any, Any](context, None, None, Some(ord), 
ser)
+  sorter.insertAll(input.map(x => (x, null)))
--- End diff --

1.In my case,  user execute a sql "select * from table sort by time limit 
1000", the k is very large, it's an extreme case. I need not change 
RDD.takeOrdered. I will limit the changes in limit.scala.
2. GuavaOrdering will sort all data in memory and then take top k. If there 
is enough memory, ExternalSorter will not spill.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15041: [SPARK-17488][CORE] TakeAndOrder will OOM when th...

[cenyuhai]

Github user cenyuhai commented on a diff in the pull request:

https://github.com/apache/spark/pull/15041#discussion_r79753174
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1384,14 +1385,15 @@ abstract class RDD[T: ClassTag](
* @param ord the implicit ordering for T
* @return an array of top elements
*/
-  def takeOrdered(num: Int)(implicit ord: Ordering[T]): Array[T] = 
withScope {
+  def takeOrdered(num: Int, ser: Serializer = SparkEnv.get.serializer)
--- End diff --

Yes, you are right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15102
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65694/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15102
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12819: [SPARK-14077][ML] Refactor NaiveBayes to support ...

[zhengruifeng]

Github user zhengruifeng commented on a diff in the pull request:

https://github.com/apache/spark/pull/12819#discussion_r79752969
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/classification/NaiveBayesSuite.scala 
---
@@ -150,6 +150,75 @@ class NaiveBayesSuite extends SparkFunSuite with 
MLlibTestSparkContext with Defa
 validateProbabilities(featureAndProbabilities, model, "multinomial")
   }
 
+  test("Naive Bayes Multinomial with weighted samples") {
+val (dataset, weightedDataset) = {
+  val nPoints = 1000
+  val piArray = Array(0.5, 0.1, 0.4).map(math.log)
+  val thetaArray = Array(
+Array(0.70, 0.10, 0.10, 0.10), // label 0
+Array(0.10, 0.70, 0.10, 0.10), // label 1
+Array(0.10, 0.10, 0.70, 0.10) // label 2
+  ).map(_.map(math.log))
+  val pi = Vectors.dense(piArray)
+  val theta = new DenseMatrix(3, 4, thetaArray.flatten, true)
+
+  val testData = generateNaiveBayesInput(piArray, thetaArray, nPoints, 
42, "multinomial")
+
+  // Let's over-sample the label-1 samples twice, label-2 samples 
triple.
+  val data1 = testData.flatMap { case labeledPoint: LabeledPoint =>
+labeledPoint.label match {
+  case 0.0 => Iterator(labeledPoint)
+  case 1.0 => Iterator(labeledPoint, labeledPoint)
+  case 2.0 => Iterator(labeledPoint, labeledPoint, labeledPoint)
+}
+  }
+
+  val rnd = new Random(8392)
+  val data2 = testData.flatMap { case LabeledPoint(label: Double, 
features: Vector) =>
--- End diff --

Good point. Of course, I will update this testsuite keep in line with other 
algorithms.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15102
  
**[Test build #65694 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65694/consoleFull)**
 for PR 15102 at commit 
[`881b206`](https://github.com/apache/spark/commit/881b20678f84b321962085a1c9dea0a7387e21c6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #11956: [SPARK-14098][SQL] Generate Java code that gets a float/...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/11956
  
Current PR #13899 does not support a case that an element is null.
For now, I am working for the followings:
1. Support all of types
2. Make structures serializable
3. Support null cases 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14897: [SPARK-17338][SQL][WIP] add global temp view

[clockfly]

Github user clockfly commented on a diff in the pull request:

https://github.com/apache/spark/pull/14897#discussion_r79752499
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -36,6 +36,7 @@ import org.apache.spark.sql.catalyst.util.StringUtils
 
 object SessionCatalog {
   val DEFAULT_DATABASE = "default"
+  val GLOBAL_TEMP_VIEW_DATABASE = "_global_temp"
--- End diff --

Maybe change the name to "_global_temporary_views"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14897: [SPARK-17338][SQL][WIP] add global temp view

[clockfly]

Github user clockfly commented on a diff in the pull request:

https://github.com/apache/spark/pull/14897#discussion_r79752478
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -142,8 +145,12 @@ class SessionCatalog(
   // 

 
   def createDatabase(dbDefinition: CatalogDatabase, ignoreIfExists: 
Boolean): Unit = {
-val qualifiedPath = 
makeQualifiedPath(dbDefinition.locationUri).toString
 val dbName = formatDatabaseName(dbDefinition.name)
+if (dbName == GLOBAL_TEMP_VIEW_DATABASE) {
--- End diff --

Since the name is very specific "_global_temp", it is unlikly that it may 
conflict with user table's name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14897: [SPARK-17338][SQL][WIP] add global temp view

[clockfly]

Github user clockfly commented on a diff in the pull request:

https://github.com/apache/spark/pull/14897#discussion_r79752423
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -142,8 +145,12 @@ class SessionCatalog(
   // 

 
   def createDatabase(dbDefinition: CatalogDatabase, ignoreIfExists: 
Boolean): Unit = {
-val qualifiedPath = 
makeQualifiedPath(dbDefinition.locationUri).toString
 val dbName = formatDatabaseName(dbDefinition.name)
+if (dbName == GLOBAL_TEMP_VIEW_DATABASE) {
--- End diff --

Is this too restrictive? If the user ask directly for  
`GLOBAL_TEMP_VIEW_DATABASE `, may be we should allow it?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15170: [MINOR][BUILD] Fix CheckStyle Error

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15170
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65691/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15170: [MINOR][BUILD] Fix CheckStyle Error

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15170
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15170: [MINOR][BUILD] Fix CheckStyle Error

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15170
  
**[Test build #65691 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65691/consoleFull)**
 for PR 15170 at commit 
[`91995aa`](https://github.com/apache/spark/commit/91995aa12685a92d033342ccc8981ea5a6968dcb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15171: [SPARK-17617][SQL] Remainder(%) expression.eval returns ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15171
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65693/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15171: [SPARK-17617][SQL] Remainder(%) expression.eval returns ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15171
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15171: [SPARK-17617][SQL] Remainder(%) expression.eval returns ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15171
  
**[Test build #65693 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65693/consoleFull)**
 for PR 15171 at commit 
[`955f49a`](https://github.com/apache/spark/commit/955f49a5bf8e99612ddce95c3a1c10ba2c1400f0).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15173: [SPARK-15698][SQL][Streaming][Follw-up]Fix FileStream so...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15173
  
**[Test build #65697 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65697/consoleFull)**
 for PR 15173 at commit 
[`793ef77`](https://github.com/apache/spark/commit/793ef77c9625dff32ee50a8e3ded15266c10d37f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15173: [SPARK-15698][SQL][Streaming][Follw-up]Fix FileSt...

[jerryshao]

GitHub user jerryshao opened a pull request:

https://github.com/apache/spark/pull/15173

[SPARK-15698][SQL][Streaming][Follw-up]Fix FileStream source and sink log 
get configuration issue

## What changes were proposed in this pull request?

This issue was introduced in the previous commit of SPARK-15698. Mistakenly 
change the way to get configuration back to original one, so here with the 
follow up PR to revert them up.


## How was this patch tested?

N/A

Ping @zsxwing , please review again, sorry to bring the inconvenience. 
Thanks a lot.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerryshao/apache-spark SPARK-15698-follow

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/15173.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15173


commit 793ef77c9625dff32ee50a8e3ded15266c10d37f
Author: jerryshao 
Date:   2016-09-21T02:12:54Z

Fix FileStream source and sink log get configuration issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14762: [SPARK-16962][CORE][SQL] Fix misaligned record accesses ...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14762
  
"The argument about speed isn't about dead code but that aligned access 
ought to be faster even on x86." -- was this validated at all?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15166: [SPARK-17513][SQL] Make StreamExecution garbage-c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15166


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15120: [SPARK-4563][core] Allow driver to advertise a different...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15120
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15120: [SPARK-4563][core] Allow driver to advertise a different...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15120
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65692/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15166: [SPARK-17513][SQL] Make StreamExecution garbage-collect ...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15166
  
merging in master/2.0. Thanks.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15172: [Spark-13331][network] Spark RPC encryption optimization

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15172
  
Can you change the title to say "AES support for over-the-wire encryption"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15120: [SPARK-4563][core] Allow driver to advertise a different...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15120
  
**[Test build #65692 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65692/consoleFull)**
 for PR 15120 at commit 
[`5137131`](https://github.com/apache/spark/commit/513713144ef36351663ec03290ec09dc99f5c01c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15090: [SPARK-17073] [SQL] generate column-level statistics

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15090
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65690/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15090: [SPARK-17073] [SQL] generate column-level statistics

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15090
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15090: [SPARK-17073] [SQL] generate column-level statistics

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15090
  
**[Test build #65690 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65690/consoleFull)**
 for PR 15090 at commit 
[`392a312`](https://github.com/apache/spark/commit/392a312dd3dd606d6cfa9f96af246d718fc0fb42).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15059: [SPARK-17506][SQL] Improve the check double value...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/15059#discussion_r79749191
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
 ---
@@ -170,11 +170,9 @@ class ArithmeticExpressionSuite extends SparkFunSuite 
with ExpressionEvalHelper
 checkEvaluation(Remainder(positiveLongLit, positiveLongLit), 0L)
 checkEvaluation(Remainder(negativeLongLit, negativeLongLit), 0L)
 
-// TODO: the following lines would fail the test due to inconsistency 
result of interpret
--- End diff --

this TODO is not fixed yet, why remove it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14995
  
**[Test build #65696 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65696/consoleFull)**
 for PR 14995 at commit 
[`58d4b7c`](https://github.com/apache/spark/commit/58d4b7c164d72db820f2819faf020013a92617ee).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14995: [Test Only][SPARK-6235][CORE]Address various 2G limits

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14995
  
**[Test build #65695 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65695/consoleFull)**
 for PR 14995 at commit 
[`5634dea`](https://github.com/apache/spark/commit/5634dea29c062415794e7bba983d7856f4719ee9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15171: [SPARK-17617][SQL] Remainder(%) expression.eval returns ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/15171
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13513: [SPARK-15698][SQL][Streaming] Add the ability to ...

[jerryshao]

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/13513#discussion_r79747197
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSinkLog.scala
 ---
@@ -79,213 +76,46 @@ object SinkFileStatus {
  * When the reader uses `allFiles` to list all files, this method only 
returns the visible files
  * (drops the deleted files).
  */
-class FileStreamSinkLog(sparkSession: SparkSession, path: String)
-  extends HDFSMetadataLog[Array[SinkFileStatus]](sparkSession, path) {
-
-  import FileStreamSinkLog._
+class FileStreamSinkLog(
+metadataLogVersion: String,
+sparkSession: SparkSession,
+path: String)
+  extends CompactibleFileStreamLog[SinkFileStatus](metadataLogVersion, 
sparkSession, path) {
 
   private implicit val formats = Serialization.formats(NoTypeHints)
 
-  /**
-   * If we delete the old files after compaction at once, there is a race 
condition in S3: other
-   * processes may see the old files are deleted but still cannot see the 
compaction file using
-   * "list". The `allFiles` handles this by looking for the next 
compaction file directly, however,
-   * a live lock may happen if the compaction happens too frequently: one 
processing keeps deleting
-   * old files while another one keeps retrying. Setting a reasonable 
cleanup delay could avoid it.
-   */
-  private val fileCleanupDelayMs = 
sparkSession.sessionState.conf.fileSinkLogCleanupDelay
+  protected override val fileCleanupDelayMs =
--- End diff --

Oh, sorry about it, will fix it now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15172: [Spark-13331][network] Spark RPC encryption optimization

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15172
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15169: [SPARK-17613] S3A base paths with no '/' at the end retu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15169
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65689/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15169: [SPARK-17613] S3A base paths with no '/' at the end retu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15169
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15169: [SPARK-17613] S3A base paths with no '/' at the end retu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15169
  
**[Test build #65689 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65689/consoleFull)**
 for PR 15169 at commit 
[`f5a9c12`](https://github.com/apache/spark/commit/f5a9c12536a16e373c33b8e84bcd07126b77ccaf).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15172: [Spark-13331][network] Spark RPC encryption optim...

[winningsix]

Github user winningsix commented on a diff in the pull request:

https://github.com/apache/spark/pull/15172#discussion_r79746570
  
--- Diff: common/network-common/pom.xml ---
@@ -97,6 +97,11 @@
   slf4j-log4j12
   test
 
+
--- End diff --

Can we move this dependency to L62 since the dependency around is test 
only. You can see the comment in L64 and L80


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14600: [SPARK-15899] [SQL] Fix the construction of the file pat...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/14600
  
@srowen Do you mind if I ask this is supposed to be closed (as it is merged 
into 2.0)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15172: [Spark-13331][network] Spark RPC encryption optim...

[winningsix]

Github user winningsix commented on a diff in the pull request:

https://github.com/apache/spark/pull/15172#discussion_r79746732
  
--- Diff: 
common/network-common/src/main/java/org/apache/spark/network/sasl/SaslServerBootstrap.java
 ---
@@ -29,7 +29,6 @@
  * authentication.
  */
 public class SaslServerBootstrap implements TransportServerBootstrap {
-
--- End diff --

No need for the changes in this file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15172: [Spark-13331][network] Spark RPC encryption optim...

[cjjnjust]

GitHub user cjjnjust reopened a pull request:

https://github.com/apache/spark/pull/15172

[Spark-13331][network] Spark RPC encryption optimization

## What changes were proposed in this pull request?

DIGEST-MD5 mechanism is used for SASL authentication and secure 
communication. DIGEST-MD5 mechanism supports 3DES, DES, and RC4 ciphers. 
However, 3DES, DES and RC4 are slow relatively.

AES provide better performance and security by design and is a replacement 
for 3DES according to NIST. Apache Common Crypto is a cryptographic library 
optimized with AES-NI, this patch employ Apache Common Crypto as enc/dec 
backend for SASL authentication and secure channel to improve spark RPC.

## How was this patch tested?

Unit tests and Integration test.






You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cjjnjust/spark shuffle_rpc_encrypt

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/15172.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15172


commit 78f4d7bcfb5830fa54b1380182f08a85fee97d92
Author: Junjie Chen 
Date:   2016-09-20T17:57:05Z

[Spark-13331] Spark RPC encryption optimization

DIGEST-MD5 mechanism is used for SASL authentication and secure 
communication. DIGEST-MD5 mechanism supports 3DES, DES, and RC4 ciphers. 
However, 3DES, DES and RC4 are slow relatively.

AES provide better performance and security by design and is a replacement 
for 3DES according to NIST. Apache Common Crypto is a cryptographic library 
optimized with AES-NI, this patch employ Apache Common Crypto as enc/dec 
backend for SASL authentication and secure channel to improve spark RPC.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15172: [Spark-13331][network] Spark RPC encryption optim...

[cjjnjust]

Github user cjjnjust closed the pull request at:

https://github.com/apache/spark/pull/15172


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12693: [SPARK-14914] Fix Resource not closed after using, mostl...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/12693
  
@taoli91 Hi. Maybe we could close this and 
https://github.com/apache/spark/pull/12696 and then start to resolve 
https://issues.apache.org/jira/browse/SPARK-17591 first step by step? If we 
could verify the tests, I guess it'd be faster to merge each PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15172: [Spark-13331][network] Spark RPC encryption optimization

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15172
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15168: [SPARK-17612][SQL] Support `DESCRIBE table PARTIT...

[skambha]

Github user skambha commented on a diff in the pull request:

https://github.com/apache/spark/pull/15168#discussion_r79746339
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 
---
@@ -341,6 +342,25 @@ class SQLQuerySuite extends QueryTest with 
SQLTestUtils with TestHiveSingleton {
 }
   }
 
+  test("describe partition") {
+withTable("partitioned_table") {
+  sql("CREATE TABLE partitioned_table (a STRING, b INT) PARTITIONED BY 
(c STRING, d STRING)")
+  sql("ALTER TABLE partitioned_table ADD PARTITION (c='Us', d=1)")
+
+  sql("DESC partitioned_table PARTITION (c='Us', d=1)")
--- End diff --

Hive supports describe based on the partition specified and will list the 
details for the particular partition when used with formatted or extended 
option. 
DESCRIBE formatted part_table partition (d='abc') 

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Describe

@dongjoon-hyun, this might be beyond the scope of this PR but this would be 
useful if there are a lot of partitions and we want to find details for a given 
partition.  What do you think? Thanks.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15172: [Spark-13331][network] Spark RPC encryption optim...

[cjjnjust]

GitHub user cjjnjust opened a pull request:

https://github.com/apache/spark/pull/15172

[Spark-13331][network] Spark RPC encryption optimization

## What changes were proposed in this pull request?

DIGEST-MD5 mechanism is used for SASL authentication and secure 
communication. DIGEST-MD5 mechanism supports 3DES, DES, and RC4 ciphers. 
However, 3DES, DES and RC4 are slow relatively.

AES provide better performance and security by design and is a replacement 
for 3DES according to NIST. Apache Common Crypto is a cryptographic library 
optimized with AES-NI, this patch employ Apache Common Crypto as enc/dec 
backend for SASL authentication and secure channel to improve spark RPC.

## How was this patch tested?

Unit tests and Integration test.






You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cjjnjust/spark shuffle_rpc_encrypt

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/15172.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15172


commit 78f4d7bcfb5830fa54b1380182f08a85fee97d92
Author: Junjie Chen 
Date:   2016-09-20T17:57:05Z

[Spark-13331] Spark RPC encryption optimization

DIGEST-MD5 mechanism is used for SASL authentication and secure 
communication. DIGEST-MD5 mechanism supports 3DES, DES, and RC4 ciphers. 
However, 3DES, DES and RC4 are slow relatively.

AES provide better performance and security by design and is a replacement 
for 3DES according to NIST. Apache Common Crypto is a cryptographic library 
optimized with AES-NI, this patch employ Apache Common Crypto as enc/dec 
backend for SASL authentication and secure channel to improve spark RPC.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12696: [SPARK-14914] Skip some test cases on Windows due to lim...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/12696
  
@taoli91 Would you be interested in 
https://issues.apache.org/jira/browse/SPARK-17591?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12696: [SPARK-14914] Skip some test cases on Windows due to lim...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/12696
  
@taoli91 Do you mind if I ask rebase this? I would like to support to 
verify this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14897: [SPARK-17338][SQL][WIP] add global temp view

[clockfly]

Github user clockfly commented on a diff in the pull request:

https://github.com/apache/spark/pull/14897#discussion_r79744373
  
--- Diff: 
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -111,7 +111,8 @@ statement
 | ALTER TABLE tableIdentifier RECOVER PARTITIONS   
#recoverPartitions
 | DROP TABLE (IF EXISTS)? tableIdentifier PURGE?   
#dropTable
 | DROP VIEW (IF EXISTS)? tableIdentifier   
#dropTable
-| CREATE (OR REPLACE)? TEMPORARY? VIEW (IF NOT EXISTS)? tableIdentifier
+| CREATE (OR REPLACE)? ((LOCAL | GLOBAL)? TEMPORARY)?
--- End diff --

Maybe not needing `LOCAL` key word?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14897: [SPARK-17338][SQL][WIP] add global temp view

[clockfly]

Github user clockfly commented on a diff in the pull request:

https://github.com/apache/spark/pull/14897#discussion_r79744336
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/ListTablesSuite.scala ---
@@ -33,7 +33,7 @@ class ListTablesSuite extends QueryTest with 
TestHiveSingleton with BeforeAndAft
   override def beforeAll(): Unit = {
 super.beforeAll()
 // The catalog in HiveContext is a case insensitive one.
-sessionState.catalog.createTempView(
+sessionState.catalog.createLocalTempView(
--- End diff --

Can we avoid renaming createLocalTempView?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/10212
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/10212
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65686/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #10212: [SPARK-12221] add cpu time to metrics

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/10212
  
**[Test build #65686 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65686/consoleFull)**
 for PR 10212 at commit 
[`2ff5bdc`](https://github.com/apache/spark/commit/2ff5bdce904f4ca55096ea4616557342003b8043).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15102
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15102
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65685/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15131: [SPARK-17577][SparkR][Core] SparkR support add fi...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/15131#discussion_r79743521
  
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1426,7 +1426,7 @@ class SparkContext(config: SparkConf) extends Logging 
with ExecutorAllocationCli
* supported for Hadoop-supported filesystems.
*/
   def addFile(path: String, recursive: Boolean): Unit = {
-val uri = new URI(path)
+val uri = new Path(path).toUri
--- End diff --

I just check the documentation and tests. It seems Windows paths are being 
tested in 
https://github.com/apache/hadoop/blob/branch-2.7.2/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestPath.java
 as `toString` it based on `URI` instance in `Path` which `toUri` will return 
directly, I guess it'd be safe.

I could not find the explicit mention about Windows path in the 
documenation https://hadoop.apache.org/docs/stable2/api/index.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15102
  
**[Test build #65685 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65685/consoleFull)**
 for PR 15102 at commit 
[`6bc2994`](https://github.com/apache/spark/commit/6bc2994f98567127137aee54a5141d98337986c5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  sealed trait ConsumerStrategy `
  * `  case class SubscribeStrategy(topics: Seq[String], kafkaParams: 
ju.Map[String, Object])`
  * `  case class SubscribePatternStrategy(`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org