[GitHub] spark issue #19294: [SPARK-21549][CORE] Respect OutputFormats with no output...

2017-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19294 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19294: [SPARK-21549][CORE] Respect OutputFormats with no output...

2017-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19294 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82131/ Test FAILed. ---

[GitHub] spark issue #19294: [SPARK-21549][CORE] Respect OutputFormats with no output...

2017-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19294 **[Test build #82131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82131/testReport)** for PR 19294 at commit [`3429de5`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #19294: [SPARK-21549][CORE] Respect OutputFormats with no output...

2017-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19294 **[Test build #82131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82131/testReport)** for PR 19294 at commit [`3429de5`](https://github.com/apache/spark/commit/34

[GitHub] spark issue #19290: [SPARK-22063][R] Upgrades lintr to latest commit sha1 ID

2017-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19290 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82129/ Test PASSed. ---

[GitHub] spark issue #19290: [SPARK-22063][R] Upgrades lintr to latest commit sha1 ID

2017-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19290 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19290: [SPARK-22063][R] Upgrades lintr to latest commit sha1 ID

2017-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19290 **[Test build #82129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82129/testReport)** for PR 19290 at commit [`ee7eb9d`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #19290: [SPARK-22063][R] Upgrades lintr to latest commit sha1 ID

2017-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19290 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19290: [SPARK-22063][R] Upgrades lintr to latest commit sha1 ID

2017-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19290 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82128/ Test PASSed. ---

[GitHub] spark issue #19290: [SPARK-22063][R] Upgrades lintr to latest commit sha1 ID

2017-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19290 **[Test build #82128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82128/testReport)** for PR 19290 at commit [`387228d`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...

2017-09-24 Thread sathiyapk
Github user sathiyapk commented on the issue: https://github.com/apache/spark/pull/19295 I pushed a new commit that addresses @wzhfy review comments.. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.or

[GitHub] spark pull request #19294: [SPARK-21549][CORE] Respect OutputFormats with no...

2017-09-24 Thread szhem
Github user szhem commented on a diff in the pull request: https://github.com/apache/spark/pull/19294#discussion_r140654204 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -57,6 +57,11 @@ class HadoopMapReduceCommitProtocol(job

[GitHub] spark issue #19294: [SPARK-21549][CORE] Respect OutputFormats with no output...

2017-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19294 **[Test build #82130 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82130/testReport)** for PR 19294 at commit [`ae0ba0a`](https://github.com/apache/spark/commit/ae

[GitHub] spark issue #19317: [SPARK-22098][CORE] Add new method aggregateByKeyLocally...

2017-09-24 Thread WeichenXu123
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/19317 @ConeyLiu Yes tree aggregate introduce extra shuffle. But it is possible to improve perf when driver total collecting data size from executors are large and there're many partitions. But I

[GitHub] spark issue #18747: [WIP][SPARK-20822][SQL] Generate code to directly get va...

2017-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18747 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #18747: [WIP][SPARK-20822][SQL] Generate code to directly get va...

2017-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18747 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82127/ Test PASSed. ---

[GitHub] spark issue #18747: [WIP][SPARK-20822][SQL] Generate code to directly get va...

2017-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18747 **[Test build #82127 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82127/testReport)** for PR 18747 at commit [`c476e87`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #19324: [SPARK-22103] Move HashAggregateExec parent consume to a...

2017-09-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19324 @juliuszsompolski Thanks for pinging me. #18931 is an attempt to separate the consume function as it can as possible. With long chain of any operators, you can have a long consume function a

[GitHub] spark pull request #19295: [SPARK-22080][SQL] Adds support for allowing user...

2017-09-24 Thread sathiyapk
Github user sathiyapk commented on a diff in the pull request: https://github.com/apache/spark/pull/19295#discussion_r140652720 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ExperimentalMethods.scala --- @@ -44,11 +44,14 @@ class ExperimentalMethods private[sql]() {

[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...

2017-09-24 Thread sathiyapk
Github user sathiyapk commented on the issue: https://github.com/apache/spark/pull/19295 @gatorsmile thanks for your comments. Here are my thoughts, thanks for correcting me if i'm wrong. (sorry for the big comment though :)) 1. This PR don't change any existing API, it adds a new

[GitHub] spark pull request #19294: [SPARK-21549][CORE] Respect OutputFormats with no...

2017-09-24 Thread szhem
Github user szhem commented on a diff in the pull request: https://github.com/apache/spark/pull/19294#discussion_r140652214 --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala --- @@ -130,17 +135,21 @@ class HadoopMapReduceCommitProtocol(

[GitHub] spark issue #19290: [SPARK-22063][R] Upgrades lintr to latest commit sha1 ID

2017-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19290 Hi @shaneknapp. I am sorry but it's me again ... Here, this PR tries to upgrade an R package, [lintr](https://github.com/jimhester/lintr) for static Code analysis for R, which is ran vi

[GitHub] spark pull request #19311: [SPARK-22083][CORE] Release locks in MemoryStore....

2017-09-24 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19311#discussion_r140651741 --- Diff: core/src/test/scala/org/apache/spark/storage/MemoryStoreSuite.scala --- @@ -407,4 +407,119 @@ class MemoryStoreSuite }) assert(m

[GitHub] spark pull request #19311: [SPARK-22083][CORE] Release locks in MemoryStore....

2017-09-24 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19311#discussion_r140651608 --- Diff: core/src/test/scala/org/apache/spark/storage/MemoryStoreSuite.scala --- @@ -407,4 +407,119 @@ class MemoryStoreSuite }) assert(m

[GitHub] spark pull request #19311: [SPARK-22083][CORE] Release locks in MemoryStore....

2017-09-24 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/19311#discussion_r140651513 --- Diff: core/src/test/scala/org/apache/spark/storage/MemoryStoreSuite.scala --- @@ -407,4 +407,119 @@ class MemoryStoreSuite }) assert(m

[GitHub] spark issue #19290: [SPARK-22063][R] Upgrades lintr to latest commit sha1 ID

2017-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19290 Ugh.. it failed to install due to permission issue ... ``` Downloading GitHub repo jimhester/lintr@5431140 from URL https://api.github.com/repos/jimhester/lintr/zipball/5431140

[GitHub] spark issue #19290: [SPARK-22063][R] Upgrades lintr to latest commit sha1 ID

2017-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19290 **[Test build #82129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82129/testReport)** for PR 19290 at commit [`ee7eb9d`](https://github.com/apache/spark/commit/ee

[GitHub] spark pull request #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest co...

2017-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19290#discussion_r140651399 --- Diff: R/pkg/R/column.R --- @@ -238,8 +238,10 @@ setMethod("between", signature(x = "Column"), #' @param x a Column. #' @param dataType a ch

[GitHub] spark pull request #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest co...

2017-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19290#discussion_r140651387 --- Diff: dev/lint-r.R --- @@ -24,10 +24,16 @@ if (! library(SparkR, lib.loc = LOCAL_LIB_LOC, logical.return = TRUE)) { stop("You should install

[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...

2017-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19290 **[Test build #82128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82128/testReport)** for PR 19290 at commit [`387228d`](https://github.com/apache/spark/commit/38

[GitHub] spark issue #18747: [WIP][SPARK-20822][SQL] Generate code to directly get va...

2017-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18747 **[Test build #82127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82127/testReport)** for PR 18747 at commit [`c476e87`](https://github.com/apache/spark/commit/c4

[GitHub] spark issue #19184: [SPARK-21971][CORE] Too many open files in Spark due to ...

2017-09-24 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/19184 @viirya @jerryshao To take a step back here. This specific issue is applicable to window operations and not to shuffle. In shuffle, you a much larger volume of data written per file

[GitHub] spark pull request #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE...

2017-09-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19307 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19307: [SPARK-22087][SPARK-14650][WIP][BUILD][REPL][CORE] Compi...

2017-09-24 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19307 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@

[GitHub] spark pull request #19277: [SPARK-22058][CORE]the BufferedInputStream will n...

2017-09-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19277 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19326: [SPARK-22107] Change as to alias in python quickstart

2017-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19326 That's fine. I believe we don't usually need a JIRA for a trivial change though. Would you mind double checking if there are similar instances in the PySpark doc? Also, it'd be great if

[GitHub] spark pull request #19332: [SPARK-22093][TESTS] Fixes `assume` in `UtilsSuit...

2017-09-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19332 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in `UtilsSuite` and ...

2017-09-24 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19332 Merged to master. Thank you @srowen and @dongjoon-hyun. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apac

[GitHub] spark issue #19294: [SPARK-21549][CORE] Respect OutputFormats with no output...

2017-09-24 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/19294 @szhem You are correct, currently it fails in the driver itself. So failures in executor are not seen - since job submission fails. With this pr, the job submission should succeed - but t

[GitHub] spark issue #19293: [SPARK-22079][SQL] Serializer in HiveOutputWriter miss l...

2017-09-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19293 **[Test build #82126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82126/testReport)** for PR 19293 at commit [`45477fb`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #19293: [SPARK-22079][SQL] Serializer in HiveOutputWriter miss l...

2017-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19293 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82126/ Test FAILed. ---

[GitHub] spark issue #19293: [SPARK-22079][SQL] Serializer in HiveOutputWriter miss l...

2017-09-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19293 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

<    1   2