[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15575 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15575 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67319/ Test PASSed. ---

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15575 **[Test build #67319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67319/consoleFull)** for PR 15575 at commit

[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-20 Thread wangmiao1981
Github user wangmiao1981 commented on a diff in the pull request: https://github.com/apache/spark/pull/15541#discussion_r84422357 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala --- @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #15568: [SPARK-18028][SQL] simplify TableFileCatalog

2016-10-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15568 @mallman , item 4 is a potential problem in the future. The current workflow is, we get the `MetastoreRelation` via `HiveMetastoreCatalog.lookupRelation`, which always lower case the database and

[GitHub] spark issue #14847: [SPARK-17254][SQL] Add StopAfter physical plan for the f...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14847 **[Test build #67324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67324/consoleFull)** for PR 14847 at commit

[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-20 Thread wangmiao1981
Github user wangmiao1981 commented on a diff in the pull request: https://github.com/apache/spark/pull/15541#discussion_r84422079 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala --- @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...

2016-10-20 Thread zhzhan
Github user zhzhan commented on the issue: https://github.com/apache/spark/pull/15541 @gatorsmile I didn't see your new comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15541: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-20 Thread wangmiao1981
Github user wangmiao1981 commented on a diff in the pull request: https://github.com/apache/spark/pull/15541#discussion_r84421949 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala --- @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #13891: [SPARK-6685][MLLIB]Use DSYRK to compute AtA in ALS

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13891 **[Test build #67323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67323/consoleFull)** for PR 13891 at commit

[GitHub] spark pull request #15568: [SPARK-18028][SQL] simplify TableFileCatalog

2016-10-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15568#discussion_r84421537 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/TableFileCatalog.scala --- @@ -102,6 +95,13 @@ class TableFileCatalog(

[GitHub] spark issue #15319: [SPARK-17733][SQL] InferFiltersFromConstraints rule neve...

2016-10-20 Thread sameeragarwal
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/15319 Thanks @jiangxb1987, this equivalence class approach looks pretty solid. I'll take a closer look tomorrow! --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #13891: [SPARK-6685][MLLIB]Use DSYRK to compute AtA in ALS

2016-10-20 Thread hqzizania
Github user hqzizania commented on the issue: https://github.com/apache/spark/pull/13891 @yanboliang So sorry for my late response. Some regression performance test results: Datasets: using

[GitHub] spark issue #15576: [SPARK-17674][SPARKR] check for warning in test output

2016-10-20 Thread wangmiao1981
Github user wangmiao1981 commented on the issue: https://github.com/apache/spark/pull/15576 Thanks for fixing this! I encountered this issue before. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...

2016-10-20 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15541 Accidentally, I deleted all my comments. You might need to check the emails to find all my comments. :) --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #15579: Added support for extra command in front of spark.

2016-10-20 Thread sheepduke
Github user sheepduke commented on the issue: https://github.com/apache/spark/pull/15579 This is rather useful sometimes because you wan tto add some extra tuning arguments like 'numactl'. Otherwise it is not even possible to achieve that. Yes it only works with YARN for

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15575 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15575 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67318/ Test PASSed. ---

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15575 **[Test build #67318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67318/consoleFull)** for PR 15575 at commit

[GitHub] spark issue #15579: Added support for extra command in front of spark.

2016-10-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15579 Please also review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark Thanks a lot! --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #15579: Added support for extra command in front of spark.

2016-10-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15579 This seems really weird to do (and also only works in YARN). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #15576: [SPARK-17674][SPARKR] check for warning in test output

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15576 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67321/ Test PASSed. ---

[GitHub] spark issue #15576: [SPARK-17674][SPARKR] check for warning in test output

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15576 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15576: [SPARK-17674][SPARKR] check for warning in test output

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15576 **[Test build #67321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67321/consoleFull)** for PR 15576 at commit

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15513 Oh, no. I will try to test each when writing the documentation. Please ignore minor incorrectness here. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15513 Do we have binary literals anyway? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15513 Then I will do as below: **Literal only** ``` a string literal. a numeric literal that defines ... a binary literal that represents ... For example, ... ```

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15513 Sure, sounds great. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15579: Added support for extra command in front of spark.

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15579 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #15579: Added support for extra command in front of spark...

2016-10-20 Thread sheepduke
GitHub user sheepduke opened a pull request: https://github.com/apache/spark/pull/15579 Added support for extra command in front of spark. ## What changes were proposed in this pull request? A minor functional change is added into yarn facility to make it possible for

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15513 How about "a string literal" vs "a string expression"? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15513 Also, will try to consolidate multiple usages and take out `_FUNC_` in extended part too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15513 @rxin, How about this? **Literal only** ``` a string type value. a numeric type value that defines ... a binary type value that defines ... For example, ... ```

[GitHub] spark issue #15551: [SPARK-18012][SQL] Simplify WriterContainer

2016-10-20 Thread tejasapatil
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/15551 @rxin : Thanks for notifying me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15569: [SPARK-18029][SQL] PruneFileSourcePartitions should not ...

2016-10-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15569 thanks for the review, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #15569: [SPARK-18029][SQL] PruneFileSourcePartitions shou...

2016-10-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15569 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15481: [SPARK-17929] [CORE] Fix deadlock when CoarseGrainedSche...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15481 **[Test build #67322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67322/consoleFull)** for PR 15481 at commit

[GitHub] spark issue #15481: [SPARK-17929] [CORE] Fix deadlock when CoarseGrainedSche...

2016-10-20 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15481 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #15576: [SPARK-17674][SPARKR] check for warning in test output

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15576 **[Test build #67321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67321/consoleFull)** for PR 15576 at commit

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread tejasapatil
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/15575 @yhuai : please see the table in this PRs description. I have added a `comment` (last column) for each entry to point out those cases. --- If your project is set up for it, you can reply to

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/15575 > I felt that there are numerous places where child's output ordering could be used but the operators don't set it Can you list them at here? --- If your project is set up for it, you can

[GitHub] spark issue #15576: [SPARK-17674][SPARKR] check for warning in test output

2016-10-20 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15576 Rebased. This should pass now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15576: [WIP][SPARK-17674][SPARKR] check for warning in test out...

2016-10-20 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15576 Test failure is intentional, it's picking up the following warnings: ``` Warnings --- 1. createDataFrame uses

[GitHub] spark pull request #15560: [SPARKR] fix warnings

2016-10-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15560 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #15578: Branch 2.0

2016-10-20 Thread wankunde
Github user wankunde closed the pull request at: https://github.com/apache/spark/pull/15578 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15560: [SPARKR] fix warnings

2016-10-20 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15560 merged to master and branch-2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15575 **[Test build #67320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67320/consoleFull)** for PR 15575 at commit

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread tejasapatil
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/15575 Agree with the planner behavior described in the last few comments (relevant code :

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15575 **[Test build #67319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67319/consoleFull)** for PR 15575 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67317/ Test PASSed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15577: [SPARK-18030][Tests] Adds more checks to collect ...

2016-10-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15577 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #67317 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67317/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #15577: [SPARK-18030][Tests] Adds more checks to collect more in...

2016-10-20 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15577 Thanks! Merging to master and 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15513 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15513 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67314/ Test PASSed. ---

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15513 **[Test build #67314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67314/consoleFull)** for PR 15513 at commit

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15513 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67315/ Test PASSed. ---

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15513 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15513 **[Test build #67315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67315/consoleFull)** for PR 15513 at commit

[GitHub] spark issue #15541: [SPARK-17637][Scheduler]Packed scheduling for Spark task...

2016-10-20 Thread zhzhan
Github user zhzhan commented on the issue: https://github.com/apache/spark/pull/15541 @rxin Can you please take a look, and let me know if you have any concern? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15513 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15513 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67313/ Test PASSed. ---

[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15513 **[Test build #67313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67313/consoleFull)** for PR 15513 at commit

[GitHub] spark issue #15578: Branch 2.0

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15578 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #15578: Branch 2.0

2016-10-20 Thread wankunde
GitHub user wankunde opened a pull request: https://github.com/apache/spark/pull/15578 Branch 2.0 You can merge this pull request into a Git repository by running: $ git pull https://github.com/wankunde/spark branch-2.0 Alternatively you can review and apply these changes as

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67316/ Test PASSed. ---

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15575 Yeah. So I don't see any exception that an unary node, if it is not `ShuffleExchange`, can have an `outputPartitioning` other than `child.outputPartitioning`. --- If your project is set up for it,

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67316/consoleFull)** for PR 14957 at commit

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/15575 Our planner decides if to add an `ShuffleExchange` by consider `outputPartitioning` and `requiredDistribution` together. If the `outputPartitioning` of the child does not satisfy the

[GitHub] spark issue #15566: [SPARK-18026][SQL] should not always lowercase partition...

2016-10-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15566 Sorry I was wrong. The root cause of SPARK-17990 is that, when we write data for a hive table, we ignore the truth that hive table is not case-preserving, and create the partition directory with

[GitHub] spark pull request #15556: [SPARK-18010][Core] Reduce work performed for bui...

2016-10-20 Thread vijoshi
Github user vijoshi commented on a diff in the pull request: https://github.com/apache/spark/pull/15556#discussion_r84413566 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala --- @@ -43,38 +43,56 @@ private[spark] class ReplayListenerBus extends

[GitHub] spark pull request #13507: [SPARK-15765][SQL][Streaming] Make continuous Par...

2016-10-20 Thread lw-lin
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/13507 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #13507: [SPARK-15765][SQL][Streaming] Make continuous Parquet wr...

2016-10-20 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/13507 I'm closing this in favor of SPARK-17924, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15575 **[Test build #67311 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67311/consoleFull)** for PR 15575 at commit

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15575 The query planner will inject exchange based on the required distribution for the child of `UnaryNodeExec`. Looks like all unary nodes have `child.outputPartitioning` as its output partition. I

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15575 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67311/ Test PASSed. ---

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15575 @yhuai let me be more clear. It is actually fairly confusing to say "this only changes if the operator doesn't shuffle data". The thing is that we are relying on each operator's output

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15575 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15575 **[Test build #67318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67318/consoleFull)** for PR 15575 at commit

[GitHub] spark pull request #15575: [SPARK-18038] [SQL] Move output partitioning defi...

2016-10-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15575#discussion_r84411683 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -96,13 +95,15 @@ trait BaseLimitExec extends UnaryExecNode with

[GitHub] spark pull request #15575: [SPARK-18038] [SQL] Move output partitioning defi...

2016-10-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15575#discussion_r84411628 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SortExec.scala --- @@ -45,6 +45,8 @@ case class SortExec( override def

[GitHub] spark issue #15481: [SPARK-17929] [CORE] Fix deadlock when CoarseGrainedSche...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15481 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67312/ Test FAILed. ---

[GitHub] spark issue #15481: [SPARK-17929] [CORE] Fix deadlock when CoarseGrainedSche...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15481 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15481: [SPARK-17929] [CORE] Fix deadlock when CoarseGrainedSche...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15481 **[Test build #67312 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67312/consoleFull)** for PR 15481 at commit

[GitHub] spark issue #15297: [SPARK-9862]Handling data skew

2016-10-20 Thread YuhuWang2002
Github user YuhuWang2002 commented on the issue: https://github.com/apache/spark/pull/15297 @tgravescs : Thank you for your response, when a single reduce task handling huge data, it's slowly and unstable. so we split one reduce task to multi- reduce task. A single reduce

[GitHub] spark issue #15575: [SPARK-18038] [SQL] Move output partitioning definition ...

2016-10-20 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/15575 `outputPartitioning` of a node will only be changed if this node shuffles data. Right now, only `ShuffleExchange` shuffles data. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #15577: [SPARK-18030][Tests] Adds more checks to collect more in...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15577 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67310/ Test PASSed. ---

[GitHub] spark issue #15577: [SPARK-18030][Tests] Adds more checks to collect more in...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15577 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15577: [SPARK-18030][Tests] Adds more checks to collect more in...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15577 **[Test build #67310 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67310/consoleFull)** for PR 15577 at commit

[GitHub] spark pull request #15575: [SPARK-18038] [SQL] Move output partitioning defi...

2016-10-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/15575#discussion_r84410677 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -96,13 +95,15 @@ trait BaseLimitExec extends UnaryExecNode with

[GitHub] spark pull request #15575: [SPARK-18038] [SQL] Move output partitioning defi...

2016-10-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/15575#discussion_r84410707 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExec.scala --- @@ -103,6 +103,8 @@ case class WindowExec( override

[GitHub] spark issue #15571: [SPARK-18034] Upgrade to MiMa 0.1.11 to fix flakiness

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15571 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67307/ Test PASSed. ---

[GitHub] spark pull request #15575: [SPARK-18038] [SQL] Move output partitioning defi...

2016-10-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/15575#discussion_r84410586 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -96,13 +95,15 @@ trait BaseLimitExec extends UnaryExecNode with

[GitHub] spark issue #15571: [SPARK-18034] Upgrade to MiMa 0.1.11 to fix flakiness

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15571 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15571: [SPARK-18034] Upgrade to MiMa 0.1.11 to fix flakiness

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15571 **[Test build #67307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67307/consoleFull)** for PR 15571 at commit

[GitHub] spark pull request #15575: [SPARK-18038] [SQL] Move output partitioning defi...

2016-10-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/15575#discussion_r84410162 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SortExec.scala --- @@ -45,6 +45,8 @@ case class SortExec( override def

[GitHub] spark issue #15566: [SPARK-18026][SQL] should not always lowercase partition...

2016-10-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15566 Actually SPARK-17990 can be fixed in this PR with a little more work. Let me do it now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #11119: [SPARK-10780][ML] Add an initial model to kmeans

2016-10-20 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/9 Related thought: if the model holds a pointer to its initialModel, then it will be serialized and shipped along with the model at prediction time. This will be inefficient for large models and even

  1   2   3   4   5   6   7   >