spark git commit: [SPARK-17269][SQL] Move finish analysis optimization stage into its own file

2016-08-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 9c0ac6b53 -> 94d52d765 [SPARK-17269][SQL] Move finish analysis optimization stage into its own file As part of breaking Optimizer.scala apart, this patch moves various finish analysis optimization stage rules into a single file. I'm su

spark git commit: [SPARK-17269][SQL] Move finish analysis optimization stage into its own file

2016-08-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master cc0caa690 -> dcefac438 [SPARK-17269][SQL] Move finish analysis optimization stage into its own file ## What changes were proposed in this pull request? As part of breaking Optimizer.scala apart, this patch moves various finish analysis opt

spark git commit: [SPARK-17270][SQL] Move object optimization rules into its own file

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master a6bca3ad0 -> cc0caa690 [SPARK-17270][SQL] Move object optimization rules into its own file ## What changes were proposed in this pull request? As part of breaking Optimizer.scala apart, this patch moves various Dataset object optimization

spark git commit: [SPARK-17266][TEST] Add empty strings to the regressionTests of PrefixComparatorsSuite

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 540e91280 -> a6bca3ad0 [SPARK-17266][TEST] Add empty strings to the regressionTests of PrefixComparatorsSuite ## What changes were proposed in this pull request? This PR adds a regression test to PrefixComparatorsSuite's "String prefix co

spark git commit: [SPARK-17244] Catalyst should not pushdown non-deterministic join conditions

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-2.0 dfdfc3092 -> 9c0ac6b53 [SPARK-17244] Catalyst should not pushdown non-deterministic join conditions ## What changes were proposed in this pull request? Given that non-deterministic expressions can be stateful, pushing them down the qu

spark git commit: [SPARK-17244] Catalyst should not pushdown non-deterministic join conditions

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master f64a1ddd0 -> 540e91280 [SPARK-17244] Catalyst should not pushdown non-deterministic join conditions ## What changes were proposed in this pull request? Given that non-deterministic expressions can be stateful, pushing them down the query

spark git commit: [SPARK-17235][SQL] Support purging of old logs in MetadataLog

2016-08-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 52feb3fbf -> dfdfc3092 [SPARK-17235][SQL] Support purging of old logs in MetadataLog ## What changes were proposed in this pull request? This patch adds a purge interface to MetadataLog, and an implementation in HDFSMetadataLog. The pu

spark git commit: [SPARK-17235][SQL] Support purging of old logs in MetadataLog

2016-08-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master a11d10f18 -> f64a1ddd0 [SPARK-17235][SQL] Support purging of old logs in MetadataLog ## What changes were proposed in this pull request? This patch adds a purge interface to MetadataLog, and an implementation in HDFSMetadataLog. The purge

spark git commit: [SPARK-17246][SQL] Add BigDecimal literal

2016-08-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 deb6a54cf -> 52feb3fbf [SPARK-17246][SQL] Add BigDecimal literal ## What changes were proposed in this pull request? This PR adds parser support for `BigDecimal` literals. If you append the suffix `BD` to a valid number then this will

spark git commit: [SPARK-17246][SQL] Add BigDecimal literal

2016-08-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master 8e5475be3 -> a11d10f18 [SPARK-17246][SQL] Add BigDecimal literal ## What changes were proposed in this pull request? This PR adds parser support for `BigDecimal` literals. If you append the suffix `BD` to a valid number then this will be i

[7/7] spark git commit: [SPARK-16967] move mesos to module

2016-08-26 Thread vanzin
[SPARK-16967] move mesos to module ## What changes were proposed in this pull request? Move Mesos code into a mvn module ## How was this patch tested? unit tests manually submitting a client mode and cluster mode job spark/mesos integration test suite Author: Michael Gummelt Closes #14637 fr

[3/7] spark git commit: [SPARK-16967] move mesos to module

2016-08-26 Thread vanzin
http://git-wip-us.apache.org/repos/asf/spark/blob/8e5475be/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala -- diff --git a/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/Mes

[1/7] spark git commit: [SPARK-16967] move mesos to module

2016-08-26 Thread vanzin
Repository: spark Updated Branches: refs/heads/master c0949dc94 -> 8e5475be3 http://git-wip-us.apache.org/repos/asf/spark/blob/8e5475be/project/MimaExcludes.scala -- diff --git a/project/MimaExcludes.scala b/project/MimaExclude

[5/7] spark git commit: [SPARK-16967] move mesos to module

2016-08-26 Thread vanzin
http://git-wip-us.apache.org/repos/asf/spark/blob/8e5475be/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala -- diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSc

[4/7] spark git commit: [SPARK-16967] move mesos to module

2016-08-26 Thread vanzin
http://git-wip-us.apache.org/repos/asf/spark/blob/8e5475be/dev/mima -- diff --git a/dev/mima b/dev/mima index c355349..11c4af2 100755 --- a/dev/mima +++ b/dev/mima @@ -24,7 +24,7 @@ set -e FWDIR="$(cd "`dirname "$0"`"/..; pwd)" c

[2/7] spark git commit: [SPARK-16967] move mesos to module

2016-08-26 Thread vanzin
http://git-wip-us.apache.org/repos/asf/spark/blob/8e5475be/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala -- diff --git a/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/Mesos

[6/7] spark git commit: [SPARK-16967] move mesos to module

2016-08-26 Thread vanzin
http://git-wip-us.apache.org/repos/asf/spark/blob/8e5475be/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala -- diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/Mesos

spark git commit: [SPARK-17207][MLLIB] fix comparing Vector bug in TestingUtils

2016-08-26 Thread dbtsai
Repository: spark Updated Branches: refs/heads/master 9812f7d53 -> c0949dc94 [SPARK-17207][MLLIB] fix comparing Vector bug in TestingUtils ## What changes were proposed in this pull request? fix comparing Vector bug in TestingUtils. There is the same bug for Matrix comparing. How to check the

spark git commit: [SPARK-17165][SQL] FileStreamSource should not track the list of seen files indefinitely

2016-08-26 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.0 6f82d2da3 -> deb6a54cf [SPARK-17165][SQL] FileStreamSource should not track the list of seen files indefinitely ## What changes were proposed in this pull request? Before this change, FileStreamSource uses an in-memory hash set to trac

spark git commit: [SPARK-17165][SQL] FileStreamSource should not track the list of seen files indefinitely

2016-08-26 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 261c55dd8 -> 9812f7d53 [SPARK-17165][SQL] FileStreamSource should not track the list of seen files indefinitely ## What changes were proposed in this pull request? Before this change, FileStreamSource uses an in-memory hash set to track th

spark git commit: [SPARK-17250][SQL] Remove HiveClient and setCurrentDatabase from HiveSessionCatalog

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master fd4ba3f62 -> 261c55dd8 [SPARK-17250][SQL] Remove HiveClient and setCurrentDatabase from HiveSessionCatalog ### What changes were proposed in this pull request? This is the first step to remove `HiveClient` from `HiveSessionState`. In the

spark git commit: [SPARK-17192][SQL] Issue Exception when Users Specify the Partitioning Columns without a Given Schema

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 188321623 -> fd4ba3f62 [SPARK-17192][SQL] Issue Exception when Users Specify the Partitioning Columns without a Given Schema ### What changes were proposed in this pull request? Address the comments by yhuai in the original PR: https://gi

spark git commit: [SPARKR][MINOR] Fix example of spark.naiveBayes

2016-08-26 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.0 27ed6d5dc -> 6f82d2da3 [SPARKR][MINOR] Fix example of spark.naiveBayes ## What changes were proposed in this pull request? The original example doesn't work because the features are not categorical. This PR fixes this by changing to a

spark git commit: [SPARKR][MINOR] Fix example of spark.naiveBayes

2016-08-26 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master 970ab8f6d -> 188321623 [SPARKR][MINOR] Fix example of spark.naiveBayes ## What changes were proposed in this pull request? The original example doesn't work because the features are not categorical. This PR fixes this by changing to anoth

spark git commit: [SPARK-17187][SQL][FOLLOW-UP] improve document of TypedImperativeAggregate

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 28ab17922 -> 970ab8f6d [SPARK-17187][SQL][FOLLOW-UP] improve document of TypedImperativeAggregate ## What changes were proposed in this pull request? improve the document to make it easier to understand and also mention window operator.

spark git commit: [SPARK-17260][MINOR] move CreateTables to HiveStrategies

2016-08-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 6063d5963 -> 28ab17922 [SPARK-17260][MINOR] move CreateTables to HiveStrategies ## What changes were proposed in this pull request? `CreateTables` rule turns a general `CreateTable` plan to `CreateHiveTableAsSelectCommand` for hive serde

spark git commit: [SPARK-16216][SQL][FOLLOWUP] Enable timestamp type tests for JSON and verify all unsupported types in CSV

2016-08-26 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 341e0e778 -> 6063d5963 [SPARK-16216][SQL][FOLLOWUP] Enable timestamp type tests for JSON and verify all unsupported types in CSV ## What changes were proposed in this pull request? This PR enables the tests for `TimestampType` for JSON an