spark git commit: [SPARK-19634][ML] Multivariate summarizer - dataframes API

2017-08-15 Thread yliang
Repository: spark Updated Branches: refs/heads/master 966083105 -> 07549b20a [SPARK-19634][ML] Multivariate summarizer - dataframes API ## What changes were proposed in this pull request? This patch adds the DataFrames API to the multivariate summarizer (mean, variance, etc.). In addition

spark git commit: [SPARK-21712][PYSPARK] Clarify type error for Column.substr()

2017-08-15 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 42b9eda80 -> 966083105 [SPARK-21712][PYSPARK] Clarify type error for Column.substr() Proposed changes: * Clarify the type error that `Column.substr()` gives. Test plan: * Tested this manually. * Test code: ```python from

spark git commit: [MINOR] Fix a typo in the method name `UserDefinedFunction.asNonNullabe`

2017-08-15 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 3f958a999 -> 42b9eda80 [MINOR] Fix a typo in the method name `UserDefinedFunction.asNonNullabe` ## What changes were proposed in this pull request? The method name `asNonNullabe` should be `asNonNullable`. ## How was this patch tested?

[1/3] spark git commit: [SPARK-21731][BUILD] Upgrade scalastyle to 0.9.

2017-08-15 Thread vanzin
Repository: spark Updated Branches: refs/heads/master cba826d00 -> 3f958a999 http://git-wip-us.apache.org/repos/asf/spark/blob/3f958a99/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala --

[2/3] spark git commit: [SPARK-21731][BUILD] Upgrade scalastyle to 0.9.

2017-08-15 Thread vanzin
http://git-wip-us.apache.org/repos/asf/spark/blob/3f958a99/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala -- diff --git

[3/3] spark git commit: [SPARK-21731][BUILD] Upgrade scalastyle to 0.9.

2017-08-15 Thread vanzin
[SPARK-21731][BUILD] Upgrade scalastyle to 0.9. This version fixes a few issues in the import order checker; it provides better error messages, and detects more improper ordering (thus the need to change a lot of files in this patch). The main fix is that it correctly complains about the order of

spark git commit: [SPARK-17742][CORE] Handle child process exit in SparkLauncher.

2017-08-15 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 14bdb25fd -> cba826d00 [SPARK-17742][CORE] Handle child process exit in SparkLauncher. Currently the launcher handle does not monitor the child spark-submit process it launches; this means that if the child exits with an error, the

spark git commit: [SPARK-18464][SQL][FOLLOWUP] support old table which doesn't store schema in table properties

2017-08-15 Thread lixiao
Repository: spark Updated Branches: refs/heads/master bc9902587 -> 14bdb25fd [SPARK-18464][SQL][FOLLOWUP] support old table which doesn't store schema in table properties ## What changes were proposed in this pull request? This is a follow-up of https://github.com/apache/spark/pull/15900 ,

spark git commit: [SPARK-19471][SQL] AggregationIterator does not initialize the generated result projection before using it

2017-08-15 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 12411b5ed -> bc9902587 [SPARK-19471][SQL] AggregationIterator does not initialize the generated result projection before using it ## What changes were proposed in this pull request? This is a follow-up PR that moves the test case in

spark git commit: [SPARK-21721][SQL][BACKPORT-2.1] Clear FileSystem deleteOnExit cache when paths are successfully removed

2017-08-15 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.1 9b749b6ce -> 6f366fbbf [SPARK-21721][SQL][BACKPORT-2.1] Clear FileSystem deleteOnExit cache when paths are successfully removed ## What changes were proposed in this pull request? Backport SPARK-21721 to branch 2.1: We put staging

spark git commit: [SPARK-21732][SQL] Lazily init hive metastore client

2017-08-15 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 0422ce06d -> 12411b5ed [SPARK-21732][SQL] Lazily init hive metastore client ## What changes were proposed in this pull request? This PR changes the codes to lazily init hive metastore client so that we can create SparkSession without

spark git commit: [SPARK-21724][SQL][DOC] Adds since information in the documentation of date functions

2017-08-15 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 4c3cf1cc5 -> 0422ce06d [SPARK-21724][SQL][DOC] Adds since information in the documentation of date functions ## What changes were proposed in this pull request? This PR adds `since` annotation in documentation so that this can be