spark git commit: [SPARK-22993][ML] Clarify HasCheckpointInterval param doc

2018-01-09 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.3 ecc24ec7f -> 2db523959 [SPARK-22993][ML] Clarify HasCheckpointInterval param doc ## What changes were proposed in this pull request? Add a note to the `HasCheckpointInterval` parameter doc that clarifies that this setting is ignored

spark git commit: [SPARK-22993][ML] Clarify HasCheckpointInterval param doc

2018-01-09 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master eaac60a1e -> 70bcc9d5a [SPARK-22993][ML] Clarify HasCheckpointInterval param doc ## What changes were proposed in this pull request? Add a note to the `HasCheckpointInterval` parameter doc that clarifies that this setting is ignored when

spark git commit: [SPARK-16060][SQL][FOLLOW-UP] add a wrapper solution for vectorized orc reader

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/master edf0a48c2 -> eaac60a1e [SPARK-16060][SQL][FOLLOW-UP] add a wrapper solution for vectorized orc reader ## What changes were proposed in this pull request? This is mostly from https://github.com/apache/spark/pull/13775 The wrapper solution

spark git commit: [SPARK-16060][SQL][FOLLOW-UP] add a wrapper solution for vectorized orc reader

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.3 162c5becc -> ecc24ec7f [SPARK-16060][SQL][FOLLOW-UP] add a wrapper solution for vectorized orc reader ## What changes were proposed in this pull request? This is mostly from https://github.com/apache/spark/pull/13775 The wrapper

spark git commit: [SPARK-22982] Remove unsafe asynchronous close() call from FileDownloadChannel

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.3 20a8c8867 -> 162c5becc [SPARK-22982] Remove unsafe asynchronous close() call from FileDownloadChannel ## What changes were proposed in this pull request? This patch fixes a severe asynchronous IO bug in Spark's Netty-based file

spark git commit: [SPARK-22982] Remove unsafe asynchronous close() call from FileDownloadChannel

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/master e59983724 -> edf0a48c2 [SPARK-22982] Remove unsafe asynchronous close() call from FileDownloadChannel ## What changes were proposed in this pull request? This patch fixes a severe asynchronous IO bug in Spark's Netty-based file transfer

svn commit: r24110 - in /dev/spark/2.3.0-SNAPSHOT-2018_01_09_22_01-20a8c88-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-01-09 Thread pwendell
Author: pwendell Date: Wed Jan 10 06:15:49 2018 New Revision: 24110 Log: Apache Spark 2.3.0-SNAPSHOT-2018_01_09_22_01-20a8c88 docs [This commit notification would consist of 1439 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23009][PYTHON] Fix for non-str col names to createDataFrame from Pandas

2018-01-09 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 7bcc26668 -> e59983724 [SPARK-23009][PYTHON] Fix for non-str col names to createDataFrame from Pandas ## What changes were proposed in this pull request? This the case when calling `SparkSession.createDataFrame` using a Pandas DataFrame

spark git commit: [SPARK-23018][PYTHON] Fix createDataFrame from Pandas timestamp series assignment

2018-01-09 Thread ueshin
Repository: spark Updated Branches: refs/heads/branch-2.3 45f5c3cee -> 20a8c8867 [SPARK-23018][PYTHON] Fix createDataFrame from Pandas timestamp series assignment ## What changes were proposed in this pull request? This fixes createDataFrame from Pandas to only assign modified timestamp

spark git commit: [SPARK-23018][PYTHON] Fix createDataFrame from Pandas timestamp series assignment

2018-01-09 Thread ueshin
Repository: spark Updated Branches: refs/heads/master 6f169ca9e -> 7bcc26668 [SPARK-23018][PYTHON] Fix createDataFrame from Pandas timestamp series assignment ## What changes were proposed in this pull request? This fixes createDataFrame from Pandas to only assign modified timestamp series

svn commit: r24109 - in /dev/spark/2.3.0-SNAPSHOT-2018_01_09_20_01-6f169ca-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-01-09 Thread pwendell
Author: pwendell Date: Wed Jan 10 04:15:06 2018 New Revision: 24109 Log: Apache Spark 2.3.0-SNAPSHOT-2018_01_09_20_01-6f169ca docs [This commit notification would consist of 1439 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [MINOR] fix a typo in BroadcastJoinSuite

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 96ba217a0 -> 6f169ca9e [MINOR] fix a typo in BroadcastJoinSuite ## What changes were proposed in this pull request? `BroadcastNestedLoopJoinExec` should be `BroadcastHashJoinExec` ## How was this patch tested? N/A Author: Wenchen Fan

spark git commit: [SPARK-23005][CORE] Improve RDD.take on small number of partitions

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.3 44763d93c -> df047bdba [SPARK-23005][CORE] Improve RDD.take on small number of partitions ## What changes were proposed in this pull request? In current implementation of RDD.take, we overestimate the number of partitions we need to

spark git commit: [SPARK-23005][CORE] Improve RDD.take on small number of partitions

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 2250cb75b -> 96ba217a0 [SPARK-23005][CORE] Improve RDD.take on small number of partitions ## What changes were proposed in this pull request? In current implementation of RDD.take, we overestimate the number of partitions we need to try

svn commit: r24105 - in /dev/spark/2.3.0-SNAPSHOT-2018_01_09_14_01-44763d9-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-01-09 Thread pwendell
Author: pwendell Date: Tue Jan 9 22:16:00 2018 New Revision: 24105 Log: Apache Spark 2.3.0-SNAPSHOT-2018_01_09_14_01-44763d9 docs [This commit notification would consist of 1439 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-22912] v2 data source support in MicroBatchExecution

2018-01-09 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-2.3 be5991902 -> 44763d93c [SPARK-22912] v2 data source support in MicroBatchExecution ## What changes were proposed in this pull request? Support for v2 data sources in microbatch streaming. ## How was this patch tested? A very basic

svn commit: r24102 - in /dev/spark/2.3.0-SNAPSHOT-2018_01_09_10_01-be59919-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-01-09 Thread pwendell
Author: pwendell Date: Tue Jan 9 18:15:38 2018 New Revision: 24102 Log: Apache Spark 2.3.0-SNAPSHOT-2018_01_09_10_01-be59919 docs [This commit notification would consist of 1439 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

[1/2] spark-website git commit: Update remaining refs to 2.2.0 as latest Spark, to 2.2.1

2018-01-09 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site 1a8adcaa8 -> ab2bcbf28 http://git-wip-us.apache.org/repos/asf/spark-website/blob/ab2bcbf2/site/powered-by.html -- diff --git a/site/powered-by.html

[2/2] spark-website git commit: Update remaining refs to 2.2.0 as latest Spark, to 2.2.1

2018-01-09 Thread srowen
Update remaining refs to 2.2.0 as latest Spark, to 2.2.1 Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/ab2bcbf2 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/ab2bcbf2 Diff:

svn commit: r24101 - in /dev/spark/2.3.0-SNAPSHOT-2018_01_09_08_01-2250cb7-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-01-09 Thread pwendell
Author: pwendell Date: Tue Jan 9 16:18:03 2018 New Revision: 24101 Log: Apache Spark 2.3.0-SNAPSHOT-2018_01_09_08_01-2250cb7 docs [This commit notification would consist of 1439 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24100 - in /dev/spark/2.3.0-SNAPSHOT-2018_01_09_06_01-be59919-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-01-09 Thread pwendell
Author: pwendell Date: Tue Jan 9 14:15:24 2018 New Revision: 24100 Log: Apache Spark 2.3.0-SNAPSHOT-2018_01_09_06_01-be59919 docs [This commit notification would consist of 1439 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-22981][SQL] Fix incorrect results of Casting Struct to String

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.3 60f6b9945 -> be5991902 [SPARK-22981][SQL] Fix incorrect results of Casting Struct to String ## What changes were proposed in this pull request? This pr fixed the issue when casting structs into strings; ``` scala> val df = Seq(((1,

spark git commit: [SPARK-22981][SQL] Fix incorrect results of Casting Struct to String

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/master f44ba910f -> 2250cb75b [SPARK-22981][SQL] Fix incorrect results of Casting Struct to String ## What changes were proposed in this pull request? This pr fixed the issue when casting structs into strings; ``` scala> val df = Seq(((1, "a"),

spark git commit: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.3 47f975b42 -> 60f6b9945 [SPARK-16060][SQL] Support Vectorized ORC Reader ## What changes were proposed in this pull request? This PR adds an ORC columnar-batch reader to native `OrcFileFormat`. Since both Spark `ColumnarBatch` and ORC

spark git commit: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 6a4206ff0 -> f44ba910f [SPARK-16060][SQL] Support Vectorized ORC Reader ## What changes were proposed in this pull request? This PR adds an ORC columnar-batch reader to native `OrcFileFormat`. Since both Spark `ColumnarBatch` and ORC

svn commit: r24098 - in /dev/spark/2.3.0-SNAPSHOT-2018_01_09_04_01-6a4206f-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-01-09 Thread pwendell
Author: pwendell Date: Tue Jan 9 12:20:25 2018 New Revision: 24098 Log: Apache Spark 2.3.0-SNAPSHOT-2018_01_09_04_01-6a4206f docs [This commit notification would consist of 1439 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r24095 - in /dev/spark/2.3.0-SNAPSHOT-2018_01_09_02_01-47f975b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-01-09 Thread pwendell
Author: pwendell Date: Tue Jan 9 10:15:09 2018 New Revision: 24095 Log: Apache Spark 2.3.0-SNAPSHOT-2018_01_09_02_01-47f975b docs [This commit notification would consist of 1439 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors

2018-01-09 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master 0959aa581 -> 6a4206ff0 [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors ## What changes were proposed in this pull request? The environment variable `SPARK_MOUNTED_CLASSPATH` is referenced in the

spark git commit: [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors

2018-01-09 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.3 e79480e5d -> 47f975b42 [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors ## What changes were proposed in this pull request? The environment variable `SPARK_MOUNTED_CLASSPATH` is referenced in the

spark git commit: [SPARK-23000] Fix Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.3 a23c07ecb -> e79480e5d [SPARK-23000] Fix Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3 ## What changes were proposed in this pull request?

spark git commit: [SPARK-23000] Fix Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3

2018-01-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 02214b094 -> 0959aa581 [SPARK-23000] Fix Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3 ## What changes were proposed in this pull request?

svn commit: r24091 - in /dev/spark/2.3.0-SNAPSHOT-2018_01_09_00_01-02214b0-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-01-09 Thread pwendell
Author: pwendell Date: Tue Jan 9 08:15:42 2018 New Revision: 24091 Log: Apache Spark 2.3.0-SNAPSHOT-2018_01_09_00_01-02214b0 docs [This commit notification would consist of 1439 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]