svn commit: r25884 - in /dev/spark/2.3.1-SNAPSHOT-2018_03_21_22_01-4da8c22-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-03-21 Thread pwendell
Author: pwendell Date: Thu Mar 22 05:21:24 2018 New Revision: 25884 Log: Apache Spark 2.3.1-SNAPSHOT-2018_03_21_22_01-4da8c22 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-23372][SQL] Writing empty struct in parquet fails during execution. It should fail earlier in the processing.

2018-03-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 95e51ff84 -> 5c9eaa6b5 [SPARK-23372][SQL] Writing empty struct in parquet fails during execution. It should fail earlier in the processing. ## What changes were proposed in this pull request? Currently we allow writing data frames with emp

spark git commit: [SPARK-23760][SQL] CodegenContext.withSubExprEliminationExprs should save/restore CSE state correctly

2018-03-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.3 c9acd46be -> 4da8c22f7 [SPARK-23760][SQL] CodegenContext.withSubExprEliminationExprs should save/restore CSE state correctly ## What changes were proposed in this pull request? Fixed `CodegenContext.withSubExprEliminationExprs()` so t

spark git commit: [SPARK-23760][SQL] CodegenContext.withSubExprEliminationExprs should save/restore CSE state correctly

2018-03-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 0604beaff -> 95e51ff84 [SPARK-23760][SQL] CodegenContext.withSubExprEliminationExprs should save/restore CSE state correctly ## What changes were proposed in this pull request? Fixed `CodegenContext.withSubExprEliminationExprs()` so that

svn commit: r25881 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_21_20_01-0604bea-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-03-21 Thread pwendell
Author: pwendell Date: Thu Mar 22 03:15:37 2018 New Revision: 25881 Log: Apache Spark 2.4.0-SNAPSHOT-2018_03_21_20_01-0604bea docs [This commit notification would consist of 1449 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

svn commit: r25880 - in /dev/spark/2.3.1-SNAPSHOT-2018_03_21_18_01-c9acd46-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-03-21 Thread pwendell
Author: pwendell Date: Thu Mar 22 01:15:41 2018 New Revision: 25880 Log: Apache Spark 2.3.1-SNAPSHOT-2018_03_21_18_01-c9acd46 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-23729][CORE] Respect URI fragment when resolving globs

2018-03-21 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.3 4b9f33ff7 -> c9acd46be [SPARK-23729][CORE] Respect URI fragment when resolving globs Firstly, glob resolution will not result in swallowing the remote name part (that is preceded by the `#` sign) in case of `--files` or `--archives` op

spark git commit: [SPARK-23729][CORE] Respect URI fragment when resolving globs

2018-03-21 Thread vanzin
Repository: spark Updated Branches: refs/heads/master a091ee676 -> 0604beaff [SPARK-23729][CORE] Respect URI fragment when resolving globs Firstly, glob resolution will not result in swallowing the remote name part (that is preceded by the `#` sign) in case of `--files` or `--archives` option

svn commit: r25874 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_21_16_01-a091ee6-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-03-21 Thread pwendell
Author: pwendell Date: Wed Mar 21 23:15:17 2018 New Revision: 25874 Log: Apache Spark 2.4.0-SNAPSHOT-2018_03_21_16_01-a091ee6 docs [This commit notification would consist of 1449 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

svn commit: r25870 - in /dev/spark/2.3.1-SNAPSHOT-2018_03_21_14_01-4b9f33f-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-03-21 Thread pwendell
Author: pwendell Date: Wed Mar 21 21:15:30 2018 New Revision: 25870 Log: Apache Spark 2.3.1-SNAPSHOT-2018_03_21_14_01-4b9f33f docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [MINOR] Fix Java lint from new JavaKolmogorovSmirnovTestSuite

2018-03-21 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 2b89e4aa2 -> a091ee676 [MINOR] Fix Java lint from new JavaKolmogorovSmirnovTestSuite ## What changes were proposed in this pull request? Fix lint-java from https://github.com/apache/spark/pull/19108 addition of JavaKolmogorovSmirnovTestSu

spark git commit: [SPARK-18580][DSTREAM][KAFKA] Add spark.streaming.backpressure.initialRate to direct Kafka streams

2018-03-21 Thread koeninger
Repository: spark Updated Branches: refs/heads/master 918c7e99a -> 2b89e4aa2 [SPARK-18580][DSTREAM][KAFKA] Add spark.streaming.backpressure.initialRate to direct Kafka streams ## What changes were proposed in this pull request? Add `spark.streaming.backpressure.initialRate` to direct Kafka S

svn commit: r25869 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_21_12_01-918c7e9-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-03-21 Thread pwendell
Author: pwendell Date: Wed Mar 21 19:15:23 2018 New Revision: 25869 Log: Apache Spark 2.4.0-SNAPSHOT-2018_03_21_12_01-918c7e9 docs [This commit notification would consist of 1449 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

svn commit: r25868 - in /dev/spark/2.3.1-SNAPSHOT-2018_03_21_10_01-1e552b3-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-03-21 Thread pwendell
Author: pwendell Date: Wed Mar 21 17:16:11 2018 New Revision: 25868 Log: Apache Spark 2.3.1-SNAPSHOT-2018_03_21_10_01-1e552b3 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-03-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 98d0ea3f6 -> 918c7e99a [SPARK-23288][SS] Fix output metrics with parquet sink ## What changes were proposed in this pull request? Output metrics were not filled when parquet sink used. This PR fixes this problem by passing a `BasicWriteJo

spark git commit: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-03-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.3 1e552b356 -> 4b9f33ff7 [SPARK-23288][SS] Fix output metrics with parquet sink ## What changes were proposed in this pull request? Output metrics were not filled when parquet sink used. This PR fixes this problem by passing a `BasicWri

spark git commit: [SPARK-23264][SQL] Fix scala.MatchError in literals.sql.out

2018-03-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.3 0b880db65 -> 1e552b356 [SPARK-23264][SQL] Fix scala.MatchError in literals.sql.out ## What changes were proposed in this pull request? To fix `scala.MatchError` in `literals.sql.out`, this pr added an entry for `CalendarIntervalType` i

spark git commit: [SPARK-23264][SQL] Fix scala.MatchError in literals.sql.out

2018-03-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 8d79113b8 -> 98d0ea3f6 [SPARK-23264][SQL] Fix scala.MatchError in literals.sql.out ## What changes were proposed in this pull request? To fix `scala.MatchError` in `literals.sql.out`, this pr added an entry for `CalendarIntervalType` in `Q

spark git commit: [SPARK-23577][SQL] Supports custom line separator for text datasource

2018-03-21 Thread wenchen
Repository: spark Updated Branches: refs/heads/master bf09f2f71 -> 8d79113b8 [SPARK-23577][SQL] Supports custom line separator for text datasource ## What changes were proposed in this pull request? This PR proposes to add `lineSep` option for a configurable line separator in text datasource

spark git commit: [SPARK-10884][ML] Support prediction on single instance for regression and classification related models

2018-03-21 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 500b21c3d -> bf09f2f71 [SPARK-10884][ML] Support prediction on single instance for regression and classification related models ## What changes were proposed in this pull request? Support prediction on single instance for regression and c

spark git commit: [SPARK-23568][ML] Use metadata numAttributes if available in Silhouette

2018-03-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master 983e8d9d6 -> 500b21c3d [SPARK-23568][ML] Use metadata numAttributes if available in Silhouette ## What changes were proposed in this pull request? Silhouette need to know the number of features. This was taken using `first` and checking t

svn commit: r25862 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_21_00_01-983e8d9-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-03-21 Thread pwendell
Author: pwendell Date: Wed Mar 21 07:17:11 2018 New Revision: 25862 Log: Apache Spark 2.4.0-SNAPSHOT-2018_03_21_00_01-983e8d9 docs [This commit notification would consist of 1449 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---