svn commit: r26754 - in /dev/spark/2.3.1-SNAPSHOT-2018_05_07_22_01-4dc6719-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-05-07 Thread pwendell
Author: pwendell Date: Tue May 8 05:16:14 2018 New Revision: 26754 Log: Apache Spark 2.3.1-SNAPSHOT-2018_05_07_22_01-4dc6719 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-24128][SQL] Mention configuration option in implicit CROSS JOIN error

2018-05-07 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.3 3a22feab4 -> 4dc6719e9 [SPARK-24128][SQL] Mention configuration option in implicit CROSS JOIN error ## What changes were proposed in this pull request? Mention `spark.sql.crossJoin.enabled` in error message when an implicit `CROSS

spark git commit: [SPARK-24128][SQL] Mention configuration option in implicit CROSS JOIN error

2018-05-07 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 0d63eb888 -> cd12c5c3e [SPARK-24128][SQL] Mention configuration option in implicit CROSS JOIN error ## What changes were proposed in this pull request? Mention `spark.sql.crossJoin.enabled` in error message when an implicit `CROSS JOIN`

spark git commit: [SPARK-23975][ML] Add support of array input for all clustering methods

2018-05-07 Thread meng
Repository: spark Updated Branches: refs/heads/master 76ecd0950 -> 0d63eb888 [SPARK-23975][ML] Add support of array input for all clustering methods ## What changes were proposed in this pull request? Add support for all of the clustering methods ## How was this patch tested? unit tests

svn commit: r26748 - in /dev/spark/2.3.1-SNAPSHOT-2018_05_07_18_01-3a22fea-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-05-07 Thread pwendell
Author: pwendell Date: Tue May 8 01:16:20 2018 New Revision: 26748 Log: Apache Spark 2.3.1-SNAPSHOT-2018_05_07_18_01-3a22fea docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r26747 - in /dev/spark/2.4.0-SNAPSHOT-2018_05_07_16_01-76ecd09-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-05-07 Thread pwendell
Author: pwendell Date: Mon May 7 23:16:03 2018 New Revision: 26747 Log: Apache Spark 2.4.0-SNAPSHOT-2018_05_07_16_01-76ecd09 docs [This commit notification would consist of 1461 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-20114][ML] spark.ml parity for sequential pattern mining - PrefixSpan

2018-05-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master f48bd6bdc -> 76ecd0950 [SPARK-20114][ML] spark.ml parity for sequential pattern mining - PrefixSpan ## What changes were proposed in this pull request? PrefixSpan API for spark.ml. New implementation instead of #20810 ## How was this

spark git commit: [SPARK-22885][ML][TEST] ML test for StructuredStreaming: spark.ml.tuning

2018-05-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 1c9c5de95 -> f48bd6bdc [SPARK-22885][ML][TEST] ML test for StructuredStreaming: spark.ml.tuning ## What changes were proposed in this pull request? ML test for StructuredStreaming: spark.ml.tuning ## How was this patch tested? N/A

spark git commit: [SPARK-23291][SPARK-23291][R][FOLLOWUP] Update SparkR migration note for

2018-05-07 Thread yliang
Repository: spark Updated Branches: refs/heads/master 56a52e0a5 -> 1c9c5de95 [SPARK-23291][SPARK-23291][R][FOLLOWUP] Update SparkR migration note for ## What changes were proposed in this pull request? This PR fixes the migration note for SPARK-23291 since it's going to backport to 2.3.1.

spark git commit: [SPARK-15750][MLLIB][PYSPARK] Constructing FPGrowth fails when no numPartitions specified in pyspark

2018-05-07 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master d83e96372 -> 56a52e0a5 [SPARK-15750][MLLIB][PYSPARK] Constructing FPGrowth fails when no numPartitions specified in pyspark ## What changes were proposed in this pull request? Change FPGrowth from private to private[spark]. If no

spark git commit: [SPARK-23291][SQL][R][BRANCH-2.3] R's substr should not reduce starting position by 1 when calling Scala API

2018-05-07 Thread yliang
Repository: spark Updated Branches: refs/heads/branch-2.3 f87785a76 -> 3a22feab4 [SPARK-23291][SQL][R][BRANCH-2.3] R's substr should not reduce starting position by 1 when calling Scala API ## What changes were proposed in this pull request? This PR backports

svn commit: r26745 - in /dev/spark/2.4.0-SNAPSHOT-2018_05_07_12_01-d83e963-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-05-07 Thread pwendell
Author: pwendell Date: Mon May 7 19:16:37 2018 New Revision: 26745 Log: Apache Spark 2.4.0-SNAPSHOT-2018_05_07_12_01-d83e963 docs [This commit notification would consist of 1460 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-24043][SQL] Interpreted Predicate should initialize nondeterministic expressions

2018-05-07 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 4e861db5f -> d83e96372 [SPARK-24043][SQL] Interpreted Predicate should initialize nondeterministic expressions ## What changes were proposed in this pull request? When creating an InterpretedPredicate instance, initialize any

svn commit: r26739 - in /dev/spark/2.4.0-SNAPSHOT-2018_05_07_05_34-4e861db-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-05-07 Thread pwendell
Author: pwendell Date: Mon May 7 12:50:22 2018 New Revision: 26739 Log: Apache Spark 2.4.0-SNAPSHOT-2018_05_07_05_34-4e861db docs [This commit notification would consist of 1460 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-16406][SQL] Improve performance of LogicalPlan.resolve

2018-05-07 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master e35ad3cad -> 4e861db5f [SPARK-16406][SQL] Improve performance of LogicalPlan.resolve ## What changes were proposed in this pull request? `LogicalPlan.resolve(...)` uses linear searches to find an attribute matching a name. This is fine

svn commit: r26737 - in /dev/spark/2.3.1-SNAPSHOT-2018_05_07_02_01-f87785a-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-05-07 Thread pwendell
Author: pwendell Date: Mon May 7 09:16:54 2018 New Revision: 26737 Log: Apache Spark 2.3.1-SNAPSHOT-2018_05_07_02_01-f87785a docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23930][SQL] Add slice function

2018-05-07 Thread ueshin
Repository: spark Updated Branches: refs/heads/master f06528015 -> e35ad3cad [SPARK-23930][SQL] Add slice function ## What changes were proposed in this pull request? The PR add the `slice` function. The behavior of the function is based on Presto's one. The function slices an array

spark git commit: [SPARK-24160][FOLLOWUP] Fix compilation failure

2018-05-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master c5981976f -> f06528015 [SPARK-24160][FOLLOWUP] Fix compilation failure ## What changes were proposed in this pull request? SPARK-24160 is causing a compilation failure (after SPARK-24143 was merged). This fixes the issue. ## How was

svn commit: r26734 - in /dev/spark/2.4.0-SNAPSHOT-2018_05_07_00_01-c598197-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-05-07 Thread pwendell
Author: pwendell Date: Mon May 7 07:17:46 2018 New Revision: 26734 Log: Apache Spark 2.4.0-SNAPSHOT-2018_05_07_00_01-c598197 docs [This commit notification would consist of 1460 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23775][TEST] Make DataFrameRangeSuite not flaky

2018-05-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.3 3f78f60cc -> f87785a76 [SPARK-23775][TEST] Make DataFrameRangeSuite not flaky ## What changes were proposed in this pull request? DataFrameRangeSuite.test("Cancelling stage in a query with Range.") stays sometimes in an infinite loop

spark git commit: [SPARK-23775][TEST] Make DataFrameRangeSuite not flaky

2018-05-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master d2aa859b4 -> c5981976f [SPARK-23775][TEST] Make DataFrameRangeSuite not flaky ## What changes were proposed in this pull request? DataFrameRangeSuite.test("Cancelling stage in a query with Range.") stays sometimes in an infinite loop and

spark git commit: [SPARK-24160] ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

2018-05-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 7564a9a70 -> d2aa859b4 [SPARK-24160] ShuffleBlockFetcherIterator should fail if it receives zero-size blocks ## What changes were proposed in this pull request? This patch modifies `ShuffleBlockFetcherIterator` so that the receipt of

spark git commit: [SPARK-23921][SQL] Add array_sort function

2018-05-07 Thread ueshin
Repository: spark Updated Branches: refs/heads/master 889f6cc10 -> 7564a9a70 [SPARK-23921][SQL] Add array_sort function ## What changes were proposed in this pull request? The PR adds the SQL function `array_sort`. The behavior of the function is based on Presto's one. The function sorts

spark git commit: [SPARK-24143] filter empty blocks when convert mapstatus to (blockId, size) pair

2018-05-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master a634d66ce -> 889f6cc10 [SPARK-24143] filter empty blocks when convert mapstatus to (blockId, size) pair ## What changes were proposed in this pull request? In current code(`MapOutputTracker.convertMapStatuses`), mapstatus are converted