svn commit: r31196 - in /dev/spark/3.0.0-SNAPSHOT-2018_11_28_19_16-8bfea86-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-28 Thread pwendell
Author: pwendell Date: Thu Nov 29 03:29:01 2018 New Revision: 31196 Log: Apache Spark 3.0.0-SNAPSHOT-2018_11_28_19_16-8bfea86 docs [This commit notification would consist of 1753 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

[1/2] spark git commit: [SPARK-26133][ML] Remove deprecated OneHotEncoder and rename OneHotEncoderEstimator to OneHotEncoder

2018-11-28 Thread dbtsai
Repository: spark Updated Branches: refs/heads/master fa0d4bf69 -> 8bfea86b1 http://git-wip-us.apache.org/repos/asf/spark/blob/8bfea86b/mllib/src/test/scala/org/apache/spark/ml/feature/OneHotEncoderSuite.scala -- diff --git a/

[2/2] spark git commit: [SPARK-26133][ML] Remove deprecated OneHotEncoder and rename OneHotEncoderEstimator to OneHotEncoder

2018-11-28 Thread dbtsai
[SPARK-26133][ML] Remove deprecated OneHotEncoder and rename OneHotEncoderEstimator to OneHotEncoder ## What changes were proposed in this pull request? We have deprecated `OneHotEncoder` at Spark 2.3.0 and introduced `OneHotEncoderEstimator`. At 3.0.0, we remove deprecated `OneHotEncoder` and

svn commit: r31190 - in /dev/spark/3.0.0-SNAPSHOT-2018_11_28_10_40-fa0d4bf-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-28 Thread pwendell
Author: pwendell Date: Wed Nov 28 18:52:58 2018 New Revision: 31190 Log: Apache Spark 3.0.0-SNAPSHOT-2018_11_28_10_40-fa0d4bf docs [This commit notification would consist of 1756 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

svn commit: r31186 - in /dev/spark/2.4.1-SNAPSHOT-2018_11_28_08_29-99a9107-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-28 Thread pwendell
Author: pwendell Date: Wed Nov 28 16:45:37 2018 New Revision: 31186 Log: Apache Spark 2.4.1-SNAPSHOT-2018_11_28_08_29-99a9107 docs [This commit notification would consist of 1476 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

svn commit: r31185 - in /dev/spark/2.3.3-SNAPSHOT-2018_11_28_08_29-96a5a127-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _

2018-11-28 Thread pwendell
Author: pwendell Date: Wed Nov 28 16:43:36 2018 New Revision: 31185 Log: Apache Spark 2.3.3-SNAPSHOT-2018_11_28_08_29-96a5a127 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] --

spark git commit: [SPARK-25829][SQL] remove duplicated map keys with last wins policy

2018-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 9fde3deab -> fa0d4bf69 [SPARK-25829][SQL] remove duplicated map keys with last wins policy ## What changes were proposed in this pull request? Currently duplicated map keys are not handled consistently. For example, map look up respects t

spark git commit: [SPARK-25989][ML] OneVsRestModel handle empty outputCols incorrectly

2018-11-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master 87bd9c75d -> 9fde3deab [SPARK-25989][ML] OneVsRestModel handle empty outputCols incorrectly ## What changes were proposed in this pull request? ignore empty output columns ## How was this patch tested? added tests Closes #22991 from zheng

spark git commit: [SPARK-25998][CORE] Change TorrentBroadcast to hold weak reference of broadcast object

2018-11-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master ce61bac1d -> 87bd9c75d [SPARK-25998][CORE] Change TorrentBroadcast to hold weak reference of broadcast object ## What changes were proposed in this pull request? This PR changes the broadcast object in TorrentBroadcast from a strong refe

spark git commit: [SPARK-26137][CORE] Use Java system property "file.separator" inste…

2018-11-28 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.3 de5f489d9 -> 96a5a127e [SPARK-26137][CORE] Use Java system property "file.separator" inste… … of hard coded "/" in DependencyUtils ## What changes were proposed in this pull request? Use Java system property "file.separator" inste

spark git commit: [SPARK-26137][CORE] Use Java system property "file.separator" inste…

2018-11-28 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.4 ac26a1dd5 -> 99a9107c9 [SPARK-26137][CORE] Use Java system property "file.separator" inste… … of hard coded "/" in DependencyUtils ## What changes were proposed in this pull request? Use Java system property "file.separator" inste

spark git commit: [SPARK-26137][CORE] Use Java system property "file.separator" inste…

2018-11-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master affe80958 -> ce61bac1d [SPARK-26137][CORE] Use Java system property "file.separator" inste… … of hard coded "/" in DependencyUtils ## What changes were proposed in this pull request? Use Java system property "file.separator" instead o

svn commit: r31184 - in /dev/spark/3.0.0-SNAPSHOT-2018_11_28_06_17-affe809-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-28 Thread pwendell
Author: pwendell Date: Wed Nov 28 14:29:36 2018 New Revision: 31184 Log: Apache Spark 3.0.0-SNAPSHOT-2018_11_28_06_17-affe809 docs [This commit notification would consist of 1756 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-26147][SQL] only pull out unevaluable python udf from join condition

2018-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.4 400d61b5d -> ac26a1dd5 [SPARK-26147][SQL] only pull out unevaluable python udf from join condition https://github.com/apache/spark/pull/22326 made a mistake that, not all python UDFs are unevaluable in join condition. Only python UDFs

spark git commit: [SPARK-26147][SQL] only pull out unevaluable python udf from join condition

2018-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 438f8fd67 -> affe80958 [SPARK-26147][SQL] only pull out unevaluable python udf from join condition ## What changes were proposed in this pull request? https://github.com/apache/spark/pull/22326 made a mistake that, not all python UDFs are

spark git commit: [SPARK-26114][CORE] ExternalSorter's readingIterator field leak

2018-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.4 9b2b0cf84 -> 400d61b5d [SPARK-26114][CORE] ExternalSorter's readingIterator field leak ## What changes were proposed in this pull request? This pull request fixes [SPARK-26114](https://issues.apache.org/jira/browse/SPARK-26114) issue

spark git commit: [SPARK-26114][CORE] ExternalSorter's readingIterator field leak

2018-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 93112e693 -> 438f8fd67 [SPARK-26114][CORE] ExternalSorter's readingIterator field leak ## What changes were proposed in this pull request? This pull request fixes [SPARK-26114](https://issues.apache.org/jira/browse/SPARK-26114) issue that

spark git commit: [SPARK-26142][SQL] Implement shuffle read metrics in SQL

2018-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 09a91d98b -> 93112e693 [SPARK-26142][SQL] Implement shuffle read metrics in SQL ## What changes were proposed in this pull request? Implement `SQLShuffleMetricsReporter` on the sql side as the customized ShuffleMetricsReporter, which exte

svn commit: r31179 - in /dev/spark/3.0.0-SNAPSHOT-2018_11_28_01_00-09a91d9-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-11-28 Thread pwendell
Author: pwendell Date: Wed Nov 28 09:13:01 2018 New Revision: 31179 Log: Apache Spark 3.0.0-SNAPSHOT-2018_11_28_01_00-09a91d9 docs [This commit notification would consist of 1756 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] ---

spark git commit: [SPARK-26021][SQL][FOLLOWUP] add test for special floating point values

2018-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 8c6871828 -> 09a91d98b [SPARK-26021][SQL][FOLLOWUP] add test for special floating point values ## What changes were proposed in this pull request? a followup of https://github.com/apache/spark/pull/23043 . Add a test to show the minor beh