spark git commit: [SPARK-24516][K8S] Change Python default to Python3

2018-10-25 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master b2e325625 -> 19ada15d1 [SPARK-24516][K8S] Change Python default to Python3 ## What changes were proposed in this pull request? As this is targeted for 3.0.0 and Python2 will be deprecated by Jan 1st, 2020, I feel it is appropriate to

spark git commit: [SPARK-24572][SPARKR] "eager execution" for R shell, IDE

2018-10-25 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master 19ada15d1 -> ddd1b1e8a [SPARK-24572][SPARKR] "eager execution" for R shell, IDE ## What changes were proposed in this pull request? Check the `spark.sql.repl.eagerEval.enabled` configuration property in SparkDataFrame `show()` method. If

svn commit: r30404 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_25_00_03-ddd1b1e-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-25 Thread pwendell
Author: pwendell Date: Thu Oct 25 07:17:50 2018 New Revision: 30404 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_25_00_03-ddd1b1e docs [This commit notification would consist of 1473 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25746][SQL] Refactoring ExpressionEncoder to get rid of flat flag

2018-10-25 Thread wenchen
Repository: spark Updated Branches: refs/heads/master ddd1b1e8a -> cb5ea201d [SPARK-25746][SQL] Refactoring ExpressionEncoder to get rid of flat flag ## What changes were proposed in this pull request? This is inspired during implementing #21732. For now `ScalaReflection` needs to consider

spark git commit: [SPARK-24794][CORE] Driver launched through rest should use all masters

2018-10-25 Thread srowen
Repository: spark Updated Branches: refs/heads/master 65c653fb4 -> 002f9c169 [SPARK-24794][CORE] Driver launched through rest should use all masters ## What changes were proposed in this pull request? In standalone cluster mode, one could launch driver with supervise mode enabled.

spark git commit: [SPARK-25347][ML][DOC] Spark datasource for image/libsvm user guide

2018-10-25 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 002f9c169 -> 6540c2f8f [SPARK-25347][ML][DOC] Spark datasource for image/libsvm user guide ## What changes were proposed in this pull request? Spark datasource for image/libsvm user guide ## How was this patch tested? Scala:

spark git commit: [BUILD] Close stale PRs

2018-10-25 Thread srowen
Repository: spark Updated Branches: refs/heads/master 3123c7f48 -> 65c653fb4 [BUILD] Close stale PRs Closes #22567 Closes #18457 Closes #21517 Closes #21858 Closes #22383 Closes #19219 Closes #22401 Closes #22811 Closes #20405 Closes #21933 Closes #22819 from srowen/ClosePRs. Authored-by:

spark git commit: [SPARK-25347][ML][DOC] Spark datasource for image/libsvm user guide

2018-10-25 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.4 d5e694805 -> a20660b6f [SPARK-25347][ML][DOC] Spark datasource for image/libsvm user guide ## What changes were proposed in this pull request? Spark datasource for image/libsvm user guide ## How was this patch tested? Scala:

svn commit: r30407 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_25_08_03-002f9c1-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-25 Thread pwendell
Author: pwendell Date: Thu Oct 25 15:17:40 2018 New Revision: 30407 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_25_08_03-002f9c1 docs [This commit notification would consist of 1473 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25808][BUILD] Upgrade jsr305 version from 1.3.9 to 3.0.0

2018-10-25 Thread srowen
Repository: spark Updated Branches: refs/heads/master cb5ea201d -> 3123c7f48 [SPARK-25808][BUILD] Upgrade jsr305 version from 1.3.9 to 3.0.0 ## What changes were proposed in this pull request? We find below warnings when build spark project: ``` [warn] *

svn commit: r30409 - in /dev/spark/2.4.1-SNAPSHOT-2018_10_25_10_02-a20660b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-25 Thread pwendell
Author: pwendell Date: Thu Oct 25 17:17:26 2018 New Revision: 30409 Log: Apache Spark 2.4.1-SNAPSHOT-2018_10_25_10_02-a20660b docs [This commit notification would consist of 1478 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r30413 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_25_16_03-46d2d2c-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-25 Thread pwendell
Author: pwendell Date: Thu Oct 25 23:17:26 2018 New Revision: 30413 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_25_16_03-46d2d2c docs [This commit notification would consist of 1474 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-24787][CORE] Revert hsync in EventLoggingListener and make FsHistoryProvider to read lastBlockBeingWritten data for logs

2018-10-25 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.4 45ed76d6f -> 1b075f26f [SPARK-24787][CORE] Revert hsync in EventLoggingListener and make FsHistoryProvider to read lastBlockBeingWritten data for logs ## What changes were proposed in this pull request? `hsync` has been added as part

svn commit: r30411 - in /dev/spark/2.4.1-SNAPSHOT-2018_10_25_14_02-45ed76d-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-25 Thread pwendell
Author: pwendell Date: Thu Oct 25 21:17:10 2018 New Revision: 30411 Log: Apache Spark 2.4.1-SNAPSHOT-2018_10_25_14_02-45ed76d docs [This commit notification would consist of 1478 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25656][SQL][DOC][EXAMPLE][BRANCH-2.4] Add a doc and examples about extra data source options

2018-10-25 Thread dongjoon
Repository: spark Updated Branches: refs/heads/branch-2.4 1b075f26f -> db121a2a1 [SPARK-25656][SQL][DOC][EXAMPLE][BRANCH-2.4] Add a doc and examples about extra data source options ## What changes were proposed in this pull request? Our current doc does not explain how we are passing the

spark git commit: [SPARK-25665][SQL][TEST] Refactor ObjectHashAggregateExecBenchmark to…

2018-10-25 Thread dongjoon
Repository: spark Updated Branches: refs/heads/master 6540c2f8f -> ccd07b736 [SPARK-25665][SQL][TEST] Refactor ObjectHashAggregateExecBenchmark to… ## What changes were proposed in this pull request? Refactor ObjectHashAggregateExecBenchmark to use main method ## How was this patch

spark git commit: [SPARK-25803][K8S] Fix docker-image-tool.sh -n option

2018-10-25 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.4 a20660b6f -> 45ed76d6f [SPARK-25803][K8S] Fix docker-image-tool.sh -n option ## What changes were proposed in this pull request? docker-image-tool.sh uses getopts in which a colon signifies that an option takes an argument. Since -n

spark git commit: [SPARK-24787][CORE] Revert hsync in EventLoggingListener and make FsHistoryProvider to read lastBlockBeingWritten data for logs

2018-10-25 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 9b98d9166 -> 46d2d2c74 [SPARK-24787][CORE] Revert hsync in EventLoggingListener and make FsHistoryProvider to read lastBlockBeingWritten data for logs ## What changes were proposed in this pull request? `hsync` has been added as part of

spark-website git commit: Use Heilmeier Catechism for SPIP template.

2018-10-25 Thread rxin
Repository: spark-website Updated Branches: refs/heads/asf-site e4b87718d -> 005a2a0d1 Use Heilmeier Catechism for SPIP template. Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/005a2a0d Tree:

svn commit: r30410 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_25_12_02-6540c2f-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-25 Thread pwendell
Author: pwendell Date: Thu Oct 25 19:17:13 2018 New Revision: 30410 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_25_12_02-6540c2f docs [This commit notification would consist of 1474 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25803][K8S] Fix docker-image-tool.sh -n option

2018-10-25 Thread vanzin
Repository: spark Updated Branches: refs/heads/master ccd07b736 -> 9b98d9166 [SPARK-25803][K8S] Fix docker-image-tool.sh -n option ## What changes were proposed in this pull request? docker-image-tool.sh uses getopts in which a colon signifies that an option takes an argument. Since -n does

svn commit: r30415 - in /dev/spark/3.0.0-SNAPSHOT-2018_10_25_20_03-72a23a6-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-25 Thread pwendell
Author: pwendell Date: Fri Oct 26 03:17:26 2018 New Revision: 30415 Log: Apache Spark 3.0.0-SNAPSHOT-2018_10_25_20_03-72a23a6 docs [This commit notification would consist of 1474 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25840][BUILD] `make-distribution.sh` should not fail due to missing LICENSE-binary

2018-10-25 Thread dongjoon
Repository: spark Updated Branches: refs/heads/branch-2.4 39e108f16 -> b739fb0d7 [SPARK-25840][BUILD] `make-distribution.sh` should not fail due to missing LICENSE-binary ## What changes were proposed in this pull request? We vote for the artifacts. All releases are in the form of the

spark git commit: [SPARK-25840][BUILD] `make-distribution.sh` should not fail due to missing LICENSE-binary

2018-10-25 Thread dongjoon
Repository: spark Updated Branches: refs/heads/master dc9b32080 -> 79f3babcc [SPARK-25840][BUILD] `make-distribution.sh` should not fail due to missing LICENSE-binary ## What changes were proposed in this pull request? We vote for the artifacts. All releases are in the form of the source

spark git commit: [SPARK-25793][ML] call SaveLoadV2_0.load for classNameV2_0

2018-10-25 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 72a23a6c4 -> dc9b32080 [SPARK-25793][ML] call SaveLoadV2_0.load for classNameV2_0 ## What changes were proposed in this pull request? The following code in BisectingKMeansModel.load calls the wrong version of load. ``` case

spark git commit: [SPARK-25819][SQL] Support parse mode option for the function `from_avro`

2018-10-25 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 79f3babcc -> 24e8c27df [SPARK-25819][SQL] Support parse mode option for the function `from_avro` ## What changes were proposed in this pull request? Current the function `from_avro` throws exception on reading corrupt records. In

spark git commit: [SPARK-25832][SQL][BRANCH-2.4] Revert newly added map related functions

2018-10-25 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.4 db121a2a1 -> a9f200e11 [SPARK-25832][SQL][BRANCH-2.4] Revert newly added map related functions ## What changes were proposed in this pull request? - Revert [SPARK-23935][SQL] Adding map_entries function:

svn commit: r30414 - in /dev/spark/2.4.1-SNAPSHOT-2018_10_25_18_02-a9f200e-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-25 Thread pwendell
Author: pwendell Date: Fri Oct 26 01:16:58 2018 New Revision: 30414 Log: Apache Spark 2.4.1-SNAPSHOT-2018_10_25_18_02-a9f200e docs [This commit notification would consist of 1478 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25772][SQL][FOLLOWUP] remove GetArrayFromMap

2018-10-25 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 46d2d2c74 -> 72a23a6c4 [SPARK-25772][SQL][FOLLOWUP] remove GetArrayFromMap ## What changes were proposed in this pull request? In https://github.com/apache/spark/pull/22745 we introduced the `GetArrayFromMap` expression. Later on I

spark git commit: [SPARK-25793][ML] call SaveLoadV2_0.load for classNameV2_0

2018-10-25 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.4 a9f200e11 -> 39e108f16 [SPARK-25793][ML] call SaveLoadV2_0.load for classNameV2_0 ## What changes were proposed in this pull request? The following code in BisectingKMeansModel.load calls the wrong version of load. ``` case

spark git commit: [MINOR][TEST][BRANCH-2.4] Regenerate golden file `datetime.sql.out`

2018-10-25 Thread dongjoon
Repository: spark Updated Branches: refs/heads/branch-2.4 b739fb0d7 -> adfd1057d [MINOR][TEST][BRANCH-2.4] Regenerate golden file `datetime.sql.out` ## What changes were proposed in this pull request? `datetime.sql.out` is a generated golden file, but it's a little bit broken during manual

svn commit: r30417 - in /dev/spark/2.4.1-SNAPSHOT-2018_10_25_22_02-eff1c50-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-25 Thread pwendell
Author: pwendell Date: Fri Oct 26 05:19:04 2018 New Revision: 30417 Log: Apache Spark 2.4.1-SNAPSHOT-2018_10_25_22_02-eff1c50 docs [This commit notification would consist of 1478 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25822][PYSPARK] Fix a race condition when releasing a Python worker

2018-10-25 Thread ueshin
Repository: spark Updated Branches: refs/heads/branch-2.3 8fbf3ee91 -> 0a05cf917 [SPARK-25822][PYSPARK] Fix a race condition when releasing a Python worker ## What changes were proposed in this pull request? There is a race condition when releasing a Python worker. If

spark git commit: [SPARK-25822][PYSPARK] Fix a race condition when releasing a Python worker

2018-10-25 Thread ueshin
Repository: spark Updated Branches: refs/heads/master 24e8c27df -> 86d469aea [SPARK-25822][PYSPARK] Fix a race condition when releasing a Python worker ## What changes were proposed in this pull request? There is a race condition when releasing a Python worker. If

spark git commit: [SPARK-25822][PYSPARK] Fix a race condition when releasing a Python worker

2018-10-25 Thread ueshin
Repository: spark Updated Branches: refs/heads/branch-2.4 adfd1057d -> eff1c5016 [SPARK-25822][PYSPARK] Fix a race condition when releasing a Python worker ## What changes were proposed in this pull request? There is a race condition when releasing a Python worker. If

spark git commit: [SPARK-25842][SQL] Deprecate rangeBetween APIs introduced in SPARK-21608

2018-10-25 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.4 eff1c5016 -> f37bceadf [SPARK-25842][SQL] Deprecate rangeBetween APIs introduced in SPARK-21608 ## What changes were proposed in this pull request? See the detailed information at https://issues.apache.org/jira/browse/SPARK-25841 on

spark git commit: [SPARK-25842][SQL] Deprecate rangeBetween APIs introduced in SPARK-21608

2018-10-25 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 86d469aea -> 89d748b33 [SPARK-25842][SQL] Deprecate rangeBetween APIs introduced in SPARK-21608 ## What changes were proposed in this pull request? See the detailed information at https://issues.apache.org/jira/browse/SPARK-25841 on why

svn commit: r30416 - in /dev/spark/2.3.3-SNAPSHOT-2018_10_25_22_02-0a05cf9-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-10-25 Thread pwendell
Author: pwendell Date: Fri Oct 26 05:17:42 2018 New Revision: 30416 Log: Apache Spark 2.3.3-SNAPSHOT-2018_10_25_22_02-0a05cf9 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]