svn commit: r28309 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_23_20_01-d2436a8-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-23 Thread pwendell
Author: pwendell Date: Tue Jul 24 03:16:29 2018 New Revision: 28309 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_23_20_01-d2436a8 docs [This commit notification would consist of 1469 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-24594][YARN] Introducing metrics for YARN

2018-07-23 Thread jshao
Repository: spark Updated Branches: refs/heads/master cfc3e1aaa -> d2436a852 [SPARK-24594][YARN] Introducing metrics for YARN ## What changes were proposed in this pull request? In this PR metrics are introduced for YARN. As up to now there was no metrics in the YARN module a new metric

svn commit: r28307 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_23_16_01-cfc3e1a-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-23 Thread pwendell
Author: pwendell Date: Mon Jul 23 23:15:57 2018 New Revision: 28307 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_23_16_01-cfc3e1a docs [This commit notification would consist of 1469 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-24339][SQL] Prunes the unused columns from child of ScriptTransformation

2018-07-23 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 61f0ca4f1 -> cfc3e1aaa [SPARK-24339][SQL] Prunes the unused columns from child of ScriptTransformation ## What changes were proposed in this pull request? Modify the strategy in ColumnPruning to add a Project between ScriptTransformation

spark git commit: [SPARK-24699][SS] Make watermarks work with Trigger.Once by saving updated watermark to commit log

2018-07-23 Thread tdas
Repository: spark Updated Branches: refs/heads/master 2edf17eff -> 61f0ca4f1 [SPARK-24699][SS] Make watermarks work with Trigger.Once by saving updated watermark to commit log ## What changes were proposed in this pull request? Streaming queries with watermarks do not work with Trigger.Once

svn commit: r28302 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_23_12_01-2edf17e-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-23 Thread pwendell
Author: pwendell Date: Mon Jul 23 19:16:31 2018 New Revision: 28302 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_23_12_01-2edf17e docs [This commit notification would consist of 1469 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-24850][SQL] fix str representation of CachedRDDBuilder

2018-07-23 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 08e315f63 -> 2edf17eff [SPARK-24850][SQL] fix str representation of CachedRDDBuilder ## What changes were proposed in this pull request? As of https://github.com/apache/spark/pull/21018, InMemoryRelation includes its cacheBuilder when

spark git commit: [SPARK-24887][SQL] Avro: use SerializableConfiguration in Spark utils to deduplicate code

2018-07-23 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 434319e73 -> 08e315f63 [SPARK-24887][SQL] Avro: use SerializableConfiguration in Spark utils to deduplicate code ## What changes were proposed in this pull request? To implement the method `buildReader` in `FileFormat`, it is required to

spark git commit: [SPARK-24802][SQL] Add a new config for Optimization Rule Exclusion

2018-07-23 Thread lixiao
Repository: spark Updated Branches: refs/heads/master ab18b02e6 -> 434319e73 [SPARK-24802][SQL] Add a new config for Optimization Rule Exclusion ## What changes were proposed in this pull request? Since Spark has provided fairly clear interfaces for adding user-defined optimization rules,

svn commit: r28297 - in /dev/spark/2.3.3-SNAPSHOT-2018_07_23_06_01-f5bc948-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-23 Thread pwendell
Author: pwendell Date: Mon Jul 23 13:16:24 2018 New Revision: 28297 Log: Apache Spark 2.3.3-SNAPSHOT-2018_07_23_06_01-f5bc948 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r28295 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_23_05_10-ab18b02-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-07-23 Thread pwendell
Author: pwendell Date: Mon Jul 23 12:27:35 2018 New Revision: 28295 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_23_05_10-ab18b02 docs [This commit notification would consist of 1469 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SQL][HIVE] Correct an assert message in function makeRDDForTable

2018-07-23 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master f59de52a2 -> ab18b02e6 [SQL][HIVE] Correct an assert message in function makeRDDForTable ## What changes were proposed in this pull request? according to the context, "makeRDDForTablePartitions" in assert message should be

spark git commit: [SQL][HIVE] Correct an assert message in function makeRDDForTable

2018-07-23 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.3 bd6bfacb2 -> f5bc94861 [SQL][HIVE] Correct an assert message in function makeRDDForTable ## What changes were proposed in this pull request? according to the context, "makeRDDForTablePartitions" in assert message should be

spark git commit: [SPARK-24883][SQL] Avro: remove implicit class AvroDataFrameWriter/AvroDataFrameReader

2018-07-23 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 8817c68f5 -> f59de52a2 [SPARK-24883][SQL] Avro: remove implicit class AvroDataFrameWriter/AvroDataFrameReader ## What changes were proposed in this pull request? As per Reynold's comment: