spark git commit: [SPARK-24948][SHS][BACKPORT-2.2] Delegate check access permissions to the file system

2018-08-07 Thread jshao
Repository: spark Updated Branches: refs/heads/branch-2.2 a5624c7ae -> 53ac8504b [SPARK-24948][SHS][BACKPORT-2.2] Delegate check access permissions to the file system ## What changes were proposed in this pull request? In `SparkHadoopUtil. checkAccessPermission`, we consider only basic

spark git commit: [SPARK-23935][SQL][FOLLOWUP] mapEntry throws org.codehaus.commons.compiler.CompileException

2018-08-07 Thread ueshin
Repository: spark Updated Branches: refs/heads/master c7a229d65 -> f08f6f431 [SPARK-23935][SQL][FOLLOWUP] mapEntry throws org.codehaus.commons.compiler.CompileException ## What changes were proposed in this pull request? This PR fixes an exception during the compilation of generated code of

svn commit: r28613 - in /dev/spark/2.3.3-SNAPSHOT-2018_08_07_22_01-9fb70f4-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-07 Thread pwendell
Author: pwendell Date: Wed Aug 8 05:15:52 2018 New Revision: 28613 Log: Apache Spark 2.3.3-SNAPSHOT-2018_08_07_22_01-9fb70f4 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r28612 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_07_20_02-5fef6e3-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-07 Thread pwendell
Author: pwendell Date: Wed Aug 8 03:16:18 2018 New Revision: 28612 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_07_20_02-5fef6e3 docs [This commit notification would consist of 1476 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25010][SQL][FOLLOWUP] Shuffle should also produce different values for each execution in streaming query.

2018-08-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 5fef6e351 -> c7a229d65 [SPARK-25010][SQL][FOLLOWUP] Shuffle should also produce different values for each execution in streaming query. ## What changes were proposed in this pull request? This is a follow-up pr of #21980. `Shuffle` can

spark git commit: [SPARK-24948][SHS][BACKPORT-2.3] Delegate check access permissions to the file system

2018-08-07 Thread jshao
Repository: spark Updated Branches: refs/heads/branch-2.3 136588e95 -> 9fb70f458 [SPARK-24948][SHS][BACKPORT-2.3] Delegate check access permissions to the file system ## What changes were proposed in this pull request? In `SparkHadoopUtil. checkAccessPermission`, we consider only basic

spark git commit: [SPARK-24251][SQL] Add AppendData logical plan.

2018-08-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master d90f1336d -> 5fef6e351 [SPARK-24251][SQL] Add AppendData logical plan. ## What changes were proposed in this pull request? This adds a new logical plan, AppendData, that was proposed in SPARK-23521: Standardize SQL logical plans. *

spark git commit: [SPARK-25045][CORE] Make `RDDBarrier.mapParititions` similar to `RDD.mapPartitions`

2018-08-07 Thread meng
Repository: spark Updated Branches: refs/heads/master 66699c5c3 -> d90f1336d [SPARK-25045][CORE] Make `RDDBarrier.mapParititions` similar to `RDD.mapPartitions` ## What changes were proposed in this pull request? Signature of the function passed to `RDDBarrier.mapPartitions()` is different

svn commit: r28611 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_07_16_02-66699c5-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-07 Thread pwendell
Author: pwendell Date: Tue Aug 7 23:15:57 2018 New Revision: 28611 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_07_16_02-66699c5 docs [This commit notification would consist of 1475 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25029][TESTS] Scala 2.12 issues: TaskNotSerializable and Janino "Two non-abstract methods ..." errors

2018-08-07 Thread srowen
Repository: spark Updated Branches: refs/heads/master f6356f9bc -> 66699c5c3 [SPARK-25029][TESTS] Scala 2.12 issues: TaskNotSerializable and Janino "Two non-abstract methods ..." errors ## What changes were proposed in this pull request? Fixes for test issues that arose after Scala 2.12

spark git commit: [SPARK-25046][SQL] Fix Alter View can excute sql like "ALTER VIEW ... AS INSERT INTO"

2018-08-07 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 8c13cb2ae -> f6356f9bc [SPARK-25046][SQL] Fix Alter View can excute sql like "ALTER VIEW ... AS INSERT INTO" ## What changes were proposed in this pull request? Alter View can excute sql like "ALTER VIEW ... AS INSERT INTO" . We should

svn commit: r28602 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_07_12_02-8c13cb2-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-07 Thread pwendell
Author: pwendell Date: Tue Aug 7 19:16:15 2018 New Revision: 28602 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_07_12_02-8c13cb2 docs [This commit notification would consist of 1475 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25031][SQL] Fix MapType schema print

2018-08-07 Thread lixiao
Repository: spark Updated Branches: refs/heads/master cb6cb3136 -> 8c13cb2ae [SPARK-25031][SQL] Fix MapType schema print ## What changes were proposed in this pull request? The PR fix the bug in `buildFormattedString` function in `MapType`, which makes the printed schema misleading. ## How

spark git commit: [SPARK-23937][SQL] Add map_filter SQL function

2018-08-07 Thread ueshin
Repository: spark Updated Branches: refs/heads/master 298e80f5c -> cb6cb3136 [SPARK-23937][SQL] Add map_filter SQL function ## What changes were proposed in this pull request? The PR adds the high order function `map_filter`, which filters the entries of a map and returns a new map which

spark git commit: [SPARK-25041][BUILD] upgrade genJavaDoc-plugin from 0.10 to 0.11

2018-08-07 Thread srowen
Repository: spark Updated Branches: refs/heads/master 1a29fec8e -> 298e80f5c [SPARK-25041][BUILD] upgrade genJavaDoc-plugin from 0.10 to 0.11 ## What changes were proposed in this pull request? This PR fixes a build error with sbt using Scala-2.12. Since [`genJavaDoc-plugin`]

spark git commit: [SPARK-24979][SQL] add AnalysisHelper#resolveOperatorsUp

2018-08-07 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 6a143e3eb -> 1a29fec8e [SPARK-24979][SQL] add AnalysisHelper#resolveOperatorsUp ## What changes were proposed in this pull request? This is a followup of https://github.com/apache/spark/pull/21822 Similar to `TreeNode`, `AnalysisHelper`

svn commit: r28600 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_07_08_02-6a143e3-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-07 Thread pwendell
Author: pwendell Date: Tue Aug 7 15:16:24 2018 New Revision: 28600 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_07_08_02-6a143e3 docs [This commit notification would consist of 1473 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-23928][TESTS][FOLLOWUP] Set seed to avoid flakiness

2018-08-07 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master b4bf8be54 -> 6a143e3eb [SPARK-23928][TESTS][FOLLOWUP] Set seed to avoid flakiness ## What changes were proposed in this pull request? The tests for shuffle can be flaky (eg.

spark git commit: [SPARK-19602][SQL] Support column resolution of fully qualified column name ( 3 part name)

2018-08-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 819c4de45 -> b4bf8be54 [SPARK-19602][SQL] Support column resolution of fully qualified column name ( 3 part name) ## What changes were proposed in this pull request? The design details is attached to the JIRA issue

svn commit: r28591 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_07_04_02-819c4de-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-07 Thread pwendell
Author: pwendell Date: Tue Aug 7 11:21:17 2018 New Revision: 28591 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_07_04_02-819c4de docs [This commit notification would consist of 1473 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-24772][SQL] Avro: support logical date type

2018-08-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 131ca146e -> 819c4de45 [SPARK-24772][SQL] Avro: support logical date type ## What changes were proposed in this pull request? Support Avro logical date type: https://avro.apache.org/docs/1.8.2/spec.html#Date ## How was this patch tested?

spark git commit: [SPARK-24005][CORE] Remove usage of Scala’s parallel collection

2018-08-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 88e0c7bbd -> 131ca146e [SPARK-24005][CORE] Remove usage of Scala’s parallel collection ## What changes were proposed in this pull request? In the PR, I propose to replace Scala parallel collections by new methods `parmap()`. The

spark git commit: [SPARK-24341][SQL] Support only IN subqueries with the same number of items per row

2018-08-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 388f5a063 -> 88e0c7bbd [SPARK-24341][SQL] Support only IN subqueries with the same number of items per row ## What changes were proposed in this pull request? Using struct types in subqueries with the `IN` clause can generate invalid

svn commit: r28588 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_07_00_02-4376362-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-07 Thread pwendell
Author: pwendell Date: Tue Aug 7 07:16:24 2018 New Revision: 28588 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_07_00_02-4376362 docs [This commit notification would consist of 1473 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-24817][CORE] Implement BarrierTaskContext.barrier()

2018-08-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 43763629f -> 388f5a063 [SPARK-24817][CORE] Implement BarrierTaskContext.barrier() ## What changes were proposed in this pull request? Implement BarrierTaskContext.barrier(), to support global sync between all the tasks in a barrier

spark git commit: [SPARK-25010][SQL] Rand/Randn should produce different values for each execution in streaming query

2018-08-07 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 4446a0b0d -> 43763629f [SPARK-25010][SQL] Rand/Randn should produce different values for each execution in streaming query ## What changes were proposed in this pull request? Like Uuid in SPARK-24896, Rand and Randn expressions now