spark git commit: [SPARK-25235][SHELL] Merge the REPL code in Scala 2.11 and 2.12 branches

2018-08-28 Thread dbtsai
Repository: spark Updated Branches: refs/heads/master 38391c9aa -> ff8dcc1d4 [SPARK-25235][SHELL] Merge the REPL code in Scala 2.11 and 2.12 branches ## What changes were proposed in this pull request? Using some reflection tricks to merge Scala 2.11 and 2.12 codebase. ## How was this patch

svn commit: r29001 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_28_20_02-38391c9-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-28 Thread pwendell
Author: pwendell Date: Wed Aug 29 03:16:01 2018 New Revision: 29001 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_28_20_02-38391c9 docs [This commit notification would consist of 1478 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25253][PYSPARK] Refactor local connection & auth code

2018-08-28 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 68ec207a3 -> 38391c9aa [SPARK-25253][PYSPARK] Refactor local connection & auth code This eliminates some duplication in the code to connect to a server on localhost to talk directly to the jvm. Also it gives consistent ipv6 and error

spark git commit: [SPARK-25260][SQL] Fix namespace handling in SchemaConverters.toAvroType

2018-08-28 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 32c8a3d7b -> 68ec207a3 [SPARK-25260][SQL] Fix namespace handling in SchemaConverters.toAvroType ## What changes were proposed in this pull request? `toAvroType` converts spark data type to avro schema. It always appends the record name

spark git commit: [MINOR] Avoid code duplication for nullable in Higher Order function

2018-08-28 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master bbbf81469 -> 32c8a3d7b [MINOR] Avoid code duplication for nullable in Higher Order function ## What changes were proposed in this pull request? Most of `HigherOrderFunction`s have the same `nullable` definition, ie. they are nullable

spark git commit: [SPARK-22357][CORE] SparkContext.binaryFiles ignore minPartitions parameter

2018-08-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master 103854028 -> bbbf81469 [SPARK-22357][CORE] SparkContext.binaryFiles ignore minPartitions parameter ## What changes were proposed in this pull request? Fix the issue that minPartitions was not used in the method. This is a simple fix and I

svn commit: r29000 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_28_16_01-1038540-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-28 Thread pwendell
Author: pwendell Date: Tue Aug 28 23:15:56 2018 New Revision: 29000 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_28_16_01-1038540 docs [This commit notification would consist of 1478 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25212][SQL] Support Filter in ConvertToLocalRelation

2018-08-28 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 7ad18ee9f -> 103854028 [SPARK-25212][SQL] Support Filter in ConvertToLocalRelation ## What changes were proposed in this pull request? Support Filter in ConvertToLocalRelation, similar to how Project works. Additionally, in Optimizer, run

svn commit: r28998 - in /dev/spark/2.3.3-SNAPSHOT-2018_08_28_14_01-306e881-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-28 Thread pwendell
Author: pwendell Date: Tue Aug 28 21:15:27 2018 New Revision: 28998 Log: Apache Spark 2.3.3-SNAPSHOT-2018_08_28_14_01-306e881 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25004][CORE] Add spark.executor.pyspark.memory limit.

2018-08-28 Thread vanzin
Repository: spark Updated Branches: refs/heads/master aff8f15c1 -> 7ad18ee9f [SPARK-25004][CORE] Add spark.executor.pyspark.memory limit. ## What changes were proposed in this pull request? This adds `spark.executor.pyspark.memory` to configure Python's address space limit,

svn commit: r28997 - in /dev/spark/2.4.0-SNAPSHOT-2018_08_28_12_01-aff8f15-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

2018-08-28 Thread pwendell
Author: pwendell Date: Tue Aug 28 19:16:55 2018 New Revision: 28997 Log: Apache Spark 2.4.0-SNAPSHOT-2018_08_28_12_01-aff8f15 docs [This commit notification would consist of 1478 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

spark git commit: [SPARK-25240][SQL] Fix for a deadlock in RECOVER PARTITIONS

2018-08-28 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 4e3f3cebe -> aff8f15c1 [SPARK-25240][SQL] Fix for a deadlock in RECOVER PARTITIONS ## What changes were proposed in this pull request? In the PR, I propose to not perform recursive parallel listening of files in the `scanPartitions`

spark git commit: [SPARK-24704][WEBUI] Fix the order of stages in the DAG graph

2018-08-28 Thread vanzin
Repository: spark Updated Branches: refs/heads/branch-2.3 8db935f97 -> 306e881b6 [SPARK-24704][WEBUI] Fix the order of stages in the DAG graph ## What changes were proposed in this pull request? Before:

spark git commit: [SPARK-23679][YARN] Setting RM_HA_URLS for AmIpFilter to avoid redirect failure in YARN mode

2018-08-28 Thread vanzin
Repository: spark Updated Branches: refs/heads/master de46df549 -> 4e3f3cebe [SPARK-23679][YARN] Setting RM_HA_URLS for AmIpFilter to avoid redirect failure in YARN mode ## What changes were proposed in this pull request? YARN `AmIpFilter` adds a new parameter "RM_HA_URLS" to support RM HA,

spark git commit: [SPARK-23997][SQL] Configurable maximum number of buckets

2018-08-28 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 1149c4efb -> de46df549 [SPARK-23997][SQL] Configurable maximum number of buckets ## What changes were proposed in this pull request? This PR implements the possibility of the user to override the maximum number of buckets when saving to a

spark git commit: [SPARK-25005][SS] Support non-consecutive offsets for Kafka

2018-08-28 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 592e3a42c -> 1149c4efb [SPARK-25005][SS] Support non-consecutive offsets for Kafka ## What changes were proposed in this pull request? As the user uses Kafka transactions to write data, the offsets in Kafka will be non-consecutive. It

spark git commit: [SPARK-25218][CORE] Fix potential resource leaks in TransportServer and SocketAuthHelper

2018-08-28 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 8198ea501 -> 592e3a42c [SPARK-25218][CORE] Fix potential resource leaks in TransportServer and SocketAuthHelper ## What changes were proposed in this pull request? Make sure TransportServer and SocketAuthHelper close the resources for