spark git commit: MAINTENANCE: Automated closing of pull requests.

Repository: spark Updated Branches: refs/heads/master 165cec9c4 - 8327df697 MAINTENANCE: Automated closing of pull requests. This commit exists to close the following pull requests on Github: Closes #2777 (close requested by 'ankurdave') Closes #2947 (close requested by 'nchammas') Closes

spark git commit: SPARK-4455 Exclude dependency on hbase-annotations module

Repository: spark Updated Branches: refs/heads/master 8327df697 - 5f5ac2daf SPARK-4455 Exclude dependency on hbase-annotations module pwendell Please take a look Author: tedyu yuzhih...@gmail.com Closes #3286 from tedyu/master and squashes the following commits: e61e610 [tedyu] SPARK-4455

spark git commit: SPARK-4455 Exclude dependency on hbase-annotations module

Repository: spark Updated Branches: refs/heads/branch-1.2 e0a20994f - 1d0fa7fb0 SPARK-4455 Exclude dependency on hbase-annotations module pwendell Please take a look Author: tedyu yuzhih...@gmail.com Closes #3286 from tedyu/master and squashes the following commits: e61e610 [tedyu]

spark git commit: [SPARK-4467] fix elements read count for ExtrenalSorter

Repository: spark Updated Branches: refs/heads/branch-1.2 1d0fa7fb0 - 9da71f865 [SPARK-4467] fix elements read count for ExtrenalSorter the elementsRead variable should be reset to 0 after each spilling Author: Tianshuo Deng td...@twitter.com Closes #3302 from

spark git commit: [SPARK-4467] fix elements read count for ExtrenalSorter

Repository: spark Updated Branches: refs/heads/master 5f5ac2daf - d75579d09 [SPARK-4467] fix elements read count for ExtrenalSorter the elementsRead variable should be reset to 0 after each spilling Author: Tianshuo Deng td...@twitter.com Closes #3302 from

spark git commit: [SPARK-4480] Avoid many small spills in external data structures (1.1)

Repository: spark Updated Branches: refs/heads/branch-1.1 e22a75923 - 16bf5f3d1 [SPARK-4480] Avoid many small spills in external data structures (1.1) This is the branch-1.1 version of #3353. This requires a separate PR because the code in master has been refactored a little to eliminate

spark git commit: [maven-release-plugin] prepare release v1.1.1-rc2

Repository: spark Updated Branches: refs/heads/branch-1.1 aa3c794f8 - 3693ae5d3 [maven-release-plugin] prepare release v1.1.1-rc2 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3693ae5d Tree:

spark git commit: [maven-release-plugin] prepare for next development iteration

Repository: spark Updated Branches: refs/heads/branch-1.1 3693ae5d3 - 1df1c1d9b [maven-release-plugin] prepare for next development iteration Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1df1c1d9 Tree:

spark git commit: [SPARK-4470] Validate number of threads in local mode

Repository: spark Updated Branches: refs/heads/branch-1.2 9da71f865 - 2fb40e1aa [SPARK-4470] Validate number of threads in local mode When running Spark locally, if number of threads is specified as 0 (e.g., `spark-submit --master local[0] ...`), the job got stuck and does not run at all. I

spark git commit: [SPARK-4482][Streaming] Disable ReceivedBlockTracker's write ahead log by default

Repository: spark Updated Branches: refs/heads/master eacc78834 - 22fc4e751 [SPARK-4482][Streaming] Disable ReceivedBlockTracker's write ahead log by default The write ahead log of ReceivedBlockTracker gets enabled as soon as checkpoint directory is set. This should not happen, as the WAL

spark git commit: [SPARK-4482][Streaming] Disable ReceivedBlockTracker's write ahead log by default

Repository: spark Updated Branches: refs/heads/branch-1.2 2fb40e1aa - ce5ea0fd6 [SPARK-4482][Streaming] Disable ReceivedBlockTracker's write ahead log by default The write ahead log of ReceivedBlockTracker gets enabled as soon as checkpoint directory is set. This should not happen, as the

spark git commit: [SPARK-4481][Streaming][Doc] Fix the wrong description of updateFunc

Repository: spark Updated Branches: refs/heads/master 22fc4e751 - 3bf7ceebb [SPARK-4481][Streaming][Doc] Fix the wrong description of updateFunc Removed `If `this` function returns None, then corresponding state key-value pair will be eliminated.` for the description of `updateFunc:

spark git commit: [HOT FIX] MiMa tests are broken

Repository: spark Updated Branches: refs/heads/master 3bf7ceebb - 0df02ca46 [HOT FIX] MiMa tests are broken This is blocking #3353 and other patches. Author: Andrew Or and...@databricks.com Closes #3371 from andrewor14/mima-hot-fix and squashes the following commits: 842d059 [Andrew Or]

spark git commit: SPARK-3962 Marked scope as provided for external projects.

Repository: spark Updated Branches: refs/heads/master 0df02ca46 - 1c938413b SPARK-3962 Marked scope as provided for external projects. Somehow maven shade plugin is set in infinite loop of creating effective pom. Author: Prashant Sharma prashan...@imaginea.com Author: Prashant Sharma

spark git commit: SPARK-3962 Marked scope as provided for external projects.

Repository: spark Updated Branches: refs/heads/branch-1.2 ce5ea0fd6 - fc73171d5 SPARK-3962 Marked scope as provided for external projects. Somehow maven shade plugin is set in infinite loop of creating effective pom. Author: Prashant Sharma prashan...@imaginea.com Author: Prashant Sharma

spark git commit: [DOC][PySpark][Streaming] Fix docstring for sphinx

Repository: spark Updated Branches: refs/heads/master 1c938413b - 9b7bbcef8 [DOC][PySpark][Streaming] Fix docstring for sphinx This commit should be merged for 1.2 release. cc tdas Author: Ken Takagiwa ugw.gi.wo...@gmail.com Closes #3311 from giwa/patch-3 and squashes the following commits:

spark git commit: [SPARK-4429][BUILD] Build for Scala 2.11 using sbt fails.

Repository: spark Updated Branches: refs/heads/master 9b7bbcef8 - f9adda9af [SPARK-4429][BUILD] Build for Scala 2.11 using sbt fails. I tried to build for Scala 2.11 using sbt with the following command: ``` $ sbt/sbt -Dscala-2.11 assembly ``` but it ends with the following error messages:

spark git commit: [SPARK-4429][BUILD] Build for Scala 2.11 using sbt fails.

Repository: spark Updated Branches: refs/heads/branch-1.2 fc73171d5 - 633d67cb7 [SPARK-4429][BUILD] Build for Scala 2.11 using sbt fails. I tried to build for Scala 2.11 using sbt with the following command: ``` $ sbt/sbt -Dscala-2.11 assembly ``` but it ends with the following error

spark git commit: [SPARK-4384] [PySpark] improve sort spilling

Repository: spark Updated Branches: refs/heads/branch-1.2 633d67cb7 - 8ecabf4b7 [SPARK-4384] [PySpark] improve sort spilling If there some big broadcasts (or other object) in Python worker, the free memory could be used for sorting will be too small, then it will keep spilling small files

spark git commit: [SPARK-4384] [PySpark] improve sort spilling

Repository: spark Updated Branches: refs/heads/master f9adda9af - 73c8ea84a [SPARK-4384] [PySpark] improve sort spilling If there some big broadcasts (or other object) in Python worker, the free memory could be used for sorting will be too small, then it will keep spilling small files into

spark git commit: [DOC][PySpark][Streaming] Fix docstring for sphinx

Repository: spark Updated Branches: refs/heads/branch-1.2 8ecabf4b7 - c4abb2eb4 [DOC][PySpark][Streaming] Fix docstring for sphinx This commit should be merged for 1.2 release. cc tdas Author: Ken Takagiwa ugw.gi.wo...@gmail.com Closes #3311 from giwa/patch-3 and squashes the following

spark git commit: [SPARK-4294][Streaming] UnionDStream stream should express the requirements in the same way as TransformedDStream

Repository: spark Updated Branches: refs/heads/branch-1.2 c4abb2eb4 - a250ca369 [SPARK-4294][Streaming] UnionDStream stream should express the requirements in the same way as TransformedDStream In class TransformedDStream: ```scala require(parents.length 0, List of DStreams to transform is

spark git commit: [SPARK-4294][Streaming] UnionDStream stream should express the requirements in the same way as TransformedDStream

Repository: spark Updated Branches: refs/heads/master 73c8ea84a - c3002c4a6 [SPARK-4294][Streaming] UnionDStream stream should express the requirements in the same way as TransformedDStream In class TransformedDStream: ```scala require(parents.length 0, List of DStreams to transform is

spark git commit: [SPARK-4495] Fix memory leak in JobProgressListener

Repository: spark Updated Branches: refs/heads/master c3002c4a6 - 04d462f64 [SPARK-4495] Fix memory leak in JobProgressListener This commit fixes a memory leak in JobProgressListener that I introduced in SPARK-2321 and adds a testing framework to ensure that it’s very difficult to

spark git commit: [SPARK-4495] Fix memory leak in JobProgressListener

Repository: spark Updated Branches: refs/heads/branch-1.2 a250ca369 - a7c64cc8f [SPARK-4495] Fix memory leak in JobProgressListener This commit fixes a memory leak in JobProgressListener that I introduced in SPARK-2321 and adds a testing framework to ensure that it’s very difficult to

spark git commit: Updating GraphX programming guide and documentation

Repository: spark Updated Branches: refs/heads/branch-1.2 a7c64cc8f - 8786ddd48 Updating GraphX programming guide and documentation This pull request revises the programming guide to reflect changes in the GraphX API as well as the deprecated mapReduceTriplets operator. Author: Joseph E.

spark git commit: Updating GraphX programming guide and documentation

Repository: spark Updated Branches: refs/heads/master 04d462f64 - 377b06820 Updating GraphX programming guide and documentation This pull request revises the programming guide to reflect changes in the GraphX API as well as the deprecated mapReduceTriplets operator. Author: Joseph E.

spark git commit: [Spark-4484] Treat maxResultSize as unlimited when set to 0; improve error message

Repository: spark Updated Branches: refs/heads/master 9ccc53c72 - 73fedf5a6 [Spark-4484] Treat maxResultSize as unlimited when set to 0; improve error message The check for maxResultSize 0 is missing, results in failures. Also, error message needs to be improved so the developers know that

spark git commit: [SPARK-4480] Avoid many small spills in external data structures

Repository: spark Updated Branches: refs/heads/branch-1.2 f21e550e3 - 4a5c3d21b [SPARK-4480] Avoid many small spills in external data structures **Summary.** Currently, we may spill many small files in `ExternalAppendOnlyMap` and `ExternalSorter`. The underlying root cause of this is

spark git commit: [SPARK-4480] Avoid many small spills in external data structures

Repository: spark Updated Branches: refs/heads/master 73fedf5a6 - 0eb4a7fb0 [SPARK-4480] Avoid many small spills in external data structures **Summary.** Currently, we may spill many small files in `ExternalAppendOnlyMap` and `ExternalSorter`. The underlying root cause of this is summarized

spark git commit: [SPARK-4446] [SPARK CORE]

Repository: spark Updated Branches: refs/heads/branch-1.2 4a5c3d21b - 83d24efb0 [SPARK-4446] [SPARK CORE] MetadataCleaner schedule task with a wrong param for delay time . Author: Leolh leosand...@gmail.com Closes #3306 from Leolh/master and squashes the following commits: 4a21f4e [Leolh]

spark git commit: [SPARK-4446] [SPARK CORE]

Repository: spark Updated Branches: refs/heads/master 0eb4a7fb0 - e216ffaea [SPARK-4446] [SPARK CORE] MetadataCleaner schedule task with a wrong param for delay time . Author: Leolh leosand...@gmail.com Closes #3306 from Leolh/master and squashes the following commits: 4a21f4e [Leolh]