spark git commit: [SPARK-10379] preserve first page in UnsafeShuffleExternalSorter

Repository: spark Updated Branches: refs/heads/branch-1.5 2fce5d880 -> b846a9dc3 [SPARK-10379] preserve first page in UnsafeShuffleExternalSorter Author: Davies Liu Closes #8543 from davies/preserve_page. (cherry picked from commit

spark git commit: [SPARK-10379] preserve first page in UnsafeShuffleExternalSorter

Repository: spark Updated Branches: refs/heads/master 3ddb9b323 -> 62b4690d6 [SPARK-10379] preserve first page in UnsafeShuffleExternalSorter Author: Davies Liu Closes #8543 from davies/preserve_page. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-10247] [CORE] improve readability of a test case in DAGSchedulerSuite

Repository: spark Updated Branches: refs/heads/master f6c447f87 -> 3ddb9b323 [SPARK-10247] [CORE] improve readability of a test case in DAGSchedulerSuite This is pretty minor, just trying to improve the readability of `DAGSchedulerSuite`, I figure every bit helps. Before whenever I read

spark git commit: Removed code duplication in ShuffleBlockFetcherIterator

Repository: spark Updated Branches: refs/heads/master 0985d2c30 -> f6c447f87 Removed code duplication in ShuffleBlockFetcherIterator Added fetchUpToMaxBytes() to prevent having to update both code blocks when a change is made. Author: Evan Racah Closes #8514 from

spark git commit: [SPARK-8707] RDD#toDebugString fails if any cached RDD has invalid partitions

Repository: spark Updated Branches: refs/heads/master 4bd85d06e -> 0985d2c30 [SPARK-8707] RDD#toDebugString fails if any cached RDD has invalid partitions Added numPartitions(evaluate: Boolean) to RDD. With "evaluate=true" the method is same with "partitions.length". With "evaluate=false",

spark git commit: [SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores

Repository: spark Updated Branches: refs/heads/master af3bc59d1 -> f0562e8cd [SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores This is a regression introduced in #4960, this commit fixes it and adds a test. tnachen andrewor14 please review, this should be an e

spark git commit: [SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores

Repository: spark Updated Branches: refs/heads/branch-1.5 bff05aaa0 -> 8cf16191f [SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores This is a regression introduced in #4960, this commit fixes it and adds a test. tnachen andrewor14 please review, this sho

spark git commit: [SPARK-10514] [MESOS] waiting for min no of total cores acquired by Spark by implementing the sufficientResourcesRegistered method

Repository: spark Updated Branches: refs/heads/master f0562e8cd -> a5ef2d060 [SPARK-10514] [MESOS] waiting for min no of total cores acquired by Spark by implementing the sufficientResourcesRegistered method spark.scheduler.minRegisteredResourcesRatio configuration parameter works for YARN

spark git commit: [SPARK-10466] [SQL] UnsafeRow SerDe exception with data spill

:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) ``` To reproduce that with code (thanks andrewor14): ```scala bin/spark-shell --master local --conf spark.shuffle.memoryFraction=0.005 --conf spark.shuffle.sort.bypassMergeThreshold=0 sc.parallelize(1 to 2 * 1000 * 1000, 10) .map { i =>

spark git commit: [SPARK-10466] [SQL] UnsafeRow SerDe exception with data spill

:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) ``` To reproduce that with code (thanks andrewor14): ```scala bin/spark-shell --master local --conf spark.shuffle.memoryFraction=0.005 --conf spark.shuffle.sort.bypassMergeThreshold=0 sc.parallelize(1 to 2 * 1000 * 1000, 10) .map { i =&

spark git commit: [SPARK-10469] [DOC] Try and document the three options

Repository: spark Updated Branches: refs/heads/master e04811137 -> a76bde9da [SPARK-10469] [DOC] Try and document the three options >From JIRA: Add documentation for tungsten-sort. >From the mailing list "I saw a new "spark.shuffle.manager=tungsten-sort" >implemented in

spark git commit: [SPARK-8167] Make tasks that fail from YARN preemption not fail job

N scheduling work. cc JoshRosen because it's similar to output commit coordination we did in the past cc andrewor14 for our discussion on how to get executor exit codes and loss reasons Author: mcheah <mch...@palantir.com> Closes #8007 from mccheah/feature/preemption-handling. Proj

spark git commit: [SPARK-10469] [DOC] Try and document the three options

Repository: spark Updated Branches: refs/heads/branch-1.5 bc70043c8 -> bff05aaa0 [SPARK-10469] [DOC] Try and document the three options >From JIRA: Add documentation for tungsten-sort. >From the mailing list "I saw a new "spark.shuffle.manager=tungsten-sort" >implemented in

spark git commit: [SPARK-9990] [SQL] Create local hash join operator

Repository: spark Updated Branches: refs/heads/master a5ef2d060 -> d88abb7e2 [SPARK-9990] [SQL] Create local hash join operator This PR includes the following changes: - Add SQLConf to LocalNode - Add HashJoinNode - Add ConvertToUnsafeNode and ConvertToSafeNode.scala to test unsafe hash join.

spark git commit: Revert "[SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores"

Repository: spark Updated Branches: refs/heads/branch-1.5 8cf16191f -> 89d351b5a Revert "[SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores" This reverts commit 8cf16191f3e3b0562f22d44b0381bea35ba511d7. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-10576] [BUILD] Move .java files out of src/main/scala

Repository: spark Updated Branches: refs/heads/master 16b6d1861 -> 4e2242bb4 [SPARK-10576] [BUILD] Move .java files out of src/main/scala Move .java files in `src/main/scala` to `src/main/java` root, except for `package-info.java` (to stay next to package.scala) Author: Sean Owen

spark git commit: [SPARK-10564] ThreadingSuite: assertion failures in threads don't fail the test (round 2)

ses #8727 from andrewor14/fix-threading-suite. (cherry picked from commit 7b6c856367b9c36348e80e83959150da9656c4dd) Signed-off-by: Andrew Or <and...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5db51f91 T

spark git commit: [SPARK-10549] scala 2.11 spark on yarn with security - Repl doesn't work

Repository: spark Updated Branches: refs/heads/master 4e2242bb4 -> ffbbc2c58 [SPARK-10549] scala 2.11 spark on yarn with security - Repl doesn't work Make this lazy so that it can set the yarn mode before creating the securityManager. Author: Tom Graves Author:

spark git commit: [SPARK-10543] [CORE] Peak Execution Memory Quantile should be Per-task Basis

28199/17dc2b84-58b8-11e5-92a8-be89ce4d29d1.png) cc andrewor14 I appreciate if you can give feedback on this since I think you introduced display of this metric. Author: Forest Fang <forest.f...@outlook.com> Closes #8726 from saurfang/stagepage. Project: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-10543] [CORE] Peak Execution Memory Quantile should be Per-task Basis

2/9828199/17dc2b84-58b8-11e5-92a8-be89ce4d29d1.png) cc andrewor14 I appreciate if you can give feedback on this since I think you introduced display of this metric. Author: Forest Fang <forest.f...@outlook.com> Closes #8726 from saurfang/stagepage. (cherry picked from commit fd1e8cddf26

spark git commit: [SPARK-9996] [SPARK-9997] [SQL] Add local expand and NestedLoopJoin operators

Repository: spark Updated Branches: refs/heads/master 64f04154e -> 217e49644 [SPARK-9996] [SPARK-9997] [SQL] Add local expand and NestedLoopJoin operators This PR is in conflict with #8535 and #8573. Will update this one when they are merged. Author: zsxwing Closes

spark git commit: [SPARK-10549] scala 2.11 spark on yarn with security - Repl doesn't work

Repository: spark Updated Branches: refs/heads/branch-1.5 a0d564a10 -> 0e1c9d9ff [SPARK-10549] scala 2.11 spark on yarn with security - Repl doesn't work Make this lazy so that it can set the yarn mode before creating the securityManager. Author: Tom Graves Author:

spark git commit: [SPARK-10594] [YARN] Remove reference to --num-executors, add --properties-file

Repository: spark Updated Branches: refs/heads/master 217e49644 -> 16b6d1861 [SPARK-10594] [YARN] Remove reference to --num-executors, add --properties-file `ApplicationMaster` no longer has the `--num-executors` flag, and had an undocumented `--properties-file` configuration option. cc

spark git commit: [SPARK-10564] ThreadingSuite: assertion failures in threads don't fail the test (round 2)

8727 from andrewor14/fix-threading-suite. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7b6c8563 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7b6c8563 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7b6c856

spark git commit: [SPARK-10548] [SPARK-10563] [SQL] Fix concurrent SQL executions

use is `SparkContext`'s local properties are inherited by default. This patch adds a way to exclude keys we don't want to be inherited, and makes SQL go through that code path. Author: Andrew Or <and...@databricks.com> Closes #8710 from andrewor14/concurrent-sql-executions. Project: htt

spark git commit: [SPARK-10548] [SPARK-10563] [SQL] Fix concurrent SQL executions / branch-1.5

563 is considered risky to backport into a maintenance release, so it is disabled by default and enabled only in SQL. Author: Andrew Or <and...@databricks.com> Closes #8721 from andrewor14/concurrent-sql-executions-1.5 and squashes the following commits: 3b9b462 [Andrew Or] Merge branch 'b

spark git commit: [SPARK-10613] [SPARK-10624] [SQL] Reduce LocalNode tests dependency on SQLContext

ks.com> Closes #8764 from andrewor14/sql-local-tests-cleanup. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/35a19f33 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/35a19f33 Diff: http://git-wip-us.apache.org/repos/

spark git commit: [SPARK-10612] [SQL] Add prepare to LocalNode.

Repository: spark Updated Branches: refs/heads/master b6e998634 -> a63cdc769 [SPARK-10612] [SQL] Add prepare to LocalNode. The idea is that we should separate the function call that does memory reservation (i.e. prepare) from the function call that consumes the input (e.g. open()), so all

spark git commit: [SPARK-10575] [SPARK CORE] Wrapped RDD.takeSample with Scope

Repository: spark Updated Branches: refs/heads/master a63cdc769 -> 99ecfa594 [SPARK-10575] [SPARK CORE] Wrapped RDD.takeSample with Scope Remove return statements in RDD.takeSample and wrap it withScope Author: vinodkc Author: vinodkc

spark git commit: [SPARK-9990] [SQL] Local hash join follow-ups

Repository: spark Updated Branches: refs/heads/master e626ac5f5 -> c2af42b5f [SPARK-9990] [SQL] Local hash join follow-ups 1. Hide `LocalNodeIterator` behind the `LocalNode#asIterator` method 2. Add tests for this Author: Andrew Or <and...@databricks.com> Closes #8708 from andrewo

spark git commit: [SPARK-10564] ThreadingSuite: assertion failures in threads don't fail the test

ing real bugs by not propagating assertion failures properly. Author: Andrew Or <and...@databricks.com> Closes #8723 from andrewor14/fix-threading-suite. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d74c6a14 Tree: h

spark git commit: [SPARK-10564] ThreadingSuite: assertion failures in threads don't fail the test

lly masking real bugs by not propagating assertion failures properly. Author: Andrew Or <and...@databricks.com> Closes #8723 from andrewor14/fix-threading-suite. (cherry picked from commit d74c6a143cbd060c25bf14a8d306841b3ec55d03) Signed-off-by: Andrew Or <and...@databricks.com> Project:

spark git commit: [SPARK-9992] [SPARK-9994] [SPARK-9998] [SQL] Implement the local TopK, sample and intersect operators

Repository: spark Updated Branches: refs/heads/master 1eede3b25 -> e626ac5f5 [SPARK-9992] [SPARK-9994] [SPARK-9998] [SQL] Implement the local TopK, sample and intersect operators This PR is in conflict with #8535. I will update this one when #8535 gets merged. Author: zsxwing

spark git commit: [SPARK-10851] [SPARKR] Exception not failing R applications (in yarn cluster mode)

Repository: spark Updated Branches: refs/heads/master 16fd2a2f4 -> c7b29ae64 [SPARK-10851] [SPARKR] Exception not failing R applications (in yarn cluster mode) The YARN backend doesn't like when user code calls System.exit, since it cannot know the exit status and thus cannot set an

spark git commit: [SPARK-10825] [CORE] [TESTS] Fix race conditions in StandaloneDynamicAllocationSuite

Repository: spark Updated Branches: refs/heads/branch-1.5 9b3014bc4 -> d54493279 [SPARK-10825] [CORE] [TESTS] Fix race conditions in StandaloneDynamicAllocationSuite Fix the following issues in StandaloneDynamicAllocationSuite: 1. It should not assume master and workers start in order 2. It

spark git commit: [SPARK-10825] [CORE] [TESTS] Fix race conditions in StandaloneDynamicAllocationSuite

Repository: spark Updated Branches: refs/heads/master 9b9fe5f7b -> dba95ea03 [SPARK-10825] [CORE] [TESTS] Fix race conditions in StandaloneDynamicAllocationSuite Fix the following issues in StandaloneDynamicAllocationSuite: 1. It should not assume master and workers start in order 2. It

spark git commit: [SPARK-10871] include number of executor failures in error msg

Repository: spark Updated Branches: refs/heads/master dba95ea03 -> b7ad54ec7 [SPARK-10871] include number of executor failures in error msg Author: Ryan Williams Closes #8939 from ryan-williams/errmsg. Project:

spark git commit: [SPARK-10871] include number of executor failures in error msg

Repository: spark Updated Branches: refs/heads/branch-1.5 d54493279 -> 3b2387368 [SPARK-10871] include number of executor failures in error msg Author: Ryan Williams Closes #8939 from ryan-williams/errmsg. (cherry picked from commit

spark git commit: [Minor] style fix for previous commit f24316e

Repository: spark Updated Branches: refs/heads/master f24316e6d -> fd61b0048 [Minor] style fix for previous commit f24316e Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fd61b004 Tree:

spark git commit: [SPARK-10695] [DOCUMENTATION] [MESOS] Fixing incorrect value informati…

Repository: spark Updated Branches: refs/heads/master f3b727c80 -> 0bd0e5bed [SPARK-10695] [DOCUMENTATION] [MESOS] Fixing incorrect value informati… …on for spark.mesos.constraints parameter. Author: Akash Mishra Closes #8816 from SleepyThread/constraint-fix.

spark git commit: [SPARK-8567] [SQL] Increase the timeout of o.a.s.sql.hive.HiveSparkSubmitSuite to 5 minutes.

Repository: spark Updated Branches: refs/heads/branch-1.5 d0e6e5312 -> 03215e3e8 [SPARK-8567] [SQL] Increase the timeout of o.a.s.sql.hive.HiveSparkSubmitSuite to 5 minutes. https://issues.apache.org/jira/browse/SPARK-8567 Looks like "SPARK-8368: includes jars passed in through --jars" is

spark git commit: [SPARK-10649] [STREAMING] Prevent inheriting job group and irrelevant job description in streaming jobs

Repository: spark Updated Branches: refs/heads/branch-1.5 f83b6e625 -> d0e6e5312 [SPARK-10649] [STREAMING] Prevent inheriting job group and irrelevant job description in streaming jobs **Note that this PR only for branch 1.5. See #8781 for the solution for Spark master.** The job group,

spark git commit: [SPARK-10458] [SPARK CORE] Added isStopped() method in SparkContext

Repository: spark Updated Branches: refs/heads/master 1fcefef06 -> f24316e6d [SPARK-10458] [SPARK CORE] Added isStopped() method in SparkContext Added isStopped() method in SparkContext Author: Madhusudanan Kandasamy Closes #8749 from kmadhugit/SPARK-10458.

spark git commit: [SQL] [MINOR] map -> foreach.

Repository: spark Updated Branches: refs/heads/branch-1.5 03215e3e8 -> a2b0fee7b [SQL] [MINOR] map -> foreach. DataFrame.explain should use foreach to print the explain content. Author: Reynold Xin Closes #8862 from rxin/map-foreach. (cherry picked from commit

spark git commit: [SQL] [MINOR] map -> foreach.

Repository: spark Updated Branches: refs/heads/master 4da32bc0e -> f3b727c80 [SQL] [MINOR] map -> foreach. DataFrame.explain should use foreach to print the explain content. Author: Reynold Xin Closes #8862 from rxin/map-foreach. Project:

spark git commit: [SPARK-10695] [DOCUMENTATION] [MESOS] Fixing incorrect value informati…

Repository: spark Updated Branches: refs/heads/branch-1.5 a2b0fee7b -> 646155e6e [SPARK-10695] [DOCUMENTATION] [MESOS] Fixing incorrect value informati… …on for spark.mesos.constraints parameter. Author: Akash Mishra Closes #8816 from

spark git commit: [SPARK-8567] [SQL] Increase the timeout of o.a.s.sql.hive.HiveSparkSubmitSuite to 5 minutes.

Repository: spark Updated Branches: refs/heads/master fd61b0048 -> 4da32bc0e [SPARK-8567] [SQL] Increase the timeout of o.a.s.sql.hive.HiveSparkSubmitSuite to 5 minutes. https://issues.apache.org/jira/browse/SPARK-8567 Looks like "SPARK-8368: includes jars passed in through --jars" is

spark git commit: [SPARK-10640] History server fails to parse TaskCommitDenied

Repository: spark Updated Branches: refs/heads/branch-1.5 118ebd405 -> 26187ab74 [SPARK-10640] History server fails to parse TaskCommitDenied ... simply because the code is missing! Author: Andrew Or <and...@databricks.com> Closes #8828 from andrewor14/task-end-reason-json.

spark git commit: [SPARK-10640] History server fails to parse TaskCommitDenied

Repository: spark Updated Branches: refs/heads/master a96ba40f7 -> 61d4c07f4 [SPARK-10640] History server fails to parse TaskCommitDenied ... simply because the code is missing! Author: Andrew Or <and...@databricks.com> Closes #8828 from andrewor14/task-end-reason-json. Proj

spark git commit: [SPARK-10640] History server fails to parse TaskCommitDenied

Repository: spark Updated Branches: refs/heads/branch-1.5 3339916ef -> 5ffd0841e [SPARK-10640] History server fails to parse TaskCommitDenied ... simply because the code is missing! Author: Andrew Or <and...@databricks.com> Closes #8828 from andrewor14/task-end-reason-json.

spark git commit: Revert "[SPARK-10640] History server fails to parse TaskCommitDenied"

Repository: spark Updated Branches: refs/heads/branch-1.5 5ffd0841e -> 118ebd405 Revert "[SPARK-10640] History server fails to parse TaskCommitDenied" This reverts commit 5ffd0841e016301807b0a008af7c3346e9f59e7a. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array

hor: Andrew Or <and...@databricks.com> Closes #8827 from andrewor14/allocate-pointer-array. (cherry picked from commit 7ff8d68cc19299e16dedfd819b9e96480fa6cf44) Signed-off-by: Andrew Or <and...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array

hor: Andrew Or <and...@databricks.com> Closes #8827 from andrewor14/allocate-pointer-array. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7ff8d68c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7ff8d68c Diff: http:

spark git commit: [SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array (round 2)

iff. Author: Andrew Or <and...@databricks.com> Closes #8888 from andrewor14/dont-track-pointer-array. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/83f6f54d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/83f6f54d D

spark git commit: [SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array (round 2)

iff. Author: Andrew Or <and...@databricks.com> Closes #8888 from andrewor14/dont-track-pointer-array. (cherry picked from commit 83f6f54d12a418f5158ee7ee985b54eef8cc1cf0) Signed-off-by: Andrew Or <and...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-12004] Preserve the RDD partitioner through RDD checkpointing

Repository: spark Updated Branches: refs/heads/branch-1.6 1cf9d3858 -> 81db8d086 [SPARK-12004] Preserve the RDD partitioner through RDD checkpointing The solution is the save the RDD partitioner in a separate file in the RDD checkpoint directory. That is, `/_partitioner`. In most cases,

spark git commit: [SPARK-12004] Preserve the RDD partitioner through RDD checkpointing

Repository: spark Updated Branches: refs/heads/master 2cef1cdfb -> 60b541ee1 [SPARK-12004] Preserve the RDD partitioner through RDD checkpointing The solution is the save the RDD partitioner in a separate file in the RDD checkpoint directory. That is, `/_partitioner`. In most cases,

spark git commit: [SPARK-12059][CORE] Avoid assertion error when unexpected state transition met in Master

Repository: spark Updated Branches: refs/heads/master 8fa3e474a -> 7bc9e1db2 [SPARK-12059][CORE] Avoid assertion error when unexpected state transition met in Master Downgrade to warning log for unexpected state transition. andrewor14 please review, thanks a lot. Author: jerryshao

spark git commit: [SPARK-12059][CORE] Avoid assertion error when unexpected state transition met in Master

Repository: spark Updated Branches: refs/heads/branch-1.6 f7ae62c45 -> 8865d87f7 [SPARK-12059][CORE] Avoid assertion error when unexpected state transition met in Master Downgrade to warning log for unexpected state transition. andrewor14 please review, thanks a lot. Author: jerryshao

spark git commit: [SPARK-12108] Make event logs smaller

eak any compatibility. Otherwise, if it is merged into 1.6.1, then we might need to add more backward compatibility handling logic (currently does not exist yet). Author: Andrew Or <and...@databricks.com> Closes #10115 from andrewor14/smaller-event-logs. (cherry picked fr

spark git commit: [SPARK-12108] Make event logs smaller

any compatibility. Otherwise, if it is merged into 1.6.1, then we might need to add more backward compatibility handling logic (currently does not exist yet). Author: Andrew Or <and...@databricks.com> Closes #10115 from andrewor14/smaller-event-logs. Project: http://git-wip-us.apache.

spark git commit: [SPARK-12081] Make unified memory manager work with small heaps

VMs, this leaves `(1024 - 300) * 0.75 = 543MB` for execution and storage. This is proposal (1) listed in the [JIRA](https://issues.apache.org/jira/browse/SPARK-12081). Author: Andrew Or <and...@databricks.com> Closes #10081 from andrewor14/unified-memory-small-heaps. (cherry picked fr

spark git commit: [SPARK-12345][MESOS] Filter SPARK_HOME when submitting Spark jobs with Mesos cluster mode.

Repository: spark Updated Branches: refs/heads/master 26d70bd2b -> ad8c1f0b8 [SPARK-12345][MESOS] Filter SPARK_HOME when submitting Spark jobs with Mesos cluster mode. SPARK_HOME is now causing problem with Mesos cluster mode since spark-submit script has been changed recently to take

spark git commit: [SPARK-12345][MESOS] Filter SPARK_HOME when submitting Spark jobs with Mesos cluster mode.

Repository: spark Updated Branches: refs/heads/branch-1.6 f81512729 -> e5b85713d [SPARK-12345][MESOS] Filter SPARK_HOME when submitting Spark jobs with Mesos cluster mode. SPARK_HOME is now causing problem with Mesos cluster mode since spark-submit script has been changed recently to take

spark git commit: [SPARK-12365][CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called

Repository: spark Updated Branches: refs/heads/master 38d9795a4 -> f590178d7 [SPARK-12365][CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called SPARK-9886 fixed ExternalBlockStore.scala This PR fixes the remaining references to

spark git commit: [SPARK-12365][CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called

Repository: spark Updated Branches: refs/heads/branch-1.6 fb02e4e3b -> 4af64385b [SPARK-12365][CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called SPARK-9886 fixed ExternalBlockStore.scala This PR fixes the remaining references to

spark git commit: [SPARK-12186][WEB UI] Send the complete request URI including the query string when redirecting.

Repository: spark Updated Branches: refs/heads/branch-1.6 4af64385b -> 154567dca [SPARK-12186][WEB UI] Send the complete request URI including the query string when redirecting. Author: Rohit Agarwal Closes #10180 from mindprince/SPARK-12186. (cherry picked from commit

spark git commit: [SPARK-12186][WEB UI] Send the complete request URI including the query string when redirecting.

Repository: spark Updated Branches: refs/heads/master f590178d7 -> fdb382275 [SPARK-12186][WEB UI] Send the complete request URI including the query string when redirecting. Author: Rohit Agarwal Closes #10180 from mindprince/SPARK-12186. Project:

spark git commit: [SPARK-12390] Clean up unused serializer parameter in BlockManager

Repository: spark Updated Branches: refs/heads/master d1508dd9b -> 97678edea [SPARK-12390] Clean up unused serializer parameter in BlockManager No change in functionality is intended. This only changes internal API. Author: Andrew Or <and...@databricks.com> Closes #10343 from a

spark git commit: [SPARK-10248][CORE] track exceptions in dagscheduler event loop in tests

Repository: spark Updated Branches: refs/heads/branch-1.6 638b89bc3 -> fb02e4e3b [SPARK-10248][CORE] track exceptions in dagscheduler event loop in tests `DAGSchedulerEventLoop` normally only logs errors (so it can continue to process more events, from other jobs). However, this is not

spark git commit: [SPARK-10248][CORE] track exceptions in dagscheduler event loop in tests

Repository: spark Updated Branches: refs/heads/master ce5fd4008 -> 38d9795a4 [SPARK-10248][CORE] track exceptions in dagscheduler event loop in tests `DAGSchedulerEventLoop` normally only logs errors (so it can continue to process more events, from other jobs). However, this is not

spark git commit: [SPARK-12386][CORE] Fix NPE when spark.executor.port is set.

Repository: spark Updated Branches: refs/heads/branch-1.6 154567dca -> 4ad08035d [SPARK-12386][CORE] Fix NPE when spark.executor.port is set. Author: Marcelo Vanzin Closes #10339 from vanzin/SPARK-12386. (cherry picked from commit

spark git commit: [SPARK-12386][CORE] Fix NPE when spark.executor.port is set.

Repository: spark Updated Branches: refs/heads/master fdb382275 -> d1508dd9b [SPARK-12386][CORE] Fix NPE when spark.executor.port is set. Author: Marcelo Vanzin Closes #10339 from vanzin/SPARK-12386. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: MAINTENANCE: Automated closing of pull requests.

Repository: spark Updated Branches: refs/heads/master 861549acd -> ce5fd4008 MAINTENANCE: Automated closing of pull requests. This commit exists to close the following pull requests on Github: Closes #1217 (requested by ankurdave, srowen) Closes #4650 (requested by andrewor14) Closes #5

spark git commit: [SPARK-12390] Clean up unused serializer parameter in BlockManager

Repository: spark Updated Branches: refs/heads/branch-1.6 881f2544e -> 88bbb5429 [SPARK-12390] Clean up unused serializer parameter in BlockManager No change in functionality is intended. This only changes internal API. Author: Andrew Or <and...@databricks.com> Closes #10343 from a

spark git commit: [HOTFIX] Compile error from commit 31b3910

Repository: spark Updated Branches: refs/heads/master 31b391019 -> 840bd2e00 [HOTFIX] Compile error from commit 31b3910 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/840bd2e0 Tree:

spark git commit: [SPARK-12130] Replace shuffleManagerClass with shortShuffleMgrNames in ExternalShuffleBlockResolver

t's comparison on the front. cc JoshRosen andrewor14 Author: Lianhui Wang <lianhuiwan...@gmail.com> Closes #10131 from lianhuiwang/spark-12130. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/369127f0 Tree: http:

spark git commit: [SPARK-4117][YARN] Spark on Yarn handle AM being told command from RM

Repository: spark Updated Branches: refs/heads/master a89e8b612 -> ca0690b5e [SPARK-4117][YARN] Spark on Yarn handle AM being told command from RM Spark on Yarn handle AM being told command from RM When RM throws ApplicationAttemptNotFoundException for allocate invocation, making the

spark git commit: [SPARK-12155][SPARK-12253] Fix executor OOM in unified memory management

tps://github.com/andrewor14/spark/blob/fix-oom/core/src/test/scala/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala#L233) that I stole from JoshRosen. **Solution.** Fix the cap on task execution memory. It should take into account the space that could have been freed by storage in addit

spark git commit: [SPARK-12155][SPARK-12253] Fix executor OOM in unified memory management

com/andrewor14/spark/blob/fix-oom/core/src/test/scala/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala#L233) that I stole from JoshRosen. **Solution.** Fix the cap on task execution memory. It should take into account the space that could have been freed by storage in addition to the curr

spark git commit: [SPARK-12251] Document and improve off-heap memory configurations

Repository: spark Updated Branches: refs/heads/branch-1.6 d0307deaa -> 9870e5c7a [SPARK-12251] Document and improve off-heap memory configurations This patch adds documentation for Spark configurations that affect off-heap memory and makes some naming and validation improvements for those

spark git commit: [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution

Repository: spark Updated Branches: refs/heads/master 442a7715a -> aec5ea000 [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution This patch fixes a bug in the eviction of storage memory by execution. ## The bug: In general, execution should be able to evict

spark git commit: [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution

Repository: spark Updated Branches: refs/heads/branch-1.6 acd462420 -> 05e441e12 [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution This patch fixes a bug in the eviction of storage memory by execution. ## The bug: In general, execution should be able to evict

spark git commit: [SPARK-12275] [SQL] No plan for BroadcastHint in some condition - 1.5 backport

Repository: spark Updated Branches: refs/heads/branch-1.5 e4cf12118 -> 0fdf5542b [SPARK-12275] [SQL] No plan for BroadcastHint in some condition - 1.5 backport backport #10265 to branch 1.5. When SparkStrategies.BasicOperators's "case BroadcastHint(child) => apply(child)" is hit, it only

spark git commit: [SPARK-12056][CORE] Part 2 Create a TaskAttemptContext only after calling setConf

Repository: spark Updated Branches: refs/heads/branch-1.6 08aa3b47e -> 9e4ac5645 [SPARK-12056][CORE] Part 2 Create a TaskAttemptContext only after calling setConf This is continuation of SPARK-12056 where change is applied to SqlNewHadoopRDD.scala andrewor14 FYI Author: tedyu <

spark git commit: [SPARK-10477][SQL] using DSL in ColumnPruningSuite to improve readability

Repository: spark Updated Branches: refs/heads/master c5b6b398d -> a89e8b612 [SPARK-10477][SQL] using DSL in ColumnPruningSuite to improve readability Author: Wenchen Fan Closes #8645 from cloud-fan/test. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-12062][CORE] Change Master to asyc rebuild UI when application completes

Repository: spark Updated Branches: refs/heads/branch-1.6 8e9a60031 -> 93095eb29 [SPARK-12062][CORE] Change Master to asyc rebuild UI when application completes This change builds the event history of completed apps asynchronously so the RPC thread will not be blocked and allow new workers

spark git commit: [SPARK-12062][CORE] Change Master to asyc rebuild UI when application completes

Repository: spark Updated Branches: refs/heads/master 8a215d233 -> c5b6b398d [SPARK-12062][CORE] Change Master to asyc rebuild UI when application completes This change builds the event history of completed apps asynchronously so the RPC thread will not be blocked and allow new workers to

spark git commit: [SPARK-12056][CORE] Part 2 Create a TaskAttemptContext only after calling setConf

Repository: spark Updated Branches: refs/heads/master 840bd2e00 -> f725b2ec1 [SPARK-12056][CORE] Part 2 Create a TaskAttemptContext only after calling setConf This is continuation of SPARK-12056 where change is applied to SqlNewHadoopRDD.scala andrewor14 FYI Author: tedyu <

spark git commit: [SPARK-9516][UI] Improvement of Thread Dump Page

Repository: spark Updated Branches: refs/heads/master c2de99a7c -> a63d9edcf [SPARK-9516][UI] Improvement of Thread Dump Page https://issues.apache.org/jira/browse/SPARK-9516 - [x] new look of Thread Dump Page - [x] click column title to sort - [x] grep - [x] search as you type squito

spark git commit: [SPARK-12351][MESOS] Add documentation about submitting Spark with mesos cluster mode.

Repository: spark Updated Branches: refs/heads/branch-1.6 9e4ac5645 -> 2c324d35a [SPARK-12351][MESOS] Add documentation about submitting Spark with mesos cluster mode. Adding more documentation about submitting jobs with mesos cluster mode. Author: Timothy Chen Closes

spark git commit: [SPARK-12351][MESOS] Add documentation about submitting Spark with mesos cluster mode.

Repository: spark Updated Branches: refs/heads/master 369127f03 -> c2de99a7c [SPARK-12351][MESOS] Add documentation about submitting Spark with mesos cluster mode. Adding more documentation about submitting jobs with mesos cluster mode. Author: Timothy Chen Closes

spark git commit: [SPARK-9026][SPARK-4514] Modifications to JobWaiter, FutureAction, and AsyncRDDActions to support non-blocking operation

Repository: spark Updated Branches: refs/heads/master a63d9edcf -> 765a48849 [SPARK-9026][SPARK-4514] Modifications to JobWaiter, FutureAction, and AsyncRDDActions to support non-blocking operation These changes rework the implementations of `SimpleFutureAction`, `ComplexFutureAction`,

spark git commit: [SPARK-10477][SQL] using DSL in ColumnPruningSuite to improve readability

Repository: spark Updated Branches: refs/heads/branch-1.6 93095eb29 -> fb08f7b78 [SPARK-10477][SQL] using DSL in ColumnPruningSuite to improve readability Author: Wenchen Fan Closes #8645 from cloud-fan/test. (cherry picked from commit

spark git commit: [SPARK-12105] [SQL] add convenient show functions

Repository: spark Updated Branches: refs/heads/master 28112657e -> 31b391019 [SPARK-12105] [SQL] add convenient show functions Author: Jean-Baptiste Onofré Closes #10130 from jbonofre/SPARK-12105. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-10123][DEPLOY] Support specifying deploy mode from configuration

Repository: spark Updated Branches: refs/heads/master 765a48849 -> 63ccdef81 [SPARK-10123][DEPLOY] Support specifying deploy mode from configuration Please help to review, thanks a lot. Author: jerryshao Closes #10195 from jerryshao/SPARK-10123. Project:

spark git commit: [SPARK-9886][CORE] Fix to use ShutdownHookManager in

Repository: spark Updated Branches: refs/heads/master 63ccdef81 -> 8a215d233 [SPARK-9886][CORE] Fix to use ShutdownHookManager in ExternalBlockStore.scala Author: Naveen Closes #10313 from naveenminchu/branch-fix-SPARK-9886. Project:

spark git commit: [SPARK-9886][CORE] Fix to use ShutdownHookManager in

Repository: spark Updated Branches: refs/heads/branch-1.6 2c324d35a -> 8e9a60031 [SPARK-9886][CORE] Fix to use ShutdownHookManager in ExternalBlockStore.scala Author: Naveen Closes #10313 from naveenminchu/branch-fix-SPARK-9886. (cherry picked from commit

spark git commit: [SPARK-12486] Worker should kill the executors more forcefully if possible.

Repository: spark Updated Branches: refs/heads/master 962aac4db -> 8f659393b [SPARK-12486] Worker should kill the executors more forcefully if possible. This patch updates the ExecutorRunner's terminate path to use the new java 8 API to terminate processes more forcefully if possible. If the

spark git commit: [SPARK-12486] Worker should kill the executors more forcefully if possible.

Repository: spark Updated Branches: refs/heads/branch-1.6 f7a322382 -> cd0203819 [SPARK-12486] Worker should kill the executors more forcefully if possible. This patch updates the ExecutorRunner's terminate path to use the new java 8 API to terminate processes more forcefully if possible. If

<    5   6   7   8   9   10   11   12   13   14   >