spark git commit: [SPARK-11667] Update dynamic allocation docs to reflect supported cluster managers

Repository: spark Updated Branches: refs/heads/branch-1.6 a98cac26f -> 782885786 [SPARK-11667] Update dynamic allocation docs to reflect supported cluster managers Author: Andrew Or <and...@databricks.com> Closes #9637 from andrewor14/update-da-docs. (cherry picked fr

spark git commit: [SPARK-2533] Add locality levels on stage summary view

Repository: spark Updated Branches: refs/heads/master 380dfcc0d -> 74c30049a [SPARK-2533] Add locality levels on stage summary view Author: Jean-Baptiste Onofré Closes #9487 from jbonofre/SPARK-2533-2. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-2533] Add locality levels on stage summary view

Repository: spark Updated Branches: refs/heads/branch-1.6 340ca9e76 -> 069591799 [SPARK-2533] Add locality levels on stage summary view Author: Jean-Baptiste Onofré Closes #9487 from jbonofre/SPARK-2533-2. (cherry picked from commit

spark git commit: [SPARK-11658] simplify documentation for PySpark combineByKey

Repository: spark Updated Branches: refs/heads/branch-1.6 782885786 -> 0dd6c2987 [SPARK-11658] simplify documentation for PySpark combineByKey Author: Chris Snow Closes #9640 from snowch/patch-3. (cherry picked from commit 68ef61bb656bd9c08239726913ca8ab271d52786)

spark git commit: [SPARK-11658] simplify documentation for PySpark combineByKey

Repository: spark Updated Branches: refs/heads/master 12a0784ac -> 68ef61bb6 [SPARK-11658] simplify documentation for PySpark combineByKey Author: Chris Snow Closes #9640 from snowch/patch-3. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-10192][CORE] simple test w/ failure involving a shared dependency

Repository: spark Updated Branches: refs/heads/master c0e48dfa6 -> 33112f9c4 [SPARK-10192][CORE] simple test w/ failure involving a shared dependency just trying to increase test coverage in the scheduler, this already works. It includes a regression test for SPARK-9809 copied some test

spark git commit: [SPARK-10192][CORE] simple test w/ failure involving a shared dependency

Repository: spark Updated Branches: refs/heads/branch-1.6 d8bfc025c -> 10272d5c9 [SPARK-10192][CORE] simple test w/ failure involving a shared dependency just trying to increase test coverage in the scheduler, this already works. It includes a regression test for SPARK-9809 copied some

spark git commit: [SPARK-11361][STREAMING] Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz

Repository: spark Updated Branches: refs/heads/master 900917541 -> 6600786dd [SPARK-11361][STREAMING] Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz Currently, when a DStream sets the scope for RDD generated by it, that scope is not allowed to be

spark git commit: [SPARK-11361][STREAMING] Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz

Repository: spark Updated Branches: refs/heads/branch-1.6 fafeca36e -> b34819c7b [SPARK-11361][STREAMING] Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz Currently, when a DStream sets the scope for RDD generated by it, that scope is not allowed to

spark git commit: [SPARK-11615] Drop @VisibleForTesting annotation

Repository: spark Updated Branches: refs/heads/branch-1.6 93ac30741 -> fafeca36e [SPARK-11615] Drop @VisibleForTesting annotation See http://search-hadoop.com/m/q3RTtjpe8r1iRbTj2 for discussion. Summary: addition of VisibleForTesting annotation resulted in spark-shell malfunctioning.

spark git commit: [MINOR] Fix typo in AggregationQuerySuite.scala

Repository: spark Updated Branches: refs/heads/master 6600786dd -> 12c7635dc [MINOR] Fix typo in AggregationQuerySuite.scala Author: Forest Fang Closes #9357 from saurfang/patch-1. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-11572] Exit AsynchronousListenerBus thread when stop() is called

Repository: spark Updated Branches: refs/heads/master 33112f9c4 -> 3e0a6cf1e [SPARK-11572] Exit AsynchronousListenerBus thread when stop() is called As vonnagy reported in the following thread: http://search-hadoop.com/m/q3RTtk982kvIow22 Attempts to join the thread in AsynchronousListenerBus

spark git commit: [SPARK-11572] Exit AsynchronousListenerBus thread when stop() is called

Repository: spark Updated Branches: refs/heads/branch-1.6 10272d5c9 -> 93ac30741 [SPARK-11572] Exit AsynchronousListenerBus thread when stop() is called As vonnagy reported in the following thread: http://search-hadoop.com/m/q3RTtk982kvIow22 Attempts to join the thread in

spark git commit: [SPARK-11615] Drop @VisibleForTesting annotation

Repository: spark Updated Branches: refs/heads/master 3e0a6cf1e -> 900917541 [SPARK-11615] Drop @VisibleForTesting annotation See http://search-hadoop.com/m/q3RTtjpe8r1iRbTj2 for discussion. Summary: addition of VisibleForTesting annotation resulted in spark-shell malfunctioning. Author:

spark git commit: [MINOR] License header formatting fix

Repository: spark Updated Branches: refs/heads/master 12c7635dc -> 745e45d5f [MINOR] License header formatting fix The header wasn't indented properly. Author: Marc Prud'hommeaux Closes #9312 from mprudhom/patch-1. Project:

spark git commit: [SPARK-10471][CORE][MESOS] prevent getting offers for unmet constraints

Repository: spark Updated Branches: refs/heads/master 88a3fdcc7 -> 5039a49b6 [SPARK-10471][CORE][MESOS] prevent getting offers for unmet constraints this change rejects offers for slaves with unmet constraints for 120s to mitigate offer starvation. this prevents mesos to send us these offers

spark git commit: [SPARK-10471][CORE][MESOS] prevent getting offers for unmet constraints

Repository: spark Updated Branches: refs/heads/branch-1.6 2459b3432 -> 74f50275e [SPARK-10471][CORE][MESOS] prevent getting offers for unmet constraints this change rejects offers for slaves with unmet constraints for 120s to mitigate offer starvation. this prevents mesos to send us these

spark git commit: [SPARK-11112] Fix Scala 2.11 compilation error in RDDInfo.scala

Repository: spark Updated Branches: refs/heads/branch-1.6 2946c85f5 -> fb469e76a [SPARK-2] Fix Scala 2.11 compilation error in RDDInfo.scala As shown in https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/job/Spark-Master-Scala211-Compile/1946/console , compilation fails with:

spark git commit: [SPARK-11112] Fix Scala 2.11 compilation error in RDDInfo.scala

Repository: spark Updated Branches: refs/heads/master 08a7a836c -> 404a28f4e [SPARK-2] Fix Scala 2.11 compilation error in RDDInfo.scala As shown in https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/job/Spark-Master-Scala211-Compile/1946/console , compilation fails with: ```

spark git commit: [SPARK-11362] [SQL] Use Spark BitSet in BroadcastNestedLoopJoin

Repository: spark Updated Branches: refs/heads/branch-1.6 fc2942d12 -> 2946c85f5 [SPARK-11362] [SQL] Use Spark BitSet in BroadcastNestedLoopJoin JIRA: https://issues.apache.org/jira/browse/SPARK-11362 We use scala.collection.mutable.BitSet in BroadcastNestedLoopJoin now. We should use

spark git commit: [SPARK-11112] DAG visualization: display RDD callsite

ricks.com> Closes #9398 from andrewor14/rdd-callsite. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7f741905 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7f741905 Diff: http://git-wip-us.apache.org/repos/as

spark git commit: [SPARK-11114][PYSPARK] add getOrCreate for SparkContext/SQLContext in Python

Repository: spark Updated Branches: refs/heads/master a1413b366 -> 232d7f8d4 [SPARK-4][PYSPARK] add getOrCreate for SparkContext/SQLContext in Python Also added SQLContext.newSession() Author: Davies Liu Closes #9122 from davies/py_create. Project:

spark git commit: [SPARK-9708][MESOS] Spark should create local temporary directories in Mesos sandbox when launched with Mesos

Repository: spark Updated Branches: refs/heads/master 232d7f8d4 -> fc26f32cf [SPARK-9708][MESOS] Spark should create local temporary directories in Mesos sandbox when launched with Mesos This is my own original work and I license this to the project under the project's open source license

spark git commit: [SPARK-11131][CORE] Fix race in worker registration protocol.

Repository: spark Updated Branches: refs/heads/master 67582132b -> 7ab0ce650 [SPARK-11131][CORE] Fix race in worker registration protocol. Because the registration RPC was not really an RPC, but a bunch of disconnected messages, it was possible for other messages to be sent before the reply

spark git commit: [SPARK-11120] Allow sane default number of executor failures when dynamically allocating in YARN

Repository: spark Updated Branches: refs/heads/master fc26f32cf -> 16906ef23 [SPARK-11120] Allow sane default number of executor failures when dynamically allocating in YARN I also added some information to container-failure error msgs about what host they failed on, which would have helped

spark git commit: [SPARK-11051][CORE] Do not allow local checkpointing after the RDD is materialized and checkpointed

Repository: spark Updated Branches: refs/heads/master 7ab0ce650 -> a1413b366 [SPARK-11051][CORE] Do not allow local checkpointing after the RDD is materialized and checkpointed JIRA: https://issues.apache.org/jira/browse/SPARK-11051 When a `RDD` is materialized and checkpointed, its

spark git commit: [SPARK-11051][CORE] Do not allow local checkpointing after the RDD is materialized and checkpointed

Repository: spark Updated Branches: refs/heads/branch-1.5 5186ec8ac -> 2195fec12 [SPARK-11051][CORE] Do not allow local checkpointing after the RDD is materialized and checkpointed JIRA: https://issues.apache.org/jira/browse/SPARK-11051 When a `RDD` is materialized and checkpointed, its

spark git commit: [SPARK-10515] When killing executor, the pending replacement executors should not be lost

Repository: spark Updated Branches: refs/heads/branch-1.5 166fdf4e3 -> 13920d5fe [SPARK-10515] When killing executor, the pending replacement executors should not be lost If the heartbeat receiver kills executors (and new ones are not registered to replace them), the idle timeout for the

spark git commit: [SPARK-11078] Ensure spilling tests actually spill

ial bugs in spilling. The size of this patch is inflated by the refactoring of `ExternalSorterSuite`, which had a lot of duplicate code and logic. Author: Andrew Or <and...@databricks.com> Closes #9124 from andrewor14/spilling-tests. Project: http://git-wip-us.apache.org/repos/asf/spark/re

spark git commit: [SPARK-10515] When killing executor, the pending replacement executors should not be lost

Repository: spark Updated Branches: refs/heads/master 723aa75a9 -> 2d000124b [SPARK-10515] When killing executor, the pending replacement executors should not be lost If the heartbeat receiver kills executors (and new ones are not registered to replace them), the idle timeout for the old

spark git commit: [SPARK-11071] [LAUNCHER] Fix flakiness in LauncherServerSuite::timeout.

Repository: spark Updated Branches: refs/heads/master b591de7c0 -> a5719804c [SPARK-11071] [LAUNCHER] Fix flakiness in LauncherServerSuite::timeout. The test could fail depending on scheduling of the various threads involved; the change removes some sources of races, while making the test a

spark git commit: fix typo bellow -> below

Repository: spark Updated Branches: refs/heads/master a5719804c -> 723aa75a9 fix typo bellow -> below Author: Britta Weber Closes #9136 from brwe/typo-bellow. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-10412] [SQL] report memory usage for tungsten sql physical operator

Repository: spark Updated Branches: refs/heads/master 3b364ff0a -> 6a2359ff1 [SPARK-10412] [SQL] report memory usage for tungsten sql physical operator https://issues.apache.org/jira/browse/SPARK-10412 some screenshots: ### aggregate: ![screen shot 2015-10-12 at 2 23 11

spark git commit: [SPARK-10887] [SQL] Build HashedRelation outside of HashJoinNode.

Repository: spark Updated Branches: refs/heads/master 2a6f614cd -> 82d275f27 [SPARK-10887] [SQL] Build HashedRelation outside of HashJoinNode. This PR refactors `HashJoinNode` to take a existing `HashedRelation`. So, we can reuse this node for both `ShuffledHashJoin` and `BroadcastHashJoin`.

spark git commit: [SPARK-7527] [CORE] Fix createNullValue to return the correct null values and REPL mode detection

Repository: spark Updated Branches: refs/heads/branch-1.4 e7c4346d0 -> e2ff49198 [SPARK-7527] [CORE] Fix createNullValue to return the correct null values and REPL mode detection The root cause of SPARK-7527 is `createNullValue` returns an incompatible value `Byte(0)` for `char` and

spark git commit: [SPARK-10851] [SPARKR] Exception not failing R applications (in yarn cluster mode)

Repository: spark Updated Branches: refs/heads/master 16fd2a2f4 -> c7b29ae64 [SPARK-10851] [SPARKR] Exception not failing R applications (in yarn cluster mode) The YARN backend doesn't like when user code calls System.exit, since it cannot know the exit status and thus cannot set an

spark git commit: [SPARK-10825] [CORE] [TESTS] Fix race conditions in StandaloneDynamicAllocationSuite

Repository: spark Updated Branches: refs/heads/branch-1.5 9b3014bc4 -> d54493279 [SPARK-10825] [CORE] [TESTS] Fix race conditions in StandaloneDynamicAllocationSuite Fix the following issues in StandaloneDynamicAllocationSuite: 1. It should not assume master and workers start in order 2. It

spark git commit: [SPARK-10825] [CORE] [TESTS] Fix race conditions in StandaloneDynamicAllocationSuite

Repository: spark Updated Branches: refs/heads/master 9b9fe5f7b -> dba95ea03 [SPARK-10825] [CORE] [TESTS] Fix race conditions in StandaloneDynamicAllocationSuite Fix the following issues in StandaloneDynamicAllocationSuite: 1. It should not assume master and workers start in order 2. It

spark git commit: [SPARK-10871] include number of executor failures in error msg

Repository: spark Updated Branches: refs/heads/master dba95ea03 -> b7ad54ec7 [SPARK-10871] include number of executor failures in error msg Author: Ryan Williams Closes #8939 from ryan-williams/errmsg. Project:

spark git commit: [SPARK-10871] include number of executor failures in error msg

Repository: spark Updated Branches: refs/heads/branch-1.5 d54493279 -> 3b2387368 [SPARK-10871] include number of executor failures in error msg Author: Ryan Williams Closes #8939 from ryan-williams/errmsg. (cherry picked from commit

spark git commit: [SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array (round 2)

iff. Author: Andrew Or <and...@databricks.com> Closes #8888 from andrewor14/dont-track-pointer-array. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/83f6f54d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/83f6f54d D

spark git commit: [SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array (round 2)

iff. Author: Andrew Or <and...@databricks.com> Closes #8888 from andrewor14/dont-track-pointer-array. (cherry picked from commit 83f6f54d12a418f5158ee7ee985b54eef8cc1cf0) Signed-off-by: Andrew Or <and...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [Minor] style fix for previous commit f24316e

Repository: spark Updated Branches: refs/heads/master f24316e6d -> fd61b0048 [Minor] style fix for previous commit f24316e Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fd61b004 Tree:

spark git commit: [SPARK-10695] [DOCUMENTATION] [MESOS] Fixing incorrect value informati…

Repository: spark Updated Branches: refs/heads/master f3b727c80 -> 0bd0e5bed [SPARK-10695] [DOCUMENTATION] [MESOS] Fixing incorrect value informati… …on for spark.mesos.constraints parameter. Author: Akash Mishra Closes #8816 from SleepyThread/constraint-fix.

spark git commit: [SPARK-8567] [SQL] Increase the timeout of o.a.s.sql.hive.HiveSparkSubmitSuite to 5 minutes.

Repository: spark Updated Branches: refs/heads/branch-1.5 d0e6e5312 -> 03215e3e8 [SPARK-8567] [SQL] Increase the timeout of o.a.s.sql.hive.HiveSparkSubmitSuite to 5 minutes. https://issues.apache.org/jira/browse/SPARK-8567 Looks like "SPARK-8368: includes jars passed in through --jars" is

spark git commit: [SPARK-10649] [STREAMING] Prevent inheriting job group and irrelevant job description in streaming jobs

Repository: spark Updated Branches: refs/heads/branch-1.5 f83b6e625 -> d0e6e5312 [SPARK-10649] [STREAMING] Prevent inheriting job group and irrelevant job description in streaming jobs **Note that this PR only for branch 1.5. See #8781 for the solution for Spark master.** The job group,

spark git commit: [SPARK-10458] [SPARK CORE] Added isStopped() method in SparkContext

Repository: spark Updated Branches: refs/heads/master 1fcefef06 -> f24316e6d [SPARK-10458] [SPARK CORE] Added isStopped() method in SparkContext Added isStopped() method in SparkContext Author: Madhusudanan Kandasamy Closes #8749 from kmadhugit/SPARK-10458.

spark git commit: [SQL] [MINOR] map -> foreach.

Repository: spark Updated Branches: refs/heads/branch-1.5 03215e3e8 -> a2b0fee7b [SQL] [MINOR] map -> foreach. DataFrame.explain should use foreach to print the explain content. Author: Reynold Xin Closes #8862 from rxin/map-foreach. (cherry picked from commit

spark git commit: [SQL] [MINOR] map -> foreach.

Repository: spark Updated Branches: refs/heads/master 4da32bc0e -> f3b727c80 [SQL] [MINOR] map -> foreach. DataFrame.explain should use foreach to print the explain content. Author: Reynold Xin Closes #8862 from rxin/map-foreach. Project:

spark git commit: [SPARK-10695] [DOCUMENTATION] [MESOS] Fixing incorrect value informati…

Repository: spark Updated Branches: refs/heads/branch-1.5 a2b0fee7b -> 646155e6e [SPARK-10695] [DOCUMENTATION] [MESOS] Fixing incorrect value informati… …on for spark.mesos.constraints parameter. Author: Akash Mishra Closes #8816 from

spark git commit: [SPARK-8567] [SQL] Increase the timeout of o.a.s.sql.hive.HiveSparkSubmitSuite to 5 minutes.

Repository: spark Updated Branches: refs/heads/master fd61b0048 -> 4da32bc0e [SPARK-8567] [SQL] Increase the timeout of o.a.s.sql.hive.HiveSparkSubmitSuite to 5 minutes. https://issues.apache.org/jira/browse/SPARK-8567 Looks like "SPARK-8368: includes jars passed in through --jars" is

spark git commit: [SPARK-10640] History server fails to parse TaskCommitDenied

Repository: spark Updated Branches: refs/heads/branch-1.5 118ebd405 -> 26187ab74 [SPARK-10640] History server fails to parse TaskCommitDenied ... simply because the code is missing! Author: Andrew Or <and...@databricks.com> Closes #8828 from andrewor14/task-end-reason-json.

spark git commit: [SPARK-10640] History server fails to parse TaskCommitDenied

Repository: spark Updated Branches: refs/heads/master a96ba40f7 -> 61d4c07f4 [SPARK-10640] History server fails to parse TaskCommitDenied ... simply because the code is missing! Author: Andrew Or <and...@databricks.com> Closes #8828 from andrewor14/task-end-reason-json. Proj

spark git commit: [SPARK-10640] History server fails to parse TaskCommitDenied

Repository: spark Updated Branches: refs/heads/branch-1.5 3339916ef -> 5ffd0841e [SPARK-10640] History server fails to parse TaskCommitDenied ... simply because the code is missing! Author: Andrew Or <and...@databricks.com> Closes #8828 from andrewor14/task-end-reason-json.

spark git commit: Revert "[SPARK-10640] History server fails to parse TaskCommitDenied"

Repository: spark Updated Branches: refs/heads/branch-1.5 5ffd0841e -> 118ebd405 Revert "[SPARK-10640] History server fails to parse TaskCommitDenied" This reverts commit 5ffd0841e016301807b0a008af7c3346e9f59e7a. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array

hor: Andrew Or <and...@databricks.com> Closes #8827 from andrewor14/allocate-pointer-array. (cherry picked from commit 7ff8d68cc19299e16dedfd819b9e96480fa6cf44) Signed-off-by: Andrew Or <and...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-10474] [SQL] Aggregation fails to allocate memory for pointer array

hor: Andrew Or <and...@databricks.com> Closes #8827 from andrewor14/allocate-pointer-array. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7ff8d68c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7ff8d68c Diff: http:

spark git commit: [SPARK-10548] [SPARK-10563] [SQL] Fix concurrent SQL executions

use is `SparkContext`'s local properties are inherited by default. This patch adds a way to exclude keys we don't want to be inherited, and makes SQL go through that code path. Author: Andrew Or <and...@databricks.com> Closes #8710 from andrewor14/concurrent-sql-executions. Project: htt

spark git commit: [SPARK-10548] [SPARK-10563] [SQL] Fix concurrent SQL executions / branch-1.5

563 is considered risky to backport into a maintenance release, so it is disabled by default and enabled only in SQL. Author: Andrew Or <and...@databricks.com> Closes #8721 from andrewor14/concurrent-sql-executions-1.5 and squashes the following commits: 3b9b462 [Andrew Or] Merge branch 'b

spark git commit: [SPARK-10613] [SPARK-10624] [SQL] Reduce LocalNode tests dependency on SQLContext

ks.com> Closes #8764 from andrewor14/sql-local-tests-cleanup. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/35a19f33 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/35a19f33 Diff: http://git-wip-us.apache.org/repos/

spark git commit: [SPARK-10612] [SQL] Add prepare to LocalNode.

Repository: spark Updated Branches: refs/heads/master b6e998634 -> a63cdc769 [SPARK-10612] [SQL] Add prepare to LocalNode. The idea is that we should separate the function call that does memory reservation (i.e. prepare) from the function call that consumes the input (e.g. open()), so all

spark git commit: [SPARK-10575] [SPARK CORE] Wrapped RDD.takeSample with Scope

Repository: spark Updated Branches: refs/heads/master a63cdc769 -> 99ecfa594 [SPARK-10575] [SPARK CORE] Wrapped RDD.takeSample with Scope Remove return statements in RDD.takeSample and wrap it withScope Author: vinodkc Author: vinodkc

spark git commit: [SPARK-10576] [BUILD] Move .java files out of src/main/scala

Repository: spark Updated Branches: refs/heads/master 16b6d1861 -> 4e2242bb4 [SPARK-10576] [BUILD] Move .java files out of src/main/scala Move .java files in `src/main/scala` to `src/main/java` root, except for `package-info.java` (to stay next to package.scala) Author: Sean Owen

spark git commit: [SPARK-10564] ThreadingSuite: assertion failures in threads don't fail the test (round 2)

ses #8727 from andrewor14/fix-threading-suite. (cherry picked from commit 7b6c856367b9c36348e80e83959150da9656c4dd) Signed-off-by: Andrew Or <and...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5db51f91 T

spark git commit: [SPARK-10549] scala 2.11 spark on yarn with security - Repl doesn't work

Repository: spark Updated Branches: refs/heads/master 4e2242bb4 -> ffbbc2c58 [SPARK-10549] scala 2.11 spark on yarn with security - Repl doesn't work Make this lazy so that it can set the yarn mode before creating the securityManager. Author: Tom Graves Author:

spark git commit: [SPARK-10543] [CORE] Peak Execution Memory Quantile should be Per-task Basis

28199/17dc2b84-58b8-11e5-92a8-be89ce4d29d1.png) cc andrewor14 I appreciate if you can give feedback on this since I think you introduced display of this metric. Author: Forest Fang <forest.f...@outlook.com> Closes #8726 from saurfang/stagepage. Project: http://git-wip-us.apache.org/repos

spark git commit: [SPARK-10543] [CORE] Peak Execution Memory Quantile should be Per-task Basis

2/9828199/17dc2b84-58b8-11e5-92a8-be89ce4d29d1.png) cc andrewor14 I appreciate if you can give feedback on this since I think you introduced display of this metric. Author: Forest Fang <forest.f...@outlook.com> Closes #8726 from saurfang/stagepage. (cherry picked from commit fd1e8cddf26

spark git commit: [SPARK-9996] [SPARK-9997] [SQL] Add local expand and NestedLoopJoin operators

Repository: spark Updated Branches: refs/heads/master 64f04154e -> 217e49644 [SPARK-9996] [SPARK-9997] [SQL] Add local expand and NestedLoopJoin operators This PR is in conflict with #8535 and #8573. Will update this one when they are merged. Author: zsxwing Closes

spark git commit: [SPARK-10549] scala 2.11 spark on yarn with security - Repl doesn't work

Repository: spark Updated Branches: refs/heads/branch-1.5 a0d564a10 -> 0e1c9d9ff [SPARK-10549] scala 2.11 spark on yarn with security - Repl doesn't work Make this lazy so that it can set the yarn mode before creating the securityManager. Author: Tom Graves Author:

spark git commit: [SPARK-10594] [YARN] Remove reference to --num-executors, add --properties-file

Repository: spark Updated Branches: refs/heads/master 217e49644 -> 16b6d1861 [SPARK-10594] [YARN] Remove reference to --num-executors, add --properties-file `ApplicationMaster` no longer has the `--num-executors` flag, and had an undocumented `--properties-file` configuration option. cc

spark git commit: [SPARK-10564] ThreadingSuite: assertion failures in threads don't fail the test (round 2)

8727 from andrewor14/fix-threading-suite. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7b6c8563 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7b6c8563 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7b6c856

spark git commit: [SPARK-9990] [SQL] Local hash join follow-ups

Repository: spark Updated Branches: refs/heads/master e626ac5f5 -> c2af42b5f [SPARK-9990] [SQL] Local hash join follow-ups 1. Hide `LocalNodeIterator` behind the `LocalNode#asIterator` method 2. Add tests for this Author: Andrew Or <and...@databricks.com> Closes #8708 from andrewo

spark git commit: [SPARK-10564] ThreadingSuite: assertion failures in threads don't fail the test

ing real bugs by not propagating assertion failures properly. Author: Andrew Or <and...@databricks.com> Closes #8723 from andrewor14/fix-threading-suite. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d74c6a14 Tree: h

spark git commit: [SPARK-10564] ThreadingSuite: assertion failures in threads don't fail the test

lly masking real bugs by not propagating assertion failures properly. Author: Andrew Or <and...@databricks.com> Closes #8723 from andrewor14/fix-threading-suite. (cherry picked from commit d74c6a143cbd060c25bf14a8d306841b3ec55d03) Signed-off-by: Andrew Or <and...@databricks.com> Project:

spark git commit: [SPARK-9992] [SPARK-9994] [SPARK-9998] [SQL] Implement the local TopK, sample and intersect operators

Repository: spark Updated Branches: refs/heads/master 1eede3b25 -> e626ac5f5 [SPARK-9992] [SPARK-9994] [SPARK-9998] [SQL] Implement the local TopK, sample and intersect operators This PR is in conflict with #8535. I will update this one when #8535 gets merged. Author: zsxwing

spark git commit: [SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores

Repository: spark Updated Branches: refs/heads/master af3bc59d1 -> f0562e8cd [SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores This is a regression introduced in #4960, this commit fixes it and adds a test. tnachen andrewor14 please review, this should be an e

spark git commit: [SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores

Repository: spark Updated Branches: refs/heads/branch-1.5 bff05aaa0 -> 8cf16191f [SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores This is a regression introduced in #4960, this commit fixes it and adds a test. tnachen andrewor14 please review, this sho

spark git commit: [SPARK-10514] [MESOS] waiting for min no of total cores acquired by Spark by implementing the sufficientResourcesRegistered method

Repository: spark Updated Branches: refs/heads/master f0562e8cd -> a5ef2d060 [SPARK-10514] [MESOS] waiting for min no of total cores acquired by Spark by implementing the sufficientResourcesRegistered method spark.scheduler.minRegisteredResourcesRatio configuration parameter works for YARN

spark git commit: [SPARK-10466] [SQL] UnsafeRow SerDe exception with data spill

:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) ``` To reproduce that with code (thanks andrewor14): ```scala bin/spark-shell --master local --conf spark.shuffle.memoryFraction=0.005 --conf spark.shuffle.sort.bypassMergeThreshold=0 sc.parallelize(1 to 2 * 1000 * 1000, 10) .map { i =>

spark git commit: [SPARK-10466] [SQL] UnsafeRow SerDe exception with data spill

:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) ``` To reproduce that with code (thanks andrewor14): ```scala bin/spark-shell --master local --conf spark.shuffle.memoryFraction=0.005 --conf spark.shuffle.sort.bypassMergeThreshold=0 sc.parallelize(1 to 2 * 1000 * 1000, 10) .map { i =&

spark git commit: [SPARK-10469] [DOC] Try and document the three options

Repository: spark Updated Branches: refs/heads/master e04811137 -> a76bde9da [SPARK-10469] [DOC] Try and document the three options >From JIRA: Add documentation for tungsten-sort. >From the mailing list "I saw a new "spark.shuffle.manager=tungsten-sort" >implemented in

spark git commit: [SPARK-8167] Make tasks that fail from YARN preemption not fail job

N scheduling work. cc JoshRosen because it's similar to output commit coordination we did in the past cc andrewor14 for our discussion on how to get executor exit codes and loss reasons Author: mcheah <mch...@palantir.com> Closes #8007 from mccheah/feature/preemption-handling. Proj

spark git commit: [SPARK-10469] [DOC] Try and document the three options

Repository: spark Updated Branches: refs/heads/branch-1.5 bc70043c8 -> bff05aaa0 [SPARK-10469] [DOC] Try and document the three options >From JIRA: Add documentation for tungsten-sort. >From the mailing list "I saw a new "spark.shuffle.manager=tungsten-sort" >implemented in

spark git commit: [SPARK-9990] [SQL] Create local hash join operator

Repository: spark Updated Branches: refs/heads/master a5ef2d060 -> d88abb7e2 [SPARK-9990] [SQL] Create local hash join operator This PR includes the following changes: - Add SQLConf to LocalNode - Add HashJoinNode - Add ConvertToUnsafeNode and ConvertToSafeNode.scala to test unsafe hash join.

spark git commit: Revert "[SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores"

Repository: spark Updated Branches: refs/heads/branch-1.5 8cf16191f -> 89d351b5a Revert "[SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores" This reverts commit 8cf16191f3e3b0562f22d44b0381bea35ba511d7. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-5945] Spark should not retry a stage infinitely on a FetchFailedException

Repository: spark Updated Branches: refs/heads/master 44948a2e9 -> 4bd85d06e [SPARK-5945] Spark should not retry a stage infinitely on a FetchFailedException The ```Stage``` class now tracks whether there were a sufficient number of consecutive failures of that stage to trigger an abort. To

spark git commit: [SPARK-10379] preserve first page in UnsafeShuffleExternalSorter

Repository: spark Updated Branches: refs/heads/branch-1.5 2fce5d880 -> b846a9dc3 [SPARK-10379] preserve first page in UnsafeShuffleExternalSorter Author: Davies Liu Closes #8543 from davies/preserve_page. (cherry picked from commit

spark git commit: [SPARK-10379] preserve first page in UnsafeShuffleExternalSorter

Repository: spark Updated Branches: refs/heads/master 3ddb9b323 -> 62b4690d6 [SPARK-10379] preserve first page in UnsafeShuffleExternalSorter Author: Davies Liu Closes #8543 from davies/preserve_page. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-10247] [CORE] improve readability of a test case in DAGSchedulerSuite

Repository: spark Updated Branches: refs/heads/master f6c447f87 -> 3ddb9b323 [SPARK-10247] [CORE] improve readability of a test case in DAGSchedulerSuite This is pretty minor, just trying to improve the readability of `DAGSchedulerSuite`, I figure every bit helps. Before whenever I read

spark git commit: Removed code duplication in ShuffleBlockFetcherIterator

Repository: spark Updated Branches: refs/heads/master 0985d2c30 -> f6c447f87 Removed code duplication in ShuffleBlockFetcherIterator Added fetchUpToMaxBytes() to prevent having to update both code blocks when a change is made. Author: Evan Racah Closes #8514 from

spark git commit: [SPARK-8707] RDD#toDebugString fails if any cached RDD has invalid partitions

Repository: spark Updated Branches: refs/heads/master 4bd85d06e -> 0985d2c30 [SPARK-8707] RDD#toDebugString fails if any cached RDD has invalid partitions Added numPartitions(evaluate: Boolean) to RDD. With "evaluate=true" the method is same with "partitions.length". With "evaluate=false",

spark git commit: [SPARK-10124] [MESOS] Fix removing queued driver in mesos cluster mode.

Repository: spark Updated Branches: refs/heads/branch-1.5 16414dae0 - a3ed2c31e [SPARK-10124] [MESOS] Fix removing queued driver in mesos cluster mode. Currently the spark applications can be queued to the Mesos cluster dispatcher, but when multiple jobs are in queue we don't handle removing

spark git commit: [SPARK-10124] [MESOS] Fix removing queued driver in mesos cluster mode.

Repository: spark Updated Branches: refs/heads/master affc8a887 - 73431d8af [SPARK-10124] [MESOS] Fix removing queued driver in mesos cluster mode. Currently the spark applications can be queued to the Mesos cluster dispatcher, but when multiple jobs are in queue we don't handle removing

spark git commit: [SPARK-9877] [CORE] Fix StandaloneRestServer NPE when submitting application

Repository: spark Updated Branches: refs/heads/branch-1.5 1ce0b01f4 - 130e06ef1 [SPARK-9877] [CORE] Fix StandaloneRestServer NPE when submitting application Detailed exception log can be seen in [SPARK-9877](https://issues.apache.org/jira/browse/SPARK-9877), the problem is when creating

spark git commit: [SPARK-9948] Fix flaky AccumulatorSuite - internal accumulators

thread so they're not guaranteed to be posted in time. This commit fixes this flakiness through a job end registration callback. Author: Andrew Or and...@databricks.com Closes #8176 from andrewor14/fix-accumulator-suite. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http

spark git commit: [SPARK-9948] Fix flaky AccumulatorSuite - internal accumulators

in a separate thread so they're not guaranteed to be posted in time. This commit fixes this flakiness through a job end registration callback. Author: Andrew Or and...@databricks.com Closes #8176 from andrewor14/fix-accumulator-suite. (cherry picked from commit 6518ef63037aa56b541927f99ad26744f91098ce

spark git commit: [SPARK-9877] [CORE] Fix StandaloneRestServer NPE when submitting application

Repository: spark Updated Branches: refs/heads/master 6518ef630 - 9407baa2a [SPARK-9877] [CORE] Fix StandaloneRestServer NPE when submitting application Detailed exception log can be seen in [SPARK-9877](https://issues.apache.org/jira/browse/SPARK-9877), the problem is when creating

spark git commit: [SPARK-9809] Task crashes because the internal accumulators are not properly initialized

Repository: spark Updated Branches: refs/heads/branch-1.5 d92568ae5 - ff3e9561d [SPARK-9809] Task crashes because the internal accumulators are not properly initialized When a stage failed and another stage was resubmitted with only part of partitions to compute, all the tasks failed with

spark git commit: [SPARK-9809] Task crashes because the internal accumulators are not properly initialized

Repository: spark Updated Branches: refs/heads/master ffa05c84f - 33bae585d [SPARK-9809] Task crashes because the internal accumulators are not properly initialized When a stage failed and another stage was resubmitted with only part of partitions to compute, all the tasks failed with error

spark git commit: [SPARK-9649] Fix flaky test MasterSuite - randomize ports

) at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389) at scala.util.Success$$anonfun$map$1.apply(Try.scala:206) at scala.util.Try$.apply(Try.scala:161) ``` Author: Andrew Or and...@databricks.com Closes #7968 from andrewor14/fix-master-flaky-test and squashes the following

<    1   2   3   4   5   6   7   8   9   10   >