spark git commit: [SPARK-9877] [CORE] Fix StandaloneRestServer NPE when submitting application

Repository: spark Updated Branches: refs/heads/master 6518ef630 - 9407baa2a [SPARK-9877] [CORE] Fix StandaloneRestServer NPE when submitting application Detailed exception log can be seen in [SPARK-9877](https://issues.apache.org/jira/browse/SPARK-9877), the problem is when creating

spark git commit: [SPARK-9809] Task crashes because the internal accumulators are not properly initialized

Repository: spark Updated Branches: refs/heads/branch-1.5 d92568ae5 - ff3e9561d [SPARK-9809] Task crashes because the internal accumulators are not properly initialized When a stage failed and another stage was resubmitted with only part of partitions to compute, all the tasks failed with

spark git commit: [SPARK-9809] Task crashes because the internal accumulators are not properly initialized

Repository: spark Updated Branches: refs/heads/master ffa05c84f - 33bae585d [SPARK-9809] Task crashes because the internal accumulators are not properly initialized When a stage failed and another stage was resubmitted with only part of partitions to compute, all the tasks failed with error

spark git commit: [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications

Repository: spark Updated Branches: refs/heads/master b85f9a242 - a807fcbe5 [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications Author: Rohit Agarwal roh...@qubole.com Closes #8088 from mindprince/SPARK-9806. Project:

spark git commit: [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications

Repository: spark Updated Branches: refs/heads/branch-1.5 2f909184e - 402c0ca9a [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications Author: Rohit Agarwal roh...@qubole.com Closes #8088 from mindprince/SPARK-9806. (cherry picked from commit

spark git commit: [SPARK-9829] [WEBUI] Display the update value for peak execution memory

Repository: spark Updated Branches: refs/heads/master a807fcbe5 - 4e3f4b934 [SPARK-9829] [WEBUI] Display the update value for peak execution memory The peak execution memory is not correct because it shows the sum of finished tasks' values when a task finishes. This PR fixes it by using the

spark git commit: [SPARK-9829] [WEBUI] Display the update value for peak execution memory

Repository: spark Updated Branches: refs/heads/branch-1.5 402c0ca9a - d9d4bdea2 [SPARK-9829] [WEBUI] Display the update value for peak execution memory The peak execution memory is not correct because it shows the sum of finished tasks' values when a task finishes. This PR fixes it by using

spark git commit: [SPARK-9575] [MESOS] Add docuemntation around Mesos shuffle service.

Repository: spark Updated Branches: refs/heads/master 5c99d8bf9 - 741a29f98 [SPARK-9575] [MESOS] Add docuemntation around Mesos shuffle service. andrewor14 Author: Timothy Chen tnac...@gmail.com Closes #7907 from tnachen/mesos_shuffle. Project: http://git-wip-us.apache.org/repos/asf/spark

spark git commit: [SPARK-8366] maxNumExecutorsNeeded should properly handle failed tasks

Repository: spark Updated Branches: refs/heads/branch-1.5 b994f8928 - 2f909184e [SPARK-8366] maxNumExecutorsNeeded should properly handle failed tasks Author: xutingjun xuting...@huawei.com Author: meiyoula 1039320...@qq.com Closes #6817 from XuTingjun/SPARK-8366. (cherry picked from commit

spark git commit: [SPARK-8366] maxNumExecutorsNeeded should properly handle failed tasks

Repository: spark Updated Branches: refs/heads/master b1581ac28 - b85f9a242 [SPARK-8366] maxNumExecutorsNeeded should properly handle failed tasks Author: xutingjun xuting...@huawei.com Author: meiyoula 1039320...@qq.com Closes #6817 from XuTingjun/SPARK-8366. Project:

spark git commit: [SPARK-8798] [MESOS] Allow additional uris to be fetched with mesos

Repository: spark Updated Branches: refs/heads/branch-1.5 93fc95934 - a2f805729 [SPARK-8798] [MESOS] Allow additional uris to be fetched with mesos Some users like to download additional files in their sandbox that they can refer to from their spark program, or even later mount these files

spark git commit: [SPARK-8798] [MESOS] Allow additional uris to be fetched with mesos

Repository: spark Updated Branches: refs/heads/master bab892328 - 5c99d8bf9 [SPARK-8798] [MESOS] Allow additional uris to be fetched with mesos Some users like to download additional files in their sandbox that they can refer to from their spark program, or even later mount these files to

spark git commit: [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications

Repository: spark Updated Branches: refs/heads/branch-1.4 6dde38026 - 89c8aea94 [SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple applications Author: Rohit Agarwal roh...@qubole.com Closes #8088 from mindprince/SPARK-9806. (cherry picked from commit

spark git commit: [SPARK-9426] [WEBUI] Job page DAG visualization is not shown

Repository: spark Updated Branches: refs/heads/master 4e3f4b934 - bab892328 [SPARK-9426] [WEBUI] Job page DAG visualization is not shown To reproduce the issue, go to the stage page and click DAG Visualization once, then go to the job page to show the job DAG visualization. You will only see

spark git commit: [SPARK-9426] [WEBUI] Job page DAG visualization is not shown

Repository: spark Updated Branches: refs/heads/branch-1.5 d9d4bdea2 - 93fc95934 [SPARK-9426] [WEBUI] Job page DAG visualization is not shown To reproduce the issue, go to the stage page and click DAG Visualization once, then go to the job page to show the job DAG visualization. You will only

spark git commit: [SPARK-9795] Dynamic allocation: avoid double counting when killing same executor twice

-9745, fixing this potential race condition seems like a strict improvement. I've added a regression test to illustrate the issue. Author: Andrew Or and...@databricks.com Closes #8078 from andrewor14/da-double-kill. (cherry picked from commit be5d1912076c2ffd21ec88611e53d3b3c59b7ecc) Signed-off

spark git commit: [SPARK-9352] [SPARK-9353] Add tests for standalone scheduling code

. Author: Andrew Or and...@databricks.com Closes #7668 from andrewor14/standalone-scheduling-tests and squashes the following commits: d852faf [Andrew Or] Add tests + fix scheduling with memory limits Conflicts: core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala

spark git commit: [SPARK-9352] [SPARK-9353] Add tests for standalone scheduling code

. Author: Andrew Or and...@databricks.com Closes #7668 from andrewor14/standalone-scheduling-tests and squashes the following commits: d852faf [Andrew Or] Add tests + fix scheduling with memory limits Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org

spark git commit: [SPARK-9352] [SPARK-9353] Add tests for standalone scheduling code

. Author: Andrew Or and...@databricks.com Closes #7668 from andrewor14/standalone-scheduling-tests and squashes the following commits: d852faf [Andrew Or] Add tests + fix scheduling with memory limits (cherry picked from commit 1cf19760d61a5a17bd175a906d34a2940141b76d) Signed-off-by: Andrew

spark git commit: [HOTFIX] Fix compile in MasterSuite

Repository: spark Updated Branches: refs/heads/branch-1.4 d0ef42c4e - 5aed92613 [HOTFIX] Fix compile in MasterSuite Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5aed9261 Tree:

spark git commit: [SPARK-8881] [SPARK-9260] Fix algorithm for scheduling executors on workers

spark.cores.max/spark.executor.cores (i.e, num_executors) num_workers, executors are not launched and the app hangs. This PR fixes and refactors the scheduling algorithm. andrewor14 Author: Nishkam Ravi nr...@cloudera.com Author: nishkamravi2 nishkamr...@gmail.com Closes #7274 from nishkamravi2

spark git commit: [SPARK-8881] [SPARK-9260] Fix algorithm for scheduling executors on workers

spark.cores.max/spark.executor.cores (i.e, num_executors) num_workers, executors are not launched and the app hangs. This PR fixes and refactors the scheduling algorithm. andrewor14 Author: Nishkam Ravi nr...@cloudera.com Author: nishkamravi2 nishkamr...@gmail.com Closes #7274 from nishkamravi2

spark git commit: [SPARK-7171] Added a method to retrieve metrics sources in TaskContext

Repository: spark Updated Branches: refs/heads/master 9a4fd875b - 31954910d [SPARK-7171] Added a method to retrieve metrics sources in TaskContext Author: Jacek Lewandowski lewandowski.ja...@gmail.com Closes #5805 from jacek-lewandowski/SPARK-7171 and squashes the following commits:

spark git commit: [SPARK-5423] [CORE] Register a TaskCompletionListener to make sure release all resources

Repository: spark Updated Branches: refs/heads/master 4f7f1ee37 - d45355ee2 [SPARK-5423] [CORE] Register a TaskCompletionListener to make sure release all resources Make `DiskMapIterator.cleanup` idempotent and register a TaskCompletionListener to make sure call `cleanup`. Author: zsxwing

spark git commit: [SPARK-9128] [CORE] Get outerclasses and objects with only one method calling in ClosureCleaner

Repository: spark Updated Branches: refs/heads/master f67da43c3 - 9a4fd875b [SPARK-9128] [CORE] Get outerclasses and objects with only one method calling in ClosureCleaner JIRA: https://issues.apache.org/jira/browse/SPARK-9128 Currently, in `ClosureCleaner`, the outerclasses and objects are

spark git commit: [SPARK-4598] [WEBUI] Task table pagination for the Stage page

Repository: spark Updated Branches: refs/heads/master 31954910d - 4f7f1ee37 [SPARK-4598] [WEBUI] Task table pagination for the Stage page This PR adds pagination for the task table to solve the scalability issue of the stage page. Here is the initial screenshot: img width=1347 alt=pagination

spark git commit: [SPARK-9036] [CORE] SparkListenerExecutorMetricsUpdate messages not included in JsonProtocol

Repository: spark Updated Branches: refs/heads/master 6592a6058 - f67da43c3 [SPARK-9036] [CORE] SparkListenerExecutorMetricsUpdate messages not included in JsonProtocol This PR implements a JSON serializer and deserializer in the JSONProtocol to handle the (de)serialization of

spark git commit: [SPARK-8911] Fix local mode endless heartbeats

the missing event that needs to be posted. Author: Andrew Or and...@databricks.com Closes #7382 from andrewor14/fix-local-heartbeat and squashes the following commits: 1258bdf [Andrew Or] Post ExecutorAdded event to local executor Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-8646] PySpark does not run on YARN if master not provided in command line

Repository: spark Updated Branches: refs/heads/master 57e9b13bf - 49351c7f5 [SPARK-8646] PySpark does not run on YARN if master not provided in command line andrewor14 davies vanzin can you take a look at this? thanks Author: Lianhui Wang lianhuiwan...@gmail.com Closes #7438 from

spark git commit: [SPARK-8119] HeartbeatReceiver should replace executors, not kill

simply wants a replacement for the expired executor. **Fix.** Differentiate between the intention to kill and the intention to replace an executor with a fresh one. More details can be found in the commit message. Author: Andrew Or and...@databricks.com Closes #7107 from andrewor14/heartbeat

spark git commit: [SPARK-6284] [MESOS] Add mesos role, principal and secret

Repository: spark Updated Branches: refs/heads/master 49351c7f5 - d86bbb4e2 [SPARK-6284] [MESOS] Add mesos role, principal and secret Mesos supports framework authentication and role to be set per framework, which the role is used to identify the framework's role which impacts the sharing

spark git commit: [SPARK-10471][CORE][MESOS] prevent getting offers for unmet constraints

Repository: spark Updated Branches: refs/heads/master 88a3fdcc7 -> 5039a49b6 [SPARK-10471][CORE][MESOS] prevent getting offers for unmet constraints this change rejects offers for slaves with unmet constraints for 120s to mitigate offer starvation. this prevents mesos to send us these offers

spark git commit: [SPARK-10471][CORE][MESOS] prevent getting offers for unmet constraints

Repository: spark Updated Branches: refs/heads/branch-1.6 2459b3432 -> 74f50275e [SPARK-10471][CORE][MESOS] prevent getting offers for unmet constraints this change rejects offers for slaves with unmet constraints for 120s to mitigate offer starvation. this prevents mesos to send us these

spark git commit: [SPARK-11112] Fix Scala 2.11 compilation error in RDDInfo.scala

Repository: spark Updated Branches: refs/heads/branch-1.6 2946c85f5 -> fb469e76a [SPARK-2] Fix Scala 2.11 compilation error in RDDInfo.scala As shown in https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/job/Spark-Master-Scala211-Compile/1946/console , compilation fails with:

spark git commit: [SPARK-11112] Fix Scala 2.11 compilation error in RDDInfo.scala

Repository: spark Updated Branches: refs/heads/master 08a7a836c -> 404a28f4e [SPARK-2] Fix Scala 2.11 compilation error in RDDInfo.scala As shown in https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/job/Spark-Master-Scala211-Compile/1946/console , compilation fails with: ```

spark git commit: [SPARK-11362] [SQL] Use Spark BitSet in BroadcastNestedLoopJoin

Repository: spark Updated Branches: refs/heads/branch-1.6 fc2942d12 -> 2946c85f5 [SPARK-11362] [SQL] Use Spark BitSet in BroadcastNestedLoopJoin JIRA: https://issues.apache.org/jira/browse/SPARK-11362 We use scala.collection.mutable.BitSet in BroadcastNestedLoopJoin now. We should use

spark git commit: [SPARK-11671] documentation code example typo

Repository: spark Updated Branches: refs/heads/branch-1.6 0dd6c2987 -> 3df6238bd [SPARK-11671] documentation code example typo Example for sqlContext.createDataDrame from pandas.DataFrame has a typo Author: Chris Snow Closes #9639 from snowch/patch-2. Project:

spark git commit: [SPARK-11670] Fix incorrect kryo buffer default value in docs

> Closes #9638 from andrewor14/fix-kryo-docs. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cf38fc75 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cf38fc75 Diff: http://git-wip-us.apache.org/repos/asf/spark/d

spark git commit: [SPARK-11670] Fix incorrect kryo buffer default value in docs

ricks.com> Closes #9638 from andrewor14/fix-kryo-docs. (cherry picked from commit cf38fc7551f4743958c2fdc7931affd672755e68) Signed-off-by: Andrew Or <and...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a

spark git commit: [SPARK-11671] documentation code example typo

Repository: spark Updated Branches: refs/heads/master f0d3b58d9 -> 380dfcc0d [SPARK-11671] documentation code example typo Example for sqlContext.createDataDrame from pandas.DataFrame has a typo Author: Chris Snow Closes #9639 from snowch/patch-2. Project:

spark git commit: [SPARK-11667] Update dynamic allocation docs to reflect supported cluster managers

Repository: spark Updated Branches: refs/heads/master cf38fc755 -> 12a0784ac [SPARK-11667] Update dynamic allocation docs to reflect supported cluster managers Author: Andrew Or <and...@databricks.com> Closes #9637 from andrewor14/update-da-docs. Project: http://git-wip-us.a

spark git commit: [SPARK-11667] Update dynamic allocation docs to reflect supported cluster managers

Repository: spark Updated Branches: refs/heads/branch-1.6 a98cac26f -> 782885786 [SPARK-11667] Update dynamic allocation docs to reflect supported cluster managers Author: Andrew Or <and...@databricks.com> Closes #9637 from andrewor14/update-da-docs. (cherry picked fr

spark git commit: [SPARK-8029] robust shuffle writer (for 1.5 branch)

Repository: spark Updated Branches: refs/heads/branch-1.5 3676d4c4d -> 330961bbf [SPARK-8029] robust shuffle writer (for 1.5 branch) Currently, all the shuffle writer will write to target path directly, the file could be corrupted by other attempt of the same partition on the same executor.

spark git commit: [SPARK-7970] Skip closure cleaning for SQL operations

Repository: spark Updated Branches: refs/heads/branch-1.6 6efe8b583 -> bcc871091 [SPARK-7970] Skip closure cleaning for SQL operations Also introduces new spark private API in RDD.scala with name 'mapPartitionsInternal' which doesn't closure cleans the RDD elements. Author: nitin goyal

spark git commit: [SPARK-7970] Skip closure cleaning for SQL operations

Repository: spark Updated Branches: refs/heads/master bdfbc1dca -> c939c70ac [SPARK-7970] Skip closure cleaning for SQL operations Also introduces new spark private API in RDD.scala with name 'mapPartitionsInternal' which doesn't closure cleans the RDD elements. Author: nitin goyal

spark git commit: [SPARKR][HOTFIX] Disable flaky SparkR package build test

Repository: spark Updated Branches: refs/heads/branch-1.6 4f8c7e18f -> bb044ec22 [SPARKR][HOTFIX] Disable flaky SparkR package build test See https://github.com/apache/spark/pull/9390#issuecomment-157160063 and https://gist.github.com/shivaram/3a2fecce60768a603dac for more information

spark git commit: [SPARKR][HOTFIX] Disable flaky SparkR package build test

Repository: spark Updated Branches: refs/heads/master fd14936be -> ea6f53e48 [SPARKR][HOTFIX] Disable flaky SparkR package build test See https://github.com/apache/spark/pull/9390#issuecomment-157160063 and https://gist.github.com/shivaram/3a2fecce60768a603dac for more information Author:

spark git commit: [SPARK-11480][CORE][WEBUI] Wrong callsite is displayed when using AsyncRDDActions#takeAsync

Repository: spark Updated Branches: refs/heads/branch-1.6 bb044ec22 -> e4abfe932 [SPARK-11480][CORE][WEBUI] Wrong callsite is displayed when using AsyncRDDActions#takeAsync When we call AsyncRDDActions#takeAsync, actually another DAGScheduler#runJob is called from another thread so we

spark git commit: [SPARK-11480][CORE][WEBUI] Wrong callsite is displayed when using AsyncRDDActions#takeAsync

Repository: spark Updated Branches: refs/heads/master ea6f53e48 -> 30f3cfda1 [SPARK-11480][CORE][WEBUI] Wrong callsite is displayed when using AsyncRDDActions#takeAsync When we call AsyncRDDActions#takeAsync, actually another DAGScheduler#runJob is called from another thread so we cannot

spark git commit: [SPARK-11710] Document new memory management model

Repository: spark Updated Branches: refs/heads/master 30f3cfda1 -> 33a0ec937 [SPARK-11710] Document new memory management model Author: Andrew Or <and...@databricks.com> Closes #9676 from andrewor14/memory-management-docs. Project: http://git-wip-us.apache.org/repos/asf/spark/re

spark git commit: [EXAMPLE][MINOR] Add missing awaitTermination in click stream example

Repository: spark Updated Branches: refs/heads/master 33a0ec937 -> bd10eb81c [EXAMPLE][MINOR] Add missing awaitTermination in click stream example Author: jerryshao Closes #9730 from jerryshao/clickstream-fix. Project:

spark git commit: [SPARK-10192][CORE] simple test w/ failure involving a shared dependency

Repository: spark Updated Branches: refs/heads/master c0e48dfa6 -> 33112f9c4 [SPARK-10192][CORE] simple test w/ failure involving a shared dependency just trying to increase test coverage in the scheduler, this already works. It includes a regression test for SPARK-9809 copied some test

spark git commit: [SPARK-10192][CORE] simple test w/ failure involving a shared dependency

Repository: spark Updated Branches: refs/heads/branch-1.6 d8bfc025c -> 10272d5c9 [SPARK-10192][CORE] simple test w/ failure involving a shared dependency just trying to increase test coverage in the scheduler, this already works. It includes a regression test for SPARK-9809 copied some

spark git commit: [SPARK-11361][STREAMING] Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz

Repository: spark Updated Branches: refs/heads/master 900917541 -> 6600786dd [SPARK-11361][STREAMING] Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz Currently, when a DStream sets the scope for RDD generated by it, that scope is not allowed to be

spark git commit: [SPARK-11361][STREAMING] Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz

Repository: spark Updated Branches: refs/heads/branch-1.6 fafeca36e -> b34819c7b [SPARK-11361][STREAMING] Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz Currently, when a DStream sets the scope for RDD generated by it, that scope is not allowed to

spark git commit: [SPARK-11615] Drop @VisibleForTesting annotation

Repository: spark Updated Branches: refs/heads/branch-1.6 93ac30741 -> fafeca36e [SPARK-11615] Drop @VisibleForTesting annotation See http://search-hadoop.com/m/q3RTtjpe8r1iRbTj2 for discussion. Summary: addition of VisibleForTesting annotation resulted in spark-shell malfunctioning.

spark git commit: [MINOR] Fix typo in AggregationQuerySuite.scala

Repository: spark Updated Branches: refs/heads/master 6600786dd -> 12c7635dc [MINOR] Fix typo in AggregationQuerySuite.scala Author: Forest Fang Closes #9357 from saurfang/patch-1. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-11572] Exit AsynchronousListenerBus thread when stop() is called

Repository: spark Updated Branches: refs/heads/master 33112f9c4 -> 3e0a6cf1e [SPARK-11572] Exit AsynchronousListenerBus thread when stop() is called As vonnagy reported in the following thread: http://search-hadoop.com/m/q3RTtk982kvIow22 Attempts to join the thread in AsynchronousListenerBus

spark git commit: [SPARK-11572] Exit AsynchronousListenerBus thread when stop() is called

Repository: spark Updated Branches: refs/heads/branch-1.6 10272d5c9 -> 93ac30741 [SPARK-11572] Exit AsynchronousListenerBus thread when stop() is called As vonnagy reported in the following thread: http://search-hadoop.com/m/q3RTtk982kvIow22 Attempts to join the thread in

spark git commit: [SPARK-11615] Drop @VisibleForTesting annotation

Repository: spark Updated Branches: refs/heads/master 3e0a6cf1e -> 900917541 [SPARK-11615] Drop @VisibleForTesting annotation See http://search-hadoop.com/m/q3RTtjpe8r1iRbTj2 for discussion. Summary: addition of VisibleForTesting annotation resulted in spark-shell malfunctioning. Author:

spark git commit: [MINOR] License header formatting fix

Repository: spark Updated Branches: refs/heads/master 12c7635dc -> 745e45d5f [MINOR] License header formatting fix The header wasn't indented properly. Author: Marc Prud'hommeaux Closes #9312 from mprudhom/patch-1. Project:

spark git commit: [SPARK-11112] DAG visualization: display RDD callsite

ricks.com> Closes #9398 from andrewor14/rdd-callsite. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7f741905 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7f741905 Diff: http://git-wip-us.apache.org/repos/as

spark git commit: [SPARK-10515] When killing executor, the pending replacement executors should not be lost

Repository: spark Updated Branches: refs/heads/branch-1.5 166fdf4e3 -> 13920d5fe [SPARK-10515] When killing executor, the pending replacement executors should not be lost If the heartbeat receiver kills executors (and new ones are not registered to replace them), the idle timeout for the

spark git commit: [SPARK-11078] Ensure spilling tests actually spill

ial bugs in spilling. The size of this patch is inflated by the refactoring of `ExternalSorterSuite`, which had a lot of duplicate code and logic. Author: Andrew Or <and...@databricks.com> Closes #9124 from andrewor14/spilling-tests. Project: http://git-wip-us.apache.org/repos/asf/spark/re

spark git commit: [SPARK-10515] When killing executor, the pending replacement executors should not be lost

Repository: spark Updated Branches: refs/heads/master 723aa75a9 -> 2d000124b [SPARK-10515] When killing executor, the pending replacement executors should not be lost If the heartbeat receiver kills executors (and new ones are not registered to replace them), the idle timeout for the old

spark git commit: [SPARK-11071] [LAUNCHER] Fix flakiness in LauncherServerSuite::timeout.

Repository: spark Updated Branches: refs/heads/master b591de7c0 -> a5719804c [SPARK-11071] [LAUNCHER] Fix flakiness in LauncherServerSuite::timeout. The test could fail depending on scheduling of the various threads involved; the change removes some sources of races, while making the test a

spark git commit: fix typo bellow -> below

Repository: spark Updated Branches: refs/heads/master a5719804c -> 723aa75a9 fix typo bellow -> below Author: Britta Weber Closes #9136 from brwe/typo-bellow. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-10412] [SQL] report memory usage for tungsten sql physical operator

Repository: spark Updated Branches: refs/heads/master 3b364ff0a -> 6a2359ff1 [SPARK-10412] [SQL] report memory usage for tungsten sql physical operator https://issues.apache.org/jira/browse/SPARK-10412 some screenshots: ### aggregate: ![screen shot 2015-10-12 at 2 23 11

spark git commit: [SPARK-11114][PYSPARK] add getOrCreate for SparkContext/SQLContext in Python

Repository: spark Updated Branches: refs/heads/master a1413b366 -> 232d7f8d4 [SPARK-4][PYSPARK] add getOrCreate for SparkContext/SQLContext in Python Also added SQLContext.newSession() Author: Davies Liu Closes #9122 from davies/py_create. Project:

spark git commit: [SPARK-9708][MESOS] Spark should create local temporary directories in Mesos sandbox when launched with Mesos

Repository: spark Updated Branches: refs/heads/master 232d7f8d4 -> fc26f32cf [SPARK-9708][MESOS] Spark should create local temporary directories in Mesos sandbox when launched with Mesos This is my own original work and I license this to the project under the project's open source license

spark git commit: [SPARK-11131][CORE] Fix race in worker registration protocol.

Repository: spark Updated Branches: refs/heads/master 67582132b -> 7ab0ce650 [SPARK-11131][CORE] Fix race in worker registration protocol. Because the registration RPC was not really an RPC, but a bunch of disconnected messages, it was possible for other messages to be sent before the reply

spark git commit: [SPARK-11120] Allow sane default number of executor failures when dynamically allocating in YARN

Repository: spark Updated Branches: refs/heads/master fc26f32cf -> 16906ef23 [SPARK-11120] Allow sane default number of executor failures when dynamically allocating in YARN I also added some information to container-failure error msgs about what host they failed on, which would have helped

spark git commit: [SPARK-11051][CORE] Do not allow local checkpointing after the RDD is materialized and checkpointed

Repository: spark Updated Branches: refs/heads/master 7ab0ce650 -> a1413b366 [SPARK-11051][CORE] Do not allow local checkpointing after the RDD is materialized and checkpointed JIRA: https://issues.apache.org/jira/browse/SPARK-11051 When a `RDD` is materialized and checkpointed, its

spark git commit: [SPARK-11051][CORE] Do not allow local checkpointing after the RDD is materialized and checkpointed

Repository: spark Updated Branches: refs/heads/branch-1.5 5186ec8ac -> 2195fec12 [SPARK-11051][CORE] Do not allow local checkpointing after the RDD is materialized and checkpointed JIRA: https://issues.apache.org/jira/browse/SPARK-11051 When a `RDD` is materialized and checkpointed, its

spark git commit: [SPARK-10887] [SQL] Build HashedRelation outside of HashJoinNode.

Repository: spark Updated Branches: refs/heads/master 2a6f614cd -> 82d275f27 [SPARK-10887] [SQL] Build HashedRelation outside of HashJoinNode. This PR refactors `HashJoinNode` to take a existing `HashedRelation`. So, we can reuse this node for both `ShuffledHashJoin` and `BroadcastHashJoin`.

spark git commit: [SPARK-7527] [CORE] Fix createNullValue to return the correct null values and REPL mode detection

Repository: spark Updated Branches: refs/heads/branch-1.4 e7c4346d0 -> e2ff49198 [SPARK-7527] [CORE] Fix createNullValue to return the correct null values and REPL mode detection The root cause of SPARK-7527 is `createNullValue` returns an incompatible value `Byte(0)` for `char` and

spark git commit: [CORE] [MINOR] change the log level to info

Repository: spark Updated Branches: refs/heads/master 5dd45bde4 - db6d57f87 [CORE] [MINOR] change the log level to info Too many logs even when set the log level to warning. Author: Cheng Hao hao.ch...@intel.com Closes #7340 from chenghao-intel/log and squashes the following commits:

spark git commit: [SPARK-8958] Dynamic allocation: change cached timeout to infinity

be very expensive and we should only allow it if the user knows what he/she is doing. FYI harishreedharan sryza. Author: Andrew Or and...@databricks.com Closes #7329 from andrewor14/da-cached-timeout and squashes the following commits: cef0b4e [Andrew Or] Change timeout to infinity Project: http

spark git commit: [SPARK-7419] [STREAMING] [TESTS] Fix CheckpointSuite.recovery with file input stream

Repository: spark Updated Branches: refs/heads/master 930fe9535 - 88bf43033 [SPARK-7419] [STREAMING] [TESTS] Fix CheckpointSuite.recovery with file input stream Fix this failure:

spark git commit: [SPARK-6287] [MESOS] Add dynamic allocation to the coarse-grained Mesos scheduler

Repository: spark Updated Branches: refs/heads/master ebdf58538 - c4830598b [SPARK-6287] [MESOS] Add dynamic allocation to the coarse-grained Mesos scheduler This is largely based on extracting the dynamic allocation parts from tnachen's #3861. Author: Iulian Dragos jagua...@gmail.com

spark git commit: [MINOR] [STREAMING] Fix log statements in ReceiverSupervisorImpl

Repository: spark Updated Branches: refs/heads/master 0b0b9ceaf - 7ce3b818f [MINOR] [STREAMING] Fix log statements in ReceiverSupervisorImpl Log statements incorrectly showed that the executor was being stopped when receiver was being stopped. Author: Tathagata Das

spark git commit: [SPARK-2017] [UI] Stage page hangs with many tasks

) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) ``` This patch compresses Jetty responses in gzip. The correct long-term fix is to add pagination. Author: Andrew Or and...@databricks.com Closes #7296 from andrewor14/gzip-jetty and squashes the following commits: a051c64 [Andrew Or] Use GZIP

spark git commit: [SPARK-2017] [UI] Stage page hangs with many tasks

) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) ``` This patch compresses Jetty responses in gzip. The correct long-term fix is to add pagination. Author: Andrew Or and...@databricks.com Closes #7296 from andrewor14/gzip-jetty and squashes the following commits: a051c64 [Andrew Or] Use GZIP

spark git commit: [SPARK-8953] SPARK_EXECUTOR_CORES is not read in SparkSubmit

Repository: spark Updated Branches: refs/heads/master 7ce3b818f - 930fe9535 [SPARK-8953] SPARK_EXECUTOR_CORES is not read in SparkSubmit The configuration ```SPARK_EXECUTOR_CORES``` won't put into ```SparkConf```, so it has no effect to the dynamic executor allocation. Author: xutingjun

spark git commit: [SPARK-7419] [STREAMING] [TESTS] Fix CheckpointSuite.recovery with file input stream

Repository: spark Updated Branches: refs/heads/branch-1.4 c04f0a5cf - dfc99715e [SPARK-7419] [STREAMING] [TESTS] Fix CheckpointSuite.recovery with file input stream Fix this failure:

spark git commit: [MINOR] Correct comments in JavaDirectKafkaWordCount

Repository: spark Updated Branches: refs/heads/branch-1.6 33ae9c74a -> 89a2ddf77 [MINOR] Correct comments in JavaDirectKafkaWordCount Author: Rohan Bhanderi Closes #9781 from RohanBhanderi/patch-3. (cherry picked from commit

spark git commit: [SPARK-11790][STREAMING][TESTS] Increase the connection timeout

Repository: spark Updated Branches: refs/heads/master e29656f8e -> 3720b1480 [SPARK-11790][STREAMING][TESTS] Increase the connection timeout Sometimes, EmbeddedZookeeper may need more than 6 seconds to setup up in a slow Jenkins worker. So just increase the timeout, it won't increase the

spark git commit: [SPARK-11790][STREAMING][TESTS] Increase the connection timeout

Repository: spark Updated Branches: refs/heads/branch-1.6 89a2ddf77 -> 3471244f7 [SPARK-11790][STREAMING][TESTS] Increase the connection timeout Sometimes, EmbeddedZookeeper may need more than 6 seconds to setup up in a slow Jenkins worker. So just increase the timeout, it won't increase the

spark git commit: [SPARK-11726] Throw exception on timeout when waiting for REST server response

Repository: spark Updated Branches: refs/heads/branch-1.6 7e17ce5b6 -> 3133d8bd1 [SPARK-11726] Throw exception on timeout when waiting for REST server response Author: Jacek Lewandowski Closes #9692 from jacek-lewandowski/SPARK-11726. Project:

spark git commit: [SPARK-9552] Add force control for killExecutors to avoid false killing for those busy executors

Repository: spark Updated Branches: refs/heads/master 928d63162 -> 965245d08 [SPARK-9552] Add force control for killExecutors to avoid false killing for those busy executors By using the dynamic allocation, sometimes it occurs false killing for those busy executors. Some executors with

spark git commit: [SPARK-9552] Add force control for killExecutors to avoid false killing for those busy executors

Repository: spark Updated Branches: refs/heads/branch-1.6 78dc07cdf -> 33ae9c74a [SPARK-9552] Add force control for killExecutors to avoid false killing for those busy executors By using the dynamic allocation, sometimes it occurs false killing for those busy executors. Some executors with

spark git commit: [MINOR] Correct comments in JavaDirectKafkaWordCount

Repository: spark Updated Branches: refs/heads/master 965245d08 -> e29656f8e [MINOR] Correct comments in JavaDirectKafkaWordCount Author: Rohan Bhanderi Closes #9781 from RohanBhanderi/patch-3. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-11726] Throw exception on timeout when waiting for REST server response

Repository: spark Updated Branches: refs/heads/master 52c734b58 -> b362d50fc [SPARK-11726] Throw exception on timeout when waiting for REST server response Author: Jacek Lewandowski Closes #9692 from jacek-lewandowski/SPARK-11726. Project:

spark git commit: [SPARK-11771][YARN][TRIVIAL] maximum memory in yarn is controlled by two params have both in error msg

Repository: spark Updated Branches: refs/heads/master 3720b1480 -> 52c734b58 [SPARK-11771][YARN][TRIVIAL] maximum memory in yarn is controlled by two params have both in error msg When we exceed the max memory tell users to increase both params instead of just the one. Author: Holden Karau

spark git commit: [SPARK-11771][YARN][TRIVIAL] maximum memory in yarn is controlled by two params have both in error msg

Repository: spark Updated Branches: refs/heads/branch-1.6 3471244f7 -> 7e17ce5b6 [SPARK-11771][YARN][TRIVIAL] maximum memory in yarn is controlled by two params have both in error msg When we exceed the max memory tell users to increase both params instead of just the one. Author: Holden

spark git commit: [SPARK-2533] Add locality levels on stage summary view

Repository: spark Updated Branches: refs/heads/master 380dfcc0d -> 74c30049a [SPARK-2533] Add locality levels on stage summary view Author: Jean-Baptiste Onofré Closes #9487 from jbonofre/SPARK-2533-2. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-2533] Add locality levels on stage summary view

Repository: spark Updated Branches: refs/heads/branch-1.6 340ca9e76 -> 069591799 [SPARK-2533] Add locality levels on stage summary view Author: Jean-Baptiste Onofré Closes #9487 from jbonofre/SPARK-2533-2. (cherry picked from commit

spark git commit: [SPARK-11658] simplify documentation for PySpark combineByKey

Repository: spark Updated Branches: refs/heads/branch-1.6 782885786 -> 0dd6c2987 [SPARK-11658] simplify documentation for PySpark combineByKey Author: Chris Snow Closes #9640 from snowch/patch-3. (cherry picked from commit 68ef61bb656bd9c08239726913ca8ab271d52786)

spark git commit: [SPARK-11658] simplify documentation for PySpark combineByKey

Repository: spark Updated Branches: refs/heads/master 12a0784ac -> 68ef61bb6 [SPARK-11658] simplify documentation for PySpark combineByKey Author: Chris Snow Closes #9640 from snowch/patch-3. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-5945] Spark should not retry a stage infinitely on a FetchFailedException

Repository: spark Updated Branches: refs/heads/master 44948a2e9 -> 4bd85d06e [SPARK-5945] Spark should not retry a stage infinitely on a FetchFailedException The ```Stage``` class now tracks whether there were a sufficient number of consecutive failures of that stage to trigger an abort. To

<    4   5   6   7   8   9   10   11   12   13   >