hor: Andrew Or
Closes #8827 from andrewor14/allocate-pointer-array.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7ff8d68c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7ff8d68c
Diff: http://git-wip-us.apache.org/repos/
rom andrewor14/sql-local-tests-cleanup.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/35a19f33
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/35a19f33
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/35a19f33
Bra
Repository: spark
Updated Branches:
refs/heads/master a63cdc769 -> 99ecfa594
[SPARK-10575] [SPARK CORE] Wrapped RDD.takeSample with Scope
Remove return statements in RDD.takeSample and wrap it withScope
Author: vinodkc
Author: vinodkc
Author: Vinod K C
Closes #8730 from vinodkc/fix_takesa
Repository: spark
Updated Branches:
refs/heads/master b6e998634 -> a63cdc769
[SPARK-10612] [SQL] Add prepare to LocalNode.
The idea is that we should separate the function call that does memory
reservation (i.e. prepare) from the function call that consumes the input (e.g.
open()), so all op
is considered risky to backport into a maintenance release, so it
is disabled by default and enabled only in SQL.
Author: Andrew Or
Closes #8721 from andrewor14/concurrent-sql-executions-1.5 and squashes the
following commits:
3b9b462 [Andrew Or] Merge branch 'branch-1.5' of github.com:apac
use is `SparkContext`'s local properties are inherited by default. This
patch adds a way to exclude keys we don't want to be inherited, and makes SQL
go through that code path.
Author: Andrew Or
Closes #8710 from andrewor14/concurrent-sql-executions.
Project: http://git-wip-us.apach
Repository: spark
Updated Branches:
refs/heads/branch-1.5 eb0cb25bb -> 5db51f911
[SPARK-10564] ThreadingSuite: assertion failures in threads don't fail the test
(round 2)
This is a follow-up patch to #8723. I missed one case there.
Author: Andrew Or
Closes #8727 from andrew
Repository: spark
Updated Branches:
refs/heads/master fd1e8cddf -> 7b6c85636
[SPARK-10564] ThreadingSuite: assertion failures in threads don't fail the test
(round 2)
This is a follow-up patch to #8723. I missed one case there.
Author: Andrew Or
Closes #8727 from andrewor14/fix-t
2/9828199/17dc2b84-58b8-11e5-92a8-be89ce4d29d1.png)
cc andrewor14 I appreciate if you can give feedback on this since I think you
introduced display of this metric.
Author: Forest Fang
Closes #8726 from saurfang/stagepage.
(cherry picked from commit fd1e8cddf2635c55fec2ac6e1f1c221c9685af0f)
S
28199/17dc2b84-58b8-11e5-92a8-be89ce4d29d1.png)
cc andrewor14 I appreciate if you can give feedback on this since I think you
introduced display of this metric.
Author: Forest Fang
Closes #8726 from saurfang/stagepage.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http:
Repository: spark
Updated Branches:
refs/heads/branch-1.5 a0d564a10 -> 0e1c9d9ff
[SPARK-10549] scala 2.11 spark on yarn with security - Repl doesn't work
Make this lazy so that it can set the yarn mode before creating the
securityManager.
Author: Tom Graves
Author: Thomas Graves
Closes #8
Repository: spark
Updated Branches:
refs/heads/master 4e2242bb4 -> ffbbc2c58
[SPARK-10549] scala 2.11 spark on yarn with security - Repl doesn't work
Make this lazy so that it can set the yarn mode before creating the
securityManager.
Author: Tom Graves
Author: Thomas Graves
Closes #8719
Repository: spark
Updated Branches:
refs/heads/master 16b6d1861 -> 4e2242bb4
[SPARK-10576] [BUILD] Move .java files out of src/main/scala
Move .java files in `src/main/scala` to `src/main/java` root, except for
`package-info.java` (to stay next to package.scala)
Author: Sean Owen
Closes #8
Repository: spark
Updated Branches:
refs/heads/master 217e49644 -> 16b6d1861
[SPARK-10594] [YARN] Remove reference to --num-executors, add --properties-file
`ApplicationMaster` no longer has the `--num-executors` flag, and had an
undocumented `--properties-file` configuration option.
cc srow
Repository: spark
Updated Branches:
refs/heads/master 64f04154e -> 217e49644
[SPARK-9996] [SPARK-9997] [SQL] Add local expand and NestedLoopJoin operators
This PR is in conflict with #8535 and #8573. Will update this one when they are
merged.
Author: zsxwing
Closes #8642 from zsxwing/expan
masking real bugs by not
propagating assertion failures properly.
Author: Andrew Or
Closes #8723 from andrewor14/fix-threading-suite.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d74c6a14
Tree: http://git-wip-us.apache.o
entially masking real bugs by not
propagating assertion failures properly.
Author: Andrew Or
Closes #8723 from andrewor14/fix-threading-suite.
(cherry picked from commit d74c6a143cbd060c25bf14a8d306841b3ec55d03)
Signed-off-by: Andrew Or
Project: http://git-wip-us.apache.org/repos/asf/spark/rep
Repository: spark
Updated Branches:
refs/heads/master e626ac5f5 -> c2af42b5f
[SPARK-9990] [SQL] Local hash join follow-ups
1. Hide `LocalNodeIterator` behind the `LocalNode#asIterator` method
2. Add tests for this
Author: Andrew Or
Closes #8708 from andrewor14/local-hash-join-follow
Repository: spark
Updated Branches:
refs/heads/master 1eede3b25 -> e626ac5f5
[SPARK-9992] [SPARK-9994] [SPARK-9998] [SQL] Implement the local TopK, sample
and intersect operators
This PR is in conflict with #8535. I will update this one when #8535 gets
merged.
Author: zsxwing
Closes #8573
Repository: spark
Updated Branches:
refs/heads/branch-1.5 8cf16191f -> 89d351b5a
Revert "[SPARK-6350] [MESOS] Fine-grained mode scheduler respects
mesosExecutor.cores"
This reverts commit 8cf16191f3e3b0562f22d44b0381bea35ba511d7.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Co
Repository: spark
Updated Branches:
refs/heads/master a5ef2d060 -> d88abb7e2
[SPARK-9990] [SQL] Create local hash join operator
This PR includes the following changes:
- Add SQLConf to LocalNode
- Add HashJoinNode
- Add ConvertToUnsafeNode and ConvertToSafeNode.scala to test unsafe hash join.
Repository: spark
Updated Branches:
refs/heads/master f0562e8cd -> a5ef2d060
[SPARK-10514] [MESOS] waiting for min no of total cores acquired by Spark by
implementing the sufficientResourcesRegistered method
spark.scheduler.minRegisteredResourcesRatio configuration parameter works for
YARN m
Repository: spark
Updated Branches:
refs/heads/branch-1.5 bff05aaa0 -> 8cf16191f
[SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores
This is a regression introduced in #4960, this commit fixes it and adds a test.
tnachen andrewor14 please review, this should be
Repository: spark
Updated Branches:
refs/heads/master af3bc59d1 -> f0562e8cd
[SPARK-6350] [MESOS] Fine-grained mode scheduler respects mesosExecutor.cores
This is a regression introduced in #4960, this commit fixes it and adds a test.
tnachen andrewor14 please review, this should be an e
ith some earlier YARN
scheduling work.
cc JoshRosen because it's similar to output commit coordination we did in the
past
cc andrewor14 for our discussion on how to get executor exit codes and loss
reasons
Author: mcheah
Closes #8007 from mccheah/feature/preemption-handling.
Project:
Repository: spark
Updated Branches:
refs/heads/branch-1.5 bc70043c8 -> bff05aaa0
[SPARK-10469] [DOC] Try and document the three options
>From JIRA:
Add documentation for tungsten-sort.
>From the mailing list "I saw a new "spark.shuffle.manager=tungsten-sort"
>implemented in
https://issues.apa
Repository: spark
Updated Branches:
refs/heads/master e04811137 -> a76bde9da
[SPARK-10469] [DOC] Try and document the three options
>From JIRA:
Add documentation for tungsten-sort.
>From the mailing list "I saw a new "spark.shuffle.manager=tungsten-sort"
>implemented in
https://issues.apache.
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
```
To reproduce that with code (thanks andrewor14):
```scala
bin/spark-shell --master local
--conf spark.shuffle.memoryFraction=0.005
--conf spark.shuffle.sort.bypassMergeThreshold=0
sc.parallelize(1 to 2 * 1000 * 1000, 10)
.map { i =&
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
```
To reproduce that with code (thanks andrewor14):
```scala
bin/spark-shell --master local
--conf spark.shuffle.memoryFraction=0.005
--conf spark.shuffle.sort.bypassMergeThreshold=0
sc.parallelize(1 to 2 * 1000 * 1000, 10)
.map { i =>
Repository: spark
Updated Branches:
refs/heads/master 22eab706f -> 47058ca5d
[SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE_PARTITIONS.key correctly for
tests
This PR fix the failed test and conflict for #8155
https://issues.apache.org/jira/browse/SPARK-9925
Closes #8155
Author: Yin Huai
Repository: spark
Updated Branches:
refs/heads/branch-1.5 4d6333597 -> 09e08dbc1
[SPARK-10454] [SPARK CORE] wait for empty event queue
Author: robbins
Closes #8605 from robbinspg/DAGSchedulerSuite-fix.
(cherry picked from commit 2e1c17553d179f2d26a165805622cc01f92081b9)
Signed-off-by: Andre
Repository: spark
Updated Branches:
refs/heads/master b087d23e2 -> 2e1c17553
[SPARK-10454] [SPARK CORE] wait for empty event queue
Author: robbins
Closes #8605 from robbinspg/DAGSchedulerSuite-fix.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.
Repository: spark
Updated Branches:
refs/heads/master 3339e6f67 -> b087d23e2
[SPARK-9669] [MESOS] Support PySpark on Mesos cluster mode.
Support running pyspark with cluster mode on Mesos!
This doesn't upload any scripts, so if running in a remote Mesos requires the
user to specify the script
Repository: spark
Updated Branches:
refs/heads/master c3c0e431a -> 3339e6f67
[SPARK-10450] [SQL] Minor improvements to readability / style / typos etc.
Author: Andrew Or
Closes #8603 from andrewor14/minor-sql-changes.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: h
Repository: spark
Updated Branches:
refs/heads/master 804a0126e -> c3c0e431a
http://git-wip-us.apache.org/repos/asf/spark/blob/c3c0e431/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ScriptTransformationSuite.scala
---
http://git-wip-us.apache.org/repos/asf/spark/blob/c3c0e431/sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala
--
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala
b/sql/
[SPARK-10176] [SQL] Show partially analyzed plans when checkAnswer fails to
analyze
This PR takes over https://github.com/apache/spark/pull/8389.
This PR improves `checkAnswer` to print the partially analyzed plan in addition
to the user friendly error message, in order to aid debugging failing
http://git-wip-us.apache.org/repos/asf/spark/blob/c3c0e431/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala
--
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala
b/sql/core/s
Repository: spark
Updated Branches:
refs/heads/master db4c130f9 -> 08b075097
[SPARK-10435] Spark submit should fail fast for Mesos cluster mode with R
It's not supported yet so we should error with a clear message.
Author: Andrew Or
Closes #8590 from andrewor14/mesos-cluster
Repository: spark
Updated Branches:
refs/heads/master 11ef32c5a -> db4c130f9
[SPARK-9591] [CORE] Job may fail for exception during getting remote block
[SPARK-9591](https://issues.apache.org/jira/browse/SPARK-9591)
When we getting the broadcast variable, we can fetch the block form several
lo
Repository: spark
Updated Branches:
refs/heads/master e62f4a46f -> 11ef32c5a
[SPARK-10430] [CORE] Added hashCode methods in AccumulableInfo and
RDDOperationScope
Author: Vinod K C
Closes #8581 from vinodkc/fix_RDDOperationScope_Hashcode.
Project: http://git-wip-us.apache.org/repos/asf/spa
Repository: spark
Updated Branches:
refs/heads/master 754f853b0 -> e62f4a46f
[SPARK-9672] [MESOS] Donât include SPARK_ENV_LOADED when passing env vars
This contribution is my original work and I license the work to the project
under the project's open source license.
Author: Pat Shields
Repository: spark
Updated Branches:
refs/heads/branch-1.5 f945b641c -> 4d6333597
[SPARK-10431] [CORE] Fix intermittent test failure. Wait for event queue to be
clear
Author: robbins
Closes #8582 from robbinspg/InputOutputMetricsSuite.
Project: http://git-wip-us.apache.org/repos/asf/spark/
Repository: spark
Updated Branches:
refs/heads/branch-1.5 f01a96713 -> f945b641c
[SPARK-9869] [STREAMING] Wait for all event notifications before asserting
results
Author: robbins
Closes #8589 from robbinspg/InputStreamSuite-fix.
(cherry picked from commit 754f853b02e9fd221f138c2446445fd56
Repository: spark
Updated Branches:
refs/heads/master d911c682f -> 754f853b0
[SPARK-9869] [STREAMING] Wait for all event notifications before asserting
results
Author: robbins
Closes #8589 from robbinspg/InputStreamSuite-fix.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Comm
Repository: spark
Updated Branches:
refs/heads/master 49aff7b9a -> d911c682f
[SPARK-10431] [CORE] Fix intermittent test failure. Wait for event queue to be
clear
Author: robbins
Closes #8582 from robbinspg/InputOutputMetricsSuite.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Repository: spark
Updated Branches:
refs/heads/master af0e3125c -> 49aff7b9a
[SPARK-10432] spark.port.maxRetries documentation is unclear
Author: Tom Graves
Closes #8585 from tgravescs/SPARK-10432.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.
Repository: spark
Updated Branches:
refs/heads/master 62b4690d6 -> 0349b5b43
[SPARK-10411] [SQL] Move visualization above explain output and hide explain by
default
New screenshots after this fix:
https://cloud.githubusercontent.com/assets/1000778/9625782/4b2dba36-518b-11e5-9104-c713ff026e3d
Repository: spark
Updated Branches:
refs/heads/branch-1.5 b846a9dc3 -> 94404ee53
[SPARK-10411] [SQL] Move visualization above explain output and hide explain by
default
New screenshots after this fix:
https://cloud.githubusercontent.com/assets/1000778/9625782/4b2dba36-518b-11e5-9104-c713ff02
Repository: spark
Updated Branches:
refs/heads/master 3ddb9b323 -> 62b4690d6
[SPARK-10379] preserve first page in UnsafeShuffleExternalSorter
Author: Davies Liu
Closes #8543 from davies/preserve_page.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apac
Repository: spark
Updated Branches:
refs/heads/branch-1.5 2fce5d880 -> b846a9dc3
[SPARK-10379] preserve first page in UnsafeShuffleExternalSorter
Author: Davies Liu
Closes #8543 from davies/preserve_page.
(cherry picked from commit 62b4690d6b3016f41292b640ac28644ef31e299d)
Signed-off-by: An
Repository: spark
Updated Branches:
refs/heads/master f6c447f87 -> 3ddb9b323
[SPARK-10247] [CORE] improve readability of a test case in DAGSchedulerSuite
This is pretty minor, just trying to improve the readability of
`DAGSchedulerSuite`, I figure every bit helps. Before whenever I read this
Repository: spark
Updated Branches:
refs/heads/master 0985d2c30 -> f6c447f87
Removed code duplication in ShuffleBlockFetcherIterator
Added fetchUpToMaxBytes() to prevent having to update both code blocks when a
change is made.
Author: Evan Racah
Closes #8514 from eracah/master.
Project:
Repository: spark
Updated Branches:
refs/heads/master 4bd85d06e -> 0985d2c30
[SPARK-8707] RDD#toDebugString fails if any cached RDD has invalid partitions
Added numPartitions(evaluate: Boolean) to RDD. With "evaluate=true" the method
is same with "partitions.length". With "evaluate=false", it
Repository: spark
Updated Branches:
refs/heads/master 44948a2e9 -> 4bd85d06e
[SPARK-5945] Spark should not retry a stage infinitely on a FetchFailedException
The ```Stage``` class now tracks whether there were a sufficient number of
consecutive failures of that stage to trigger an abort.
To
Repository: spark
Updated Branches:
refs/heads/branch-1.5 16414dae0 -> a3ed2c31e
[SPARK-10124] [MESOS] Fix removing queued driver in mesos cluster mode.
Currently the spark applications can be queued to the Mesos cluster dispatcher,
but when multiple jobs are in queue we don't handle removing
Repository: spark
Updated Branches:
refs/heads/master affc8a887 -> 73431d8af
[SPARK-10124] [MESOS] Fix removing queued driver in mesos cluster mode.
Currently the spark applications can be queued to the Mesos cluster dispatcher,
but when multiple jobs are in queue we don't handle removing job
Repository: spark
Updated Branches:
refs/heads/branch-1.5 1ce0b01f4 -> 130e06ef1
[SPARK-9877] [CORE] Fix StandaloneRestServer NPE when submitting application
Detailed exception log can be seen in
[SPARK-9877](https://issues.apache.org/jira/browse/SPARK-9877), the problem is
when creating `St
Repository: spark
Updated Branches:
refs/heads/master 6518ef630 -> 9407baa2a
[SPARK-9877] [CORE] Fix StandaloneRestServer NPE when submitting application
Detailed exception log can be seen in
[SPARK-9877](https://issues.apache.org/jira/browse/SPARK-9877), the problem is
when creating `Standa
ate thread
so they're not guaranteed to be posted in time. This commit fixes this
flakiness through a job end registration callback.
Author: Andrew Or
Closes #8176 from andrewor14/fix-accumulator-suite.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http:/
n a separate thread
so they're not guaranteed to be posted in time. This commit fixes this
flakiness through a job end registration callback.
Author: Andrew Or
Closes #8176 from andrewor14/fix-accumulator-suite.
(cherry picked from commit 6518ef63037aa56b541927f99ad26744f91098ce)
Signed-off-by
Repository: spark
Updated Branches:
refs/heads/branch-1.5 d92568ae5 -> ff3e9561d
[SPARK-9809] Task crashes because the internal accumulators are not properly
initialized
When a stage failed and another stage was resubmitted with only part of
partitions to compute, all the tasks failed with e
Repository: spark
Updated Branches:
refs/heads/master ffa05c84f -> 33bae585d
[SPARK-9809] Task crashes because the internal accumulators are not properly
initialized
When a stage failed and another stage was resubmitted with only part of
partitions to compute, all the tasks failed with error
ort.scala:393)
at
akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
at scala.util.Try$.apply(Try.scala:161)
```
Author: Andrew Or
Closes #7968 from andrewor14/fix-master-flaky-test and squashes the follo
Repository: spark
Updated Branches:
refs/heads/branch-1.5 883c7d35f -> 30460206f
[SPARK-9649] Fix MasterSuite, third time's a charm
This particular test did not load the default configurations so
it continued to start the REST server, which causes port bind
exceptions.
Project: http://git-wi
Repository: spark
Updated Branches:
refs/heads/master 65fec798c -> 8815ba2f6
[SPARK-9649] Fix MasterSuite, third time's a charm
This particular test did not load the default configurations so
it continued to start the REST server, which causes port bind
exceptions.
Project: http://git-wip-us
Repository: spark
Updated Branches:
refs/heads/branch-1.5 ca39c9e91 -> 4b547b91d
[SPARK-9870] Disable driver UI and Master REST server in SparkSubmitSuite
I think that we should pass additional configuration flags to disable the
driver UI and Master REST server in SparkSubmitSuite and HiveSpa
Repository: spark
Updated Branches:
refs/heads/master f4bc01f1f -> 7b13ed27c
[SPARK-9870] Disable driver UI and Master REST server in SparkSubmitSuite
I think that we should pass additional configuration flags to disable the
driver UI and Master REST server in SparkSubmitSuite and HiveSparkSu
it's not clear whether this fixes SPARK-9745, fixing this potential race
condition seems like a strict improvement. I've added a regression test to
illustrate the issue.
Author: Andrew Or
Closes #8078 from andrewor14/da-double-kill.
(cherry picked from commit be5d1912076c2ffd21ec886
7;s not clear whether this fixes SPARK-9745, fixing this potential race
condition seems like a strict improvement. I've added a regression test to
illustrate the issue.
Author: Andrew Or
Closes #8078 from andrewor14/da-double-kill.
Project: http://git-wip-us.apache.org/repos/asf/spar
Repository: spark
Updated Branches:
refs/heads/branch-1.5 a2f805729 -> 5dd0c5cd6
[SPARK-9575] [MESOS] Add docuemntation around Mesos shuffle service.
andrewor14
Author: Timothy Chen
Closes #7907 from tnachen/mesos_shuffle.
(cherry picked from com
Repository: spark
Updated Branches:
refs/heads/master 5c99d8bf9 -> 741a29f98
[SPARK-9575] [MESOS] Add docuemntation around Mesos shuffle service.
andrewor14
Author: Timothy Chen
Closes #7907 from tnachen/mesos_shuffle.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Com
Repository: spark
Updated Branches:
refs/heads/branch-1.5 93fc95934 -> a2f805729
[SPARK-8798] [MESOS] Allow additional uris to be fetched with mesos
Some users like to download additional files in their sandbox that they can
refer to from their spark program, or even later mount these files t
Repository: spark
Updated Branches:
refs/heads/master bab892328 -> 5c99d8bf9
[SPARK-8798] [MESOS] Allow additional uris to be fetched with mesos
Some users like to download additional files in their sandbox that they can
refer to from their spark program, or even later mount these files to an
Repository: spark
Updated Branches:
refs/heads/master 4e3f4b934 -> bab892328
[SPARK-9426] [WEBUI] Job page DAG visualization is not shown
To reproduce the issue, go to the stage page and click DAG Visualization once,
then go to the job page to show the job DAG visualization. You will only see
Repository: spark
Updated Branches:
refs/heads/branch-1.5 d9d4bdea2 -> 93fc95934
[SPARK-9426] [WEBUI] Job page DAG visualization is not shown
To reproduce the issue, go to the stage page and click DAG Visualization once,
then go to the job page to show the job DAG visualization. You will only
Repository: spark
Updated Branches:
refs/heads/master a807fcbe5 -> 4e3f4b934
[SPARK-9829] [WEBUI] Display the update value for peak execution memory
The peak execution memory is not correct because it shows the sum of finished
tasks' values when a task finishes.
This PR fixes it by using the
Repository: spark
Updated Branches:
refs/heads/branch-1.5 402c0ca9a -> d9d4bdea2
[SPARK-9829] [WEBUI] Display the update value for peak execution memory
The peak execution memory is not correct because it shows the sum of finished
tasks' values when a task finishes.
This PR fixes it by using
Repository: spark
Updated Branches:
refs/heads/branch-1.4 6dde38026 -> 89c8aea94
[SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple
applications
Author: Rohit Agarwal
Closes #8088 from mindprince/SPARK-9806.
(cherry picked from commit a807fcbe50b2ce18751d80d39e9d21842f7da
Repository: spark
Updated Branches:
refs/heads/branch-1.5 2f909184e -> 402c0ca9a
[SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple
applications
Author: Rohit Agarwal
Closes #8088 from mindprince/SPARK-9806.
(cherry picked from commit a807fcbe50b2ce18751d80d39e9d21842f7da
Repository: spark
Updated Branches:
refs/heads/master b85f9a242 -> a807fcbe5
[SPARK-9806] [WEB UI] Don't share ReplayListenerBus between multiple
applications
Author: Rohit Agarwal
Closes #8088 from mindprince/SPARK-9806.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit:
Repository: spark
Updated Branches:
refs/heads/master b1581ac28 -> b85f9a242
[SPARK-8366] maxNumExecutorsNeeded should properly handle failed tasks
Author: xutingjun
Author: meiyoula <1039320...@qq.com>
Closes #6817 from XuTingjun/SPARK-8366.
Project: http://git-wip-us.apache.org/repos/asf
Repository: spark
Updated Branches:
refs/heads/branch-1.5 b994f8928 -> 2f909184e
[SPARK-8366] maxNumExecutorsNeeded should properly handle failed tasks
Author: xutingjun
Author: meiyoula <1039320...@qq.com>
Closes #6817 from XuTingjun/SPARK-8366.
(cherry picked from commit b85f9a242a12e8096
Repository: spark
Updated Branches:
refs/heads/branch-1.5 725e5c7a4 -> 1067c7369
[HOTFIX] Fix style error caused by ef961ed48a4f45447f0e0ad256b040c7ab2d78d9
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1067c736
Tree: ht
Repository: spark
Updated Branches:
refs/heads/master 017b5de07 -> 736af95bd
[HOTFIX] Fix style error caused by 017b5de
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/736af95b
Tree: http://git-wip-us.apache.org/repos/asf/
Repository: spark
Updated Branches:
refs/heads/branch-1.5 47e473550 -> 2ad75d99f
[SPARK-9731] Standalone scheduling incorrect cores if spark.executor.cores is
not set
The issue only happens if `spark.executor.cores` is not set and executor memory
is set to a high value.
For example, if we ha
Repository: spark
Updated Branches:
refs/heads/master c564b2744 -> ef062c159
[SPARK-9731] Standalone scheduling incorrect cores if spark.executor.cores is
not set
The issue only happens if `spark.executor.cores` is not set and executor memory
is set to a high value.
For example, if we have a
ode. There should be no
functional differences between this and #7820.
Author: Timothy Chen
Author: Andrew Or
Closes #7881 from andrewor14/tim-cleanup-mesos-shuffle and squashes the
following commits:
8894f7d [Andrew Or] Clean up code
2a5fa10 [Andrew Or] Merge branch 'mesos_shuffle_c
rst introduced in YARN since 1.2, and then extended to Mesos coarse-grained
mode recently. Today, it is finally supported in standalone mode as well!
I tested this locally and it works as expected. This is WIP because unit tests
are coming.
Author: Andrew Or
Closes #7532 from andrewor14/standal
353.
Author: Andrew Or
Closes #7668 from andrewor14/standalone-scheduling-tests and squashes the
following commits:
d852faf [Andrew Or] Add tests + fix scheduling with memory limits
Conflicts:
core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
Project: http://git-
Repository: spark
Updated Branches:
refs/heads/branch-1.4 d0ef42c4e -> 5aed92613
[HOTFIX] Fix compile in MasterSuite
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5aed9261
Tree: http://git-wip-us.apache.org/repos/asf/spa
353.
Author: Andrew Or
Closes #7668 from andrewor14/standalone-scheduling-tests and squashes the
following commits:
d852faf [Andrew Or] Add tests + fix scheduling with memory limits
(cherry picked from commit 1cf19760d61a5a17bd175a906d34a2940141b76d)
Signed-off-by: Andrew Or
Confli
353.
Author: Andrew Or
Closes #7668 from andrewor14/standalone-scheduling-tests and squashes the
following commits:
d852faf [Andrew Or] Add tests + fix scheduling with memory limits
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/com
hen
spark.cores.max/spark.executor.cores (i.e, num_executors) < num_workers,
executors are not launched and the app hangs. This PR fixes and refactors the
scheduling algorithm.
andrewor14
Author: Nishkam Ravi
Author: nishkamravi2
Closes #7274 from nishkamravi2/master_scheduler and squash
hen
spark.cores.max/spark.executor.cores (i.e, num_executors) < num_workers,
executors are not launched and the app hangs. This PR fixes and refactors the
scheduling algorithm.
andrewor14
Author: Nishkam Ravi
Author: nishkamravi2
Closes #7274 from nishkamravi2/master_scheduler and squashes the fol
Repository: spark
Updated Branches:
refs/heads/master 4f7f1ee37 -> d45355ee2
[SPARK-5423] [CORE] Register a TaskCompletionListener to make sure release all
resources
Make `DiskMapIterator.cleanup` idempotent and register a TaskCompletionListener
to make sure call `cleanup`.
Author: zsxwing
Repository: spark
Updated Branches:
refs/heads/master 31954910d -> 4f7f1ee37
[SPARK-4598] [WEBUI] Task table pagination for the Stage page
This PR adds pagination for the task table to solve the scalability issue of
the stage page. Here is the initial screenshot:
https://cloud.githubuserconte
Repository: spark
Updated Branches:
refs/heads/master 9a4fd875b -> 31954910d
[SPARK-7171] Added a method to retrieve metrics sources in TaskContext
Author: Jacek Lewandowski
Closes #5805 from jacek-lewandowski/SPARK-7171 and squashes the following
commits:
ed20bda [Jacek Lewandowski] SPARK
Repository: spark
Updated Branches:
refs/heads/master f67da43c3 -> 9a4fd875b
[SPARK-9128] [CORE] Get outerclasses and objects with only one method calling
in ClosureCleaner
JIRA: https://issues.apache.org/jira/browse/SPARK-9128
Currently, in `ClosureCleaner`, the outerclasses and objects are
Repository: spark
Updated Branches:
refs/heads/master 6592a6058 -> f67da43c3
[SPARK-9036] [CORE] SparkListenerExecutorMetricsUpdate messages not included in
JsonProtocol
This PR implements a JSON serializer and deserializer in the JSONProtocol to
handle the (de)serialization of SparkListener
601 - 700 of 1712 matches
Mail list logo
