Github user li-zhihui closed the pull request at:
https://github.com/apache/spark/pull/5451
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/5451#issuecomment-91730112
@srowen @vanzin Thanks for your comments.
I add a new commit as your comments.
---
If your project is set up for it, you can reply to this email and have your
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/5451#issuecomment-91732744
@srowen I guess the exception maybe make user feel puzzled because it's
unexpected, although it doesn't effect application execution.
---
If your project is set up
GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/5255
Fix string interpolator error in HeartbeatReceiver
Error log before fixed
code15/03/29 10:07:25 ERROR YarnScheduler: Lost an executor 24 (already
removed): Executor heartbeat timed out after
Github user li-zhihui closed the pull request at:
https://github.com/apache/spark/pull/1462
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1462#discussion_r25405036
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -62,6 +62,11 @@ private[spark] class
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1462#discussion_r25405040
--- Diff: docs/configuration.md ---
@@ -831,7 +831,7 @@ Apart from these, the following properties are also
available, and may be useful
td0/td
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-76123507
Add some new commits to fix code conflict and some issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/4781
Modify default value description for
spark.scheduler.minRegisteredResourcesRatio on docs.
Because the configuration is not supported in mesos mode now.
See https://github.com/apache/spark
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-74014949
@pwendell Do we need the feature in mesos mode? I am pleasure to update it.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r19321179
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -322,14 +322,14 @@ private[spark] class Executor(
// Fetch missing
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-60346005
@andrewor14 I guess the failure is non-interrelated with the patch. But I
don't know why failed again, can you give me some advise?
---
If your project is set up
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-60025907
@andrewor14 more comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r18564683
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -313,15 +313,84 @@ private[spark] object Utils extends Logging
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r18564686
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -313,15 +313,84 @@ private[spark] object Utils extends Logging
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r18564689
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -313,15 +313,84 @@ private[spark] object Utils extends Logging
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-56126098
@andrewor14 any more comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-55852899
@andrewor14 @JoshRosen
I am not sure if the test failure is related to the patch. Can you have a
look at the failure? Or just retest it?
---
If your project
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-55856239
@JoshRosen thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r17643781
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -313,15 +313,83 @@ private[spark] object Utils extends Logging
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r17643825
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -322,14 +322,14 @@ private[spark] class Executor(
// Fetch missing
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r17578863
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -313,15 +313,84 @@ private[spark] object Utils extends Logging
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r17524360
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -313,15 +313,84 @@ private[spark] object Utils extends Logging
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r17524413
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -805,11 +805,12 @@ class SparkContext(config: SparkConf) extends Logging
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r17524670
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -313,15 +313,84 @@ private[spark] object Utils extends Logging
Github user li-zhihui closed the pull request at:
https://github.com/apache/spark/pull/1684
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-55084516
Thanks @chenghao-intel It seems the sql unit test failure is fixed.
@andrewor14 Can you retest this again?
---
If your project is set up for it, you can reply
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r17278282
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -313,14 +313,74 @@ private[spark] object Utils extends Logging
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-54914557
@andrewor14
In yarn mode, these cache files will be clean up automatically, and in
standalone mode, it's not handled.
Now in standalone mode, application
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r17278861
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -313,14 +313,74 @@ private[spark] object Utils extends Logging
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-54600822
@JoshRosen @andrewor14
I use codeurl.hashCode + timestamp/code as codecachedFileName/code,
I believe it is impossible that existing codeurl.hashCode/code collision
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r17097313
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -317,13 +317,58 @@ private[spark] object Utils extends Logging
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-54408600
@JoshRosen @andrewor14
I test the patch in yarn mode, and the codelocalDir/code is a
per-application temporary directory in this mode. Now I know
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-53519868
@JoshRosen do you have time to review it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-53229624
Rollback old commits, add a new commit base on latest code.
@pwendell @tgravescs @kayousterhout @tnachen
---
If your project is set up for it, you can reply
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-52873144
@JoshRosen any more comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-52873228
@tgravescs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51420203
@tgravescs
Test failure because FileServerSuite:Build timed out (after 120
minutes), but I guess the patch is irrelevant to the issue.
Can you ask
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51282316
@tgravescs got it.
But, codeYarnClusterSchedulerBackend/code use the same strategy (code
and comment).
https://github.com/apache/spark/blob/master/yarn
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51283355
@tgravescs I rollback previous commit and add a new commit just update
comment.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-51285402
@tgravescs done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-51051752
@JoshRosen added comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-51147815
Thaks @JoshRosen sorry I missed the important operation (and I missed
codeFileUtil.chmod(targetFile.getAbsolutePath, a+x)/code too).
I add a new commit
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15738382
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
---
@@ -40,6 +41,10 @@ private[spark] class
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15738384
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -47,19 +47,19 @@ class
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-51011136
As @pwendell says, the configuration is disable in standalone mode. And in
the worst situation, it sleep
codespark.scheduler.maxRegisteredResourcesWaitingTime/code
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1616#issuecomment-51011405
@JoshRosen more comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-51019363
@pwendell removing support for this in standalone mode is just keeping
totalExpectedExecutors zero.
https://github.com/li-zhihui/spark/commit
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-50962993
Maybe we should think the feature in standalone mode and mesos mode
together.
Is it necessary in mesos mode? https://github.com/apache/spark/pull/1462
GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/1684
Fix conflict between code and doc in YarnClientSchedulerBackend.scala
Doc say: system properties override environment variables.
https://github.com/apache/spark/blob/master/yarn/common/src
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1684#issuecomment-50837342
@tgravescs can you please to have a look at this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-50850584
@tgravescs @kayousterhout can you close this PR before code frozen of 1.1
release? Otherwise, it would result in incompatible configuration property name
because
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1616#discussion_r15447773
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -317,6 +317,28 @@ private[spark] object Utils extends Logging
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-50423094
@tnachen I add a new PR to try to fix the issue,
https://github.com/apache/spark/pull/1525
---
If your project is set up for it, you can reply to this email and have
GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/1616
[SPARK-2713] Executors of same application in same host should only
download files jars once
If Spark lunched multiple executors in one host for one application, every
executor would download
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-49836422
I add a new commit, @tgravescs @markhamstra
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15272948
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -268,14 +264,18 @@ class
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15325405
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
---
@@ -36,6 +36,7 @@ private[spark] class
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-49957064
@kayousterhout I think use totalExpectedCores and totalExpectedExecutors
replace totalExpectedResources is a good idea, thanks.
---
If your project is set up
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-49960668
@tgravescs @kayousterhout I add a new commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/1525
Fix race condition at SchedulerBackend.isReady in standalone mode
In SPARK-1946(PR #900), configuration
codespark.scheduler.minRegisteredExecutorsRatio/code was introduced.
However
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1525#issuecomment-49714878
@kayousterhout @tgravescs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15268735
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -47,19 +47,19 @@ class
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15268755
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
---
@@ -108,4 +108,8 @@ private[spark] class
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15268932
--- Diff: docs/configuration.md ---
@@ -707,21 +707,22 @@ Apart from these, the following properties are also
available, and may be useful
/td
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/1525#discussion_r15269102
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -47,19 +47,19 @@ class
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-49690527
Sorry @tgravescs @kayousterhout I am not aware of the issue's seriousness
at that time. thanks @kayousterhout for your coach.
---
If your project is set up
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-49572335
@tgravescs
I tested it on a cluster with mesos-0.18.1(fine-grained and
coarse-grained), it work well.
I think you are right. In fact, user don't have
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1462#issuecomment-49284642
@tgravescs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-48869936
@tgravescs add a commit according to comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14813843
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -46,9 +46,19 @@ class
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-48714143
Thanks @tgravescs
I will file a new jira for handling mesos and follow it after the PR merged.
---
If your project is set up for it, you can reply to this email
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-47490304
@tgravescs @kayousterhout
It will lead to a logic deadlock in yarn-cluster mode, if waitBackendReady
is in TaskSchedulerImpl.start.
How about move
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14280510
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-47331430
@tgravescs @kayousterhout
I move waitBackendReady back to submitTasks method, because it
(waitBackendReady in start method) dose not work on yarn-cluster mode
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14231480
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -244,6 +255,17 @@ class
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-47204723
@tgravescs @kayousterhout
I add a new commit
* Move waitBackendReady to TaskSchedulerImpl.start
* Code refactor by @kayousterhout 's comments
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14232018
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
---
@@ -0,0 +1,51 @@
+/*
+ * Licensed
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14279974
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14280169
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14280444
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14225172
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
---
@@ -0,0 +1,51 @@
+/*
+ * Licensed
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14225319
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -244,6 +255,17 @@ class
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14225397
--- Diff:
yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -164,6 +164,7 @@ class ApplicationMaster(args
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14225520
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
---
@@ -95,6 +95,7 @@ private[spark] class
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-47188350
@tgravescs @kayousterhout
How about move waitBackendReady to TaskSchedulerImpl.start. It will be
called only once at spark initialization.
---
If your project
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14166951
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -46,9 +46,14 @@ class
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-47061333
@tgravescs
I add a new commit that move waitBackendReady from DAGScheduler.submitStage
to TaskSchedulerImpl.submitTasks, for 2 reasons
* Optimize
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14060589
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterSchedulerBackend.scala
---
@@ -0,0 +1,51 @@
+/*
+ * Licensed
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r14010613
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -225,6 +232,17 @@ class
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-46655979
Thanks @tgravescs
I add a new commit.
* code style
* default minRegisteredRatio = 0 in yarn mode
* driver get --num-executors in yarn/alpha
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-46657630
@tgravescs @mridulm
In my test case(4 nodes, 128 executors), it need 25 seconds to register all
executors.
Now maxRegisteredWaitingTime = 10 senconds, I think
GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/1131
[SPARK-2193] Improve tasks preferrd locality by sorting tasks partial or...
Now, the last executor(s) maybe not get itâs preferred task(s), although
these tasks have build
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/1131#issuecomment-46540777
@mridulm
for example:
2 tasks(task_x, task_y), 2 executors(host1, host2)
task_x.preferredLocations = [host2, host3, host1]
task_y.preferredLocations
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r13851868
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
---
@@ -77,6 +77,12 @@ private[spark] class
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-46291702
@tgravescs I add a commit support yarn-cluster.
A little issue, the YarnClusterSchedulerBackend can't get --num-executors
as totalExecutors currently(spark
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r13895415
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -225,6 +232,17 @@ class
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r13793268
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -48,6 +48,10 @@ class
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r13740596
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -48,6 +48,10 @@ class
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r13688559
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -431,6 +431,16 @@ private[spark] class TaskSchedulerImpl
1 - 100 of 106 matches
Mail list logo
