Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/634#discussion_r15173511
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
---
@@ -30,6 +30,11 @@ private[spark] class
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/634#discussion_r15173596
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientClusterScheduler.scala
---
@@ -37,14 +37,4 @@ private[spark] class
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-49643611
Looks good. +1, Thanks @sryza.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/634
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/634#discussion_r15156184
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientClusterScheduler.scala
---
@@ -37,14 +37,4 @@ private[spark] class
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/634#discussion_r15156221
--- Diff:
yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
---
@@ -30,6 +30,11 @@ private[spark] class
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-49554989
Updated patch sets the min registered executors ratio to .8 for YARN
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-49555111
QA tests have started for PR 634. This patch merges cleanly. brView
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16877/consoleFull
---
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-49558003
QA results for PR 634:br- This patch PASSES unit tests.br- This patch
merges cleanlybr- This patch adds no public classesbrbrFor more
information see test
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-49429447
He had increased it to 30 seconds based on some experiments he did. I
guess it depends on how well the scheduler recovers from starting with fewer
executors. I know
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-49126877
That's a good point. Changing it for YARN seems like the right thing, and
80% sounds reasonable to me.
Another thing is the wait time. Previously it was 6
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/634#discussion_r14940726
--- Diff:
yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -416,19 +407,8 @@ object ApplicationMaster extends Logging
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/634#discussion_r14940751
--- Diff:
yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -370,7 +359,6 @@ object ApplicationMaster extends Logging
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-49048527
I just thought of something bad here. Now the default behavior is to wait
for 0% and we have removed the bit of sleep there so if someone is just using
the default
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-48957597
@sryza I merged in SPARK-1946, would you want to update this to handle both
cluster mode and client mode. I think we can actually also remove the
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-48969752
QA tests have started for PR 634. This patch merges cleanly. brView
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16642/consoleFull
---
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-48971649
Posted a patch that takes out the sleeps and waitForInitialAllocations
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-48971773
QA tests have started for PR 634. This patch merges cleanly. brView
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16644/consoleFull
---
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-48976630
QA results for PR 634:br- This patch PASSES unit tests.br- This patch
merges cleanlybr- This patch adds no public classesbrbrFor more
information see test
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-48978149
QA results for PR 634:br- This patch PASSES unit tests.br- This patch
merges cleanlybr- This patch adds no public classesbrbrFor more
information see test
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-45267979
@sryza can we close this and just do SPARK-1453 and/or SPARK-1946
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-45274561
I guess we could keep this around incase we do SPARK-1946
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42172774
Btw I dont recall exactly why the actual value of 3 seconds is there; I
think earlier spark used to crash in case there are no containers available
when job is going to
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42188695
Note I had filed SPARK-1453 to make the wait configurable, see the
discussions there. I'm fine with removing this once the other one is
configurable. I would
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42209580
I like the idea of making the first wait configurable and taking out the
second. For the second, we could possibly still wait for the executors we've
launched to register.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42256611
@sryza Feel free to take over that jira to make it configurable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/634
SPARK-1707. Remove unnecessary 3 second sleep in YarnClusterScheduler
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sryza/spark sandy-spark-1707
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42141909
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42141915
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42142889
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14647/
---
If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42142888
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42146148
@sryza @mridulm what is this sleep actually for? We should really have
in-code documentation for things like this.
If it's necessary to sleep like this, could we
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42149808
Ah I see - sorry I didn't read the JIRA.
So since we don't know a-priori how many executors to expect, I don't think
we can wait on any particular condition.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42149840
@sryza @mridulm if we do decide we want to have a new type of waiting here,
maybe we could add `spark.scheduler.initialWait` or something to give time to
allow executors
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42159221
Keep in mind this sleep is after we've already waited to receive a set of
containers from the YARN ResourceManager and sent RPCs to the NodeManagers to
start them. So we
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42159371
@sryza Ah I see, in that case, we could wait on the specific condition, but
what if e.g. there is a straggler and one container takes a long time to launch
(or never
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42159417
Yeah, I think we'd want some (probably configurable) max wait time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/634#issuecomment-42159450
Seems reasonable - still curious exactly what the case @mridulm is dealing
with is. I guess it's just the case where the containers take a long time to
launch (once
38 matches
Mail list logo
