[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018590#comment-15018590 ] Naganarasimha G R commented on YARN-3784: - Thanks for considering [~sunilg], May be if i am informed with the usage of the preemption timeout then i could understand better. IIUC if its for to take better decision, then assume a scenario such that AM's heartbeat interval is 3 seconds and the preemption timeout is 5 seconds, and assume just after a am heartbeat container was targeted to be preempted then in the next hearbeat that is after 3 seconds it will be informed that a container will be timed out in 5 seconds instead of 2 seconds which i feel is of little help or might even lead to wrong decisions!. Hence suggested timestamp instead of just the timeout value. Thoughts? Correct me if i am wrong ! > Indicate preemption timout along with the list of containers to AM > (preemption message) > --- > > Key: YARN-3784 > URL: https://issues.apache.org/jira/browse/YARN-3784 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch, > 0003-YARN-3784.patch, 0004-YARN-3784.patch > > > Currently during preemption, AM is notified with a list of containers which > are marked for preemption. Introducing a timeout duration also along with > this container list so that AM can know how much time it will get to do a > graceful shutdown to its containers (assuming one of preemption policy is > loaded in AM). > This will help in decommissioning NM scenarios, where NM will be > decommissioned after a timeout (also killing containers on it). This timeout > will be helpful to indicate AM that those containers can be killed by RM > forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019226#comment-15019226 ] Sunil G commented on YARN-3784: --- As mentioned earlier, preemption timeout for a container can vary. (PCPP or decommission). Assume container1 is marked for preemption at time frame X. After that container2 at time frame X+1. Similarly container3 at X+2. If AM heartbeat interval is 3sec, next heartbeat will come at X+3. Hence there is an elapsed time for each 3 containers which are to be preempted in the scheduler wait queue. Like container1 has 3sec, container2 has 2sec and container3 has 1sec. If this elapsed time can be subtracted from each containers proposed timeout, a more realistic and correct timeout will reach AM. This can help in taking a better decision whether to a graceful checkpoint preemption or do some immediate local copy of some o/p file etc. I ll raise these AM side improvements as needed after this ticket. Also this is a good metric in AM, user can also get this information. If any containers crossed it's timeout while waiting for AM heartbeat like u mentioned, I ll mark as - 1 or 0 to indicate to AM that a possible action is already taken in RM mostly, and AM need not to do any graceful preemption on that. This will also be an improvement in AM side which is already in plan. I ll be handling this in the MR side ticket. Does this make sense? > Indicate preemption timout along with the list of containers to AM > (preemption message) > --- > > Key: YARN-3784 > URL: https://issues.apache.org/jira/browse/YARN-3784 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch, > 0003-YARN-3784.patch, 0004-YARN-3784.patch > > > Currently during preemption, AM is notified with a list of containers which > are marked for preemption. Introducing a timeout duration also along with > this container list so that AM can know how much time it will get to do a > graceful shutdown to its containers (assuming one of preemption policy is > loaded in AM). > This will help in decommissioning NM scenarios, where NM will be > decommissioned after a timeout (also killing containers on it). This timeout > will be helpful to indicate AM that those containers can be killed by RM > forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[
https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15020191#comment-15020191
]
Naganarasimha G R commented on YARN-3784:
-
Thanks [~sunilg], for updating offline on this issue, basically i was missing
the part that in FiCaSchedulerApp you are subtracting the elapsed time and
hence you were able send the effective timeout to the AM during Hearbeat, but i
can see following issues with current approach than having probable timestamp
(current time + preemption timeout) during the creation of PreemptionContainer
and share this to AM
* There can be a small delta between actual timeout value and the time when it
can actually timeout
* some additional loops during creation of response during heartbeat response
(though not a thing of high performance impact but nevertheless can be avoided
)
* Avoid additional storage of {{containersWithFirstNotifyTime}} in
FiCaSchedulerApp
But may current approach is more simpler for users to understand a numerical
value than a timestamp!, thoughts from others ?
Also few issues with the patch :
* possible leak in {{containersWithFirstNotifyTime}} as remove is not being
called?
* can there be a case where {{containersWithFirstNotifyTime}} be not filled in
for a preempted container ? if not i feel additional if check {{if
(containersWithFirstNotifyTime.containsKey(c))}} in the for loop is not
required.
> Indicate preemption timout along with the list of containers to AM
> (preemption message)
> ---
>
> Key: YARN-3784
> URL: https://issues.apache.org/jira/browse/YARN-3784
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch,
> 0003-YARN-3784.patch, 0004-YARN-3784.patch
>
>
> Currently during preemption, AM is notified with a list of containers which
> are marked for preemption. Introducing a timeout duration also along with
> this container list so that AM can know how much time it will get to do a
> graceful shutdown to its containers (assuming one of preemption policy is
> loaded in AM).
> This will help in decommissioning NM scenarios, where NM will be
> decommissioned after a timeout (also killing containers on it). This timeout
> will be helpful to indicate AM that those containers can be killed by RM
> forcefully after the timeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[
https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15020210#comment-15020210
]
Sunil G commented on YARN-3784:
---
Thanks [~Naganarasimha Garla] for the comments.
As discussed offline, this patch is proposing a timeout with each
to-be-preempted container to AM. As mentioned above, this timeout is derived by
calculating,
{noformat}
(Timeout associated with Each container) - (Total Timewait happened for this
container in the queue for AM heartbeat)
{noformat}
For eg: if 15 sec is the proposed timeout for a container to get a hard kill
from RM and 3 sec is the AM heartbeat interval, then in worst case 12sec will
be timeout proposed to AM.
So few points here based on the comments above.
- 12 seconds to timeout is a meaningful unit to AM to understand that the
specified container will be preempted.
- A Delta in terms of milli seconds may happen as the time taken to reach
heartbeat response from RM to AM. As I see this, this may not very big in
terms of considerable time unit. I would like to get an opinion from community
on this part.
If this DELTA is not much considerable amount of unit, then giving a
{{timeout}} to AM is more meaningful and easily understandable. Else we may
give a time unit like {{time from epoch time + timeout}}, which AM needs to
re-translate by looking at current time in AM's end and take a difference (a
LONG value will be passed in heartbeat). This is also point to note as the
information passed is not very direct or understandable like "going to get
timeout in X milli seconds".
As mentioned by Naga, this latter option will definitely help in avoiding few
extra logic in RM end (calculation of timeout difference) but comes with a cost
of added complexity/less clarity in the information passed. [~leftnoteasy] and
[~djp], could you also pls share your views on this.
[~Naganarasimha Garla], I will see the other comments based on the feedback for
the first points. Thank You!!
> Indicate preemption timout along with the list of containers to AM
> (preemption message)
> ---
>
> Key: YARN-3784
> URL: https://issues.apache.org/jira/browse/YARN-3784
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch,
> 0003-YARN-3784.patch, 0004-YARN-3784.patch
>
>
> Currently during preemption, AM is notified with a list of containers which
> are marked for preemption. Introducing a timeout duration also along with
> this container list so that AM can know how much time it will get to do a
> graceful shutdown to its containers (assuming one of preemption policy is
> loaded in AM).
> This will help in decommissioning NM scenarios, where NM will be
> decommissioned after a timeout (also killing containers on it). This timeout
> will be helpful to indicate AM that those containers can be killed by RM
> forcefully after the timeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15013390#comment-15013390 ] Sunil G commented on YARN-3784: --- Hi [~Naganarasimha Garla] Thank you for bringing in the thoughts, nothing is too late. :) Yes, preemption is now only happening from PCPP. After YARN-3224, preemption will also happen from Decommissioning cases. So timeout b/w forceful killing of container is a varying entity. And in ideal cases, we assume to be greater than AM heartbeat interval (usual cases, it will be). Still we can hit with this corner cases. In existing system, AM skips such requests for preemption as containers might have been killed. This ticket is adding one more information which is timeout, along with list of containers. So its an added information for AM, but can skipped if containers are preempted already by AM. I will see whether I can mark a timeout as 0 or -1 to indicate that those are already passed timeout. Hope this clarify the question. Pls let me know if its ok. > Indicate preemption timout along with the list of containers to AM > (preemption message) > --- > > Key: YARN-3784 > URL: https://issues.apache.org/jira/browse/YARN-3784 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch, > 0003-YARN-3784.patch, 0004-YARN-3784.patch > > > Currently during preemption, AM is notified with a list of containers which > are marked for preemption. Introducing a timeout duration also along with > this container list so that AM can know how much time it will get to do a > graceful shutdown to its containers (assuming one of preemption policy is > loaded in AM). > This will help in decommissioning NM scenarios, where NM will be > decommissioned after a timeout (also killing containers on it). This timeout > will be helpful to indicate AM that those containers can be killed by RM > forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010394#comment-15010394 ] Sunil G commented on YARN-3784: --- Test case failures are unrelated here. [~djp] [~leftnoteasy], could you please help to check the changes here. > Indicate preemption timout along with the list of containers to AM > (preemption message) > --- > > Key: YARN-3784 > URL: https://issues.apache.org/jira/browse/YARN-3784 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch, > 0003-YARN-3784.patch, 0004-YARN-3784.patch > > > Currently during preemption, AM is notified with a list of containers which > are marked for preemption. Introducing a timeout duration also along with > this container list so that AM can know how much time it will get to do a > graceful shutdown to its containers (assuming one of preemption policy is > loaded in AM). > This will help in decommissioning NM scenarios, where NM will be > decommissioned after a timeout (also killing containers on it). This timeout > will be helpful to indicate AM that those containers can be killed by RM > forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[
https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001145#comment-15001145
]
Hadoop QA commented on YARN-3784:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 32s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 3s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
22s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 37s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
trunk has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 46s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 12s
{color} | {color:red} Patch generated 5 new checkstyle issues in root (total
was 392, now 389). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
introduced 1 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 26s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 30s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 51s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 4s
{color} | {color:green} hadoop-sls in the patch passed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 23s
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970802#comment-14970802 ] Junping Du commented on YARN-3784: -- bq. AM protocol changes are handled in this ticket along with CS change. I feel its better to track FairScheduler changes separately and it can be linked to this. The plan sounds reasonable. YARN-3224 depends on this JIRA. [~sunilg], I am reviewing your patch now but it sounds like the patch is already stale. Can you update the patch with latest trunk? Thanks! > Indicate preemption timout along with the list of containers to AM > (preemption message) > --- > > Key: YARN-3784 > URL: https://issues.apache.org/jira/browse/YARN-3784 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch > > > Currently during preemption, AM is notified with a list of containers which > are marked for preemption. Introducing a timeout duration also along with > this container list so that AM can know how much time it will get to do a > graceful shutdown to its containers (assuming one of preemption policy is > loaded in AM). > This will help in decommissioning NM scenarios, where NM will be > decommissioned after a timeout (also killing containers on it). This timeout > will be helpful to indicate AM that those containers can be killed by RM > forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970872#comment-14970872 ] Sunil G commented on YARN-3784: --- Thank you [~djp] Sure, I will rebase the patch soon. > Indicate preemption timout along with the list of containers to AM > (preemption message) > --- > > Key: YARN-3784 > URL: https://issues.apache.org/jira/browse/YARN-3784 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch > > > Currently during preemption, AM is notified with a list of containers which > are marked for preemption. Introducing a timeout duration also along with > this container list so that AM can know how much time it will get to do a > graceful shutdown to its containers (assuming one of preemption policy is > loaded in AM). > This will help in decommissioning NM scenarios, where NM will be > decommissioned after a timeout (also killing containers on it). This timeout > will be helpful to indicate AM that those containers can be killed by RM > forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745512#comment-14745512 ] Sunil G commented on YARN-3784: --- Hi [~adhoot] AM protocol changes are handled in this ticket along with CS change. I feel its better to track FairScheduler changes separately and it can be linked to this. > Indicate preemption timout along with the list of containers to AM > (preemption message) > --- > > Key: YARN-3784 > URL: https://issues.apache.org/jira/browse/YARN-3784 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch > > > Currently during preemption, AM is notified with a list of containers which > are marked for preemption. Introducing a timeout duration also along with > this container list so that AM can know how much time it will get to do a > graceful shutdown to its containers (assuming one of preemption policy is > loaded in AM). > This will help in decommissioning NM scenarios, where NM will be > decommissioned after a timeout (also killing containers on it). This timeout > will be helpful to indicate AM that those containers can be killed by RM > forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744006#comment-14744006 ] Anubhav Dhoot commented on YARN-3784: - Hi [~sunilg] this does not include support for FairScheduler. Are we planning to add that here or have a separate jira to track that work? Thx > Indicate preemption timout along with the list of containers to AM > (preemption message) > --- > > Key: YARN-3784 > URL: https://issues.apache.org/jira/browse/YARN-3784 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch > > > Currently during preemption, AM is notified with a list of containers which > are marked for preemption. Introducing a timeout duration also along with > this container list so that AM can know how much time it will get to do a > graceful shutdown to its containers (assuming one of preemption policy is > loaded in AM). > This will help in decommissioning NM scenarios, where NM will be > decommissioned after a timeout (also killing containers on it). This timeout > will be helpful to indicate AM that those containers can be killed by RM > forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[
https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632406#comment-14632406
]
Sunil G commented on YARN-3784:
---
Thank you [~leftnoteasy] for sharing the comments.
I have tried to do the same in second patch. Please correct me if its not the
way as you thought.
{code}
+ public synchronized void addPreemptContainer(ContainerId cont, long timeout) {
// ignore already completed containers
if (liveContainers.containsKey(cont)) {
+containersToPreempt.put(cont, timeout);
+ containersWithFirstNotifyTime.put(cont, clock.getTime());
{code}
Here along with maxWaitTime, I also put the time in another map with key as
containerID.
Now when AM heartbeat comes {{getAllocation}} will be called. Below code is
added in {{getAllocation}}
{code}
+Long currTime = clock.getTime();
+for (ContainerId c : containersToPreempt.keySet()) {
+ if (containersWithFirstNotifyTime.containsKey(c)) {
+Long timeout = containersToPreempt.get(c);
+containersToPreempt.put(c, timeout
+- (currTime - containersWithFirstNotifyTime.get(c)));
+ }
+}
{code}
This may happen in another time frame. Hence I have tried to find the {{delta}}
duration which is elapsed from time of preempt-container reporting to AM
fetching the same. This {{delta}} is reduced from the real {{maxWaitTime}},
which will help to find the possible max time that container will get till it
is forcefully released from RM. This is calculated for each container.
Could you please check this and let me know if I missed any point.
Indicate preemption timout along with the list of containers to AM
(preemption message)
---
Key: YARN-3784
URL: https://issues.apache.org/jira/browse/YARN-3784
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch
Currently during preemption, AM is notified with a list of containers which
are marked for preemption. Introducing a timeout duration also along with
this container list so that AM can know how much time it will get to do a
graceful shutdown to its containers (assuming one of preemption policy is
loaded in AM).
This will help in decommissioning NM scenarios, where NM will be
decommissioned after a timeout (also killing containers on it). This timeout
will be helpful to indicate AM that those containers can be killed by RM
forcefully after the timeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631671#comment-14631671 ] Wangda Tan commented on YARN-3784: -- Hi [~sunilg], Thanks for your comments, I will post cancel-preemption event related comments to YARN-3937 soon. Indicate preemption timout along with the list of containers to AM (preemption message) --- Key: YARN-3784 URL: https://issues.apache.org/jira/browse/YARN-3784 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sunil G Assignee: Sunil G Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch Currently during preemption, AM is notified with a list of containers which are marked for preemption. Introducing a timeout duration also along with this container list so that AM can know how much time it will get to do a graceful shutdown to its containers (assuming one of preemption policy is loaded in AM). This will help in decommissioning NM scenarios, where NM will be decommissioned after a timeout (also killing containers on it). This timeout will be helpful to indicate AM that those containers can be killed by RM forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631691#comment-14631691 ] Wangda Tan commented on YARN-3784: -- And also about this patch, same as commented by [~chris.douglas]. I found timeout sent to AM is maxWaitTime, which I think should be how much time till the container preempted. Maybe one solution is compute a absolute time for each to-be-preempted containers, and timeout will be computed when AM is pulling these information. Indicate preemption timout along with the list of containers to AM (preemption message) --- Key: YARN-3784 URL: https://issues.apache.org/jira/browse/YARN-3784 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sunil G Assignee: Sunil G Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch Currently during preemption, AM is notified with a list of containers which are marked for preemption. Introducing a timeout duration also along with this container list so that AM can know how much time it will get to do a graceful shutdown to its containers (assuming one of preemption policy is loaded in AM). This will help in decommissioning NM scenarios, where NM will be decommissioned after a timeout (also killing containers on it). This timeout will be helpful to indicate AM that those containers can be killed by RM forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630167#comment-14630167 ] Wangda Tan commented on YARN-3784: -- Beyond timeout, another thing we may need consider is: after a container is removed from to-be-preempted list, should we notify scheduler/AM about that? This could happen if other applications release containers, or other queues/applications cancel resource requests. Now proportionalCPP can notify scheduler many times for a same container, if we have to-preempt/remove-from-to-preempt event, we can also reduce number of messages send to scheduler (which could cause YARN-3508 happens). Indicate preemption timout along with the list of containers to AM (preemption message) --- Key: YARN-3784 URL: https://issues.apache.org/jira/browse/YARN-3784 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sunil G Assignee: Sunil G Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch Currently during preemption, AM is notified with a list of containers which are marked for preemption. Introducing a timeout duration also along with this container list so that AM can know how much time it will get to do a graceful shutdown to its containers (assuming one of preemption policy is loaded in AM). This will help in decommissioning NM scenarios, where NM will be decommissioned after a timeout (also killing containers on it). This timeout will be helpful to indicate AM that those containers can be killed by RM forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[ https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630794#comment-14630794 ] Sunil G commented on YARN-3784: --- Yes [~leftnoteasy] Thank you for sharing your thoughts. I f I understood you correctly, there are chances that to-be-preempted container will reside in FicaSchedulerApp till allocate call comes from AM. Within this duration, there are chances that some more containers got free or cancelled its resource requests. Due to this, we may remove this container from this to-be-preempted list. I feel we can have a remove-from-to-preempt in scheduler, and propportionalCPP can notify the app when such scenario occurs. This can be added as a new argument to AM response also. I will separate this improvement in to another ticket. From your second point, I feel we can keep a getter api (synchronized) for to-be-preempted containers which is present in FicaSchedulerApp (scheduler level). With this api, proportionalCPP can have look whether the container which is newly identified to preempt is already reported as to-be-preempted container in app level. If so, proportionalCPP need not have to raise another event to scheduler. I ll separate this if its ok. Indicate preemption timout along with the list of containers to AM (preemption message) --- Key: YARN-3784 URL: https://issues.apache.org/jira/browse/YARN-3784 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sunil G Assignee: Sunil G Attachments: 0001-YARN-3784.patch, 0002-YARN-3784.patch Currently during preemption, AM is notified with a list of containers which are marked for preemption. Introducing a timeout duration also along with this container list so that AM can know how much time it will get to do a graceful shutdown to its containers (assuming one of preemption policy is loaded in AM). This will help in decommissioning NM scenarios, where NM will be decommissioned after a timeout (also killing containers on it). This timeout will be helpful to indicate AM that those containers can be killed by RM forcefully after the timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[
https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608543#comment-14608543
]
Sunil G commented on YARN-3784:
---
Thankyou [~chris.douglas] for the comments.
I will update a patch correcting these problems. Regarding below point,
bq.If containers are preempted for multiple causes (e.g., over-capacity, NM
decommission), then the time to preempt could vary widely
My concern also was same. Currently preemption message will look like below.
{noformat}
message PreemptionContractProto {
repeated PreemptionResourceRequestProto resource = 1;
repeated PreemptionContainerProto container = 2;
+ optional int64 timeout = 3;
}
message PreemptionContainerProto {
optional ContainerIdProto id = 1;
}
{noformat}
I have added {{timeout}} per message level. I can try attaching it per
container level as an optional parameter. One potential bottleneck is,
different preemption events(ProportionalCPP, Decommission etc) can come to
Application at different time. And {{allocate}} call from
ApplicationMasterService may hit after some secs to fetch to be preempted
containers. Hence there can be some elapsed time already lost for few
containers. We can subtract and then send to AM, but will it overload scheduler
if many containers are marked for preemption (storing last update time per
container level)?
Indicate preemption timout along with the list of containers to AM
(preemption message)
---
Key: YARN-3784
URL: https://issues.apache.org/jira/browse/YARN-3784
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-YARN-3784.patch
Currently during preemption, AM is notified with a list of containers which
are marked for preemption. Introducing a timeout duration also along with
this container list so that AM can know how much time it will get to do a
graceful shutdown to its containers (assuming one of preemption policy is
loaded in AM).
This will help in decommissioning NM scenarios, where NM will be
decommissioned after a timeout (also killing containers on it). This timeout
will be helpful to indicate AM that those containers can be killed by RM
forcefully after the timeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3784) Indicate preemption timout along with the list of containers to AM (preemption message)
[
https://issues.apache.org/jira/browse/YARN-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606205#comment-14606205
]
Chris Douglas commented on YARN-3784:
-
Minor:
- Docs for timeout don't include units
- Many whitespace changes in {{FiCaSchedulerApp}}
- change nested if to {{}} at:
{noformat}
+if (this.preemptionTimeout != 0) {
+ if (timeout this.preemptionTimeout) {
{noformat}
- Would it be possible to test more than the timeout reported is non-zero? If
this used a {{Clock}} instead of calling {{System.currentTimeMillis}} directly,
the unit test could be easier to write...
If containers are preempted for multiple causes (e.g., over-capacity, NM
decommission), then the time to preempt could vary widely. The ProportionalCPP
also limits the preempted capacity per round, so a global timeout will be very
pessimistic. Would it make sense to change {{timeout}} to be {{nextkill}}? More
general solutions would be significantly more work...
Indicate preemption timout along with the list of containers to AM
(preemption message)
---
Key: YARN-3784
URL: https://issues.apache.org/jira/browse/YARN-3784
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-YARN-3784.patch
Currently during preemption, AM is notified with a list of containers which
are marked for preemption. Introducing a timeout duration also along with
this container list so that AM can know how much time it will get to do a
graceful shutdown to its containers (assuming one of preemption policy is
loaded in AM).
This will help in decommissioning NM scenarios, where NM will be
decommissioned after a timeout (also killing containers on it). This timeout
will be helpful to indicate AM that those containers can be killed by RM
forcefully after the timeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
