[jira] [Commented] (MAPREDUCE-6541) Exclude pending reducer memory when calculating available mapper slots from headroom to avoid deadlock
[ https://issues.apache.org/jira/browse/MAPREDUCE-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996221#comment-14996221 ] Varun Saxena commented on MAPREDUCE-6541: - I think it should be scheduled reducer's memory which should be excluded instead of pending. > Exclude pending reducer memory when calculating available mapper slots from > headroom to avoid deadlock > --- > > Key: MAPREDUCE-6541 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6541 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Varun Saxena > > We saw a MR deadlock recently: > - When NM restarted by framework without enable recovery, containers running > on these nodes will be identified as "ABORTED", and MR AM will try to > reschedule "ABORTED" mapper containers. > - Since such lost mappers are "ABORTED" container, MR AM gives normal mapper > priority (priority=20) to such mapper requests. If there's any pending > reducer (priority=10) at the same time, mapper requests need to wait for > reducer requests satisfied. > - In our test, one mapper needs 700+ MB, reducer needs 1000+ MB, and RM > available resource = mapper-request = (700+ MB), only one job was running in > the system so scheduler cannot allocate more reducer containers AND MR-AM > thinks there're enough headroom for mapper so reducer containers will not be > preempted. > MAPREDUCE-6302 can solve most of the problems, but in the other hand, I think > we may need to exclude pending reducers resource when calculating > #available-mapper-slots from headroom. Which we can avoid excessive reducer > preemption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6541) Exclude pending reducer memory when calculating available mapper slots from headroom to avoid deadlock
[ https://issues.apache.org/jira/browse/MAPREDUCE-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994678#comment-14994678 ] Wangda Tan commented on MAPREDUCE-6541: --- [~Naganarasimha], that's also different: MAPREDUCE-6514 is to fix bug in existing reducer preemption logic (remove reducer request locally but not notify RM). This is an enhancement of how to calculate available mapper slots. > Exclude pending reducer memory when calculating available mapper slots from > headroom to avoid deadlock > --- > > Key: MAPREDUCE-6541 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6541 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wangda Tan > > We saw a MR deadlock recently: > - When NM restarted by framework without enable recovery, containers running > on these nodes will be identified as "ABORTED", and MR AM will try to > reschedule "ABORTED" mapper containers. > - Since such lost mappers are "ABORTED" container, MR AM gives normal mapper > priority (priority=20) to such mapper requests. If there's any pending > reducer (priority=10) at the same time, mapper requests need to wait for > reducer requests satisfied. > - In our test, one mapper needs 700+ MB, reducer needs 1000+ MB, and RM > available resource = mapper-request = (700+ MB), only one job was running in > the system so scheduler cannot allocate more reducer containers AND MR-AM > thinks there're enough headroom for mapper so reducer containers will not be > preempted. > MAPREDUCE-6302 can solve most of the problems, but in the other hand, I think > we may need to exclude pending reducers resource when calculating > #available-mapper-slots from headroom. Which we can avoid excessive reducer > preemption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6541) Exclude pending reducer memory when calculating available mapper slots from headroom to avoid deadlock
[
https://issues.apache.org/jira/browse/MAPREDUCE-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994732#comment-14994732
]
Varun Saxena commented on MAPREDUCE-6541:
-
Assigned it to myself as I am working on 2 similar JIRAs'.
Wangda kindly reassign if you will be working on it.
Haven't thought through all the cases but on the face of it, this makes sense.
Because reducers will have higher priority so whether mapper has enough
headroom needs to take into account that pending reducers will be assigned
resources before it. Which will be done if we exclude pending reducers
resources. If I am not wrong you are talking about below condition in
RMContainerAllocator#preemptReducesIfNeeded
{code}
// The pending mappers haven't been waiting for too long. Let us see if
// the headroom can fit a mapper.
Resource availableResourceForMap = getAvailableResources();
if
(ResourceCalculatorUtils.computeAvailableContainers(availableResourceForMap,
mapResourceRequest, getSchedulerResourceTypes()) > 0) {
// the available headroom is enough to run a mapper
return;
}
{code}
Coming to MAPREDUCE-6514, in addition to updating ask, it will also consider
whether to ramp up reduces or not if maps are hanging in scheduleReduces().
Will update JIRA description accordingly.
> Exclude pending reducer memory when calculating available mapper slots from
> headroom to avoid deadlock
> ---
>
> Key: MAPREDUCE-6541
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6541
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Varun Saxena
>
> We saw a MR deadlock recently:
> - When NM restarted by framework without enable recovery, containers running
> on these nodes will be identified as "ABORTED", and MR AM will try to
> reschedule "ABORTED" mapper containers.
> - Since such lost mappers are "ABORTED" container, MR AM gives normal mapper
> priority (priority=20) to such mapper requests. If there's any pending
> reducer (priority=10) at the same time, mapper requests need to wait for
> reducer requests satisfied.
> - In our test, one mapper needs 700+ MB, reducer needs 1000+ MB, and RM
> available resource = mapper-request = (700+ MB), only one job was running in
> the system so scheduler cannot allocate more reducer containers AND MR-AM
> thinks there're enough headroom for mapper so reducer containers will not be
> preempted.
> MAPREDUCE-6302 can solve most of the problems, but in the other hand, I think
> we may need to exclude pending reducers resource when calculating
> #available-mapper-slots from headroom. Which we can avoid excessive reducer
> preemption.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6541) Exclude pending reducer memory when calculating available mapper slots from headroom to avoid deadlock
[ https://issues.apache.org/jira/browse/MAPREDUCE-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994674#comment-14994674 ] Naganarasimha G R commented on MAPREDUCE-6541: -- Hi [~wangda], Then will it be similar to MAPREDUCE-6514 ? > Exclude pending reducer memory when calculating available mapper slots from > headroom to avoid deadlock > --- > > Key: MAPREDUCE-6541 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6541 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wangda Tan > > We saw a MR deadlock recently: > - When NM restarted by framework without enable recovery, containers running > on these nodes will be identified as "ABORTED", and MR AM will try to > reschedule "ABORTED" mapper containers. > - Since such lost mappers are "ABORTED" container, MR AM gives normal mapper > priority (priority=20) to such mapper requests. If there's any pending > reducer (priority=10) at the same time, mapper requests need to wait for > reducer requests satisfied. > - In our test, one mapper needs 700+ MB, reducer needs 1000+ MB, and RM > available resource = mapper-request = (700+ MB), only one job was running in > the system so scheduler cannot allocate more reducer containers AND MR-AM > thinks there're enough headroom for mapper so reducer containers will not be > preempted. > MAPREDUCE-6302 can solve most of the problems, but in the other hand, I think > we may need to exclude pending reducers resource when calculating > #available-mapper-slots from headroom. Which we can avoid excessive reducer > preemption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6541) Exclude pending reducer memory when calculating available mapper slots from headroom to avoid deadlock
[ https://issues.apache.org/jira/browse/MAPREDUCE-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994582#comment-14994582 ] Wangda Tan commented on MAPREDUCE-6541: --- [~Naganarasimha], think for pointing me MAPREDUCE-6513, I think they're similar issues, but the proposal maybe a little different: - MAPREDUCE-6513 is trying to make retried mappers has higher priority. - This JIRA is trying to exclude pending reducer memory so reducer preemption will involve earlier. I'm linking the two JIRAs. > Exclude pending reducer memory when calculating available mapper slots from > headroom to avoid deadlock > --- > > Key: MAPREDUCE-6541 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6541 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wangda Tan > > We saw a MR deadlock recently: > - When NM restarted by framework without enable recovery, containers running > on these nodes will be identified as "ABORTED", and MR AM will try to > reschedule "ABORTED" mapper containers. > - Since such lost mappers are "ABORTED" container, MR AM gives normal mapper > priority (priority=20) to such mapper requests. If there's any pending > reducer (priority=10) at the same time, mapper requests need to wait for > reducer requests satisfied. > - In our test, one mapper needs 700+ MB, reducer needs 1000+ MB, and RM > available resource = mapper-request = (700+ MB), only one job was running in > the system so scheduler cannot allocate more reducer containers AND MR-AM > thinks there're enough headroom for mapper so reducer containers will not be > preempted. > MAPREDUCE-6302 can solve most of the problems, but in the other hand, I think > we may need to exclude pending reducers resource when calculating > #available-mapper-slots from headroom. Which we can avoid excessive reducer > preemption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6541) Exclude pending reducer memory when calculating available mapper slots from headroom to avoid deadlock
[ https://issues.apache.org/jira/browse/MAPREDUCE-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994785#comment-14994785 ] Wangda Tan commented on MAPREDUCE-6541: --- Thanks for sharing your thoughts [~varun_saxena]. And thanks for taking this, please go ahead. Reconsidered these issues, I think 3 fixes are all required: - MAPREDUCE-6513: failed/killed mappers should have higher priority - MAPREDUCE-6514: reducer preemption should also cleanup resource requests in RM side. - And also this one. I think previous two are more important, this is just an optimization. +[~vinodkv]. > Exclude pending reducer memory when calculating available mapper slots from > headroom to avoid deadlock > --- > > Key: MAPREDUCE-6541 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6541 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Varun Saxena > > We saw a MR deadlock recently: > - When NM restarted by framework without enable recovery, containers running > on these nodes will be identified as "ABORTED", and MR AM will try to > reschedule "ABORTED" mapper containers. > - Since such lost mappers are "ABORTED" container, MR AM gives normal mapper > priority (priority=20) to such mapper requests. If there's any pending > reducer (priority=10) at the same time, mapper requests need to wait for > reducer requests satisfied. > - In our test, one mapper needs 700+ MB, reducer needs 1000+ MB, and RM > available resource = mapper-request = (700+ MB), only one job was running in > the system so scheduler cannot allocate more reducer containers AND MR-AM > thinks there're enough headroom for mapper so reducer containers will not be > preempted. > MAPREDUCE-6302 can solve most of the problems, but in the other hand, I think > we may need to exclude pending reducers resource when calculating > #available-mapper-slots from headroom. Which we can avoid excessive reducer > preemption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6541) Exclude pending reducer memory when calculating available mapper slots from headroom to avoid deadlock
[ https://issues.apache.org/jira/browse/MAPREDUCE-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994507#comment-14994507 ] Wangda Tan commented on MAPREDUCE-6541: --- +[~jlowe], [~kasha]. > Exclude pending reducer memory when calculating available mapper slots from > headroom to avoid deadlock > --- > > Key: MAPREDUCE-6541 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6541 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wangda Tan > > We saw a MR deadlock recently: > - When NM restarted by framework without enable recovery, containers running > on these nodes will be identified as "ABORTED", and MR AM will try to > reschedule "ABORTED" mapper containers. > - Since such lost mappers are "ABORTED" container, MR AM gives normal mapper > priority (priority=20) to such mapper requests. If there's any pending > reducer (priority=10) at the same time, mapper requests need to wait for > reducer requests satisfied. > - In our test, one mapper needs 700+ MB, reducer needs 1000+ MB, and RM > available resource = mapper-request = (700+ MB), only one job was running in > the system so scheduler cannot allocate more reducer containers AND MR-AM > thinks there're enough headroom for mapper so reducer containers will not be > preempted. > MAPREDUCE-6302 can solve most of the problems, but in the other hand, I think > we may need to exclude pending reducers resource when calculating > #available-mapper-slots from headroom. Which we can avoid excessive reducer > preemption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6541) Exclude pending reducer memory when calculating available mapper slots from headroom to avoid deadlock
[ https://issues.apache.org/jira/browse/MAPREDUCE-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994565#comment-14994565 ] Naganarasimha G R commented on MAPREDUCE-6541: -- Hi [~wangda], Case seems to be similar with MAPREDUCE-6513 ? > Exclude pending reducer memory when calculating available mapper slots from > headroom to avoid deadlock > --- > > Key: MAPREDUCE-6541 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6541 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Wangda Tan > > We saw a MR deadlock recently: > - When NM restarted by framework without enable recovery, containers running > on these nodes will be identified as "ABORTED", and MR AM will try to > reschedule "ABORTED" mapper containers. > - Since such lost mappers are "ABORTED" container, MR AM gives normal mapper > priority (priority=20) to such mapper requests. If there's any pending > reducer (priority=10) at the same time, mapper requests need to wait for > reducer requests satisfied. > - In our test, one mapper needs 700+ MB, reducer needs 1000+ MB, and RM > available resource = mapper-request = (700+ MB), only one job was running in > the system so scheduler cannot allocate more reducer containers AND MR-AM > thinks there're enough headroom for mapper so reducer containers will not be > preempted. > MAPREDUCE-6302 can solve most of the problems, but in the other hand, I think > we may need to exclude pending reducers resource when calculating > #available-mapper-slots from headroom. Which we can avoid excessive reducer > preemption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
