[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF

2017-04-21 Thread Andrew Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated YARN-6193:
--
Fix Version/s: 3.0.0-alpha3

> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6193.000.patch, YARN-6193.001.patch, 
> YARN-6193.002.patch, YARN-6193.003.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container 
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this 
> leads to verifying both resources instead of just the dominant resource. 
> Ideally, the check should be based on policy. 
> Note that current implementation of 
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF

2017-02-17 Thread Karthik Kambatla (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-6193:
---
Attachment: YARN-6193.003.patch

> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch, YARN-6193.001.patch, 
> YARN-6193.002.patch, YARN-6193.003.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container 
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this 
> leads to verifying both resources instead of just the dominant resource. 
> Ideally, the check should be based on policy. 
> Note that current implementation of 
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF

2017-02-16 Thread Karthik Kambatla (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-6193:
---
Attachment: YARN-6193.002.patch

Most test failures correspond to tests that mock FairScheduler. 
TestFairSchedulerPreemption is a timeout issue. Fixed them all in the 002.patch.

> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch, YARN-6193.001.patch, 
> YARN-6193.002.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container 
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this 
> leads to verifying both resources instead of just the dominant resource. 
> Ideally, the check should be based on policy. 
> Note that current implementation of 
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF

2017-02-15 Thread Karthik Kambatla (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-6193:
---
Attachment: YARN-6193.001.patch

> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch, YARN-6193.001.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container 
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this 
> leads to verifying both resources instead of just the dominant resource. 
> Ideally, the check should be based on policy. 
> Note that current implementation of 
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF

2017-02-14 Thread Karthik Kambatla (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-6193:
---
Issue Type: Sub-task  (was: Bug)
Parent: YARN-5990

> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container 
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this 
> leads to verifying both resources instead of just the dominant resource. 
> Ideally, the check should be based on policy. 
> Note that current implementation of 
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF

2017-02-14 Thread Karthik Kambatla (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-6193:
---
Attachment: YARN-6193.000.patch

> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container 
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this 
> leads to verifying both resources instead of just the dominant resource. 
> Ideally, the check should be based on policy. 
> Note that current implementation of 
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF

2017-02-14 Thread Karthik Kambatla (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-6193:
---
Description: 
{{FSAppAttempt#canContainerBePreempted}} verifies preempting a container 
doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this 
leads to verifying both resources instead of just the dominant resource. 
Ideally, the check should be based on policy. 

Note that current implementation of 
{{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken. 

  was:{{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} considers 
all resources instead of just dominant resource. Ideally, it should use the 
DominantResourceCalculator.


> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container 
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this 
> leads to verifying both resources instead of just the dominant resource. 
> Ideally, the check should be based on policy. 
> Note that current implementation of 
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF

2017-02-14 Thread Karthik Kambatla (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-6193:
---
Summary: FairScheduler might not trigger preemption when using DRF  (was: 
FSAppAttempt#canContainerBePreempted considers all resources instead of just 
dominant resource)

> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} considers all 
> resources instead of just dominant resource. Ideally, it should use the 
> DominantResourceCalculator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org