[jira] [Updated] (YARN-4333) Fair scheduler should support preemption within queue
[ https://issues.apache.org/jira/browse/YARN-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4333: --- Issue Type: Sub-task (was: Improvement) Parent: YARN-4752 > Fair scheduler should support preemption within queue > - > > Key: YARN-4333 > URL: https://issues.apache.org/jira/browse/YARN-4333 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Tao Jie >Assignee: Tao Jie > Attachments: YARN-4333.001.patch, YARN-4333.002.patch, > YARN-4333.003.patch > > > Now each app in fair scheduler is allocated its fairshare, however fairshare > resource is not ensured even if fairSharePreemption is enabled. > Consider: > 1, When the cluster is idle, we submit app1 to queueA,which takes maxResource > of queueA. > 2, Then the cluster becomes busy, but app1 does not release any resource, > queueA resource usage is over its fairshare > 3, Then we submit app2(maybe with higher priority) to queueA. Now app2 has > its own fairshare, but could not obtain any resource, since queueA is still > over its fairshare and resource will not assign to queueA anymore. Also, > preemption is not triggered in this case. > So we should allow preemption within queue, when app is starved for fairshare. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4333) Fair scheduler should support preemption within queue
[ https://issues.apache.org/jira/browse/YARN-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated YARN-4333: -- Attachment: YARN-4333.003.patch > Fair scheduler should support preemption within queue > - > > Key: YARN-4333 > URL: https://issues.apache.org/jira/browse/YARN-4333 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Tao Jie >Assignee: Tao Jie > Attachments: YARN-4333.001.patch, YARN-4333.002.patch, > YARN-4333.003.patch > > > Now each app in fair scheduler is allocated its fairshare, however fairshare > resource is not ensured even if fairSharePreemption is enabled. > Consider: > 1, When the cluster is idle, we submit app1 to queueA,which takes maxResource > of queueA. > 2, Then the cluster becomes busy, but app1 does not release any resource, > queueA resource usage is over its fairshare > 3, Then we submit app2(maybe with higher priority) to queueA. Now app2 has > its own fairshare, but could not obtain any resource, since queueA is still > over its fairshare and resource will not assign to queueA anymore. Also, > preemption is not triggered in this case. > So we should allow preemption within queue, when app is starved for fairshare. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4333) Fair scheduler should support preemption within queue
[ https://issues.apache.org/jira/browse/YARN-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated YARN-4333: -- Attachment: YARN-4333.002.patch > Fair scheduler should support preemption within queue > - > > Key: YARN-4333 > URL: https://issues.apache.org/jira/browse/YARN-4333 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Tao Jie >Assignee: Tao Jie > Attachments: YARN-4333.001.patch, YARN-4333.002.patch > > > Now each app in fair scheduler is allocated its fairshare, however fairshare > resource is not ensured even if fairSharePreemption is enabled. > Consider: > 1, When the cluster is idle, we submit app1 to queueA,which takes maxResource > of queueA. > 2, Then the cluster becomes busy, but app1 does not release any resource, > queueA resource usage is over its fairshare > 3, Then we submit app2(maybe with higher priority) to queueA. Now app2 has > its own fairshare, but could not obtain any resource, since queueA is still > over its fairshare and resource will not assign to queueA anymore. Also, > preemption is not triggered in this case. > So we should allow preemption within queue, when app is starved for fairshare. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4333) Fair scheduler should support preemption within queue
[ https://issues.apache.org/jira/browse/YARN-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4333: - Assignee: Tao Jie > Fair scheduler should support preemption within queue > - > > Key: YARN-4333 > URL: https://issues.apache.org/jira/browse/YARN-4333 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Tao Jie >Assignee: Tao Jie > Attachments: YARN-4333.001.patch > > > Now each app in fair scheduler is allocated its fairshare, however fairshare > resource is not ensured even if fairSharePreemption is enabled. > Consider: > 1, When the cluster is idle, we submit app1 to queueA,which takes maxResource > of queueA. > 2, Then the cluster becomes busy, but app1 does not release any resource, > queueA resource usage is over its fairshare > 3, Then we submit app2(maybe with higher priority) to queueA. Now app2 has > its own fairshare, but could not obtain any resource, since queueA is still > over its fairshare and resource will not assign to queueA anymore. Also, > preemption is not triggered in this case. > So we should allow preemption within queue, when app is starved for fairshare. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4333) Fair scheduler should support preemption within queue
[ https://issues.apache.org/jira/browse/YARN-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated YARN-4333: -- Attachment: YARN-4333.001.patch > Fair scheduler should support preemption within queue > - > > Key: YARN-4333 > URL: https://issues.apache.org/jira/browse/YARN-4333 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Tao Jie > Attachments: YARN-4333.001.patch > > > Now each app in fair scheduler is allocated its fairshare, however fairshare > resource is not ensured even if fairSharePreemption is enabled. > Consider: > 1, When the cluster is idle, we submit app1 to queueA,which takes maxResource > of queueA. > 2, Then the cluster becomes busy, but app1 does not release any resource, > queueA resource usage is over its fairshare > 3, Then we submit app2(maybe with higher priority) to queueA. Now app2 has > its own fairshare, but could not obtain any resource, since queueA is still > over its fairshare and resource will not assign to queueA anymore. Also, > preemption is not triggered in this case. > So we should allow preemption within queue, when app is starved for fairshare. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4333) Fair scheduler should support preemption within queue
[ https://issues.apache.org/jira/browse/YARN-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feng Yuan updated YARN-4333: Issue Type: Improvement (was: Bug) > Fair scheduler should support preemption within queue > - > > Key: YARN-4333 > URL: https://issues.apache.org/jira/browse/YARN-4333 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Tao Jie > > Now each app in fair scheduler is allocated its fairshare, however fairshare > resource is not ensured even if fairSharePreemption is enabled. > Consider: > 1, When the cluster is idle, we submit app1 to queueA,which takes maxResource > of queueA. > 2, Then the cluster becomes busy, but app1 does not release any resource, > queueA resource usage is over its fairshare > 3, Then we submit app2(maybe with higher priority) to queueA. Now app2 has > its own fairshare, but could not obtain any resource, since queueA is still > over its fairshare and resource will not assign to queueA anymore. Also, > preemption is not triggered in this case. > So we should allow preemption within queue, when app is starved for fairshare. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4333) Fair scheduler should support preemption within queue
[ https://issues.apache.org/jira/browse/YARN-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated YARN-4333: -- Affects Version/s: 2.6.0 > Fair scheduler should support preemption within queue > - > > Key: YARN-4333 > URL: https://issues.apache.org/jira/browse/YARN-4333 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Tao Jie > > Now each app in fair scheduler is allocated its fairshare, however fairshare > resource is not ensured even if fairSharePreemption is enabled. > Consider: > 1, When the cluster is idle, we submit app1 to queueA,which takes maxResource > of queueA. > 2, Then the cluster becomes busy, but app1 does not release any resource, > queueA resource usage is over its fairshare > 3, Then we submit app2(maybe with higher priority) to queueA. Now app2 has > its own fairshare, but could not obtain any resource, since queueA is still > over its fairshare and resource will not assign to queueA anymore. Also, > preemption is not triggered in this case. > So we should allow preemption within queue, when app is starved for fairshare. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
