[jira] [Updated] (YARN-10257) FS-CS converter: skip increment properties for mem/vcores and fix DRF check

2024-02-11 Thread Shilun Fan (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shilun Fan updated YARN-10257:
--
  Component/s: fs-cs
   fairscheduler
 Target Version/s: 3.3.0, 3.4.0
Affects Version/s: 3.3.0
   3.4.0

> FS-CS converter: skip increment properties for mem/vcores and fix DRF check
> ---
>
> Key: YARN-10257
> URL: https://issues.apache.org/jira/browse/YARN-10257
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler, fs-cs
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0, 3.4.0
>
> Attachments: YARN-10257-001.patch, YARN-10257-002.patch
>
>
> Two issues have been discovered during fs2cs testing:
> 1. The conversion of allocation increment properties are not needed:
> {{yarn.scheduler.increment-allocation-mb}}
> {{yarn.scheduler.increment-allocation-vcores}}
> {{yarn.resource-types.memory-mb.increment-allocation}}
> {{yarn.resource-types.vcores.increment-allocation}}
> 2. The following piece of code is incorrect - the default scheduling policy 
> can be different from DRF, which is a problem if DRF is used everywhere else:
> {code}
>   private boolean isDrfUsed(FairScheduler fs) {
> FSQueue rootQueue = fs.getQueueManager().getRootQueue();
> AllocationConfiguration allocConf = fs.getAllocationConfiguration();
> String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();
> if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
>   return true;
> } else {
>   return isDrfUsedOnQueueLevel(rootQueue);
> }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10257) FS-CS converter: skip increment properties for mem/vcores and fix DRF check

2021-05-16 Thread Wei-Chiu Chuang (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated YARN-10257:
---
Fix Version/s: (was: 3.3.1)
   3.3.0

> FS-CS converter: skip increment properties for mem/vcores and fix DRF check
> ---
>
> Key: YARN-10257
> URL: https://issues.apache.org/jira/browse/YARN-10257
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0, 3.4.0
>
> Attachments: YARN-10257-001.patch, YARN-10257-002.patch
>
>
> Two issues have been discovered during fs2cs testing:
> 1. The conversion of allocation increment properties are not needed:
> {{yarn.scheduler.increment-allocation-mb}}
> {{yarn.scheduler.increment-allocation-vcores}}
> {{yarn.resource-types.memory-mb.increment-allocation}}
> {{yarn.resource-types.vcores.increment-allocation}}
> 2. The following piece of code is incorrect - the default scheduling policy 
> can be different from DRF, which is a problem if DRF is used everywhere else:
> {code}
>   private boolean isDrfUsed(FairScheduler fs) {
> FSQueue rootQueue = fs.getQueueManager().getRootQueue();
> AllocationConfiguration allocConf = fs.getAllocationConfiguration();
> String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();
> if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
>   return true;
> } else {
>   return isDrfUsedOnQueueLevel(rootQueue);
> }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10257) FS-CS converter: skip increment properties for mem/vcores and fix DRF check

2020-05-05 Thread Szilard Nemeth (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated YARN-10257:
--
Fix Version/s: 3.3.1

> FS-CS converter: skip increment properties for mem/vcores and fix DRF check
> ---
>
> Key: YARN-10257
> URL: https://issues.apache.org/jira/browse/YARN-10257
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10257-001.patch, YARN-10257-002.patch
>
>
> Two issues have been discovered during fs2cs testing:
> 1. The conversion of allocation increment properties are not needed:
> {{yarn.scheduler.increment-allocation-mb}}
> {{yarn.scheduler.increment-allocation-vcores}}
> {{yarn.resource-types.memory-mb.increment-allocation}}
> {{yarn.resource-types.vcores.increment-allocation}}
> 2. The following piece of code is incorrect - the default scheduling policy 
> can be different from DRF, which is a problem if DRF is used everywhere else:
> {code}
>   private boolean isDrfUsed(FairScheduler fs) {
> FSQueue rootQueue = fs.getQueueManager().getRootQueue();
> AllocationConfiguration allocConf = fs.getAllocationConfiguration();
> String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();
> if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
>   return true;
> } else {
>   return isDrfUsedOnQueueLevel(rootQueue);
> }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10257) FS-CS converter: skip increment properties for mem/vcores and fix DRF check

2020-05-05 Thread Szilard Nemeth (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated YARN-10257:
--
Fix Version/s: 3.4.0

> FS-CS converter: skip increment properties for mem/vcores and fix DRF check
> ---
>
> Key: YARN-10257
> URL: https://issues.apache.org/jira/browse/YARN-10257
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10257-001.patch, YARN-10257-002.patch
>
>
> Two issues have been discovered during fs2cs testing:
> 1. The conversion of allocation increment properties are not needed:
> {{yarn.scheduler.increment-allocation-mb}}
> {{yarn.scheduler.increment-allocation-vcores}}
> {{yarn.resource-types.memory-mb.increment-allocation}}
> {{yarn.resource-types.vcores.increment-allocation}}
> 2. The following piece of code is incorrect - the default scheduling policy 
> can be different from DRF, which is a problem if DRF is used everywhere else:
> {code}
>   private boolean isDrfUsed(FairScheduler fs) {
> FSQueue rootQueue = fs.getQueueManager().getRootQueue();
> AllocationConfiguration allocConf = fs.getAllocationConfiguration();
> String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();
> if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
>   return true;
> } else {
>   return isDrfUsedOnQueueLevel(rootQueue);
> }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10257) FS-CS converter: skip increment properties for mem/vcores and fix DRF check

2020-05-05 Thread Peter Bacsko (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated YARN-10257:

Attachment: YARN-10257-002.patch

> FS-CS converter: skip increment properties for mem/vcores and fix DRF check
> ---
>
> Key: YARN-10257
> URL: https://issues.apache.org/jira/browse/YARN-10257
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10257-001.patch, YARN-10257-002.patch
>
>
> Two issues have been discovered during fs2cs testing:
> 1. The conversion of allocation increment properties are not needed:
> {{yarn.scheduler.increment-allocation-mb}}
> {{yarn.scheduler.increment-allocation-vcores}}
> {{yarn.resource-types.memory-mb.increment-allocation}}
> {{yarn.resource-types.vcores.increment-allocation}}
> 2. The following piece of code is incorrect - the default scheduling policy 
> can be different from DRF, which is a problem if DRF is used everywhere else:
> {code}
>   private boolean isDrfUsed(FairScheduler fs) {
> FSQueue rootQueue = fs.getQueueManager().getRootQueue();
> AllocationConfiguration allocConf = fs.getAllocationConfiguration();
> String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();
> if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
>   return true;
> } else {
>   return isDrfUsedOnQueueLevel(rootQueue);
> }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10257) FS-CS converter: skip increment properties for mem/vcores and fix DRF check

2020-05-04 Thread Peter Bacsko (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated YARN-10257:

Attachment: YARN-10257-001.patch

> FS-CS converter: skip increment properties for mem/vcores and fix DRF check
> ---
>
> Key: YARN-10257
> URL: https://issues.apache.org/jira/browse/YARN-10257
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10257-001.patch
>
>
> Two issues have been discovered during fs2cs testing:
> 1. The conversion of allocation increment properties are not needed:
> {{yarn.scheduler.increment-allocation-mb}}
> {{yarn.scheduler.increment-allocation-vcores}}
> {{yarn.resource-types.memory-mb.increment-allocation}}
> {{yarn.resource-types.vcores.increment-allocation}}
> 2. The following piece of code is incorrect - the default scheduling policy 
> can be different from DRF, which is a problem if DRF is used everywhere else:
> {code}
>   private boolean isDrfUsed(FairScheduler fs) {
> FSQueue rootQueue = fs.getQueueManager().getRootQueue();
> AllocationConfiguration allocConf = fs.getAllocationConfiguration();
> String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();
> if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
>   return true;
> } else {
>   return isDrfUsedOnQueueLevel(rootQueue);
> }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10257) FS-CS converter: skip increment properties for mem/vcores and fix DRF check

2020-05-04 Thread Peter Bacsko (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated YARN-10257:

Description: 
Two issues have been discovered during fs2cs testing:

1. The conversion of allocation increment properties are not needed:

{{yarn.scheduler.increment-allocation-mb}}
{{yarn.scheduler.increment-allocation-vcores}}
{{yarn.resource-types.memory-mb.increment-allocation}}
{{yarn.resource-types.vcores.increment-allocation}}

2. The following piece of code is incorrect - the default scheduling policy can 
be different from DRF, which is a problem if DRF is used everywhere else:

{code}
  private boolean isDrfUsed(FairScheduler fs) {
FSQueue rootQueue = fs.getQueueManager().getRootQueue();
AllocationConfiguration allocConf = fs.getAllocationConfiguration();

String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();

if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
  return true;
} else {
  return isDrfUsedOnQueueLevel(rootQueue);
}
  }
{code}

  was:
Two issues have been discovered during fs2cs testing:

1. The value of two properties are not checked:

{{yarn.scheduler.increment-allocation-mb}}
{{yarn.scheduler.increment-allocation-vcores}}

Although these two are marked as deprecated, they're still in use and must be 
handled.

2. The following piece of code is incorrect - the default scheduling policy can 
be different from DRF, which is a problem if DRF is used everywhere else:

{code}
  private boolean isDrfUsed(FairScheduler fs) {
FSQueue rootQueue = fs.getQueueManager().getRootQueue();
AllocationConfiguration allocConf = fs.getAllocationConfiguration();

String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();

if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
  return true;
} else {
  return isDrfUsedOnQueueLevel(rootQueue);
}
  }
{code}


> FS-CS converter: skip increment properties for mem/vcores and fix DRF check
> ---
>
> Key: YARN-10257
> URL: https://issues.apache.org/jira/browse/YARN-10257
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> Two issues have been discovered during fs2cs testing:
> 1. The conversion of allocation increment properties are not needed:
> {{yarn.scheduler.increment-allocation-mb}}
> {{yarn.scheduler.increment-allocation-vcores}}
> {{yarn.resource-types.memory-mb.increment-allocation}}
> {{yarn.resource-types.vcores.increment-allocation}}
> 2. The following piece of code is incorrect - the default scheduling policy 
> can be different from DRF, which is a problem if DRF is used everywhere else:
> {code}
>   private boolean isDrfUsed(FairScheduler fs) {
> FSQueue rootQueue = fs.getQueueManager().getRootQueue();
> AllocationConfiguration allocConf = fs.getAllocationConfiguration();
> String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();
> if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
>   return true;
> } else {
>   return isDrfUsedOnQueueLevel(rootQueue);
> }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10257) FS-CS converter: skip increment properties for mem/vcores and fix DRF check

2020-05-04 Thread Peter Bacsko (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated YARN-10257:

Summary: FS-CS converter: skip increment properties for mem/vcores and fix 
DRF check  (was: FS-CS converter: check deprecated increment properties for 
mem/vcores and fix DRF check)

> FS-CS converter: skip increment properties for mem/vcores and fix DRF check
> ---
>
> Key: YARN-10257
> URL: https://issues.apache.org/jira/browse/YARN-10257
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> Two issues have been discovered during fs2cs testing:
> 1. The value of two properties are not checked:
> {{yarn.scheduler.increment-allocation-mb}}
> {{yarn.scheduler.increment-allocation-vcores}}
> Although these two are marked as deprecated, they're still in use and must be 
> handled.
> 2. The following piece of code is incorrect - the default scheduling policy 
> can be different from DRF, which is a problem if DRF is used everywhere else:
> {code}
>   private boolean isDrfUsed(FairScheduler fs) {
> FSQueue rootQueue = fs.getQueueManager().getRootQueue();
> AllocationConfiguration allocConf = fs.getAllocationConfiguration();
> String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();
> if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
>   return true;
> } else {
>   return isDrfUsedOnQueueLevel(rootQueue);
> }
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org