[jira] [Updated] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated YARN-4066: --- Attachment: YARN-4066-3.patch The formatting issues have been taken care of, sorry about that. > Large number of queues choke fair scheduler > --- > > Key: YARN-4066 > URL: https://issues.apache.org/jira/browse/YARN-4066 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Johan Gustavsson >Assignee: Johan Gustavsson > Attachments: YARN-4066-2.patch, YARN-4066-3.patch, yarn-4066-1.patch > > > Due to synchronization and all the loops performed during queue creation, > setting a large amount of queues (12000+) will completely choke the > scheduler. To deal with this some optimization to > "QueueManager.updateAllocationConfiguration(AllocationConfiguration > queueConf)" should be done to reduce the amount of unnesecary loops. The > attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated YARN-4066: --- Attachment: YARN-4066-2.patch > Large number of queues choke fair scheduler > --- > > Key: YARN-4066 > URL: https://issues.apache.org/jira/browse/YARN-4066 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Johan Gustavsson >Assignee: Johan Gustavsson > Attachments: YARN-4066-2.patch, yarn-4066-1.patch > > > Due to synchronization and all the loops performed during queue creation, > setting a large amount of queues (12000+) will completely choke the > scheduler. To deal with this some optimization to > "QueueManager.updateAllocationConfiguration(AllocationConfiguration > queueConf)" should be done to reduce the amount of unnesecary loops. The > attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4066: --- Assignee: Johan Gustavsson > Large number of queues choke fair scheduler > --- > > Key: YARN-4066 > URL: https://issues.apache.org/jira/browse/YARN-4066 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Johan Gustavsson >Assignee: Johan Gustavsson > Attachments: yarn-4066-1.patch > > > Due to synchronization and all the loops performed during queue creation, > setting a large amount of queues (12000+) will completely choke the > scheduler. To deal with this some optimization to > "QueueManager.updateAllocationConfiguration(AllocationConfiguration > queueConf)" should be done to reduce the amount of unnesecary loops. The > attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated YARN-4066: --- Attachment: yarn-4066-1.patch Large number of queues choke fair scheduler --- Key: YARN-4066 URL: https://issues.apache.org/jira/browse/YARN-4066 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 2.7.1 Reporter: Johan Gustavsson Attachments: yarn-4066-1.patch Due to synchronization and all the loops performed during queue creation, setting a large amount of queues (12000+) will completely choke the scheduler. To deal with this some optimization to QueueManager.updateAllocationConfiguration(AllocationConfiguration queueConf) should be done to reduce the amount of unnesecary loops. The attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
