[jira] [Created] (YARN-4891) Failed/Running/Killed tasks view not working on running jobs (webui)
Johan Gustavsson created YARN-4891: -- Summary: Failed/Running/Killed tasks view not working on running jobs (webui) Key: YARN-4891 URL: https://issues.apache.org/jira/browse/YARN-4891 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, webapp Reporter: Johan Gustavsson Priority: Minor When a job is running clicking any of the links within the box marked below: !https://i.gyazo.com/1ee42be833324712a7dd2a88f2458ee7.png! Results in an empty view like the following: !https://i.gyazo.com/46a91b32b1a6d6e1b886b5c5305c6d7a.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4583) Resource manager should purge generic history data when using FileSystemApplicationHistoryStore
Johan Gustavsson created YARN-4583: -- Summary: Resource manager should purge generic history data when using FileSystemApplicationHistoryStore Key: YARN-4583 URL: https://issues.apache.org/jira/browse/YARN-4583 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.7.1, 2.7.0, 2.4.1 Reporter: Johan Gustavsson Assignee: Johan Gustavsson Init's current state when enabling `yarn.timeline-service.generic-application-history.enabled` and setting `yarn.timeline-service.generic-application-history.store-class` to `org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore` files keep building up in dir until it reaches max files for dir. There should be a way to set the RM to purge these files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4583) Resource manager should purge generic history data when using FileSystemApplicationHistoryStore
[ https://issues.apache.org/jira/browse/YARN-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated YARN-4583: --- Attachment: YARN-4583.patch > Resource manager should purge generic history data when using > FileSystemApplicationHistoryStore > --- > > Key: YARN-4583 > URL: https://issues.apache.org/jira/browse/YARN-4583 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.4.1, 2.7.0, 2.7.1 >Reporter: Johan Gustavsson >Assignee: Johan Gustavsson > Attachments: YARN-4583.patch > > > Init's current state when enabling > `yarn.timeline-service.generic-application-history.enabled` and setting > `yarn.timeline-service.generic-application-history.store-class` to > `org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore` > files keep building up in dir until it reaches max files for dir. There > should be a way to set the RM to purge these files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4583) Resource manager should purge generic history data when using FileSystemApplicationHistoryStore
[ https://issues.apache.org/jira/browse/YARN-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093563#comment-15093563 ] Johan Gustavsson commented on YARN-4583: Added a path pending review > Resource manager should purge generic history data when using > FileSystemApplicationHistoryStore > --- > > Key: YARN-4583 > URL: https://issues.apache.org/jira/browse/YARN-4583 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.4.1, 2.7.0, 2.7.1 >Reporter: Johan Gustavsson > Attachments: YARN-4583.patch > > > In it's current state when enabling > `yarn.timeline-service.generic-application-history.enabled` and setting > `yarn.timeline-service.generic-application-history.store-class` to > `org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore` > files keep building up in dir until it reaches max files for dir. There > should be a way to set the RM to purge these files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4583) Resource manager should purge generic history data when using FileSystemApplicationHistoryStore
[ https://issues.apache.org/jira/browse/YARN-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated YARN-4583: --- Assignee: (was: Johan Gustavsson) > Resource manager should purge generic history data when using > FileSystemApplicationHistoryStore > --- > > Key: YARN-4583 > URL: https://issues.apache.org/jira/browse/YARN-4583 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.4.1, 2.7.0, 2.7.1 >Reporter: Johan Gustavsson > Attachments: YARN-4583.patch > > > Init's current state when enabling > `yarn.timeline-service.generic-application-history.enabled` and setting > `yarn.timeline-service.generic-application-history.store-class` to > `org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore` > files keep building up in dir until it reaches max files for dir. There > should be a way to set the RM to purge these files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4583) Resource manager should purge generic history data when using FileSystemApplicationHistoryStore
[ https://issues.apache.org/jira/browse/YARN-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated YARN-4583: --- Description: In it's current state when enabling `yarn.timeline-service.generic-application-history.enabled` and setting `yarn.timeline-service.generic-application-history.store-class` to `org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore` files keep building up in dir until it reaches max files for dir. There should be a way to set the RM to purge these files. (was: Init's current state when enabling `yarn.timeline-service.generic-application-history.enabled` and setting `yarn.timeline-service.generic-application-history.store-class` to `org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore` files keep building up in dir until it reaches max files for dir. There should be a way to set the RM to purge these files.) > Resource manager should purge generic history data when using > FileSystemApplicationHistoryStore > --- > > Key: YARN-4583 > URL: https://issues.apache.org/jira/browse/YARN-4583 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.4.1, 2.7.0, 2.7.1 >Reporter: Johan Gustavsson > Attachments: YARN-4583.patch > > > In it's current state when enabling > `yarn.timeline-service.generic-application-history.enabled` and setting > `yarn.timeline-service.generic-application-history.store-class` to > `org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore` > files keep building up in dir until it reaches max files for dir. There > should be a way to set the RM to purge these files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4583) Resource manager should purge generic history data when using FileSystemApplicationHistoryStore
[ https://issues.apache.org/jira/browse/YARN-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated YARN-4583: --- Attachment: YARN-4583.001.patch Fixed formatting according to QA output > Resource manager should purge generic history data when using > FileSystemApplicationHistoryStore > --- > > Key: YARN-4583 > URL: https://issues.apache.org/jira/browse/YARN-4583 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.4.1, 2.7.0, 2.7.1 >Reporter: Johan Gustavsson > Attachments: YARN-4583.001.patch, YARN-4583.patch > > > In it's current state when enabling > `yarn.timeline-service.generic-application-history.enabled` and setting > `yarn.timeline-service.generic-application-history.store-class` to > `org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore` > files keep building up in dir until it reaches max files for dir. There > should be a way to set the RM to purge these files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4583) Resource manager should purge generic history data when using FileSystemApplicationHistoryStore
[ https://issues.apache.org/jira/browse/YARN-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095342#comment-15095342 ] Johan Gustavsson commented on YARN-4583: If FileSystemWriter is deprecated, you can go ahead and void this. I originally wrote this patch for 2.4.1 then noticed there were no similar function in trunk so I ported it and submitted it. I am planning on using ATS ones I upgrade to 2.7.* but haven't had time to look into the setup yet. > Resource manager should purge generic history data when using > FileSystemApplicationHistoryStore > --- > > Key: YARN-4583 > URL: https://issues.apache.org/jira/browse/YARN-4583 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.4.1, 2.7.0, 2.7.1 >Reporter: Johan Gustavsson > Attachments: YARN-4583.001.patch, YARN-4583.patch > > > In it's current state when enabling > `yarn.timeline-service.generic-application-history.enabled` and setting > `yarn.timeline-service.generic-application-history.store-class` to > `org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore` > files keep building up in dir until it reaches max files for dir. There > should be a way to set the RM to purge these files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14910113#comment-14910113 ] Johan Gustavsson commented on YARN-4066: Thanks for re-running this. Looking at the tests in place I see no need to introduce new tests for this patch, also this patch doesn't break the tests in place. > Large number of queues choke fair scheduler > --- > > Key: YARN-4066 > URL: https://issues.apache.org/jira/browse/YARN-4066 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Johan Gustavsson >Assignee: Johan Gustavsson > Attachments: YARN-4066-2.patch, YARN-4066-3.patch, yarn-4066-1.patch > > > Due to synchronization and all the loops performed during queue creation, > setting a large amount of queues (12000+) will completely choke the > scheduler. To deal with this some optimization to > "QueueManager.updateAllocationConfiguration(AllocationConfiguration > queueConf)" should be done to reduce the amount of unnesecary loops. The > attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated YARN-4066: --- Attachment: YARN-4066-3.patch The formatting issues have been taken care of, sorry about that. > Large number of queues choke fair scheduler > --- > > Key: YARN-4066 > URL: https://issues.apache.org/jira/browse/YARN-4066 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Johan Gustavsson >Assignee: Johan Gustavsson > Attachments: YARN-4066-2.patch, YARN-4066-3.patch, yarn-4066-1.patch > > > Due to synchronization and all the loops performed during queue creation, > setting a large amount of queues (12000+) will completely choke the > scheduler. To deal with this some optimization to > "QueueManager.updateAllocationConfiguration(AllocationConfiguration > queueConf)" should be done to reduce the amount of unnesecary loops. The > attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated YARN-4066: --- Attachment: YARN-4066-2.patch > Large number of queues choke fair scheduler > --- > > Key: YARN-4066 > URL: https://issues.apache.org/jira/browse/YARN-4066 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Johan Gustavsson >Assignee: Johan Gustavsson > Attachments: YARN-4066-2.patch, yarn-4066-1.patch > > > Due to synchronization and all the loops performed during queue creation, > setting a large amount of queues (12000+) will completely choke the > scheduler. To deal with this some optimization to > "QueueManager.updateAllocationConfiguration(AllocationConfiguration > queueConf)" should be done to reduce the amount of unnesecary loops. The > attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905783#comment-14905783 ] Johan Gustavsson commented on YARN-4066: Thanks for the feedback Karthik, I'll try to have a fixed patch ready by next week. > Large number of queues choke fair scheduler > --- > > Key: YARN-4066 > URL: https://issues.apache.org/jira/browse/YARN-4066 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Johan Gustavsson >Assignee: Johan Gustavsson > Attachments: yarn-4066-1.patch > > > Due to synchronization and all the loops performed during queue creation, > setting a large amount of queues (12000+) will completely choke the > scheduler. To deal with this some optimization to > "QueueManager.updateAllocationConfiguration(AllocationConfiguration > queueConf)" should be done to reduce the amount of unnesecary loops. The > attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746323#comment-14746323 ] Johan Gustavsson commented on YARN-4066: Please let me know if there is anything I can do to continue the process of merging this > Large number of queues choke fair scheduler > --- > > Key: YARN-4066 > URL: https://issues.apache.org/jira/browse/YARN-4066 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.1 >Reporter: Johan Gustavsson > Attachments: yarn-4066-1.patch > > > Due to synchronization and all the loops performed during queue creation, > setting a large amount of queues (12000+) will completely choke the > scheduler. To deal with this some optimization to > "QueueManager.updateAllocationConfiguration(AllocationConfiguration > queueConf)" should be done to reduce the amount of unnesecary loops. The > attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706035#comment-14706035 ] Johan Gustavsson commented on YARN-4066: As I done seem to be able to edit the above comment and the tree ended up weird I'll repast it below root: 1 q1: veryhigh high default low verylow Large number of queues choke fair scheduler --- Key: YARN-4066 URL: https://issues.apache.org/jira/browse/YARN-4066 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 2.7.1 Reporter: Johan Gustavsson Attachments: yarn-4066-1.patch Due to synchronization and all the loops performed during queue creation, setting a large amount of queues (12000+) will completely choke the scheduler. To deal with this some optimization to QueueManager.updateAllocationConfiguration(AllocationConfiguration queueConf) should be done to reduce the amount of unnesecary loops. The attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706031#comment-14706031 ] Johan Gustavsson commented on YARN-4066: Basically its a tree as follows ranging from 1 to 16000. For each user group there is one general queue and one with weight divided sub queues. root - 1 - q1 - veryhigh - high - default - low - verylow Large number of queues choke fair scheduler --- Key: YARN-4066 URL: https://issues.apache.org/jira/browse/YARN-4066 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 2.7.1 Reporter: Johan Gustavsson Attachments: yarn-4066-1.patch Due to synchronization and all the loops performed during queue creation, setting a large amount of queues (12000+) will completely choke the scheduler. To deal with this some optimization to QueueManager.updateAllocationConfiguration(AllocationConfiguration queueConf) should be done to reduce the amount of unnesecary loops. The attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4066) Large number of queues choke fair scheduler
Johan Gustavsson created YARN-4066: -- Summary: Large number of queues choke fair scheduler Key: YARN-4066 URL: https://issues.apache.org/jira/browse/YARN-4066 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 2.7.1 Reporter: Johan Gustavsson Due to synchronization and all the loops performed during queue creation, setting a large amount of queues (12000+) will completely choke the scheduler. To deal with this some optimization to QueueManager.updateAllocationConfiguration(AllocationConfiguration queueConf) should be done to reduce the amount of unnesecary loops. The attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated YARN-4066: --- Attachment: yarn-4066-1.patch Large number of queues choke fair scheduler --- Key: YARN-4066 URL: https://issues.apache.org/jira/browse/YARN-4066 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 2.7.1 Reporter: Johan Gustavsson Attachments: yarn-4066-1.patch Due to synchronization and all the loops performed during queue creation, setting a large amount of queues (12000+) will completely choke the scheduler. To deal with this some optimization to QueueManager.updateAllocationConfiguration(AllocationConfiguration queueConf) should be done to reduce the amount of unnesecary loops. The attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4066) Large number of queues choke fair scheduler
[ https://issues.apache.org/jira/browse/YARN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704334#comment-14704334 ] Johan Gustavsson commented on YARN-4066: Sorry, I re-uploaded it and I think it should be visible now Large number of queues choke fair scheduler --- Key: YARN-4066 URL: https://issues.apache.org/jira/browse/YARN-4066 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 2.7.1 Reporter: Johan Gustavsson Attachments: yarn-4066-1.patch Due to synchronization and all the loops performed during queue creation, setting a large amount of queues (12000+) will completely choke the scheduler. To deal with this some optimization to QueueManager.updateAllocationConfiguration(AllocationConfiguration queueConf) should be done to reduce the amount of unnesecary loops. The attached patch have been tested to work with atleast 96000 queues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
