[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-2144: -- Attachment: YARN-2144.patch Looks good overall, did some minor edits myself. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Attachment: YARN-2144.patch Updated findbugs exclude file, suppressed warning of mixed synchronized/unsynchronized accessing rmContext. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Attachment: YARN-2144.patch Removed null checking of never happened blocks. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Attachment: YARN-2144.patch > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Attachment: YARN-2144.patch > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Attachment: YARN-2144.patch Sorry I forgot attaching patch yesterday, attached now. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Description: There should be easy-to-read logs when preemption does occur. RM logs should have following properties: * Logs are retrievable when an application is still running and often flushed. * Can distinguish between AM container preemption and task container preemption with container ID shown. * Should be INFO level log. was: There should be easy-to-read logs when preemption does occur. 1. For debugging purpose, RM should log this. 2. For administrative purpose, RM webpage should have a page to show recent preemption events. RM logs should have following properties: * Logs are retrievable when an application is still running and often flushed. * Can distinguish between AM container preemption and task container preemption with container ID shown. * Should be INFO level log. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Attachment: YARN-2144.patch Rebased patch to latest trunk. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Attachment: YARN-2144.patch Attached new patch fixed test failures. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Attachment: YARN-2144.patch Re-add patch to trigger jenkins building. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Attachment: AM-page-preemption-info.png > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-2144: - Attachment: YARN-2144.patch I’ve attached a patch contains changes to show preemption information on RM app page and RM log, 1) log style: {code} 2014-06-16 10:45:22,247 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Non-AM container preempted, appId=appattempt_1402886643897_0002_01, containerId=container_1402886643897_0002_01_04 {code} {code} 2014-06-16 10:45:22,247 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: AM container preempted, appId=appattempt_1402886643897_0002_01, containerId=container_1402886643897_0002_01_01 {code} 2) Info in app page: See Not Included, 1) Persist preemption info across RM restart/HA. 2) FairScheduler related changes to show preemption info on RM app page is not covered in this patch. Any feedback is welcome! Thanks > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)