[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: YARN-3884.0008.patch > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch, > YARN-3884.0003.patch, YARN-3884.0004.patch, YARN-3884.0005.patch, > YARN-3884.0006.patch, YARN-3884.0007.patch, YARN-3884.0008.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: (was: YARN-3384.0007.patch) > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch, > YARN-3884.0003.patch, YARN-3884.0004.patch, YARN-3884.0005.patch, > YARN-3884.0006.patch, YARN-3884.0007.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedReso
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: YARN-3884.0007.patch > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch, > YARN-3884.0003.patch, YARN-3884.0004.patch, YARN-3884.0005.patch, > YARN-3884.0006.patch, YARN-3884.0007.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, >
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: YARN-3384.0007.patch [~varun_saxena] Thank you for review. Attaching patch after checkstyle fix > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3384.0007.patch, > YARN-3884.0002.patch, YARN-3884.0003.patch, YARN-3884.0004.patch, > YARN-3884.0005.patch, YARN-3884.0006.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resour
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: YARN-3884.0006.patch > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch, > YARN-3884.0003.patch, YARN-3884.0004.patch, YARN-3884.0005.patch, > YARN-3884.0006.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: (was: YARN-3884.0006.patch) > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch, > YARN-3884.0003.patch, YARN-3884.0004.patch, YARN-3884.0005.patch, > YARN-3884.0006.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacit
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: YARN-3884.0006.patch Attaching patch after rebase > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch, > YARN-3884.0003.patch, YARN-3884.0004.patch, YARN-3884.0005.patch, > YARN-3884.0006.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResourc
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: YARN-3884.0005.patch Attaching patch handling comments > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch, > YARN-3884.0003.patch, YARN-3884.0004.patch, YARN-3884.0005.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: YARN-3884.0004.patch > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch, > YARN-3884.0003.patch, YARN-3884.0004.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > nu
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: YARN-3884.0003.patch Attaching patch after handling patch > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch, > YARN-3884.0003.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125,
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: YARN-3884.0002.patch Issue of happening because on removal of reserved container the {{RMContainerEvent}} is not currently send. *Two scenarios are handled as part of this jira* # Container state change from reserved to kill. # Container state change from reserved to released Sending {{RMContainerEvent}} on above two cases.Also additional detailed logging is added to track internal unreserve cases. Attaching patch handling the same. > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg, YARN-3884.0002.patch > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_14358499
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated YARN-3884: Component/s: (was: timelineserver) > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 usedCapacity=2.0317461 absoluteUsedCapacity=0.8125 > used= cluster= >
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3884: - Labels: oct16-easy (was: ) > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 usedCapacity=2.0317461 absoluteUsedCapacity=0.8125 > us
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3884: - Component/s: resourcemanager > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineserver > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: oct16-easy > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 usedCapacity=2.0317461 absoluteUsedCapacity=0.8125 > u
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Component/s: timelineserver > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 usedCapacity=2.0317461 absoluteUsedCapacity=0.8125 > used= cluster= > 2015-07-02 20:45:32,174
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Attachment: Test Result-Container status.jpg Test Result Snapshot attached > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg, Test Result-Container status.jpg > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 usedCapacity=2.0317461 absoluteUsedCapacity=0.8125 > used= cluster= > 2015-07-02 20:
[jira] [Updated] (YARN-3884) RMContainerImpl transition from RESERVED to KILL apphistory status not updated
[ https://issues.apache.org/jira/browse/YARN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3884: --- Summary: RMContainerImpl transition from RESERVED to KILL apphistory status not updated (was: Capacity scheduler for DROP_RESERVATION event container status not updated) > RMContainerImpl transition from RESERVED to KILL apphistory status not updated > -- > > Key: YARN-3884 > URL: https://issues.apache.org/jira/browse/YARN-3884 > Project: Hadoop YARN > Issue Type: Bug > Environment: Suse11 Sp3 >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3884.patch, Apphistory Container Status.jpg, > Elapsed Time.jpg > > > Setup > === > 1 NM 3072 16 cores each > Steps to reproduce > === > 1.Submit apps to Queue 1 with 512 mb 1 core > 2.Submit apps to Queue 2 with 512 mb and 5 core > lots of containers get reserved and unreserved in this case > {code} > 2015-07-02 20:45:31,169 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0002_01_13 Container Transitioned from NEW to > RESERVED > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=1.6410257, absoluteUsedCapacity=0.65625, numApps=1, > numContainers=5 usedCapacity=1.6410257 absoluteUsedCapacity=0.65625 > used= cluster= > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.QueueA stats: QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 > 2015-07-02 20:45:31,170 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.96875 > absoluteUsedCapacity=0.96875 used= > cluster= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from NEW to > ALLOCATED > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=dsperf > OPERATION=AM Allocated ContainerTARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1435849994778_0001 > CONTAINERID=container_e24_1435849994778_0001_01_14 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: > Assigned container container_e24_1435849994778_0001_01_14 of capacity > on host host-10-19-92-117:64318, which has 6 > containers, used and available > after allocation > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > assignedContainer application attempt=appattempt_1435849994778_0001_01 > container=Container: [ContainerId: > container_e24_1435849994778_0001_01_14, NodeId: host-10-19-92-117:64318, > NodeHttpAddress: host-10-19-92-117:65321, Resource: , > Priority: 20, Token: null, ] queue=default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.0846906, absoluteUsedCapacity=0.4166, numApps=1, > numContainers=5 clusterResource= > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > Re-sorting assigned queue: root.default stats: default: capacity=0.2, > absoluteCapacity=0.2, usedResources=, > usedCapacity=2.5016286, absoluteUsedCapacity=0.5, numApps=1, numContainers=6 > 2015-07-02 20:45:31,191 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=1.0 absoluteUsedCapacity=1.0 > used= cluster= > 2015-07-02 20:45:32,143 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e24_1435849994778_0001_01_14 Container Transitioned from > ALLOCATED to ACQUIRED > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application application_1435849994778_0002 > on node: host-10-19-92-143:64318 > 2015-07-02 20:45:32,174 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: > Reserved container application=application_1435849994778_0002 > resource= queue=QueueA: capacity=0.4, > absoluteCapacity=0.4, usedResources=, > usedCapacity=2.0317461, absoluteUsedCapacity=0.8125, numApps=1, > numContainers=6 usedCapacity=2.0317461