[
https://issues.apache.org/jira/browse/YARN-8138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zian Chen updated YARN-8138:
----------------------------
Description:
Add unit test to validate queue priority preemption works under node partition.
Test configuration:
queue A (capacity=50, priority=1)
queue B (capacity=50, priority=2)
both have accessible-node-labels set to x
A.accessible-node-labels.x.capacity = 50
B.accessible-node-labels.x.capacity = 50
Along with this pre-emption related properties have been set.
Test steps:
- Submit an application A1 to B, with am-container = container = 4096, no. of
containers = 4
- Submit an application A2 to A, with am-container = 1024, container = 2048,
no of containers = (NUM_NM-1)
- Kill application A1
- Submit an application A3 to B with am-container=container=5210, no. of
containers=NUM_NM
- Expectation is that containers are pre-empted from application A2 to A3 but
there is no container pre-emption happening
was:
There seems to be an issue with pre-emption when using node labels with queue
priority.
Test configuration:
queue A (capacity=50, priority=1)
queue B (capacity=50, priority=2)
both have accessible-node-labels set to x
A.accessible-node-labels.x.capacity = 50
B.accessible-node-labels.x.capacity = 50
Along with this pre-emption related properties have been set.
Test steps:
- Set NM memory = 6000MB and containerMemory = 750MB
- Submit an application A1 to B, with am-container = container =
(6000-750-1500), no. of containers = 2
- Submit an application A2 to A, with am-container = 750, container = 1500, no
of containers = (NUM_NM-1)
- Kill application A1
- Submit an application A3 to B with am-container=container=5000, no. of
containers=3
- Expectation is that containers are pre-empted from application A2 to A3 but
there is no container pre-emption happening
Container pre-emption is stuck with the message in the RM log,
{noformat}
2018-02-02 11:41:36,974 INFO capacity.CapacityScheduler
(CapacityScheduler.java:tryCommit(2673)) - Allocation proposal accepted
2018-02-02 11:41:36,984 INFO capacity.CapacityScheduler
(CapacityScheduler.java:allocateContainerOnSingleNode(1391)) - Trying to
fulfill reservation for application application_1517571510094_0003 on node:
XXXXXXXXXX:25454
2018-02-02 11:41:36,984 INFO allocator.AbstractContainerAllocator
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(97)) -
Reserved container application=application_1517571510094_0003
resource=<memory:3072, vCores:1>
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@3f04848e
cluster=<memory:18000, vCores:3>
2018-02-02 11:41:36,984 INFO capacity.CapacityScheduler
(CapacityScheduler.java:tryCommit(2673)) - Allocation proposal accepted
2018-02-02 11:41:36,984 INFO capacity.CapacityScheduler
(CapacityScheduler.java:allocateContainerOnSingleNode(1391)) - Trying to
fulfill reservation for application application_1517571510094_0003 on node:
XXXXXXXXXX:25454
2018-02-02 11:41:36,984 INFO allocator.AbstractContainerAllocator
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(97)) -
Reserved container application=application_1517571510094_0003
resource=<memory:3072, vCores:1>
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@3f04848e
cluster=<memory:18000, vCores:3>
2018-02-02 11:41:36,984 INFO capacity.CapacityScheduler
(CapacityScheduler.java:tryCommit(2673)) - Allocation proposal accepted
2018-02-02 11:41:36,994 INFO capacity.CapacityScheduler
(CapacityScheduler.java:allocateContainerOnSingleNode(1391)) - Trying to
fulfill reservation for application application_1517571510094_0003 on node:
XXXXXXXXXX:25454
2018-02-02 11:41:36,995 INFO allocator.AbstractContainerAllocator
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(97)) -
Reserved container application=application_1517571510094_0003
resource=<memory:3072, vCores:1>
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@3f04848e
cluster=<memory:18000, vCores:3>{noformat}
> Add unit test to validate queue priority preemption works under node
> partition.
> -------------------------------------------------------------------------------
>
> Key: YARN-8138
> URL: https://issues.apache.org/jira/browse/YARN-8138
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Charan Hebri
> Assignee: Zian Chen
> Priority: Minor
> Attachments: YARN-8138.001.patch, YARN-8138.002.patch
>
>
> Add unit test to validate queue priority preemption works under node
> partition.
> Test configuration:
> queue A (capacity=50, priority=1)
> queue B (capacity=50, priority=2)
> both have accessible-node-labels set to x
> A.accessible-node-labels.x.capacity = 50
> B.accessible-node-labels.x.capacity = 50
> Along with this pre-emption related properties have been set.
> Test steps:
> - Submit an application A1 to B, with am-container = container = 4096, no.
> of containers = 4
> - Submit an application A2 to A, with am-container = 1024, container = 2048,
> no of containers = (NUM_NM-1)
> - Kill application A1
> - Submit an application A3 to B with am-container=container=5210, no. of
> containers=NUM_NM
> - Expectation is that containers are pre-empted from application A2 to A3
> but there is no container pre-emption happening
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
