Hi all,
I have a problem with CapacityScheduler container reservations.
My cluster has the following specs:
Hadoop version: 2.7.1
*Queues:
yarn.scheduler.capacity.root.default.capacity 20
yarn.scheduler.capacity.root.samza.capacity 40
yarn.scheduler.capacity.root.loaders.capacity 40
yarn.scheduler.capacity.root.loaders.fast.capacity 60
yarn.scheduler.capacity.root.loaders.slow.capacity 40
*Number Nodemanagers: 3
**Nodemanagers:
—>rbcx5qyowm23.redborder.cluster
Total Vmem allocated for Containers 53.89 GB
Vmem enforcement enabled true
Total Pmem allocated for Container 5.39 GB
—>rbbxvxmciv15.redborder.cluster
Total Vmem allocated for Containers 125.45 GB
Vmem enforcement enabled true
Total Pmem allocated for Container 12.54 GB
—>rbbxdz032p71.redborder.cluster
Total Vmem allocated for Containers 79.11 GB
Vmem enforcement enabled true
Total Pmem allocated for Container 7.91 GB
**Total cluster memory: 25.84GB
This is my problem:
I start the ResourceManager and all the Nodemanagers(3). At this point I have
25.84GB free at the cluster as I described above. Now I submit 4 Samza
applications. These applications are correctly uploaded and they start running.
Samza applications are applications that never ends so they are indefinitely
running. That’s normal. So at this point I have:
10GB Used
15.84GB Free.
Detailed for each node:
rbcx5qyowm23.redborder.cluster Used: 2GB Free: 3.39GB
rbbxvxmciv15.redborder.cluster Used: 3GB Free: 9.54GB
rbbxdz032p71.redborder.cluster Used: 5GB Free: 2.91GB
At this point I upload an oozie application. Oozie have a workflow. Oozie
submit the oozielauncher application which is the responsible to upload the
applications defined at the workflow. So at this point I have 4 samza
applications, 1 oozie launcher and 1 application uploaded by oozie launcher
(this application uploaded by oozie launcher has an AM and a container uploaded
by this AM). My problem appear at this point.
On Hadoop Scheduler webpage I see:
4 Samza applications Running.
1 Oozie launcher Running
1 PigLatin application (submitted by oozielauncher) (this application never
ends and it should!).
**When I inspect the PigLatin application:
YarnApplicationState: RUNNING: AM has registered with RM and started running.
The AM container is allocated and running.
It has made an “Off Switch Request” requesting a Container of 2GB Ram.
At this point the cluster is:
rbcx5qyowm23.redborder.cluster Used: 4GB Free: 1.39GB
rbbxvxmciv15.redborder.cluster Used: 3GB Free: 9.54GB
rbbxdz032p71.redborder.cluster Used: 7GB Free: 933MB
The container requested by AM is never allocated although there is 9.54GB free
at rbbxvxmciv15.redborder.cluster.
Both the oozie launcher and the application are submitted on
“yarn.scheduler.capacity.root.loaders.fast” and it have enough free space to
run the container.
So I started to see the logs. I enabled CapacityScheduler and ResourceManager
DEBUG log level.
Thats what I have found on ResourceManager logs:
**AM is correctly running as I saw at ResourceManager web
2016-03-14_10:37:32.01418 16/03/14 10:37:32 INFO
resourcemanager.ApplicationMasterService: AM registration
appattempt_1457951280710_0006_000001
2016-03-14_10:37:32.01423 16/03/14 10:37:32 INFO resourcemanager.RMAuditLogger:
USER=oozie IP=10.0.107.16 OPERATION=Register App Master
TARGET=ApplicationMasterService RESULT=SUCCESS
APPID=application_1457951280710_0006
APPATTEMPTID=appattempt_1457951280710_0006_000001
2016-03-14_10:37:32.01426 16/03/14 10:37:32 INFO attempt.RMAppAttemptImpl:
appattempt_1457951280710_0006_000001 State change from LAUNCHED to RUNNING
2016-03-14_10:37:32.01427 16/03/14 10:37:32 INFO rmapp.RMAppImpl:
application_1457951280710_0006 State change from ACCEPTED to RUNNING
** It reserves a Container at rbcx5qyowm23.redborder.cluster (I remember that
at this state this node has 1.39GB free)
2016-03-14_10:37:33.65925 16/03/14 10:37:33 INFO rmcontainer.RMContainerImpl:
container_1457951280710_0006_01_000002 Container Transitioned from NEW to
RESERVED
2016-03-14_10:37:33.65934 16/03/14 10:37:33 INFO capacity.LeafQueue: Reserved
container application=application_1457951280710_0006 resource=<memory:2048,
vCores:1> queue=fast: capacity=0.6, absoluteCapacity=0.24000001,
usedResources=<memory:4096, vCores:3>, usedCapacity=0.6449378,
absoluteUsedCapacity=0.15477045, numApps=2, numContainers=3
usedCapacity=0.6449378 absoluteUsedCapacity=0.15477045 used=<memory:4096,
vCores:3> cluster=<memory:26465, vCores:45>
2016-03-14_10:37:33.65936 16/03/14 10:37:33 INFO capacity.ParentQueue:
Re-sorting assigned queue: root.loaders.fast stats: fast: capacity=0.6,
absoluteCapacity=0.24000001, usedResources=<memory:6144, vCores:4>,
usedCapacity=0.9674067, absoluteUsedCapacity=0.23215568, numApps=2,
numContainers=4
2016-03-14_10:37:33.65937 16/03/14 10:37:33 INFO capacity.ParentQueue:
assignedContainer queue=loaders usedCapacity=0.5803892
absoluteUsedCapacity=0.23215568 used=<memory:6144, vCores:4>
cluster=<memory:26465, vCores:45>
2016-03-14_10:37:33.65939 16/03/14 10:37:33 INFO capacity.ParentQueue:
Re-sorting assigned queue: root.loaders stats: loaders: numChildQueue= 2,
capacity=0.4, absoluteCapacity=0.4, usedResources=<memory:6144,
vCores:4>usedCapacity=0.5803892, numApps=2, numContainers=4
2016-03-14_10:37:33.65940 16/03/14 10:37:33 INFO capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.6190818
absoluteUsedCapacity=0.6190818 used=<memory:16384, vCores:12>
cluster=<memory:26465, vCores:45>
**And now, It’s indefinitely logging:
2016-03-14_10:37:35.66182 16/03/14 10:37:35 INFO capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1457951280710_0006 on
node: rbcx5qyowm23.redborder.cluster:45454
2016-03-14_10:37:35.66228 16/03/14 10:37:35 INFO capacity.LeafQueue: Reserved
container application=application_1457951280710_0006 resource=<memory:2048,
vCores:1> queue=fast: capacity=0.6, absoluteCapacity=0.24000001,
usedResources=<memory:6144, vCores:4>, usedCapacity=0.9674067,
absoluteUsedCapacity=0.23215568, numApps=2, numContainers=4
usedCapacity=0.9674067 absoluteUsedCapacity=0.23215568 used=<memory:6144,
vCores:4> cluster=<memory:26465, vCores:45>
2016-03-14_10:37:35.66235 16/03/14 10:37:35 INFO capacity.CapacityScheduler:
Skipping scheduling since node rbcx5qyowm23.redborder.cluster:45454 is reserved
by application appattempt_1457951280710_0006_000001
**Its always trying to allocate the reserved container at
rbcx5qyowm23.redborder.cluster but it doesn’t try to allocate it at
rbbxvxmciv15.redborder.cluster (I remember that at this point this node have
9.54GB free). So, as it is trying to allocate the container at
rbcx5qyowm23.redborder.cluster and rbcx5qyowm23.redborder.cluster is running
Samza applications which never end, the reservation is stucked at this point.
Always trying to allocate the container in a node that is running applications
which will never free the resources. If I kill the oozie yarn application it,
at some submissions, runs ok, but when this situation happen again (reservation
at rbbxvxmciv15.redborder.cluster) it have a deadlock again.
I have inspected the CapacityScheduler code and I saw that there is a
“reservationsContinueLooking” variable that enable looking on other nodes
although a reservation is made for a node.
At the log appear that this variable is true:
2016-03-14_10:28:00.66642 16/03/14 10:28:00 INFO capacity.LeafQueue:
Initializing fast
2016-03-14_10:28:00.66643 capacity = 0.6 [= (float) configuredCapacity / 100 ]
2016-03-14_10:28:00.66644 asboluteCapacity = 0.24000001 [=
parentAbsoluteCapacity * capacity ]
2016-03-14_10:28:00.66645 maxCapacity = 0.6 [= configuredMaxCapacity ]
2016-03-14_10:28:00.66645 absoluteMaxCapacity = 0.24000001 [= 1.0
maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100
otherwise ]
2016-03-14_10:28:00.66646 userLimit = 100 [= configuredUserLimit ]
2016-03-14_10:28:00.66648 userLimitFactor = 1.0 [= configuredUserLimitFactor ]
2016-03-14_10:28:00.66648 maxApplications = 2400 [=
configuredMaximumSystemApplicationsPerQueue or
(int)(configuredMaximumSystemApplications * absoluteCapacity)]
2016-03-14_10:28:00.66649 maxApplicationsPerUser = 2400 [=
(int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ]
2016-03-14_10:28:00.66650 usedCapacity = 0.0 [= usedResourcesMemory /
(clusterResourceMemory * absoluteCapacity)]
2016-03-14_10:28:00.66650 absoluteUsedCapacity = 0.0 [= usedResourcesMemory /
clusterResourceMemory]
2016-03-14_10:28:00.66651 maxAMResourcePerQueuePercent = 1.0 [=
configuredMaximumAMResourcePercent ]
2016-03-14_10:28:00.66651 minimumAllocationFactor = 0.875 [=
(float)(maximumAllocationMemory - minimumAllocationMemory) /
maximumAllocationMemory ]
2016-03-14_10:28:00.66653 maximumAllocation = <memory:8192, vCores:32> [=
configuredMaxAllocation ]
2016-03-14_10:28:00.66653 numContainers = 0 [= currentNumContainers ]
2016-03-14_10:28:00.66654 state = RUNNING [= configuredState ]
2016-03-14_10:28:00.66654 acls = ADMINISTER_QUEUE: SUBMIT_APP: [=
configuredAcls ]
2016-03-14_10:28:00.66654 nodeLocalityDelay = 10
2016-03-14_10:28:00.66655 labels=*,
2016-03-14_10:28:00.66655 nodeLocalityDelay = 10
2016-03-14_10:28:00.66656 reservationsContinueLooking = true
2016-03-14_10:28:00.66656 preemptionDisabled = true
(As i see, “reservationsContinueLooking” is true by default at fast queue)
At the code I notice:
// if reservation continous looking enabled, check to see if could we
// potentially use this node instead of a reserved node if the application
// has reserved containers.
// TODO, now only consider reservation cases when the node has no label
if (this.reservationsContinueLooking
&& label.equals(RMNodeLabelsManager.NO_LABEL)
But my nodes have not labels.
Thank you in advance.
Regards.
Pablo Casares Crespo
Big Data Developer
[email protected]
+34 667735764 | +34 955 601 160
SAN FRANCISCO - SEVILLE - MADRID
This email, including attachments, is intended exclusively for its addressee.
It contains information that is CONFIDENTIAL whose disclosure is prohibited by
law and may be covered by legal privilege. If you have received this email in
error, please notify the sender and delete it from your system.
Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su
destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida
por la ley o puede estar sometida a secreto profesional. Si ha recibido este
mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su
destrucción.
