Jagadish created YARN-8034:
------------------------------
Summary: Clarification on preferredHost request with
relaxedLocality
Key: YARN-8034
URL: https://issues.apache.org/jira/browse/YARN-8034
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jagadish
I work on Apache Samza, a stateful stream-processing framework that leverages
Yarn for resource management. The Samza AM requests resources on specific hosts
to schedule stateful jobs. We set relaxLocality = true in these requests we
make to Yarn. Often we have observed that we don't get containers on the hosts
that we requested them on and the Yarn RM returns containers on arbitrary
hosts.
Do you know what the behavior of the FairScheduler/CapacityScheduler is when
setting "relaxLocality = true".I did play around by setting a high value for
yarn.scheduler.capacity.node-locality-delay but it did not seem to matter.
However, when setting relaxLocality = false, we get resources on the exact
hosts we requested on.
The behavior I want from Yarn is "Honor locality to the best possible extent
and only return a container on an arbitrary host if the requested host is
down". Is there a way to accomplish this?
If you can point me to the Scheduler code, I'm happy to look at it as well. For
context, we have continuous scheduling enabled in our clusters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
