[
https://issues.apache.org/jira/browse/YARN-8034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16407398#comment-16407398
]
Jagadish commented on YARN-8034:
--------------------------------
Hey [~jlowe] and [~kkaranasos]:
Thanks for the pointers. It helped me dive-deep into the internals of how the
Capacity Scheduler actually works. Currently, our YARN clusters are on version
2.7 - So, we cannot leverage YARN-6344 just yet. Hence, we'll stick to making
our app-master request resources with strict locality (by setting relaxLocality
= false).
While the "localityWaitFactor" is certainly at play when used with the
CapacityScheduler, I'd like to understand how it affects the FairScheduler. The
code you pointed out: FsAppAttempt#getLocalityWaitFactor appears to be un-used
in the 2.7 branch of Hadoop. There's however,
FicaSchedulerApp#getLocalityWaitFactor but its usage seems restricted to the
capacity scheduler though. Is there something else I am missing?
Here's the setup:
We have FairScheduler with "continuous scheduling" turned on, with a
node-locality delay = 5s, rack-locality-delay = 5s, node-locality-threshold =
-1 and rack-locality-threshold = -1.
I requested one container on a preferred host with relaxLocality = true but
Yarn appears to return a container on an arbitrary host instantly. Is there
some other scheduler tunable I'm missing? Any pointers to the source code are
greatly appreciated!
Thanks [~jlowe] and [~kkaranasos]
> Clarification on preferredHost request with relaxedLocality
> -----------------------------------------------------------
>
> Key: YARN-8034
> URL: https://issues.apache.org/jira/browse/YARN-8034
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Jagadish
> Priority: Major
>
> I work on Apache Samza, a stateful stream-processing framework that leverages
> Yarn for resource management. The Samza AM requests resources on specific
> hosts to schedule stateful jobs. We set relaxLocality = true in these
> requests we make to Yarn. Often we have observed that we don't get containers
> on the hosts that we requested them on and the Yarn RM returns containers on
> arbitrary hosts.
> Do you know what the behavior of the FairScheduler/CapacityScheduler is when
> setting "relaxLocality = true".I did play around by setting a high value for
> yarn.scheduler.capacity.node-locality-delay but it did not seem to matter.
> However, when setting relaxLocality = false, we get resources on the exact
> hosts we requested on.
> The behavior I want from Yarn is "Honor locality to the best possible extent
> and only return a container on an arbitrary host if the requested host is
> down". Is there a way to accomplish this?
> If you can point me to the Scheduler code, I'm happy to look at it as well.
> For context, we have continuous scheduling enabled in our clusters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
