[
https://issues.apache.org/jira/browse/YARN-8034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16402040#comment-16402040
]
Jagadish commented on YARN-8034:
--------------------------------
Thank you [~jlowe] for your recommendations. Very useful.
>> The host could be down, full of other containers, unhealthy, etc.
I did have some follow-up questions so that I understand the behavior of this
config better.
- I setup an experiment with the Samza AM to request 1 container (with 1G
memory and 1 vcore, host = our-preferred-host, relaxLocality = true and rack =
null.)
- I observed that the Yarn RM immediately returns a container on a different
host in the next second after the request was made.
- I am able to repro' this 100% of the time across multiple runs (and across
multiple hosts in our cluster) which makes me wonder if the preferredHost is
ignored if we set relaxLocality = true? I did verify that all nodes were
healthy in the cluster.
- For more context, I'm able to observe this behavior with both the
fair-scheduler and the capacity-scheduler with "continuous scheduling" enabled
in both cases. So, I'm not sure if that matters.
FWIW, with relaxLocality = false, the RM returns containers on the exact hosts
that we requested them on. I'm happy to submit a documentation patch to Hadoop
so that we make everyone's life better when using Yarn for stateful apps :-)
>> The node locality delay gives admins some control over how patiently the RM
>> will wait for locality.
Our node locality delay is configured to the number of nodes in the cluster. I
did try increasing it to an arbitrary high number and it did not seem to affect
the results of the above experiment. Are there other knobs at play I'm missing?
>> Yes, although it will require some work on the Samza AM's part. Samza's AM
>> can make requests for specific nodes with relaxLocality=false, but it also
>> should monitor the updatedNodes field of each AllocateResponse. The RM will
>> notify applications in that response when a node becomes unusable or becomes
>> usable again. The Samza AM can cancel and resubmit a request (either for a
>> different host or with relaxLocality=true) when a node trying to be
>> allocated becomes unusable.
Our original approach was for the Samza AM re-submit the request with
relaxedLocality = true after waiting for some timeout. Thank you for your
helpful recommendations.
> Clarification on preferredHost request with relaxedLocality
> -----------------------------------------------------------
>
> Key: YARN-8034
> URL: https://issues.apache.org/jira/browse/YARN-8034
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Jagadish
> Priority: Major
>
> I work on Apache Samza, a stateful stream-processing framework that leverages
> Yarn for resource management. The Samza AM requests resources on specific
> hosts to schedule stateful jobs. We set relaxLocality = true in these
> requests we make to Yarn. Often we have observed that we don't get containers
> on the hosts that we requested them on and the Yarn RM returns containers on
> arbitrary hosts.
> Do you know what the behavior of the FairScheduler/CapacityScheduler is when
> setting "relaxLocality = true".I did play around by setting a high value for
> yarn.scheduler.capacity.node-locality-delay but it did not seem to matter.
> However, when setting relaxLocality = false, we get resources on the exact
> hosts we requested on.
> The behavior I want from Yarn is "Honor locality to the best possible extent
> and only return a container on an arbitrary host if the requested host is
> down". Is there a way to accomplish this?
> If you can point me to the Scheduler code, I'm happy to look at it as well.
> For context, we have continuous scheduling enabled in our clusters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
