Hai Lu created SAMZA-2601:
-----------------------------
Summary: Continuous heavy logging when YARN does not allocate
resources for a container
Key: SAMZA-2601
URL: https://issues.apache.org/jira/browse/SAMZA-2601
Project: Samza
Issue Type: Bug
Reporter: Hai Lu
Assignee: Hai Lu
If YARN does not immediately allocate resources for a container and host
affinity is disabled for the job, then the container allocator will spam the
logs with messages about the request, even if the request is considered
expired. This is because the allocator loops over the pending requests to check
if they have been allocated, but there is no delay between loop iterations, so
it might just keep checking the same pending request over and over as fast as
possible. A metric is also incremented in this flow, so that metric is
extremely high, which is probably not the intention.
One case we saw is that if YARN does not have enough capacity, then it will not
return resources, so the AM gets into this logging loop:
Handling assignment for Processor ID:
Did not find any allocated containers for running Processor ID:
...
Functionally, I did not observe impact, since YARN is out of capacity anyways,
so the job isn't going to be deployed in any case. However, the AM basically
goes into an infinite polling loop if YARN is out of capacity, and this will
fill up disk with logs very quickly (3GB/s).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
