[
https://issues.apache.org/jira/browse/YARN-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Ma updated YARN-11402:
-----------------------------
Description:
Tens of thousands of meaningless logs are frequently printed during
ResourceManager startup and recover container.
As we know, ResourceManager will always keep 10k application information by
default. In our very big scale cluster, it is very usual that resourcemanager
try to recover the containers which already finished and does not exist in
ResourceManager but still reported by nodemanager.
Under this case, below logs will be frequently printed, more importantly, this
log is meaningless, in real production setups, the maintainers actually more
care about which containers are properly recovered or killed not the ones are
skipped.
The related code are as follows,
!screenshot-1.png!
So we move the log into function killOrphanContainerOnNode().
!screenshot-2.png!
Only the containers to be killed need to be loged which is vital for trouble
shooting to distinguish whether the containers are kill by hadoop inner
mechanism or by users themselves.
was:
Tens of thousands of meaningless logs are frequently printed during
ResourceManager startup and recover container.
As we know, ResourceManager will always keep 10k application information by
default. In our very big scale cluster, it is very usual that resourcemanager
try to recover the containers which already finished and does not exist in
ResourceManager but still reported by nodemanager.
Under this case, below logs will be frequently printed, more importantly, this
log is meaningless, in real production setups, the maintainers actually more
care about which containers are properly recovered or killed not the ones are
skipped.
The related code are as follows,
!screenshot-1.png!
> Meaningless logs are frequently printed during ResourceManager startup and
> recover container.
> ---------------------------------------------------------------------------------------------
>
> Key: YARN-11402
> URL: https://issues.apache.org/jira/browse/YARN-11402
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Daniel Ma
> Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> Tens of thousands of meaningless logs are frequently printed during
> ResourceManager startup and recover container.
> As we know, ResourceManager will always keep 10k application information by
> default. In our very big scale cluster, it is very usual that resourcemanager
> try to recover the containers which already finished and does not exist in
> ResourceManager but still reported by nodemanager.
> Under this case, below logs will be frequently printed, more importantly,
> this log is meaningless, in real production setups, the maintainers actually
> more care about which containers are properly recovered or killed not the
> ones are skipped.
> The related code are as follows,
> !screenshot-1.png!
> So we move the log into function killOrphanContainerOnNode().
> !screenshot-2.png!
> Only the containers to be killed need to be loged which is vital for trouble
> shooting to distinguish whether the containers are kill by hadoop inner
> mechanism or by users themselves.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
