Shuyi Zhang created YARN-11407:
----------------------------------
Summary: short-live job log aggregation blocked on LRA
Key: YARN-11407
URL: https://issues.apache.org/jira/browse/YARN-11407
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Shuyi Zhang
Currently, log aggregation threads are scheduled based on applications start
time. This means later applications that start later but finish earlier, will
be blocked on log aggregation, until the previous running jobs finish and
release a log aggregation thread for it to use. To resolve this bottleneck, we
can decouple the initialization from the thread scheduling of the
ApplogAggregator:
Initializing an AppLogAggregator on application start
But scheduling it in the thread pool on application finish
This schedules log aggregation threads based on finish time instead application
start time, so prevents having a pileup of jobs waiting for log aggregation,
especially when waiting for long-running jobs to finish.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
