[
https://issues.apache.org/jira/browse/AURORA-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mehrdad Nurolahzade updated AURORA-1837:
----------------------------------------
Description:
Current implementation of {{TaskHistoryPrunner}} registers all inactive tasks
upon terminal _state_ change for pruning.
{{TaskHistoryPrunner::registerInactiveTask()}} uses a delay executor to
schedule the process of pruning _task_s. However, we have noticed most of
pruning takes place after scheduler recovers from a fail-over.
Modify {{TaskHistoryPruner}} to a design similar to {{JobUpdateHistoryPruner}}:
# Instead of registering delay executor's upon terminal task state transitions,
have it wake up on preconfigured intervals, find all terminal state tasks that
meet pruning criteria and delete them.
# Make the initial task history pruning delay configurable so that it does not
hamper scheduler upon start.
was:
TaskHistoryPrunner registers all inactive tasks upon _state_ change for
pruning.
TaskHistoryPrunner::registerInactiveTask uses delay executor to schedule the
process of prunning _task_s and _job_s. This is totally reasonable since
pruning in not critical and can be done when the load on the scheduler is low.
Once pruning tasks, a delay is used in the first pruning phase
(shutdownOnError) but in the second one seems to be instant. This has caused
problems when lots of tasks are changing state and the load on the scheduler is
high (for instance during scheduler restore).
to do items:
1. investigate if we can add a delay to all executions, and what the delays
should be.
2. investigate if executions can be suppressed based on the load on the
scheduler.
> Improve task history pruning
> ----------------------------
>
> Key: AURORA-1837
> URL: https://issues.apache.org/jira/browse/AURORA-1837
> Project: Aurora
> Issue Type: Task
> Reporter: Reza Motamedi
> Priority: Minor
> Labels: scheduler
>
> Current implementation of {{TaskHistoryPrunner}} registers all inactive tasks
> upon terminal _state_ change for pruning.
> {{TaskHistoryPrunner::registerInactiveTask()}} uses a delay executor to
> schedule the process of pruning _task_s. However, we have noticed most of
> pruning takes place after scheduler recovers from a fail-over.
> Modify {{TaskHistoryPruner}} to a design similar to
> {{JobUpdateHistoryPruner}}:
> # Instead of registering delay executor's upon terminal task state
> transitions, have it wake up on preconfigured intervals, find all terminal
> state tasks that meet pruning criteria and delete them.
> # Make the initial task history pruning delay configurable so that it does
> not hamper scheduler upon start.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
