-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59699/
-----------------------------------------------------------
(Updated June 1, 2017, 11:48 p.m.)
Review request for Aurora, David McLaughlin, Santhosh Kumar, Stephan Erb, and
Zameer Manji.
Changes
-------
Refactored the createDeleteEvent code to ensure that the tasks deleted from
TaskStore are consistent with the tasks in PubsubEvent.
Bugs: AURORA-1929
https://issues.apache.org/jira/browse/AURORA-1929
Repository: aurora
Description
-------
Improve task history pruning by batch deleting tasks.
The `'aurora_admin prune_tasks'` endpoint seems to be very slow when the
cluster has a large number of inactive tasks.
This CR batches all removeTasks operations and execute them all at once to
avoid additional cost of coalescing. The fix will also benefit implicit task
history pruning since it has similar underlying implementation. See
https://issues.apache.org/jira/browse/AURORA-1929 for more information and
details.
Diffs (updated)
-----
src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java
73878210f9028901fda3b08e66c6a63c24260d35
Diff: https://reviews.apache.org/r/59699/diff/4/
Changes: https://reviews.apache.org/r/59699/diff/3-4/
Testing
-------
__unit_tests:__
./build-support/jenkins/build.sh
No unit tests were created for this patch since it does not add new
functionalities or alter the interface, but improves the efficiency of the
existing code.
__e2e tests:__
Attached was a screenshot of the task history pruning benchmark obtained from a
scale test in Twitter's test cluster.
- Before applying this patch, the task history pruning takes ~30 minutes on
130K tasks.
- After applying the patch, the pruning takes ~1 minute.
File Attachments
----------------
task_history_pruning_benchmark.png
https://reviews.apache.org/media/uploaded/files/2017/06/01/74eb5104-d338-4530-abd2-b82fbdc6bf84__task_history_pruning_benchmark.png
Thanks,
Kai Huang
