> On June 1, 2017, 10:46 p.m., David McLaughlin wrote: > > src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java > > Line 376 (original), 377 (patched) > > <https://reviews.apache.org/r/59699/diff/3/?file=1739138#file1739138line379> > > > > I wonder if it's using the taskIds from the event? Since they have been > > selected from the database and might not match the taskIds we passed in. > > This logic was there in the previous patch too. This would ensure > > consistency between taskStore.deleteTasks and the taskIds we broadcast as > > deleted. > > Kai Huang wrote: > Previously taskStore.deleteTasks() is not using the taskIds from the > event. But it doesn't hurt to ensure the consistency here.
If you look at the old code for deleteTasks, before it processes each task it first selects them all from the taskStore (kind of what is happening in the event code). - David ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59699/#review176694 ----------------------------------------------------------- On June 1, 2017, 10:41 p.m., Kai Huang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59699/ > ----------------------------------------------------------- > > (Updated June 1, 2017, 10:41 p.m.) > > > Review request for Aurora, David McLaughlin and Santhosh Kumar. > > > Bugs: AURORA-1929 > https://issues.apache.org/jira/browse/AURORA-1929 > > > Repository: aurora > > > Description > ------- > > Improve task history pruning by batch deleting tasks. > > The `'aurora_admin prune_tasks'` endpoint seems to be very slow when the > cluster has a large number of inactive tasks. > > This CR batches all removeTasks operations and execute them all at once to > avoid additional cost of coalescing. The fix will also benefit implicit task > history pruning since it has similar underlying implementation. See > https://issues.apache.org/jira/browse/AURORA-1929 for more information and > details. > > > Diffs > ----- > > src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java > 73878210f9028901fda3b08e66c6a63c24260d35 > > > Diff: https://reviews.apache.org/r/59699/diff/3/ > > > Testing > ------- > > __unit_tests:__ > > ./build-support/jenkins/build.sh > > No unit tests were created for this patch since it does not add new > functionalities or alter the interface, but improves the efficiency of the > existing code. > > __e2e tests:__ > > Attached was a screenshot of the task history pruning benchmark obtained from > a scale test in Twitter's test cluster. > > - Before applying this patch, the task history pruning takes ~30 minutes on > 130K tasks. > > - After applying the patch, the pruning takes ~1 minute. > > > File Attachments > ---------------- > > task_history_pruning_benchmark.png > > https://reviews.apache.org/media/uploaded/files/2017/06/01/74eb5104-d338-4530-abd2-b82fbdc6bf84__task_history_pruning_benchmark.png > > > Thanks, > > Kai Huang > >
