----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59699/#review176567 -----------------------------------------------------------
src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java Line 388 (original), 411 (patched) <https://reviews.apache.org/r/59699/#comment249950> I'm not sure this is the right approach. I don't think we want to skip writing transactions to the replicated log. All of the APIs down to the storage layer already support batching so seems like we just want to move the delete command outside of the code that generates pubsub events. E.g. call taskStore.deleteTasks(taskIds) here when we receive the batch from the prune tasks endpoint: https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java#L371 And then remove the taskStore.deleteTasks from the PubsubEvent factory method (have no idea why it's in there). - David McLaughlin On June 1, 2017, 12:18 a.m., Kai Huang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59699/ > ----------------------------------------------------------- > > (Updated June 1, 2017, 12:18 a.m.) > > > Review request for Aurora, David McLaughlin and Santhosh Kumar. > > > Bugs: AURORA-1929 > https://issues.apache.org/jira/browse/AURORA-1929 > > > Repository: aurora > > > Description > ------- > > Improve task history pruning by batch deleting tasks. > > The `'aurora_admin prune_tasks'` endpoint seems to be very slow when the > cluster has a large number of inactive tasks. > > This CR batches all removeTasks operations and execute them all at once to > avoid additional cost of coalescing. The fix will also benefit implicit task > history pruning since it has similar underlying implementation. See > https://issues.apache.org/jira/browse/AURORA-1929 for more information and > details. > > > Diffs > ----- > > src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java > 73878210f9028901fda3b08e66c6a63c24260d35 > > > Diff: https://reviews.apache.org/r/59699/diff/1/ > > > Testing > ------- > > ./build-support/jenkins/build.sh > > > Thanks, > > Kai Huang > >
