-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55217/
-----------------------------------------------------------
Review request for Aurora, Joshua Cohen and Stephan Erb.
Bugs: AURORA-1847
https://issues.apache.org/jira/browse/AURORA-1847
Repository: aurora
Description
-------
If scheduler is configured to run with the `MemTaskStore` every hit on
scheduler landing page (`/scheduler`) causes a call to
`MemTaskStore.getJobKeys()` through `ReadOnlyScheduler.getRoleSummary()`.
The implementation of `MemTaskStore.getJobKeys()` is currently very inefficient
as it requires a sequential scan of the task store and mapping to their
respective job keys. In Twitter clusters this method is currently taking half a
second per call (`mem_storage_get_job_keys`).
This patch eliminates the sequential scan and mapping to job key by simply
returning an immutable copy of the key set of the existing secondary index
`job`.
Diffs
-----
src/jmh/java/org/apache/aurora/benchmark/TaskStoreBenchmarks.java
f2f00b92bf901c438e40bbc177e9f5a112be1bbc
src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java
fc272ddb45be8b2f55f01c3d54f7fa9058202c0b
Diff: https://reviews.apache.org/r/55217/diff/
Testing
-------
Using the following modified version of existing
`TaskStoreBenchmarks.MemFetchTasksBenchmark` which benchmarks
`TaskStore.getJobKeys()`:
```java
public static class MemFetchTasksBenchmark extends AbstractFetchTasksBenchmark {
@Setup(Level.Trial)
@Override
public void setUp() {
storage = Guice.createInjector(
Modules.combine(
DbModule.testModuleWithWorkQueue(PLAIN, Optional.of(new
InMemStoresModule(PLAIN))),
new AbstractModule() {
@Override
protected void configure() {
bind(StatsProvider.class).toInstance(new FakeStatsProvider());
bind(Clock.class).toInstance(new FakeClock());
}
}))
.getInstance(Storage.class);
}
@Setup(Level.Iteration)
public void setUpIteration() {
createTasks(numTasks);
}
@TearDown(Level.Iteration)
public void tearDownIteration() {
deleteTasks();
}
@Benchmark
public Iterable<IJobKey> run() {
return storage.read(store -> store.getTaskStore().getJobKeys());
}
}
```
Benchmark results BEFORE patch:
```
Benchmark (numTasks) Mode Cnt Score
Error Units
TaskStoreBenchmarks.MemFetchTasksBenchmark.run 10000 thrpt 5 572.761
± 168.865 ops/s
TaskStoreBenchmarks.MemFetchTasksBenchmark.run 50000 thrpt 5 80.697
± 8.516 ops/s
TaskStoreBenchmarks.MemFetchTasksBenchmark.run 100000 thrpt 5 25.102
± 3.244 ops/s
```
Benchmark results AFTER patch:
```
Benchmark (numTasks) Mode Cnt
Score Error Units
TaskStoreBenchmarks.MemFetchTasksBenchmark.run 10000 thrpt 5
327336.998 ± 10207.402 ops/s
TaskStoreBenchmarks.MemFetchTasksBenchmark.run 50000 thrpt 5
320506.958 ± 23430.527 ops/s
TaskStoreBenchmarks.MemFetchTasksBenchmark.run 100000 thrpt 5
287962.695 ± 51917.245 ops/s
```
Thanks,
Mehrdad Nurolahzade
