> On Jan. 5, 2017, 11:38 a.m., Dmitriy Shirchenko wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java,
> > lines 117-126
> > <https://reviews.apache.org/r/55217/diff/1/?file=1597677#file1597677line117>
> >
> > :nit any reason why you didn't extract second index declration on
> > 117:120
I initially did but PMD did not like it, so I let it be:
`src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java:99:
Perhaps 'hostIndex' could be replaced by a local variable.
:pmdMain FAILED`
- Mehrdad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55217/#review160620
-----------------------------------------------------------
On Jan. 5, 2017, 10:59 a.m., Mehrdad Nurolahzade wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55217/
> -----------------------------------------------------------
>
> (Updated Jan. 5, 2017, 10:59 a.m.)
>
>
> Review request for Aurora, Joshua Cohen and Stephan Erb.
>
>
> Bugs: AURORA-1847
> https://issues.apache.org/jira/browse/AURORA-1847
>
>
> Repository: aurora
>
>
> Description
> -------
>
> If scheduler is configured to run with the `MemTaskStore` every hit on
> scheduler landing page (`/scheduler`) causes a call to
> `MemTaskStore.getJobKeys()` through `ReadOnlyScheduler.getRoleSummary()`.
>
> The implementation of `MemTaskStore.getJobKeys()` is currently very
> inefficient as it requires a sequential scan of the task store and mapping to
> their respective job keys. In Twitter clusters this method is currently
> taking half a second per call (`mem_storage_get_job_keys`).
>
> This patch eliminates the sequential scan and mapping to job key by simply
> returning an immutable copy of the key set of the existing secondary index
> `job`.
>
>
> Diffs
> -----
>
> src/jmh/java/org/apache/aurora/benchmark/TaskStoreBenchmarks.java
> f2f00b92bf901c438e40bbc177e9f5a112be1bbc
> src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java
> fc272ddb45be8b2f55f01c3d54f7fa9058202c0b
>
> Diff: https://reviews.apache.org/r/55217/diff/
>
>
> Testing
> -------
>
> Using the following modified version of existing
> `TaskStoreBenchmarks.MemFetchTasksBenchmark` which benchmarks
> `TaskStore.getJobKeys()`:
> ```java
> public static class MemFetchTasksBenchmark extends
> AbstractFetchTasksBenchmark {
> @Setup(Level.Trial)
> @Override
> public void setUp() {
> storage = Guice.createInjector(
> Modules.combine(
> DbModule.testModuleWithWorkQueue(PLAIN, Optional.of(new
> InMemStoresModule(PLAIN))),
> new AbstractModule() {
> @Override
> protected void configure() {
> bind(StatsProvider.class).toInstance(new
> FakeStatsProvider());
> bind(Clock.class).toInstance(new FakeClock());
> }
> }))
> .getInstance(Storage.class);
>
> }
>
> @Setup(Level.Iteration)
> public void setUpIteration() {
> createTasks(numTasks);
> }
>
> @TearDown(Level.Iteration)
> public void tearDownIteration() {
> deleteTasks();
> }
>
> @Benchmark
> public Iterable<IJobKey> run() {
> return storage.read(store -> store.getTaskStore().getJobKeys());
> }
> }
> ```
>
> Benchmark results BEFORE patch:
> ```
> Benchmark (numTasks) Mode Cnt
> Score Error Units
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run 10000 thrpt 5
> 572.761 ± 168.865 ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run 50000 thrpt 5
> 80.697 ± 8.516 ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run 100000 thrpt 5
> 25.102 ± 3.244 ops/s
> ```
>
> Benchmark results AFTER patch:
> ```
> Benchmark (numTasks) Mode Cnt
> Score Error Units
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run 10000 thrpt 5
> 327336.998 ± 10207.402 ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run 50000 thrpt 5
> 320506.958 ± 23430.527 ops/s
> TaskStoreBenchmarks.MemFetchTasksBenchmark.run 100000 thrpt 5
> 287962.695 ± 51917.245 ops/s
> ```
>
>
> Thanks,
>
> Mehrdad Nurolahzade
>
>
