Re: Review Request 35672: DbTaskStore perf: optimize queries scoped to a task ID.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35672/#review88804 --- Ship it! Ship It! - Maxim Khutornenko On June 19, 2015, 10:43 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35672/ --- (Updated June 19, 2015, 10:43 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1298 https://issues.apache.org/jira/browse/AURORA-1298 Repository: aurora Description --- DbTaskStore perf: optimize queries scoped to a task ID. Diffs - src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java cceac8ab14243c7806c48cf5a8d4c1175d7004b8 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 9b30b01e3a9ecdf368910a7270f0d6fed911b2de src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 8270407f97f44991dcfa47263c4287c58ac558f1 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 4b67f6ba03d299ed3de73bb5ea69d949364835b3 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 7c27f374b1143c82131c19448c3236f3dfb96667 Diff: https://reviews.apache.org/r/35672/diff/ Testing --- Scheduling benchmark results with MemTaskStore: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 541318.648 ± 25645.908 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 57782.604 ± 1896.717 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 104040.842 ±42.494 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 55.713 ± 1.078 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 55.160 ± 1.434 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 54.352 ± 3.150 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 53.149 ± 1.026 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 56678.838 ± 1818.328 ops/s ``` With DbTaskStore before this change: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 42070.261 ± 846.323 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 19909.569 ± 516.249 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 2956.306 ± 24.422 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 54.233 ± 1.455 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 54.970 ± 1.027 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 53.516 ± 1.172 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 45.404 ± 2.238 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 16391.602 ± 752.267 ops/s ``` With DbTaskStore after this change: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 149589.434 ± 3553.202 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 29268.634 ± 1072.366 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 103187.037 ± 25.248 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 56.504 ±0.575 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 54.710 ±1.396 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 54.777 ±1.244 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 45.155 ±1.602 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 23351.751 ± 509.439 ops/s ``` While we lack formal acceptance criteria for DbTaskStore on these benchmarks, i believe these should be considered acceptable. Thanks, Bill Farner
Re: Review Request 35672: DbTaskStore perf: optimize queries scoped to a task ID.
On June 19, 2015, 11:50 p.m., Maxim Khutornenko wrote: src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java, line 127 https://reviews.apache.org/r/35672/diff/1/?file=988557#file988557line127 Curious, why special treating whitespacing here and not in other places? Bill Farner wrote: This is a bulk move of the predicate, so i don't have a good answer. Happy to leave a TODO/ticket to re-evaluate, but i'd like to avoid bundling that with this change. A TODO would be nice to follow up that we are not missing anything here. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35672/#review88605 --- On June 19, 2015, 10:43 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35672/ --- (Updated June 19, 2015, 10:43 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1298 https://issues.apache.org/jira/browse/AURORA-1298 Repository: aurora Description --- DbTaskStore perf: optimize queries scoped to a task ID. Diffs - src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java cceac8ab14243c7806c48cf5a8d4c1175d7004b8 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 9b30b01e3a9ecdf368910a7270f0d6fed911b2de src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 8270407f97f44991dcfa47263c4287c58ac558f1 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 4b67f6ba03d299ed3de73bb5ea69d949364835b3 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 7c27f374b1143c82131c19448c3236f3dfb96667 Diff: https://reviews.apache.org/r/35672/diff/ Testing --- Scheduling benchmark results with MemTaskStore: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 541318.648 ± 25645.908 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 57782.604 ± 1896.717 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 104040.842 ±42.494 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 55.713 ± 1.078 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 55.160 ± 1.434 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 54.352 ± 3.150 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 53.149 ± 1.026 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 56678.838 ± 1818.328 ops/s ``` With DbTaskStore before this change: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 42070.261 ± 846.323 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 19909.569 ± 516.249 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 2956.306 ± 24.422 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 54.233 ± 1.455 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 54.970 ± 1.027 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 53.516 ± 1.172 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 45.404 ± 2.238 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 16391.602 ± 752.267 ops/s ``` With DbTaskStore after this change: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 149589.434 ± 3553.202 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 29268.634 ± 1072.366 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 103187.037 ± 25.248 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 56.504 ±0.575 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 54.710 ±1.396 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 54.777 ±1.244 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 45.155 ±1.602 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmark
Re: Review Request 35672: DbTaskStore perf: optimize queries scoped to a task ID.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35672/ --- (Updated June 22, 2015, 4:51 p.m.) Review request for Aurora and Maxim Khutornenko. Changes --- Added TODO. Bugs: AURORA-1298 https://issues.apache.org/jira/browse/AURORA-1298 Repository: aurora Description --- DbTaskStore perf: optimize queries scoped to a task ID. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java cceac8ab14243c7806c48cf5a8d4c1175d7004b8 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 9b30b01e3a9ecdf368910a7270f0d6fed911b2de src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 8270407f97f44991dcfa47263c4287c58ac558f1 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 4b67f6ba03d299ed3de73bb5ea69d949364835b3 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 7c27f374b1143c82131c19448c3236f3dfb96667 Diff: https://reviews.apache.org/r/35672/diff/ Testing --- Scheduling benchmark results with MemTaskStore: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 541318.648 ± 25645.908 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 57782.604 ± 1896.717 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 104040.842 ±42.494 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 55.713 ± 1.078 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 55.160 ± 1.434 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 54.352 ± 3.150 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 53.149 ± 1.026 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 56678.838 ± 1818.328 ops/s ``` With DbTaskStore before this change: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 42070.261 ± 846.323 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 19909.569 ± 516.249 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 2956.306 ± 24.422 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 54.233 ± 1.455 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 54.970 ± 1.027 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 53.516 ± 1.172 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 45.404 ± 2.238 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 16391.602 ± 752.267 ops/s ``` With DbTaskStore after this change: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 149589.434 ± 3553.202 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 29268.634 ± 1072.366 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 103187.037 ± 25.248 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 56.504 ±0.575 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 54.710 ±1.396 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 54.777 ±1.244 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 45.155 ±1.602 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 23351.751 ± 509.439 ops/s ``` While we lack formal acceptance criteria for DbTaskStore on these benchmarks, i believe these should be considered acceptable. Thanks, Bill Farner
Re: Review Request 35672: DbTaskStore perf: optimize queries scoped to a task ID.
On June 19, 2015, 11:50 p.m., Maxim Khutornenko wrote: src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java, line 127 https://reviews.apache.org/r/35672/diff/1/?file=988557#file988557line127 Curious, why special treating whitespacing here and not in other places? This is a bulk move of the predicate, so i don't have a good answer. Happy to leave a TODO/ticket to re-evaluate, but i'd like to avoid bundling that with this change. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35672/#review88605 --- On June 19, 2015, 10:43 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35672/ --- (Updated June 19, 2015, 10:43 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1298 https://issues.apache.org/jira/browse/AURORA-1298 Repository: aurora Description --- DbTaskStore perf: optimize queries scoped to a task ID. Diffs - src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java cceac8ab14243c7806c48cf5a8d4c1175d7004b8 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 9b30b01e3a9ecdf368910a7270f0d6fed911b2de src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 8270407f97f44991dcfa47263c4287c58ac558f1 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 4b67f6ba03d299ed3de73bb5ea69d949364835b3 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 7c27f374b1143c82131c19448c3236f3dfb96667 Diff: https://reviews.apache.org/r/35672/diff/ Testing --- Scheduling benchmark results with MemTaskStore: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 541318.648 ± 25645.908 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 57782.604 ± 1896.717 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 104040.842 ±42.494 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 55.713 ± 1.078 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 55.160 ± 1.434 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 54.352 ± 3.150 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 53.149 ± 1.026 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 56678.838 ± 1818.328 ops/s ``` With DbTaskStore before this change: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 42070.261 ± 846.323 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 19909.569 ± 516.249 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 2956.306 ± 24.422 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 54.233 ± 1.455 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 54.970 ± 1.027 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 53.516 ± 1.172 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 45.404 ± 2.238 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 16391.602 ± 752.267 ops/s ``` With DbTaskStore after this change: ``` ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 149589.434 ± 3553.202 ops/s InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 29268.634 ± 1072.366 ops/s LimitConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 103187.037 ± 25.248 ops/s PreemptorSlotSearchBenchmark.runBenchmark1 thrpt 10 56.504 ±0.575 ops/s PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt 10 54.710 ±1.396 ops/s PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt 10 54.777 ±1.244 ops/s PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt 10 45.155 ±1.602 ops/s ValueConstraintMismatchSchedulingBenchmark.runBenchmarkN/A thrpt 10 23351.751 ± 509.439 ops/s ``` While we lack formal acceptance
