> On Aug. 1, 2018, 12:42 a.m., Benjamin Mahler wrote: > > A couple of comments on the benchmark information before looking at the > > code, these probably belong on the previous review, but since the numbers > > are only shown in this one I'll leave these here: > > > > * Can we compare percentiles (e.g. min, q1, q3, max) across the approaches > > instead of averages? i.e. how much better does min,q1,q3,max get? Averages > > are generally a poor fit for performance data because it doesn't tell us > > about the distribution (e.g. if we make p90 3x worse for a 10% benefit to > > average that's not ok), we can include p50 if we're interested in the > > half-way point. > > * Can you include the cpu model of the box you ran this on? I'm interested > > in how many physical/virtual cores there are. > > * Can you also include the regular state query benchmark measurements to > > make sure we're not regressing too much on the single request case? (no > > need to get the non-optimized build numbers). > > * Some of the numbers don't look very good, e.g. Before `[min: > > 1.578161651secs, max: 8.789315237secs]` After: `[4.047655443secs, > > 6.00752698secs]`. Can we see the distribution here? Do you understand > > exactly why the lowest measurement is so much higher? Looking at the > > non-optimized numbers, the minimum didn't get worse? Is the data highly > > variable between runs? > > * Can you also include perf data for the optimized run? > > http://mesos.apache.org/documentation/latest/performance-profiling/ > > Alexander Rukletsov wrote: > * Sure. Updated https://reviews.apache.org/r/68131/ (see also preparatory > work in https://reviews.apache.org/r/68224/ and > https://reviews.apache.org/r/68225/). > * Done. > * Will do, stay tuned. > * This is fine and expected. Due to the batching there is an extra defer > in the master queue, which affects the response time of the first request. > Nothing to worry about, IMO. > * Will do, stay tuned.
> This is fine and expected. Due to the batching there is an extra defer in the > master queue, which affects the response time of the first request. Nothing > to worry about, IMO. Looking at the numbers for the single request benchmark case (thanks for posting those), the batching overhead only seems to be 0.15% or 150ms (10.946s -> 11.096s). This means that it's not just the extra defer that's causing an 85% or 1.5 second slowdown (1.512s -> 2.820s) in the minimum request processing time. It must be something else, like the request is now ending up in a batch (probably with more than 4 requests so that we're entering hyperthreading territory). Let's make sure we understand why it's happening. - Benjamin ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68132/#review206711 ----------------------------------------------------------- On Aug. 7, 2018, 12:11 p.m., Alexander Rukletsov wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68132/ > ----------------------------------------------------------- > > (Updated Aug. 7, 2018, 12:11 p.m.) > > > Review request for mesos, Benno Evers and Benjamin Mahler. > > > Bugs: MESOS-9122 > https://issues.apache.org/jira/browse/MESOS-9122 > > > Repository: mesos > > > Description > ------- > > With this patch handlers for '/state' requests are not scheduled > directly after authorization, but are accumulated and then scheduled > for later parallel processing. > > This approach allows, if there are N '/state' requests in the Master's > mailbox and T is the request response time, to block the Master actor > only once for time O(T) instead of blocking it for time N*T prior to > this patch. > > This batching technique reduces both the time Master is spending > answering '/state' requests and the average request response time > in presence of multiple requests in the Master's mailbox. However, > for seldom '/state' requests that don't accumulate in the Master's > mailbox, the response time might increase due to an added trip > through the mailbox. > > The change preserves the read-your-writes consistency model. > > > Diffs > ----- > > src/master/http.cpp d43fbd689598612ec5946b46e2fa5e7f5e22cfa8 > src/master/master.hpp 209b998db8d2bad7a3812df44f0939458f48eb11 > > > Diff: https://reviews.apache.org/r/68132/diff/2/ > > > Testing > ------- > > `make check` on Mac OS 10.13.5 and various Linux distros. > > Run `MasterStateQueryLoad_BENCHMARK_Test.v0State` benchmark and > `MasterStateQuery_BENCHMARK_Test.GetState`, see below. > > **Setup** > Processor: Intel i7-4980HQ 2.8 GHz with 6 MB on-chip L3 cache and 128 MB L4 > cache (Crystalwell) > Total Number of Cores: 4 > Total Number of Cores: 8 > L2 Cache (per Core): 256 KB > > Compiler: Apple LLVM version 9.1.0 (clang-902.0.39.2) > Optimization: -O2 > > **MasterStateQuery_BENCHMARK_Test.GetState, v0 '/state' response time** > > setup | no batching | > batching > ---------------------------------------------------------|-------------|---------- > 1000 agents, 10000 running, and 10000 completed tasks | 146.496ms | > 158.319ms > 10000 agents, 100000 running, and 100000 completed tasks | 1.795s | > 1.899s > 20000 agents, 200000 running, and 200000 completed tasks | 3.742s | > 4.427s > 40000 agents, 400000 running, and 400000 completed tasks | 10.946s | > 11.096s > > **MasterStateQueryLoad_BENCHMARK_Test.v0State, setup 1** > Test setup 1: 100 agents with a total of 10000 running tasks and 10000 > completed tasks; 50 '/state' and '/flags' requests will be sent in parallel > with 200ms interval, i.e., total **50 measurements** per endpoint. > > /flags | no batching | batching /state | no batching | batching > ------------------------------- * -------------------------------- > min | 1.598ms | 1.475ms min | 100.627ms | 105.383ms > p25 | 2.370ms | 2.452ms p25 | 102.206ms | 107.184ms > p50 | 2.520ms | 2.562ms p50 | 103.213ms | 108.468ms > p75 | 2.623ms | 2.665ms p75 | 104.100ms | 109.808ms > p90 | 2.803ms | 2.731ms p90 | 106.079ms | 111.043ms > max | 84.957ms | 2.934ms max | 153.438ms | 154.636ms > > **MasterStateQueryLoad_BENCHMARK_Test.v0State, setup 2** > Test setup 2: 1000 agents with a total of 100000 running tasks and 100000 > completed tasks; 10 '/state' and '/flags' requests will be sent in parallel > with 200ms interval, i.e., total **10 measurements** per endpoint. > > /flags | no batching | batching /state | no batching | batching > -------------------------------- * ------------------------------- > min | 2.309ms | 1.579ms min | 1.512s | 2.820s > p25 | 1.547s | 373.609ms p25 | 3.262s | 3.588s > p50 | 3.189s | 831.261ms p50 | 5.052s | 4.253s > p75 | 5.346s | 2.215s p75 | 6.846s | 4.510s > p90 | 5.854s | 2.351s p90 | 7.883s | 4.705s > max | 7.237s | 2.444s max | 8.517s | 4.934s > > > Thanks, > > Alexander Rukletsov > >
