----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50205/#review144499 -----------------------------------------------------------
Ship it! The test itself LGTM so I'll commit it. However when I ran it it does show some performance regression on the basic scalars. ## Head Resources benchmark on top of 6ce156b2bed7edecb27ce50f68d262ea4ab4f88e ``` [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 Took 72228us to perform 50000 'total += r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 Took 101574us to perform 50000 'total -= r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 Took 251715us to perform 50000 'total = total + r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 Took 293414us to perform 50000 'total = total - r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 ``` ## Pre-shared resources Resources benchmark on top of 48a492cd9d7d0a194735b9b4107a35b489c596e1 ``` [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 Took 48583us to perform 50000 'total += r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 Took 86166us to perform 50000 'total -= r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 Took 233148us to perform 50000 'total = total + r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 Took 265719us to perform 50000 'total = total - r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 ``` 'total += r' run time sees 40% jump. (I've run these 10 times and took an average, the avg is 48685.4us (pre-shared resources) vs. 68421.7us (post-shared resources). The overhead seems to be due we call `addable()` and `subtractable()` twice for 'total += r' and 'total -= r'. After I remove one of the calls, the numbers become: Resources benchmarks on top of 6ce156b2bed7edecb27ce50f68d262ea4ab4f88e but with addable()/subtractable() tweaks ``` [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 Took 48955us to perform 50000 'total += r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 Took 85126us to perform 50000 'total -= r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 Took 226796us to perform 50000 'total = total + r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 Took 266480us to perform 50000 'total = total - r' operations on cpus(*):1; gpus(*):1; mem(*):128; disk(*):256 ``` Let's address this issue. :) - Jiang Yan Xu On July 30, 2016, 12:03 a.m., Anindya Sinha wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50205/ > ----------------------------------------------------------- > > (Updated July 30, 2016, 12:03 a.m.) > > > Review request for mesos, Klaus Ma and Jiang Yan Xu. > > > Bugs: MESOS-4892 > https://issues.apache.org/jira/browse/MESOS-4892 > > > Repository: mesos > > > Description > ------- > > Enhanced benchmark test for resources to include shared resources. > > > Diffs > ----- > > src/tests/resources_tests.cpp 4111e080b84079e100b731c9a56861b204f17388 > > Diff: https://reviews.apache.org/r/50205/diff/ > > > Testing > ------- > > Tests passed. Results for resources benchmark is as follows: > > Minimal impact seen in Resources arithmetic with the Resources refactor > changes to incorporate shared resources. > > With shared resources patch (note that 4th test below is for shared resources > for scalars) > > [----------] 4 tests from ResourcesOperators/Resources_BENCHMARK_Test > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 (806 ms) > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1 > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1 (17032 > ms) > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2 > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2 (1048 > ms) > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/3 > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/3 (940 ms) > [----------] 4 tests from ResourcesOperators/Resources_BENCHMARK_Test (19826 > ms total) > > HEAD > > [----------] 3 tests from ResourcesOperators/Resources_BENCHMARK_Test > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 (726 ms) > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1 > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1 (17413 > ms) > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2 > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2 (1026 > ms) > [----------] 3 tests from ResourcesOperators/Resources_BENCHMARK_Test (19165 > ms total) > > Output from stdout is: > > [==========] Running 4 tests from 1 test case. > [----------] Global test environment set-up. > [----------] 4 tests from ResourcesOperators/Resources_BENCHMARK_Test > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 > Took 72635us to perform 50000 'total += r' operations on cpus(*):1; > gpus(*):1; mem(*):128; disk(*):256 > Took 116027us to perform 50000 'total -= r' operations on cpus(*):1; > gpus(*):1; mem(*):128; disk(*):256 > Took 283213us to perform 50000 'total = total + r' operations on cpus(*):1; > gpus(*):1; mem(*):128; disk(*):256 > Took 333641us to perform 50000 'total = total - r' operations on cpus(*):1; > gpus(*):1; mem(*):128; disk(*):256 > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 (806 ms) > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1 > Took 4.103766secs to perform 10 'total += r' operations on cpus(0, > principal_0, {key_0: value_0}):1; gpus(... > Took 4.310104secs to perform 10 'total -= r' operations on cpus(0, > principal_0, {key_0: value_0}):1; gpus(... > Took 4186ms to perform 10 'total = total + r' operations on cpus(0, > principal_0, {key_0: value_0}):1; gpus(... > Took 4.380353secs to perform 10 'total = total - r' operations on cpus(0, > principal_0, {key_0: value_0}):1; gpus(... > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1 (17032 > ms) > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2 > Took 483119us to perform 1000 'total += r' operations on ports(*):[1-2, 4-5, > 7-8, 10-11, 13-14, 16-17, 1... > Took 2271us to perform 1000 'total -= r' operations on ports(*):[1-2, 4-5, > 7-8, 10-11, 13-14, 16-17, 1... > Took 559844us to perform 1000 'total = total + r' operations on > ports(*):[1-2, 4-5, 7-8, 10-11, 13-14, 16-17, 1... > Took 2423us to perform 1000 'total = total - r' operations on ports(*):[1-2, > 4-5, 7-8, 10-11, 13-14, 16-17, 1... > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2 (1048 > ms) > [ RUN ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/3 > Took 61629us to perform 50000 'total += r' operations on cpus(*):1; > mem(*):128; disk(test)[persistentId:... > Took 98664us to perform 50000 'total -= r' operations on cpus(*):1; > mem(*):128; disk(test)[persistentId:... > Took 348041us to perform 50000 'total = total + r' operations on cpus(*):1; > mem(*):128; disk(test)[persistentId:... > Took 431057us to perform 50000 'total = total - r' operations on cpus(*):1; > mem(*):128; disk(test)[persistentId:... > [ OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/3 (940 ms) > [----------] 4 tests from ResourcesOperators/Resources_BENCHMARK_Test (19826 > ms total) > > [----------] Global test environment tear-down > [==========] 4 tests from 1 test case ran. (19836 ms total) > [ PASSED ] 4 tests. > > > Thanks, > > Anindya Sinha > >
