-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18758/
-----------------------------------------------------------
Review request for mesos, Benjamin Hindman and Vinod Kone.
Bugs: MESOS-764
https://issues.apache.org/jira/browse/MESOS-764
Repository: mesos-git
Description
-------
The bottleneck in the Registrar is the copying of the Registry. For a large
number of slaves and operations, this becomes prohibitively expensive. This
patch therefore eliminates copying when applying the operations. Optimizations
to Future also eliminated copying in _update.
Further optimizations will be explored for larger clusters, this was currently
tested with 10,000 slaves.
Diffs
-----
src/master/registrar.cpp 37337c07b24a96e71910b7c83085d159361a1188
Diff: https://reviews.apache.org/r/18758/diff/
Testing
-------
Prior to this patch and the Future::get optimizations, the performance
benchmark I wrote took over an hour for 10,000 slaves.
With these optimizations 10,000 slaves can be handled quite efficiently:
Key numbers with gcc 4.1.2 on a 2.4GHz 12 core machine:
I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in
10.04498586secs
I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves
in 9.829011689secs
I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in
22.715892ms
I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in
3.577429332secs
[ RUN ] RegistrarTest.DISABLED_performance
I0304 23:36:55.685456 33261 registrar.cpp:218] Recovering registrar
I0304 23:36:55.801048 33252 registrar.cpp:381] Attempting to update the
'registry'
Applied 1 operations in: 1.235673ms
Stored Registry in: 36.199255ms
I0304 23:36:55.837477 33271 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 66813ns
I0304 23:36:55.972564 33266 registrar.cpp:381] Attempting to update the
'registry'
Applied 1 operations in: 1.573455ms
Stored Registry in: 30.215382ms
I0304 23:36:56.002907 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 113129ns
I0304 23:36:56.003094 33266 registrar.cpp:381] Attempting to update the
'registry'
Applied 146 operations in: 8.579203ms
Stored Registry in: 174.710644ms
I0304 23:36:56.179323 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 146 promises in: 4.656175ms
I0304 23:36:56.184154 33266 registrar.cpp:381] Attempting to update the
'registry'
Applied 574 operations in: 73.828182ms
Stored Registry in: 527.540062ms
I0304 23:36:56.744689 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 574 promises in: 13.873625ms
I0304 23:36:56.760572 33266 registrar.cpp:381] Attempting to update the
'registry'
Applied 2400 operations in: 907.070761ms
Stored Registry in: 268.604653ms
I0304 23:36:57.778074 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 2400 promises in: 69.867013ms
I0304 23:36:57.858157 33266 registrar.cpp:381] Attempting to update the
'registry'
Applied 6879 operations in: 7.910334133secs
Stored Registry in: 990.759579ms
I0304 23:37:05.795042 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 6879 promises in: 221.701612ms
I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in
10.04498586secs
I0304 23:37:06.125613 33266 registrar.cpp:381] Attempting to update the
'registry'
Applied 1 operations in: 852.076566ms
Stored Registry in: 1.384153523secs
I0304 23:37:07.576215 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 191.668241ms
I0304 23:37:07.768013 33266 registrar.cpp:381] Attempting to update the
'registry'
Applied 993 operations in: 923.901654ms
Stored Registry in: 953.378897ms
I0304 23:37:08.827229 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 993 promises in: 164.437762ms
I0304 23:37:08.992169 33266 registrar.cpp:381] Attempting to update the
'registry'
Applied 9006 operations in: 6.081593589secs
Stored Registry in: 896.142182ms
I0304 23:37:15.397238 33257 registrar.cpp:425] Successfully updated 'registry'
Transitioned 9006 promises in: 427.267246ms
I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves
in 9.829011689secs
I0304 23:37:15.876085 33257 registrar.cpp:218] Recovering registrar
I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in
22.715892ms
I0304 23:37:15.876656 33264 registrar.cpp:381] Attempting to update the
'registry'
Applied 1 operations in: 819.303422ms
Stored Registry in: 1.383689258secs
I0304 23:37:17.329473 33264 registrar.cpp:425] Successfully updated
'registry'
Transitioned 1 promises in: 189.505006ms
I0304 23:37:17.541276 33264 registrar.cpp:381] Attempting to update the
'registry'
Applied 1 operations in: 921.761221ms
Stored Registry in: 986.688995ms
I0304 23:37:18.588706 33264 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 179.771104ms
I0304 23:37:18.768606 33264 registrar.cpp:381] Attempting to update the
'registry'
Applied 9998 operations in: 441.652539ms
Stored Registry in: 209.866548ms
I0304 23:37:19.298931 33265 registrar.cpp:425] Successfully updated 'registry'
Transitioned 9998 promises in: 154.775647ms
I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in
3.577429332secs
[ OK ] RegistrarTest.DISABLED_performance (23791 ms)
Thanks,
Ben Mahler
