Today I am investigating a multi-thread (OpenMP) performance issue in
GraphicsMagick and noticed that it appears to be due to the memory
allocator (default or libumem) provided by Illumos.
It is my distinct impression that libc is calling into libumem in the
default compilation mode.
What I noticed was that if libmtmalloc is used the performance
improves (due to improved concurrency) by almost a factor of 5:
Using the default allocator:
% gm benchmark -duration 10 convert -size 4000x3000 tile:model.pnm -wave 25x150
null:
Results: 40 threads 13 iter 74.63s user 10.122100s total 1.284 iter/s 0.174
iter/cpu
Using libumem:
% LD_PRELOAD_64=libumem.so.1 gm benchmark -duration 10 convert -size 4000x3000
tile:model.pnm -wave 25x150 null:
Results: 40 threads 13 iter 77.28s user 10.226807s total 1.271 iter/s 0.168
iter/cpu
Using mtmalloc:
% LD_PRELOAD_64=libmtmalloc.so.1 gm benchmark -duration 10 convert -size
4000x3000 tile:model.pnm -wave 25x150 null:
Results: 40 threads 64 iter 246.82s user 10.148286s total 6.306 iter/s 0.259
iter/cpu
Is this huge difference in performance due to mtmalloc expected? I
thought that modern libumem was supposed to make up most of the
difference.
These memory allocation statistics were gathered on a Linux system for
a similar test run (but with 24 threads rather than 40):
Memory usage summary: heap total: 13818648999, heap peak: 242173550, stack
peak: 35328
total calls total memory failed calls
malloc| 4445 13818641439 0
realloc| 1 200 0 (nomove:0, dec:0, free:0)
calloc| 23 7360 0
free| 4533 13818557536
Histogram for block sizes:
0-15 360 8% =============
16-31 399 8% ==============
32-47 3 <1%
48-63 58 1% ==
64-79 2 <1%
80-95 2 <1%
96-111 2 <1%
112-127 3 <1%
128-143 236 5% ========
160-175 1 <1%
192-207 60 1% ==
208-223 2 <1%
256-271 514 11% ==================
320-335 23 <1%
384-399 1368 30% ==================================================
432-447 1 <1%
544-559 229 5% ========
768-783 1 <1%
1328-1343 1 <1%
2512-2527 1 <1%
4096-4111 2 <1%
6528-6543 1 <1%
6864-6879 228 5% ========
8192-8207 114 2% ====
8240-8255 57 1% ==
8528-8543 571 12% ====================
32000-32015 57 1% ==
32816-32831 1 <1%
large 172 3% ======
Bob
--
Bob Friesenhahn
[email protected], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
------------------------------------------
illumos: illumos-discuss
Permalink:
https://illumos.topicbox.com/groups/discuss/T30dd2eceb8a069b3-Ma12ef72fb3c962a118edd772
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription
