Hi,
here is a re-spin of the QP buffers memory allocation optimization patches in
which QP buffers are allocated using mmap() regardless of the page size.
Changes V1 -> V2:
----------------
- Use mmap whatever the page size, not only with 64K pages.
libmthca and libmlx4 allocate QP buffers using posix_memalign(), which
results in big memory wastage on architectures with 64K pages.
Replacing posix_memalign() with mmap() allows to fix this (more description
in the patches themselves).
Now, for some numbers, a micro benchmark I wrote shows the heap usage and
the number of mmaped pages used with posix_memalign() and mmap() respectively
for 1000, 2000, up to 8000 QP.
MTHCA
posix_memalign mmap
QP heap mmaped(pages) heap mmaped(pages)
1000 838736 2988 576512 1000
2000 1751216 5973 1161264 2000
3000 2598144 8961 1746016 3000
4000 3510656 11946 2330704 4000
5000 4357616 14934 2915440 5000
6000 5270080 17919 3500176 6000
7000 6117056 20907 4084912 7000
8000 6963968 23895 4669632 8000
MLX4
posix_memalign mmap
QP heap mmaped(pages) heap mmaped(pages)
1000 1469424 2982 1010544 1003
2000 2994048 5958 2010752 2003
3000 4518672 8934 3010960 3003
4000 5969520 11913 4002960 4003
5000 7494176 14889 5003168 5003
6000 8953248 17868 6003376 6003
7000 10477856 20844 7003584 7003
8000 12002496 23820 8003792 8003
This patchset consists in 3 patches:
1. Optimize memory allocation of QP buffers for libmthca
2. Optimize memory allocation of QP buffers for libmlx4
3. Fix the fixes patches for libmlx4 after having applied the
previous patch.
Sebastien Dugue
_______________________________________________
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
