Hello.
As a result of today's community meeting I start dedicated ML thread for
gathering memory management issues together to make it possible to
summarize them and construct some plan what to do next.
Very important notice: I'm not the active GlusterFS developer, but
gained excessive experience with GlusterFS in the past at previous work,
and the main issue that was chasing me all the time was memory leaking.
Consider this as a request for action from GlusterFS customer,
apparently approved by Kaushal and Amye during last meeting :).
So, here go key points.
1) Almost all nasty and obvious memory leaks have been successfully
fixed during the last year, and that allowed me to run GlusterFS in
production at previous work for almost all types of workload except one
— dovecot mail storage. The specific of this workload is that it
involved huge amount of files, and I assume this to be kinda of edge
case unhiding some dark corners of GlusterFS memory management. I was
able to provide Nithya with Valgrind+Massif memory profiling results and
test case, and that helped her to prepare at least 1 extra fix (and more
to come AFAIK), which has some deal with readdirp-related code.
Nevertheless, it is reported that this is not the major source of
leaking. Nithya suspect that memory gets fragmented heavily due to lots
of small allocations, and memory pools cannot cope with this kind of
fragmentation under constant load.
Related BZs:
* https://bugzilla.redhat.com/show_bug.cgi?id=1369364
* https://bugzilla.redhat.com/show_bug.cgi?id=1380249
People involved:
* nbalacha, could you please provide more info on your findings?
2) Meanwhile, Jeff goes on with brick multiplexing feature, facing some
issue with memory management too and blaming memory pools for that.
Related ML email:
*
http://www.gluster.org/pipermail/gluster-devel/2016-October/051118.html
*
http://www.gluster.org/pipermail/gluster-devel/2016-October/051160.html
People involved:
* jdarcy, have you discussed this outside of ML? It seems your email
didn't get proper attention.
3) We had brief discussion with obnox and anoopcs on #gluster-meeting
and #gluster-dev regarding jemalloc and talloc. obnox believes that we
may use both, jemalloc for substituting malloc/free, talloc for
rewriting memory management for GlusterFS properly.
Related logs:
*
https://botbot.me/freenode/gluster-dev/2016-10-26/?msg=75501394&page=2
People involved:
* obnox, could you share your ideas on this?
To summarize:
1) we need key devs involved in memory management to share their ideas;
2) using production-proven memory allocators and memory pools
implementation is desired;
3) someone should manage the workflow of reconstructing memory
management.
Feel free to add anything I've missed.
Regards,
Oleksandr
_______________________________________________
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
