on 14/01/2014 18:47 Matthew Ahrens said the following: > > > On Tue, Jan 14, 2014 at 6:44 AM, Andriy Gapon <[email protected] > <mailto:[email protected]>> wrote: > > > There is a FreeBSD mailing list report that seems to point to a potential > issue > with buffers used for L2ARC compression. > > This message in the thread should explain my suspicions: > http://thread.gmane.org/gmane.os.freebsd.current/155757/focus=155781 > > As I mentioned in that thread, the data buffers allocated for L2ARC > compression > are not accounted in ARC consumption stats nor via any other counter, so > it is > not easy to monitor how much memory is used by them. > >From a quick look at the code I can not see how the buffers could be > leaked or > kept in prolonged usage. On the other hand, the buffers are tied to l2arc > header objects and arc header objects, which are placed on a special list > of an > l2arc device, so I do not preclude a possibility of some complex behavior > arising there. I do not understand the code well enough, so I can only > vague > suspicions about possible interaction with ARC buffer eviction / release. > > In any case, it would be nice to help the user having the problem to > diagnose > it. I am quite unfamiliar with the code, so it's hard for me to come up > with > any suggestions. > > > It sounds like you suspect that b_tmp_cdata is not being freed.
Yes. > While the l2arc > is implicated (problem went away when turning it off), it doesn't seem that we > have a smoking gun pointing to the allocation from l2arc_compress_buf() just > yet. (It did a lot of allocations, but it may have also done a lot of frees.) Well, for me a smoking gun was lots of data in zio_data_buf_131072, much more than the total ARC size. I checked the code and all allocations from zio_data_buf_* also update ARC size stats. All except b_tmp_cdata allocation, that is. > You could check if there are a lot of l2arc_buf_hdr_t's with b_tmp_cdata set. > If not, does FreeBSD have something like ::findleaks? (On illumos, it's part > of > MDB. It finds allocations with no pointers to them, based on inferred type > information.) Unfortunately I don't think that examining _all_ l2arc_buf_hdr_t's is easy on FreeBSD and there is nothing like ::findleaks too :-( > P.S. As I was finishing writing this I noticed that > l2arc_do_free_on_write() > does not seem to handle the temporary compression buffers. Not sure if > this > could be a problem. > > > It looks like l2arc_write_done() calls l2arc_release_cdata_buf() before > calling > l2arc_release_cdata_buf(), so I don't think this would be a problem. I was worried about buffers that get put on l2arc_free_on_write list in arc_buf_data_free() in the case when an L2 write is still ongoing. More specifically, it seems that arc_hdr_destroy() would remove an arc_buf_hdr_t object from l2ad_buflist if it's there before calling arc_buf_data_free(). And thus l2arc_write_done() won't see that buffer. There are also a couple of other places in the code where a arc_buf_hdr_t object can be removed from l2ad_buflist: arc_release and l2arc_evict. But for me it is hard to say whether that could lead to a leak of b_tmp_cdata. -- Andriy Gapon _______________________________________________ developer mailing list [email protected] http://lists.open-zfs.org/mailman/listinfo/developer
