> On 23 Oct 2016, at 19:39, Ben RUBSON <[email protected]> wrote: > >> On 23 Oct 2016, at 19:24, Matthew Ahrens <[email protected] >> <mailto:[email protected]>> wrote: >> >> On Sun, Oct 23, 2016 at 8:20 AM, Ben RUBSON <[email protected] >> <mailto:[email protected]>> wrote: >> Hello, >> >> I found that some buffers that could be L2ARC eligible are not flagged such, >> leading to some performance impact. >> >> As a test I ran the same IO workload 10 times in a raw. >> It is a metadata only workload (files listing). >> l2arc_noprefetch=0. >> >> Here is the activity of my pool disks during this test : >> http://img15.hostingpics.net/pics/958365941.png >> <http://img15.hostingpics.net/pics/958365941.png> >> We can see that the first files' listing takes about 3 hours, during which >> the SSD cache is fed. >> The other runs take about 35 minutes each, read IOs are made from SSD cache >> but also from HDD disks, cache is no more fed. >> >> After some debugging, I found that the buffers which are not L2-backed are >> metadata buffers, as the ones which are in the cache, but are not >> ARC_FLAG_L2CACHE flagged. >> I then quickly modified ZFS code to add ARC_FLAG_L2CACHE to each header, and >> here is the result : >> http://img15.hostingpics.net/pics/538922792.png >> <http://img15.hostingpics.net/pics/538922792.png> >> All IOs now come from the cache, leading to a nice performance improvement. >> Perfect. >> >> So I must now understand why some buffers are not ARC_FLAG_L2CACHE flagged. >> >> Sounds good. I noticed that when prefetching (dmu_zfetch(), >> dmu_prefetch()), we don't set L2CACHE, but if we later do a demand read (via >> dbuf_read_impl()), it will pass ARC_FLAG_L2CACHE, and arc_read() should add >> the flag to the arc_buf_hdr_t. So it isn't obvious to me where we are >> failing to set the flag. > > I also though that it was due to prefetched buffers, but I also had > no-prefetched buffers in this case.
I made some accounting to get a better idea. After the first files' listing, so from the 2nd to the 10th files' listing, all along this window, ZFS evicts from the concerned pool : - 00574722 (03.8%) buffers !HDR_PREFETCH(hdr) && !HDR_L2CACHE(hdr) && HDR_ISTYPE_METADATA(hdr) ; - 14621913 (96.2%) buffers HDR_PREFETCH(hdr) && !HDR_L2CACHE(hdr) && HDR_ISTYPE_METADATA(hdr) ; To verify that all these prefetch buffers were not useless, I made the same test with vfs.zfs.prefetch_disable=1. I get almost the same results, with of course a slightly different distribution (8.8% / 91.2%). (I discovered in the code that vfs.zfs.prefetch_disable=1 does not disable all prefetch methods, and that the remaining methods prefetch buffers which will be used) OK, so it's strange that we have some buffers which are not prefetch (so are demand ?) and which are not ARC_FLAG_L2CACHE flagged. And why the prefetch ones do not get this flag later on. > And I think that even if we prefetch, we should set L2CACHE, as > l2arc_noprefetch tunable is here to control whether or not we want to have > prefetched buffers L2-backed. > (preference applied in arc_read_done()) > >> Going deeper into the code, ARC_FLAG_L2CACHE is set by arc_read(), depending >> on its arc_flags parameter. >> arc_read() is called 22 times in ZFS code, only 2 calls are made with >> arc_flags containing ARC_FLAG_L2CACHE. >> I modified dbuf.c so that each one of its 4 arc_read() calls use >> ARC_FLAG_L2CACHE, it improved the situation. >> >> My question is then, can't we make all the arc_read() calls use >> ARC_FLAG_L2CACHE, when the pool has a L2 cache ? >> >> I assume you mean to set L2CACHE if DBUF_IS_CACHEABLE(). That may be a >> viable solution but I'd first like to understand what is causing the problem. > > L2 :) Set L2CACHE if DBUF_IS_L2CACHEABLE(), as for the first arc_read() call > in dbuf.c. Matt, how would you use DBUF_IS_L2CACHEABLE() in the 3 other functions where arc_read() is used ? In other words, how would you get a dmu_buf_impl_t var to feed DBUF_IS_L2CACHEABLE() in the following functions of dbuf.c ? - dbuf_issue_final_prefetch - dbuf_prefetch_indirect_done - dbuf_prefetch Other question, to you think we should pay attention to arc_read() calls made in the other files ? - dmu_objset.c (it already uses DMU_OS_IS_L2CACHEABLE so it's OK for this file) - dmu_diff.c - dmu_traverse.c - dsl_scan.c - zil.c Thank you very much ! Ben ------------------------------------------- openzfs-developer Archives: https://www.listbox.com/member/archive/274414/=now RSS Feed: https://www.listbox.com/member/archive/rss/274414/28015062-cce53afa Modify Your Subscription: https://www.listbox.com/member/?member_id=28015062&id_secret=28015062-f966d51c Powered by Listbox: http://www.listbox.com
