> On 02 Nov 2016, at 22:07, Ben RUBSON <[email protected]> wrote: > >> >> On 23 Oct 2016, at 19:39, Ben RUBSON <[email protected] >> <mailto:[email protected]>> wrote: >> >>> On 23 Oct 2016, at 19:24, Matthew Ahrens <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> On Sun, Oct 23, 2016 at 8:20 AM, Ben RUBSON <[email protected] >>> <mailto:[email protected]>> wrote: >>> Hello, >>> >>> I found that some buffers that could be L2ARC eligible are not flagged >>> such, leading to some performance impact. >>> >>> As a test I ran the same IO workload 10 times in a raw. >>> It is a metadata only workload (files listing). >>> l2arc_noprefetch=0. >>> >>> Here is the activity of my pool disks during this test : >>> http://img15.hostingpics.net/pics/958365941.png >>> <http://img15.hostingpics.net/pics/958365941.png> >>> We can see that the first files' listing takes about 3 hours, during which >>> the SSD cache is fed. >>> The other runs take about 35 minutes each, read IOs are made from SSD cache >>> but also from HDD disks, cache is no more fed. >>> >>> After some debugging, I found that the buffers which are not L2-backed are >>> metadata buffers, as the ones which are in the cache, but are not >>> ARC_FLAG_L2CACHE flagged. >>> I then quickly modified ZFS code to add ARC_FLAG_L2CACHE to each header, >>> and here is the result : >>> http://img15.hostingpics.net/pics/538922792.png >>> <http://img15.hostingpics.net/pics/538922792.png> >>> All IOs now come from the cache, leading to a nice performance improvement. >>> Perfect. >>> >>> So I must now understand why some buffers are not ARC_FLAG_L2CACHE flagged. >>> >>> Sounds good. I noticed that when prefetching (dmu_zfetch(), >>> dmu_prefetch()), we don't set L2CACHE, but if we later do a demand read >>> (via dbuf_read_impl()), it will pass ARC_FLAG_L2CACHE, and arc_read() >>> should add the flag to the arc_buf_hdr_t. So it isn't obvious to me where >>> we are failing to set the flag. >> >> I also though that it was due to prefetched buffers, but I also had >> no-prefetched buffers in this case. > > I made some accounting to get a better idea. > After the first files' listing, so from the 2nd to the 10th files' listing, > all along this window, ZFS evicts from the concerned pool : > - 00574722 (03.8%) buffers !HDR_PREFETCH(hdr) && !HDR_L2CACHE(hdr) && > HDR_ISTYPE_METADATA(hdr) ; > - 14621913 (96.2%) buffers HDR_PREFETCH(hdr) && !HDR_L2CACHE(hdr) && > HDR_ISTYPE_METADATA(hdr) ; > > To verify that all these prefetch buffers were not useless, I made the same > test with vfs.zfs.prefetch_disable=1. > I get almost the same results, with of course a slightly different > distribution (8.8% / 91.2%). > (I discovered in the code that vfs.zfs.prefetch_disable=1 does not disable > all prefetch methods, and that the remaining methods prefetch buffers which > will be used) > > OK, so it's strange that we have some buffers which are not prefetch (so are > demand ?) and which are not ARC_FLAG_L2CACHE flagged. > And why the prefetch ones do not get this flag later on. > >> And I think that even if we prefetch, we should set L2CACHE, as >> l2arc_noprefetch tunable is here to control whether or not we want to have >> prefetched buffers L2-backed. >> (preference applied in arc_read_done()) >> >>> Going deeper into the code, ARC_FLAG_L2CACHE is set by arc_read(), >>> depending on its arc_flags parameter. >>> arc_read() is called 22 times in ZFS code, only 2 calls are made with >>> arc_flags containing ARC_FLAG_L2CACHE. >>> I modified dbuf.c so that each one of its 4 arc_read() calls use >>> ARC_FLAG_L2CACHE, it improved the situation. >>> >>> My question is then, can't we make all the arc_read() calls use >>> ARC_FLAG_L2CACHE, when the pool has a L2 cache ? >>> >>> I assume you mean to set L2CACHE if DBUF_IS_CACHEABLE(). That may be a >>> viable solution but I'd first like to understand what is causing the >>> problem. >> >> L2 :) Set L2CACHE if DBUF_IS_L2CACHEABLE(), as for the first arc_read() call >> in dbuf.c. > > Matt, how would you use DBUF_IS_L2CACHEABLE() in the 3 other functions where > arc_read() is used ? > In other words, how would you get a dmu_buf_impl_t var to feed > DBUF_IS_L2CACHEABLE() in the following functions of dbuf.c ? > - dbuf_issue_final_prefetch > - dbuf_prefetch_indirect_done > - dbuf_prefetch
I just worked on this and opened a PR : https://github.com/openzfs/openzfs/pull/222 Discussion can go-on there. Thx ! > Other question, to you think we should pay attention to arc_read() calls made > in the other files ? > - dmu_objset.c (it already uses DMU_OS_IS_L2CACHEABLE so it's OK for this > file) > - dmu_diff.c > - dmu_traverse.c > - dsl_scan.c > - zil.c ------------------------------------------- openzfs-developer Archives: https://www.listbox.com/member/archive/274414/=now RSS Feed: https://www.listbox.com/member/archive/rss/274414/28015062-cce53afa Modify Your Subscription: https://www.listbox.com/member/?member_id=28015062&id_secret=28015062-f966d51c Powered by Listbox: http://www.listbox.com
