> On 23 Oct 2016, at 19:24, Matthew Ahrens <[email protected]> wrote: > > On Sun, Oct 23, 2016 at 8:20 AM, Ben RUBSON <[email protected] > <mailto:[email protected]>> wrote: > Hello, > > I found that some buffers that could be L2ARC eligible are not flagged such, > leading to some performance impact. > > As a test I ran the same IO workload 10 times in a raw. > It is a metadata only workload (files listing). > l2arc_noprefetch=0. > > Here is the activity of my pool disks during this test : > http://img15.hostingpics.net/pics/958365941.png > <http://img15.hostingpics.net/pics/958365941.png> > We can see that the first files' listing takes about 3 hours, during which > the SSD cache is fed. > The other runs take about 35 minutes each, read IOs are made from SSD cache > but also from HDD disks, cache is no more fed. > > After some debugging, I found that the buffers which are not L2-backed are > metadata buffers, as the ones which are in the cache, but are not > ARC_FLAG_L2CACHE flagged. > I then quickly modified ZFS code to add ARC_FLAG_L2CACHE to each header, and > here is the result : > http://img15.hostingpics.net/pics/538922792.png > <http://img15.hostingpics.net/pics/538922792.png> > All IOs now come from the cache, leading to a nice performance improvement. > Perfect. > > So I must now understand why some buffers are not ARC_FLAG_L2CACHE flagged. > > Sounds good. I noticed that when prefetching (dmu_zfetch(), dmu_prefetch()), > we don't set L2CACHE, but if we later do a demand read (via > dbuf_read_impl()), it will pass ARC_FLAG_L2CACHE, and arc_read() should add > the flag to the arc_buf_hdr_t. So it isn't obvious to me where we are > failing to set the flag.
I also though that it was due to prefetched buffers, but I also had no-prefetched buffers in this case. And I think that even if we prefetch, we should set L2CACHE, as l2arc_noprefetch tunable is here to control whether or not we want to have prefetched buffers L2-backed. (preference applied in arc_read_done()) > Going deeper into the code, ARC_FLAG_L2CACHE is set by arc_read(), depending > on its arc_flags parameter. > arc_read() is called 22 times in ZFS code, only 2 calls are made with > arc_flags containing ARC_FLAG_L2CACHE. > I modified dbuf.c so that each one of its 4 arc_read() calls use > ARC_FLAG_L2CACHE, it improved the situation. > > My question is then, can't we make all the arc_read() calls use > ARC_FLAG_L2CACHE, when the pool has a L2 cache ? > > I assume you mean to set L2CACHE if DBUF_IS_CACHEABLE(). That may be a > viable solution but I'd first like to understand what is causing the problem. L2 :) Set L2CACHE if DBUF_IS_L2CACHEABLE(), as for the first arc_read() call in dbuf.c. > --matt Thank you for your feedback Matt ! > Thank you very much, > > Ben ------------------------------------------- openzfs-developer Archives: https://www.listbox.com/member/archive/274414/=now RSS Feed: https://www.listbox.com/member/archive/rss/274414/28015062-cce53afa Modify Your Subscription: https://www.listbox.com/member/?member_id=28015062&id_secret=28015062-f966d51c Powered by Listbox: http://www.listbox.com
