On Sun, Oct 23, 2016 at 8:20 AM, Ben RUBSON <[email protected]> wrote:
> Hello, > > I found that some buffers that could be L2ARC eligible are not flagged > such, leading to some performance impact. > > As a test I ran the same IO workload 10 times in a raw. > It is a metadata only workload (files listing). > l2arc_noprefetch=0. > > Here is the activity of my pool disks during this test : > http://img15.hostingpics.net/pics/958365941.png > We can see that the first files' listing takes about 3 hours, during which > the SSD cache is fed. > The other runs take about 35 minutes each, read IOs are made from SSD > cache but also from HDD disks, cache is no more fed. > > After some debugging, I found that the buffers which are not L2-backed are > metadata buffers, as the ones which are in the cache, but are not > ARC_FLAG_L2CACHE flagged. > I then quickly modified ZFS code to add ARC_FLAG_L2CACHE to each header, > and here is the result : > http://img15.hostingpics.net/pics/538922792.png > All IOs now come from the cache, leading to a nice performance > improvement. Perfect. > > So I must now understand why some buffers are not ARC_FLAG_L2CACHE flagged. > Sounds good. I noticed that when prefetching (dmu_zfetch(), dmu_prefetch()), we don't set L2CACHE, but if we later do a demand read (via dbuf_read_impl()), it will pass ARC_FLAG_L2CACHE, and arc_read() should add the flag to the arc_buf_hdr_t. So it isn't obvious to me where we are failing to set the flag. > Going deeper into the code, ARC_FLAG_L2CACHE is set by arc_read(), > depending on its arc_flags parameter. > arc_read() is called 22 times in ZFS code, only 2 calls are made with > arc_flags containing ARC_FLAG_L2CACHE. > I modified dbuf.c so that each one of its 4 arc_read() calls use > ARC_FLAG_L2CACHE, it improved the situation. > > My question is then, can't we make all the arc_read() calls use > ARC_FLAG_L2CACHE, when the pool has a L2 cache ? > I assume you mean to set L2CACHE if DBUF_IS_CACHEABLE(). That may be a viable solution but I'd first like to understand what is causing the problem. --matt > > Thank you very much, > > Ben > ------------------------------------------- openzfs-developer Archives: https://www.listbox.com/member/archive/274414/=now RSS Feed: https://www.listbox.com/member/archive/rss/274414/28015062-cce53afa Modify Your Subscription: https://www.listbox.com/member/?member_id=28015062&id_secret=28015062-f966d51c Powered by Listbox: http://www.listbox.com
