> One thing I tried is to not allocate new entries and return NFS4ERR_DELAY in
> the
> hope that the increased refcnt at LRU is temporary. This worked for some time;
> but then I hit a case where I see all the entries at the LRU of L1 has a
> refcnt of 2
> and the subsequent entries have a refcnt of 1. All L2's were empty. I
> realized that
> whenever a new entry is created, the refcnt is 2 and it is put at the LRU.
> Also
> promotions from L2 moves them to LRU of L1. So it is likely that many threads
> may end up finding no entries at LRU and end allocating new entries.
>
> Then I tried another experiment: Invoke lru_wake_thread() when the number of
> entries is greater than entries_hiwat; but still allocate a new entry for the
> current thread. This worked. I had to make a change in lru_run() to allow
> demotion in case of 'entries > entries_hiwat' in addition to max FD check. The
> side effect would be that it will close FDs and demote to L2. Almost all of
> these
> FDs are opened in the context of setattr/getattr; so attributes are already in
> cache and FDs are probably useless until the cache expires. I think your
> idea of
> moving further down the lane may be a better approach.
>
> I will try your suggestion next. With 1023 lanes, it is unlikely that all
> lanes will
> have an active entry.
Thanks for the explanation. You are observing in practice something I
considered in theory...
I like the idea of demoting entries when entries > entries_hiwat, Matt, Daniel,
do you see any negative side effects to that?
About the open fds, my thinking is shifting to not keeping an fd open for
getattr/settattr. If the global fd is already open use it, otherwise, just open
a temp fd for the operation. With NFS v4 clients, that will virtually eliminate
global fd usage and for V3 clients will mean the global fd is only open for
files the client is doing I/O on.
Frank
> Thanks,
> Pradeep
>
> On 4/3/18, Daniel Gryniewicz <d...@redhat.com> wrote:
> > So, the way this is supposed to work is that getting a ref when the
> > ref is 1 is always an LRU_REQ_INITIAL ref, so that moves it to the
> > MRU. At that point, further refs don't move it around in the queue,
> > just increment the refcount. This should be the case, because
> > mdcache_new_entry() and mdcache_find_keyed() both get an INITIAL ref,
> > and all other refs require you to already have a pointer to the entry
> > (and therefore a ref).
> >
> > Can you try something, since you have a reproducer? It seems that,
> > with
> > 1.7 million files, 17 lanes may be a bit low. Can you try with
> > something ridiculously large, like 1023, and see if that makes a
> > difference?
> >
> > I suspect we'll have to add logic to move further down the lanes if
> > futility hits.
> >
> > Daniel
> >
> > On 04/02/2018 12:30 PM, Pradeep wrote:
> >> We discussed this a while ago. I'm running into this again with 2.6.0.
> >> Here is a snapshot of the lru_state (I set the max entries to 10):
> >>
> >> {entries_hiwat = 200000, entries_used = 1772870, chunks_hiwat =
> >> 100000, chunks_used = 16371, lru_reap_l1 = 8116842,
> >> lru_reap_l2 = 1637334, lru_reap_failed = 1637334, attr_from_cache
> >> = 31917512, attr_from_cache_for_client = 5975849,
> >> fds_system_imposed = 1048576, fds_hard_limit = 1038090, fds_hiwat
> >> = 943718, fds_lowat = 524288, futility = 0, per_lane_work = 50,
> >> biggest_window = 419430, prev_fd_count = 0, prev_time =
> >> 1522647830, caching_fds = true}
> >>
> >> As you can see it has grown well beyond the limlt set (1.7 million vs
> >> 200K max size). lru_reap_failed indicates number of times the reap
> >> failed from L1 and L2.
> >> I'm wondering what can cause the reap to fail once it reaches a
> >> steady state. It appears to me that the entry at LRU (head of the
> >> queue) is actually being used (refcnt > 1) and there are entries in
> >> the queue with refcnt == 1. But those are not being looked at. My
> >> understanding is that if an entry is accessed, it must move to MRU
> >> (tail of the queue). Any idea why the entry at LRU can have a refcnt > 1?
> >>
> >> This can happen if the refcnt is incremented without QLOCK and if
> >> lru_reap_impl() is called at the same time from another thread, it
> >> will skip the first entry and return NULL. This was done in
> >> _mdcache_lru_ref() which could cause the refcnt on the head of the
> >> queue to be incremented while some other thread looks at it holding a
> >> QLOCK. I tried moving the increment/dequeue in _mdcache_lru_ref()
> >> inside QLOCK; but that did not help.
> >>
> >> Also if "get_ref()" is called for the entry at the LRU for some
> >> reason, it will just increment refcnt and return. I think the
> >> assumption is that by the time "get_ref() is called, the entry is supposed
> >> to be
> out of LRU.
> >>
> >>
> >> Thanks,
> >> Pradeep
> >>
> >
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
