> Right, this is reaping. I was thinking it was the lane thread. Reaping only > looks at the single LRU of each queue. We should probably look at some > small number of each lane, like 2 or 3. > > Frank, this, in combination with the PIN lane, it probably the issue.
Yea, that would be a problem, also, consider that v4 will have much less use of global fd under support_ex, and thus the fewer open files to trigger less lru_run_lane events. One option would be to promote anything that got state on it to MRU, persistent state should (mostly) only happen if the object is going to be referenced in subsequent NFS requests anyway (the one exception would be "touch" to create a file that is done in a single compound (I don't know if the client actually succeeds in optimizing that into a single compound). Also, the vector operations those research folks have been working on would create a scenario where a file is accessed with state and done in a single compound, which may not deserve promotion to MRU of L1 (but one way to handle that is check on the put ref that decrements to refcount 1 if at that point there is state on the file, if so, promote to MRU of L1). Other than objects with persistent state, we SHOULD only have #threads * 2 (or maybe 3 at most) active references, so as long as we set up the number of lanes and everything right, we should still be able to reap. Frank > Daniel > > On 08/11/2017 11:21 AM, Pradeep wrote: > > Hi Daniel, > > > > I'm testing with 2.5.1. I haven't changed those parameters. Those > > parameters only affect once you are in lru_run_lane(), right? Since > > the FDs are lower than low-watermark, it never calls lru_run_lane(). > > > > Thanks, > > Pradeep > > > > On Fri, Aug 11, 2017 at 5:43 AM, Daniel Gryniewicz <d...@redhat.com > > <mailto:d...@redhat.com>> wrote: > > > > Have you set Reaper_Work? Have you changed LRU_N_Q_LANES? (and > > which version of Ganesha?) > > > > Daniel > > > > On 08/10/2017 07:12 PM, Pradeep wrote: > > > > Debugged this a little more. It appears that the entries that > > can be reaped are not at the LRU position (head) of the L1 > > queue. So those can be free'd later by lru_run(). I don't see it > > happening either for some reason. > > > > (gdb) p LRU[1].L1 > > $29 = {q = {next = 0x7fb459e71960, prev = 0x7fb3ec3c0d30}, id = > > LRU_ENTRY_L1, size = 260379} > > > > head of the list is an entry with refcnt 2; but there are > > several entries with refcnt 1. > > > > (gdb) p *(mdcache_lru_t *)0x7fb459e71960 > > $30 = {q = {next = 0x7fb43ddea8a0, prev = 0x7d68a0 <LRU+224>}, > > qid = LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 1, cf = 2} > > (gdb) p *(mdcache_lru_t *)0x7fb43ddea8a0 > > $31 = {q = {next = 0x7fb3f041f9a0, prev = 0x7fb459e71960}, qid = > > LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 1, cf = 0} > > (gdb) p *(mdcache_lru_t *)0x7fb3f041f9a0 > > $32 = {q = {next = 0x7fb466960200, prev = 0x7fb43ddea8a0}, qid = > > LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 1, cf = 0} > > (gdb) p *(mdcache_lru_t *)0x7fb466960200 > > $33 = {q = {next = 0x7fb451e20570, prev = 0x7fb3f041f9a0}, qid = > > LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 1, cf = 1} > > > > The entries with refcnt 1 are moved to L2 by the background > > thread (lru_run). However it does it only of the open file count > > is greater than low water mark. In my case, the open_fd_count is > > not high; so lru_run() doesn't call lru_run_lane() to demote > > those entries to L2. What is the best approach to handle this > > scenario? > > > > Thanks, > > Pradeep > > > > > > > > On Mon, Aug 7, 2017 at 6:08 AM, Daniel Gryniewicz > > <d...@redhat.com <mailto:d...@redhat.com> > > <mailto:d...@redhat.com <mailto:d...@redhat.com>>> wrote: > > > > It never has been. In cache_inode, a pin-ref kept it from > > being > > reaped, now any ref beyond 1 keeps it. > > > > On Fri, Aug 4, 2017 at 1:31 PM, Frank Filz > > <ffilz...@mindspring.com <mailto:ffilz...@mindspring.com> > > <mailto:ffilz...@mindspring.com > > <mailto:ffilz...@mindspring.com>>> wrote: > > >> I'm hitting a case where mdcache keeps growing well > > beyond the > > high water > > >> mark. Here is a snapshot of the lru_state: > > >> > > >> 1 = {entries_hiwat = 100000, entries_used = 2306063, > > chunks_hiwat = > > > 100000, > > >> chunks_used = 16462, > > >> > > >> It has grown to 2.3 million entries and each entry is > > ~1.6K. > > >> > > >> I looked at the first entry in lane 0, L1 queue: > > >> > > >> (gdb) p LRU[0].L1 > > >> $9 = {q = {next = 0x7fad64256f00, prev = > > 0x7faf21a1bc00}, id = > > >> LRU_ENTRY_L1, size = 254628} > > >> (gdb) p (mdcache_entry_t *)(0x7fad64256f00-1024) > > >> $10 = (mdcache_entry_t *) 0x7fad64256b00 > > >> (gdb) p $10->lru > > >> $11 = {q = {next = 0x7fad65ea0f00, prev = 0x7d67c0 > > <LRU>}, qid = > > >> LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 0, cf = 0} > > >> (gdb) p $10->fh_hk.inavl > > >> $13 = true > > > > > > The refcount 2 prevents reaping. > > > > > > There could be a refcount leak. > > > > > > Hmm, though, I thought the entries_hwmark was a hard > > limit, guess > > not... > > > > > > Frank > > > > > >> Lane 1: > > >> (gdb) p LRU[1].L1 > > >> $18 = {q = {next = 0x7fad625c0300, prev = > > 0x7faec08c5100}, id = > > >> LRU_ENTRY_L1, size = 253006} > > >> (gdb) p (mdcache_entry_t *)(0x7fad625c0300 - 1024) > > >> $21 = (mdcache_entry_t *) 0x7fad625bff00 > > >> (gdb) p $21->lru > > >> $22 = {q = {next = 0x7fad66fce600, prev = 0x7d68a0 > > <LRU+224>}, qid = > > >> LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 1, cf = 1} > > >> > > >> (gdb) p $21->fh_hk.inavl > > >> $24 = true > > >> > > >> As per LRU_ENTRY_RECLAIMABLE(), these entry should be > > reclaimable. Not > > >> sure why it is not able to claim it. Any ideas? > > >> > > >> Thanks, > > >> Pradeep > > >> > > >> > > > > > > > > > ---------------------------------------------------------------------------- > > > -- > > >> Check out the vibrant tech community on one of the > > world's most > > engaging > > >> tech sites, Slashdot.org! http://sdm.link/slashdot > > >> _______________________________________________ > > >> Nfs-ganesha-devel mailing list > > >> Nfs-ganesha-devel@lists.sourceforge.net > > <mailto:Nfs-ganesha-devel@lists.sourceforge.net> > > <mailto:Nfs-ganesha-devel@lists.sourceforge.net > > <mailto:Nfs-ganesha-devel@lists.sourceforge.net>> > > >> > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel> > > > > <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel>> > > > > > > > > > --- > > > This email has been checked for viruses by Avast > > antivirus software. > > > https://www.avast.com/antivirus > > <https://www.avast.com/antivirus> > > <https://www.avast.com/antivirus > <https://www.avast.com/antivirus>> > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > Check out the vibrant tech community on one of the > > world's most > > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > _______________________________________________ > > > Nfs-ganesha-devel mailing list > > > Nfs-ganesha-devel@lists.sourceforge.net > > <mailto:Nfs-ganesha-devel@lists.sourceforge.net> > > <mailto:Nfs-ganesha-devel@lists.sourceforge.net > > <mailto:Nfs-ganesha-devel@lists.sourceforge.net>> > > > > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel> > > > > <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel>> > > > > > > > > --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
