yes, this is one of the things I thought we might parameterise On Fri, Aug 11, 2017 at 11:52 AM, Daniel Gryniewicz <d...@redhat.com> wrote: > Right, this is reaping. I was thinking it was the lane thread. Reaping > only looks at the single LRU of each queue. We should probably look at some > small number of each lane, like 2 or 3. > > Frank, this, in combination with the PIN lane, it probably the issue. > > Daniel > > On 08/11/2017 11:21 AM, Pradeep wrote: >> >> Hi Daniel, >> >> I'm testing with 2.5.1. I haven't changed those parameters. Those >> parameters only affect once you are in lru_run_lane(), right? Since the FDs >> are lower than low-watermark, it never calls lru_run_lane(). >> >> Thanks, >> Pradeep >> >> On Fri, Aug 11, 2017 at 5:43 AM, Daniel Gryniewicz <d...@redhat.com >> <mailto:d...@redhat.com>> wrote: >> >> Have you set Reaper_Work? Have you changed LRU_N_Q_LANES? (and >> which version of Ganesha?) >> >> Daniel >> >> On 08/10/2017 07:12 PM, Pradeep wrote: >> >> Debugged this a little more. It appears that the entries that >> can be reaped are not at the LRU position (head) of the L1 >> queue. So those can be free'd later by lru_run(). I don't see it >> happening either for some reason. >> >> (gdb) p LRU[1].L1 >> $29 = {q = {next = 0x7fb459e71960, prev = 0x7fb3ec3c0d30}, id = >> LRU_ENTRY_L1, size = 260379} >> >> head of the list is an entry with refcnt 2; but there are >> several entries with refcnt 1. >> >> (gdb) p *(mdcache_lru_t *)0x7fb459e71960 >> $30 = {q = {next = 0x7fb43ddea8a0, prev = 0x7d68a0 <LRU+224>}, >> qid = LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 1, cf = 2} >> (gdb) p *(mdcache_lru_t *)0x7fb43ddea8a0 >> $31 = {q = {next = 0x7fb3f041f9a0, prev = 0x7fb459e71960}, qid = >> LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 1, cf = 0} >> (gdb) p *(mdcache_lru_t *)0x7fb3f041f9a0 >> $32 = {q = {next = 0x7fb466960200, prev = 0x7fb43ddea8a0}, qid = >> LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 1, cf = 0} >> (gdb) p *(mdcache_lru_t *)0x7fb466960200 >> $33 = {q = {next = 0x7fb451e20570, prev = 0x7fb3f041f9a0}, qid = >> LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 1, cf = 1} >> >> The entries with refcnt 1 are moved to L2 by the background >> thread (lru_run). However it does it only of the open file count >> is greater than low water mark. In my case, the open_fd_count is >> not high; so lru_run() doesn't call lru_run_lane() to demote >> those entries to L2. What is the best approach to handle this >> scenario? >> >> Thanks, >> Pradeep >> >> >> >> On Mon, Aug 7, 2017 at 6:08 AM, Daniel Gryniewicz >> <d...@redhat.com <mailto:d...@redhat.com> >> <mailto:d...@redhat.com <mailto:d...@redhat.com>>> wrote: >> >> It never has been. In cache_inode, a pin-ref kept it from >> being >> reaped, now any ref beyond 1 keeps it. >> >> On Fri, Aug 4, 2017 at 1:31 PM, Frank Filz >> <ffilz...@mindspring.com <mailto:ffilz...@mindspring.com> >> <mailto:ffilz...@mindspring.com >> >> <mailto:ffilz...@mindspring.com>>> wrote: >> >> I'm hitting a case where mdcache keeps growing well >> beyond the >> high water >> >> mark. Here is a snapshot of the lru_state: >> >> >> >> 1 = {entries_hiwat = 100000, entries_used = 2306063, >> chunks_hiwat = >> > 100000, >> >> chunks_used = 16462, >> >> >> >> It has grown to 2.3 million entries and each entry is >> ~1.6K. >> >> >> >> I looked at the first entry in lane 0, L1 queue: >> >> >> >> (gdb) p LRU[0].L1 >> >> $9 = {q = {next = 0x7fad64256f00, prev = >> 0x7faf21a1bc00}, id = >> >> LRU_ENTRY_L1, size = 254628} >> >> (gdb) p (mdcache_entry_t *)(0x7fad64256f00-1024) >> >> $10 = (mdcache_entry_t *) 0x7fad64256b00 >> >> (gdb) p $10->lru >> >> $11 = {q = {next = 0x7fad65ea0f00, prev = 0x7d67c0 >> <LRU>}, qid = >> >> LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 0, cf = 0} >> >> (gdb) p $10->fh_hk.inavl >> >> $13 = true >> > >> > The refcount 2 prevents reaping. >> > >> > There could be a refcount leak. >> > >> > Hmm, though, I thought the entries_hwmark was a hard >> limit, guess >> not... >> > >> > Frank >> > >> >> Lane 1: >> >> (gdb) p LRU[1].L1 >> >> $18 = {q = {next = 0x7fad625c0300, prev = >> 0x7faec08c5100}, id = >> >> LRU_ENTRY_L1, size = 253006} >> >> (gdb) p (mdcache_entry_t *)(0x7fad625c0300 - 1024) >> >> $21 = (mdcache_entry_t *) 0x7fad625bff00 >> >> (gdb) p $21->lru >> >> $22 = {q = {next = 0x7fad66fce600, prev = 0x7d68a0 >> <LRU+224>}, qid = >> >> LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 1, cf = 1} >> >> >> >> (gdb) p $21->fh_hk.inavl >> >> $24 = true >> >> >> >> As per LRU_ENTRY_RECLAIMABLE(), these entry should be >> reclaimable. Not >> >> sure why it is not able to claim it. Any ideas? >> >> >> >> Thanks, >> >> Pradeep >> >> >> >> >> > >> >> ---------------------------------------------------------------------------- >> > -- >> >> Check out the vibrant tech community on one of the >> world's most >> engaging >> >> tech sites, Slashdot.org! http://sdm.link/slashdot >> >> _______________________________________________ >> >> Nfs-ganesha-devel mailing list >> >> Nfs-ganesha-devel@lists.sourceforge.net >> <mailto:Nfs-ganesha-devel@lists.sourceforge.net> >> <mailto:Nfs-ganesha-devel@lists.sourceforge.net >> <mailto:Nfs-ganesha-devel@lists.sourceforge.net>> >> >> >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >> <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel> >> >> <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >> <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel>> >> > >> > >> > --- >> > This email has been checked for viruses by Avast >> antivirus software. >> > https://www.avast.com/antivirus >> <https://www.avast.com/antivirus> >> <https://www.avast.com/antivirus >> <https://www.avast.com/antivirus>> >> > >> > >> > >> >> ------------------------------------------------------------------------------ >> > Check out the vibrant tech community on one of the >> world's most >> > engaging tech sites, Slashdot.org! >> http://sdm.link/slashdot >> > _______________________________________________ >> > Nfs-ganesha-devel mailing list >> > Nfs-ganesha-devel@lists.sourceforge.net >> <mailto:Nfs-ganesha-devel@lists.sourceforge.net> >> <mailto:Nfs-ganesha-devel@lists.sourceforge.net >> <mailto:Nfs-ganesha-devel@lists.sourceforge.net>> >> > >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >> <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel> >> >> <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >> <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel>> >> >> >> >> > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
