Right, this is reaping. I was thinking it was the lane thread. Reaping only looks at the single LRU of each queue. We should probably look at some small number of each lane, like 2 or 3.

Frank, this, in combination with the PIN lane, it probably the issue.

Daniel

On 08/11/2017 11:21 AM, Pradeep wrote:
Hi Daniel,

I'm testing with 2.5.1. I haven't changed those parameters. Those parameters only affect once you are in lru_run_lane(), right? Since the FDs are lower than low-watermark, it never calls lru_run_lane().

Thanks,
Pradeep

On Fri, Aug 11, 2017 at 5:43 AM, Daniel Gryniewicz <d...@redhat.com <mailto:d...@redhat.com>> wrote:

    Have you set Reaper_Work?  Have you changed LRU_N_Q_LANES?  (and
    which version of Ganesha?)

    Daniel

    On 08/10/2017 07:12 PM, Pradeep wrote:

        Debugged this a little more. It appears that the entries that
        can be reaped are not at the LRU position (head) of the L1
        queue. So those can be free'd later by lru_run(). I don't see it
        happening either for some reason.

        (gdb) p LRU[1].L1
        $29 = {q = {next = 0x7fb459e71960, prev = 0x7fb3ec3c0d30}, id =
        LRU_ENTRY_L1, size = 260379}

        head of the list is an entry with refcnt 2; but there are
        several entries with refcnt 1.

        (gdb) p *(mdcache_lru_t *)0x7fb459e71960
        $30 = {q = {next = 0x7fb43ddea8a0, prev = 0x7d68a0 <LRU+224>},
        qid = LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 1, cf = 2}
        (gdb) p *(mdcache_lru_t *)0x7fb43ddea8a0
        $31 = {q = {next = 0x7fb3f041f9a0, prev = 0x7fb459e71960}, qid =
        LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 1, cf = 0}
        (gdb) p *(mdcache_lru_t *)0x7fb3f041f9a0
        $32 = {q = {next = 0x7fb466960200, prev = 0x7fb43ddea8a0}, qid =
        LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 1, cf = 0}
        (gdb) p *(mdcache_lru_t *)0x7fb466960200
        $33 = {q = {next = 0x7fb451e20570, prev = 0x7fb3f041f9a0}, qid =
        LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 1, cf = 1}

        The entries with refcnt 1 are moved to L2 by the background
        thread (lru_run). However it does it only of the open file count
        is greater than low water mark. In my case, the open_fd_count is
        not high; so lru_run() doesn't call lru_run_lane() to demote
        those entries to L2. What is the best approach to handle this
        scenario?

        Thanks,
        Pradeep



        On Mon, Aug 7, 2017 at 6:08 AM, Daniel Gryniewicz
        <d...@redhat.com <mailto:d...@redhat.com>
        <mailto:d...@redhat.com <mailto:d...@redhat.com>>> wrote:

             It never has been.  In cache_inode, a pin-ref kept it from
        being
             reaped, now any ref beyond 1 keeps it.

             On Fri, Aug 4, 2017 at 1:31 PM, Frank Filz
        <ffilz...@mindspring.com <mailto:ffilz...@mindspring.com>
             <mailto:ffilz...@mindspring.com
        <mailto:ffilz...@mindspring.com>>> wrote:
              >> I'm hitting a case where mdcache keeps growing well
        beyond the
             high water
              >> mark. Here is a snapshot of the lru_state:
              >>
              >> 1 = {entries_hiwat = 100000, entries_used = 2306063,
        chunks_hiwat =
              > 100000,
              >> chunks_used = 16462,
              >>
              >> It has grown to 2.3 million entries and each entry is
        ~1.6K.
              >>
              >> I looked at the first entry in lane 0, L1 queue:
              >>
              >> (gdb) p LRU[0].L1
              >> $9 = {q = {next = 0x7fad64256f00, prev =
        0x7faf21a1bc00}, id =
              >> LRU_ENTRY_L1, size = 254628}
              >> (gdb) p (mdcache_entry_t *)(0x7fad64256f00-1024)
              >> $10 = (mdcache_entry_t *) 0x7fad64256b00
              >> (gdb) p $10->lru
              >> $11 = {q = {next = 0x7fad65ea0f00, prev = 0x7d67c0
        <LRU>}, qid =
              >> LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 0, cf = 0}
              >> (gdb) p $10->fh_hk.inavl
              >> $13 = true
              >
              > The refcount 2 prevents reaping.
              >
              > There could be a refcount leak.
              >
              > Hmm, though, I thought the entries_hwmark was a hard
        limit, guess
             not...
              >
              > Frank
              >
              >> Lane 1:
              >> (gdb) p LRU[1].L1
              >> $18 = {q = {next = 0x7fad625c0300, prev =
        0x7faec08c5100}, id =
              >> LRU_ENTRY_L1, size = 253006}
              >> (gdb) p (mdcache_entry_t *)(0x7fad625c0300 - 1024)
              >> $21 = (mdcache_entry_t *) 0x7fad625bff00
              >> (gdb) p $21->lru
              >> $22 = {q = {next = 0x7fad66fce600, prev = 0x7d68a0
        <LRU+224>}, qid =
              >> LRU_ENTRY_L1, refcnt = 2, flags = 0, lane = 1, cf = 1}
              >>
              >> (gdb) p $21->fh_hk.inavl
              >> $24 = true
              >>
              >> As per LRU_ENTRY_RECLAIMABLE(), these entry should be
             reclaimable. Not
              >> sure why it is not able to claim it. Any ideas?
              >>
              >> Thanks,
              >> Pradeep
              >>
              >>
              >
----------------------------------------------------------------------------
              > --
              >> Check out the vibrant tech community on one of the
        world's most
             engaging
              >> tech sites, Slashdot.org! http://sdm.link/slashdot
              >> _______________________________________________
              >> Nfs-ganesha-devel mailing list
              >> Nfs-ganesha-devel@lists.sourceforge.net
        <mailto:Nfs-ganesha-devel@lists.sourceforge.net>
             <mailto:Nfs-ganesha-devel@lists.sourceforge.net
        <mailto:Nfs-ganesha-devel@lists.sourceforge.net>>
              >>
        https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
        <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel>
<https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
        <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel>>
              >
              >
              > ---
              > This email has been checked for viruses by Avast
        antivirus software.
              > https://www.avast.com/antivirus
        <https://www.avast.com/antivirus>
        <https://www.avast.com/antivirus <https://www.avast.com/antivirus>>
              >
              >
              >
------------------------------------------------------------------------------
              > Check out the vibrant tech community on one of the
        world's most
              > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
              > _______________________________________________
              > Nfs-ganesha-devel mailing list
              > Nfs-ganesha-devel@lists.sourceforge.net
        <mailto:Nfs-ganesha-devel@lists.sourceforge.net>
             <mailto:Nfs-ganesha-devel@lists.sourceforge.net
        <mailto:Nfs-ganesha-devel@lists.sourceforge.net>>
              >
        https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
        <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel>
<https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
        <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel>>






------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel

Reply via email to