Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

OK, given GlusterFS v3.7.6 with the following patches:

===
Kaleb S KEITHLEY (1):
  fuse: use-after-free fix in fuse-bridge, revisited

Pranith Kumar K (1):
  mount/fuse: Fix use-after-free crash

Soumya Koduri (3):
  gfapi: Fix inode nlookup counts
  inode: Retire the inodes from the lru list in inode_table_destroy
  upcall: free the xdr* allocations
===

I've repeated "rsync" test under Valgrind, and here is Valgrind output:

https://gist.github.com/f8e0151a6878cacc9b1a

I see DHT-related leaks.

On понеділок, 25 січня 2016 р. 02:46:32 EET Oleksandr Natalenko wrote:
> Also, I've repeated the same "find" test again, but with glusterfs process
> launched under valgrind. And here is valgrind output:
> 
> https://gist.github.com/097afb01ebb2c5e9e78d
> 
> On неділя, 24 січня 2016 р. 09:33:00 EET Mathieu Chateau wrote:
> > Thanks for all your tests and times, it looks promising :)
> > 
> > 
> > Cordialement,
> > Mathieu CHATEAU
> > http://www.lotp.fr
> > 
> > 2016-01-23 22:30 GMT+01:00 Oleksandr Natalenko :
> > > OK, now I'm re-performing tests with rsync + GlusterFS v3.7.6 + the
> > > following
> > > patches:
> > > 
> > > ===
> > > 
> > > Kaleb S KEITHLEY (1):
> > >   fuse: use-after-free fix in fuse-bridge, revisited
> > > 
> > > Pranith Kumar K (1):
> > >   mount/fuse: Fix use-after-free crash
> > > 
> > > Soumya Koduri (3):
> > >   gfapi: Fix inode nlookup counts
> > >   inode: Retire the inodes from the lru list in inode_table_destroy
> > >   upcall: free the xdr* allocations
> > > 
> > > ===
> > > 
> > > I run rsync from one GlusterFS volume to another. While memory started
> > > from
> > > under 100 MiBs, it stalled at around 600 MiBs for source volume and does
> > > not
> > > grow further. As for target volume it is ~730 MiBs, and that is why I'm
> > > going
> > > to do several rsync rounds to see if it grows more (with no patches bare
> > > 3.7.6
> > > could consume more than 20 GiBs).
> > > 
> > > No "kernel notifier loop terminated" message so far for both volumes.
> > > 
> > > Will report more in several days. I hope current patches will be
> > > incorporated
> > > into 3.7.7.
> > > 
> > > On пʼятниця, 22 січня 2016 р. 12:53:36 EET Kaleb S. KEITHLEY wrote:
> > > > On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
> > > > > On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
> > > > >> I presume by this you mean you're not seeing the "kernel notifier
> > > > >> loop
> > > > >> terminated" error in your logs.
> > > > > 
> > > > > Correct, but only with simple traversing. Have to test under rsync.
> > > > 
> > > > Without the patch I'd get "kernel notifier loop terminated" within a
> > > > few
> > > > minutes of starting I/O.  With the patch I haven't seen it in 24 hours
> > > > of beating on it.
> > > > 
> > > > >> Hmmm.  My system is not leaking. Last 24 hours the RSZ and VSZ are
> > > 
> > > > >> stable:
> > > http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/longe
> > > v
> > > 
> > > > >> ity /client.out
> > > > > 
> > > > > What ops do you perform on mounted volume? Read, write, stat? Is
> > > > > that
> > > > > 3.7.6 + patches?
> > > > 
> > > > I'm running an internally developed I/O load generator written by a
> > > > guy
> > > > on our perf team.
> > > > 
> > > > it does, create, write, read, rename, stat, delete, and more.
> 
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Here are the results of "rsync" test. I've got 2 volumes — source and target — 
performing multiple files rsyncing from one volume to another.

Source volume:

===
root 22259  3.5  1.5 1204200 771004 ?  Ssl  Jan23 109:42 /usr/sbin/
glusterfs --volfile-server=glusterfs.example.com --volfile-id=source /mnt/net/
glusterfs/source
===

One may see that memory consumption of source volume is not that high as with 
"find" test. Here is source volume client statedump: https://gist.github.com/
ef5b798859219e739aeb

Here is source volume info: https://gist.github.com/3d2f32e7346df9333004

Target volume:

===
root 22200 23.8  6.9 3983676 3456252 ? Ssl  Jan23 734:57 /usr/sbin/
glusterfs --volfile-server=glusterfs.example.com --volfile-id=target /mnt/net/
glusterfs/target
===

Here is target volume info: https://gist.github.com/c9de01168071575b109e

Target volume RAM consumption is very high (more than 3 GiBs). Here is client 
statedump too: https://gist.github.com/31e43110eaa4da663435

I see huge DHT-related memory usage, e.g.:

===
[cluster/distribute.asterisk_records-dht - usage-type gf_common_mt_mem_pool 
memusage]
size=725575592
num_allocs=7552486
max_size=725575836
max_num_allocs=7552489
total_allocs=90843958

[cluster/distribute.asterisk_records-dht - usage-type gf_common_mt_char 
memusage]
size=586404954
num_allocs=7572836
max_size=586405157
max_num_allocs=7572839
total_allocs=80463096
===

Ideas?

On понеділок, 25 січня 2016 р. 02:46:32 EET Oleksandr Natalenko wrote:
> Also, I've repeated the same "find" test again, but with glusterfs process
> launched under valgrind. And here is valgrind output:
> 
> https://gist.github.com/097afb01ebb2c5e9e78d
> 
> On неділя, 24 січня 2016 р. 09:33:00 EET Mathieu Chateau wrote:
> > Thanks for all your tests and times, it looks promising :)
> > 
> > 
> > Cordialement,
> > Mathieu CHATEAU
> > http://www.lotp.fr
> > 
> > 2016-01-23 22:30 GMT+01:00 Oleksandr Natalenko :
> > > OK, now I'm re-performing tests with rsync + GlusterFS v3.7.6 + the
> > > following
> > > patches:
> > > 
> > > ===
> > > 
> > > Kaleb S KEITHLEY (1):
> > >   fuse: use-after-free fix in fuse-bridge, revisited
> > > 
> > > Pranith Kumar K (1):
> > >   mount/fuse: Fix use-after-free crash
> > > 
> > > Soumya Koduri (3):
> > >   gfapi: Fix inode nlookup counts
> > >   inode: Retire the inodes from the lru list in inode_table_destroy
> > >   upcall: free the xdr* allocations
> > > 
> > > ===
> > > 
> > > I run rsync from one GlusterFS volume to another. While memory started
> > > from
> > > under 100 MiBs, it stalled at around 600 MiBs for source volume and does
> > > not
> > > grow further. As for target volume it is ~730 MiBs, and that is why I'm
> > > going
> > > to do several rsync rounds to see if it grows more (with no patches bare
> > > 3.7.6
> > > could consume more than 20 GiBs).
> > > 
> > > No "kernel notifier loop terminated" message so far for both volumes.
> > > 
> > > Will report more in several days. I hope current patches will be
> > > incorporated
> > > into 3.7.7.
> > > 
> > > On пʼятниця, 22 січня 2016 р. 12:53:36 EET Kaleb S. KEITHLEY wrote:
> > > > On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
> > > > > On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
> > > > >> I presume by this you mean you're not seeing the "kernel notifier
> > > > >> loop
> > > > >> terminated" error in your logs.
> > > > > 
> > > > > Correct, but only with simple traversing. Have to test under rsync.
> > > > 
> > > > Without the patch I'd get "kernel notifier loop terminated" within a
> > > > few
> > > > minutes of starting I/O.  With the patch I haven't seen it in 24 hours
> > > > of beating on it.
> > > > 
> > > > >> Hmmm.  My system is not leaking. Last 24 hours the RSZ and VSZ are
> > > 
> > > > >> stable:
> > > http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/longe
> > > v
> > > 
> > > > >> ity /client.out
> > > > > 
> > > > > What ops do you perform on mounted volume? Read, write, stat? Is
> > > > > that
> > > > > 3.7.6 + patches?
> > > > 
> > > > I'm running an internally developed I/O load generator written by a
> > > > guy
> > > > on our perf team.
> > > > 
> > > > it does, create, write, read, rename, stat, delete, and more.
> 
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Also, I've repeated the same "find" test again, but with glusterfs process 
launched under valgrind. And here is valgrind output:

https://gist.github.com/097afb01ebb2c5e9e78d

On неділя, 24 січня 2016 р. 09:33:00 EET Mathieu Chateau wrote:
> Thanks for all your tests and times, it looks promising :)
> 
> 
> Cordialement,
> Mathieu CHATEAU
> http://www.lotp.fr
> 
> 2016-01-23 22:30 GMT+01:00 Oleksandr Natalenko :
> > OK, now I'm re-performing tests with rsync + GlusterFS v3.7.6 + the
> > following
> > patches:
> > 
> > ===
> > 
> > Kaleb S KEITHLEY (1):
> >   fuse: use-after-free fix in fuse-bridge, revisited
> > 
> > Pranith Kumar K (1):
> >   mount/fuse: Fix use-after-free crash
> > 
> > Soumya Koduri (3):
> >   gfapi: Fix inode nlookup counts
> >   inode: Retire the inodes from the lru list in inode_table_destroy
> >   upcall: free the xdr* allocations
> > 
> > ===
> > 
> > I run rsync from one GlusterFS volume to another. While memory started
> > from
> > under 100 MiBs, it stalled at around 600 MiBs for source volume and does
> > not
> > grow further. As for target volume it is ~730 MiBs, and that is why I'm
> > going
> > to do several rsync rounds to see if it grows more (with no patches bare
> > 3.7.6
> > could consume more than 20 GiBs).
> > 
> > No "kernel notifier loop terminated" message so far for both volumes.
> > 
> > Will report more in several days. I hope current patches will be
> > incorporated
> > into 3.7.7.
> > 
> > On пʼятниця, 22 січня 2016 р. 12:53:36 EET Kaleb S. KEITHLEY wrote:
> > > On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
> > > > On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
> > > >> I presume by this you mean you're not seeing the "kernel notifier
> > > >> loop
> > > >> terminated" error in your logs.
> > > > 
> > > > Correct, but only with simple traversing. Have to test under rsync.
> > > 
> > > Without the patch I'd get "kernel notifier loop terminated" within a few
> > > minutes of starting I/O.  With the patch I haven't seen it in 24 hours
> > > of beating on it.
> > > 
> > > >> Hmmm.  My system is not leaking. Last 24 hours the RSZ and VSZ are
> > 
> > > >> stable:
> > http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/longev
> > 
> > > >> ity /client.out
> > > > 
> > > > What ops do you perform on mounted volume? Read, write, stat? Is that
> > > > 3.7.6 + patches?
> > > 
> > > I'm running an internally developed I/O load generator written by a guy
> > > on our perf team.
> > > 
> > > it does, create, write, read, rename, stat, delete, and more.


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

BTW, am I the only one who sees in

max_size=4294965480

almost 2^32? Could that be integer overflow?

On неділя, 24 січня 2016 р. 13:23:55 EET Oleksandr Natalenko wrote:
> The leak definitely remains. I did "find /mnt/volume -type d" over GlusterFS
> volume, with mentioned patches applied and without "kernel notifier loop
> terminated" message, but "glusterfs" process consumed ~4GiB of RAM after
> "find" finished.
> 
> Here is statedump:
> 
> https://gist.github.com/10cde83c63f1b4f1dd7a
> 
> I see the following:
> 
> ===
> [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
> size=4235109959
> num_allocs=2
> max_size=4294965480
> max_num_allocs=3
> total_allocs=4533524
> ===
> 
> ~4GiB, right?
> 
> Pranith, Kaleb?
> 
> On неділя, 24 січня 2016 р. 09:33:00 EET Mathieu Chateau wrote:
> > Thanks for all your tests and times, it looks promising :)
> > 
> > 
> > Cordialement,
> > Mathieu CHATEAU
> > http://www.lotp.fr
> > 
> > 2016-01-23 22:30 GMT+01:00 Oleksandr Natalenko :
> > > OK, now I'm re-performing tests with rsync + GlusterFS v3.7.6 + the
> > > following
> > > patches:
> > > 
> > > ===
> > > 
> > > Kaleb S KEITHLEY (1):
> > >   fuse: use-after-free fix in fuse-bridge, revisited
> > > 
> > > Pranith Kumar K (1):
> > >   mount/fuse: Fix use-after-free crash
> > > 
> > > Soumya Koduri (3):
> > >   gfapi: Fix inode nlookup counts
> > >   inode: Retire the inodes from the lru list in inode_table_destroy
> > >   upcall: free the xdr* allocations
> > > 
> > > ===
> > > 
> > > I run rsync from one GlusterFS volume to another. While memory started
> > > from
> > > under 100 MiBs, it stalled at around 600 MiBs for source volume and does
> > > not
> > > grow further. As for target volume it is ~730 MiBs, and that is why I'm
> > > going
> > > to do several rsync rounds to see if it grows more (with no patches bare
> > > 3.7.6
> > > could consume more than 20 GiBs).
> > > 
> > > No "kernel notifier loop terminated" message so far for both volumes.
> > > 
> > > Will report more in several days. I hope current patches will be
> > > incorporated
> > > into 3.7.7.
> > > 
> > > On пʼятниця, 22 січня 2016 р. 12:53:36 EET Kaleb S. KEITHLEY wrote:
> > > > On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
> > > > > On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
> > > > >> I presume by this you mean you're not seeing the "kernel notifier
> > > > >> loop
> > > > >> terminated" error in your logs.
> > > > > 
> > > > > Correct, but only with simple traversing. Have to test under rsync.
> > > > 
> > > > Without the patch I'd get "kernel notifier loop terminated" within a
> > > > few
> > > > minutes of starting I/O.  With the patch I haven't seen it in 24 hours
> > > > of beating on it.
> > > > 
> > > > >> Hmmm.  My system is not leaking. Last 24 hours the RSZ and VSZ are
> > > 
> > > > >> stable:
> > > http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/longe
> > > v
> > > 
> > > > >> ity /client.out
> > > > > 
> > > > > What ops do you perform on mounted volume? Read, write, stat? Is
> > > > > that
> > > > > 3.7.6 + patches?
> > > > 
> > > > I'm running an internally developed I/O load generator written by a
> > > > guy
> > > > on our perf team.
> > > > 
> > > > it does, create, write, read, rename, stat, delete, and more.
> 
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

The leak definitely remains. I did "find /mnt/volume -type d" over GlusterFS 
volume, with mentioned patches applied and without "kernel notifier loop 
terminated" message, but "glusterfs" process consumed ~4GiB of RAM after 
"find" finished.

Here is statedump:

https://gist.github.com/10cde83c63f1b4f1dd7a

I see the following:

===
[mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
size=4235109959
num_allocs=2
max_size=4294965480
max_num_allocs=3
total_allocs=4533524
===

~4GiB, right?

Pranith, Kaleb?

On неділя, 24 січня 2016 р. 09:33:00 EET Mathieu Chateau wrote:
> Thanks for all your tests and times, it looks promising :)
> 
> 
> Cordialement,
> Mathieu CHATEAU
> http://www.lotp.fr
> 
> 2016-01-23 22:30 GMT+01:00 Oleksandr Natalenko :
> > OK, now I'm re-performing tests with rsync + GlusterFS v3.7.6 + the
> > following
> > patches:
> > 
> > ===
> > 
> > Kaleb S KEITHLEY (1):
> >   fuse: use-after-free fix in fuse-bridge, revisited
> > 
> > Pranith Kumar K (1):
> >   mount/fuse: Fix use-after-free crash
> > 
> > Soumya Koduri (3):
> >   gfapi: Fix inode nlookup counts
> >   inode: Retire the inodes from the lru list in inode_table_destroy
> >   upcall: free the xdr* allocations
> > 
> > ===
> > 
> > I run rsync from one GlusterFS volume to another. While memory started
> > from
> > under 100 MiBs, it stalled at around 600 MiBs for source volume and does
> > not
> > grow further. As for target volume it is ~730 MiBs, and that is why I'm
> > going
> > to do several rsync rounds to see if it grows more (with no patches bare
> > 3.7.6
> > could consume more than 20 GiBs).
> > 
> > No "kernel notifier loop terminated" message so far for both volumes.
> > 
> > Will report more in several days. I hope current patches will be
> > incorporated
> > into 3.7.7.
> > 
> > On пʼятниця, 22 січня 2016 р. 12:53:36 EET Kaleb S. KEITHLEY wrote:
> > > On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
> > > > On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
> > > >> I presume by this you mean you're not seeing the "kernel notifier
> > > >> loop
> > > >> terminated" error in your logs.
> > > > 
> > > > Correct, but only with simple traversing. Have to test under rsync.
> > > 
> > > Without the patch I'd get "kernel notifier loop terminated" within a few
> > > minutes of starting I/O.  With the patch I haven't seen it in 24 hours
> > > of beating on it.
> > > 
> > > >> Hmmm.  My system is not leaking. Last 24 hours the RSZ and VSZ are
> > 
> > > >> stable:
> > http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/longev
> > 
> > > >> ity /client.out
> > > > 
> > > > What ops do you perform on mounted volume? Read, write, stat? Is that
> > > > 3.7.6 + patches?
> > > 
> > > I'm running an internally developed I/O load generator written by a guy
> > > on our perf team.
> > > 
> > > it does, create, write, read, rename, stat, delete, and more.


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

OK, now I'm re-performing tests with rsync + GlusterFS v3.7.6 + the following 
patches:

===
Kaleb S KEITHLEY (1):
  fuse: use-after-free fix in fuse-bridge, revisited

Pranith Kumar K (1):
  mount/fuse: Fix use-after-free crash

Soumya Koduri (3):
  gfapi: Fix inode nlookup counts
  inode: Retire the inodes from the lru list in inode_table_destroy
  upcall: free the xdr* allocations
===

I run rsync from one GlusterFS volume to another. While memory started from 
under 100 MiBs, it stalled at around 600 MiBs for source volume and does not 
grow further. As for target volume it is ~730 MiBs, and that is why I'm going 
to do several rsync rounds to see if it grows more (with no patches bare 3.7.6 
could consume more than 20 GiBs).

No "kernel notifier loop terminated" message so far for both volumes.

Will report more in several days. I hope current patches will be incorporated 
into 3.7.7.

On пʼятниця, 22 січня 2016 р. 12:53:36 EET Kaleb S. KEITHLEY wrote:
> On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
> > On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
> >> I presume by this you mean you're not seeing the "kernel notifier loop
> >> terminated" error in your logs.
> > 
> > Correct, but only with simple traversing. Have to test under rsync.
> 
> Without the patch I'd get "kernel notifier loop terminated" within a few
> minutes of starting I/O.  With the patch I haven't seen it in 24 hours
> of beating on it.
> 
> >> Hmmm.  My system is not leaking. Last 24 hours the RSZ and VSZ are
> >> stable:
> >> http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/longev
> >> ity /client.out
> > 
> > What ops do you perform on mounted volume? Read, write, stat? Is that
> > 3.7.6 + patches?
> 
> I'm running an internally developed I/O load generator written by a guy
> on our perf team.
> 
> it does, create, write, read, rename, stat, delete, and more.


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Kaleb S. KEITHLEY]

On 01/22/2016 01:20 PM, Joe Julian wrote:
> 
> 
> On 01/22/16 09:53, Kaleb S. KEITHLEY wrote:
>> On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
>>> On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
 I presume by this you mean you're not seeing the "kernel notifier loop
 terminated" error in your logs.
>>> Correct, but only with simple traversing. Have to test under rsync.
>> Without the patch I'd get "kernel notifier loop terminated" within a few
>> minutes of starting I/O.  With the patch I haven't seen it in 24 hours
>> of beating on it.
>>
 Hmmm.  My system is not leaking. Last 24 hours the RSZ and VSZ are
 stable:
 http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/longevity

 /client.out
>>> What ops do you perform on mounted volume? Read, write, stat? Is that
>>> 3.7.6 +
>>> patches?
>> I'm running an internally developed I/O load generator written by a guy
  


>> on our perf team.
>>
>> it does, create, write, read, rename, stat, delete, and more.
>>
> Github link?

I looked for one before posting. I don't think he has shared it.

-- 

Kaleb
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Kaleb S. KEITHLEY]

On 01/22/2016 12:15 PM, Oleksandr Natalenko wrote:
> OK, compiles and runs well now,

I presume by this you mean you're not seeing the "kernel notifier loop
terminated" error in your logs.

> but still leaks. 

Hmmm.  My system is not leaking. Last 24 hours the RSZ and VSZ are
stable:
http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/longevity/client.out

-- 

Kaleb
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Kaleb S. KEITHLEY]

On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
> On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
>> I presume by this you mean you're not seeing the "kernel notifier loop
>> terminated" error in your logs.
> 
> Correct, but only with simple traversing. Have to test under rsync.

Without the patch I'd get "kernel notifier loop terminated" within a few
minutes of starting I/O.  With the patch I haven't seen it in 24 hours
of beating on it.

> 
>> Hmmm.  My system is not leaking. Last 24 hours the RSZ and VSZ are
>> stable:
>> http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/longevity
>> /client.out
> 
> What ops do you perform on mounted volume? Read, write, stat? Is that 3.7.6 + 
> patches?

I'm running an internally developed I/O load generator written by a guy
on our perf team.

it does, create, write, read, rename, stat, delete, and more.

-- 

Kaleb
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

I perform the tests using 1) rsync (massive copy of millions of files); 2) 
find (simple tree traversing).

To check if memory leak happens, I use find tool. I've performed two 
traversing (w/ and w/o fopen-keep-cache=off) with remount between them, but I 
didn't encounter "kernel notifier loop terminated" message during both 
traversing as well as before unmounting volume. Nevertheless, memory still 
leaks (at least up to 3 GiB in each case), so I believe invalidation requests 
are not the case.

I've also checked logs for the volume where I do rsync, and the message 
"kernel notifier loop terminated" happens somewhere in the middle of rsyncing, 
not before unmounting. But the memory starts leaking on rsync start as well, 
not just after "kernel notifier loop terminated" message. So, I believe, 
"kernel notifier loop terminated" is not the case again.

Also, I've tried to implement quick and dirty GlusterFS FUSE client using API 
(see https://github.com/pfactum/xglfs), and with latest patches from this 
thread (http://review.gluster.org/#/c/13096/, http://review.gluster.org/#/c/
13125/ and http://review.gluster.org/#/c/13232/) my FUSE client does not leak 
on tree traversing. So, I believe, this should be related to GlusterFS FUSE 
implementation.

How could I debug memory leak better?

On четвер, 21 січня 2016 р. 10:32:32 EET Xavier Hernandez wrote:
> If this message appears way before the volume is unmounted, can you try
> to start the volume manually using this command and repeat the tests ?
> 
> glusterfs --fopen-keep-cache=off --volfile-server=
> --volfile-id=/ 
> 
> This will prevent invalidation requests to be sent to the kernel, so
> there shouldn't be any memory leak even if the worker thread exits
> prematurely.
> 
> If that solves the problem, we could try to determine the cause of the
> premature exit and solve it.
> 
> Xavi
> 
> On 20/01/16 10:08, Oleksandr Natalenko wrote:
> > Yes, there are couple of messages like this in my logs too (I guess one
> > message per each remount):
> > 
> > ===
> > [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
> > glusterfs-fuse: kernel notifier loop terminated
> > ===
> > 
> > On середа, 20 січня 2016 р. 09:51:23 EET Xavier Hernandez wrote:
> >> I'm seeing a similar problem with 3.7.6.
> >> 
> >> This latest statedump contains a lot of gf_fuse_mt_invalidate_node_t
> >> objects in fuse. Looking at the code I see they are used to send
> >> invalidations to kernel fuse, however this is done in a separate thread
> >> that writes a log message when it exits. On the system I'm seeing the
> >> memory leak, I can see that message in the log files:
> >> 
> >> [2016-01-18 23:04:55.384873] I [fuse-bridge.c:3875:notify_kernel_loop]
> >> 0-glusterfs-fuse: kernel notifier loop terminated
> >> 
> >> But the volume is still working at this moment, so any future inode
> >> invalidations will leak memory because it was this thread that should
> >> release it.
> >> 
> >> Can you check if you also see this message in the mount log ?
> >> 
> >> It seems that this thread terminates if write returns any error
> >> different than ENOENT. I'm not sure if there could be any other error
> >> that can cause this.
> >> 
> >> Xavi
> >> 
> >> On 20/01/16 00:13, Oleksandr Natalenko wrote:
> >>> Here is another RAM usage stats and statedump of GlusterFS mount
> >>> approaching to just another OOM:
> >>> 
> >>> ===
> >>> root 32495  1.4 88.3 4943868 1697316 ? Ssl  Jan13 129:18
> >>> /usr/sbin/
> >>> glusterfs --volfile-server=server.example.com --volfile-id=volume
> >>> /mnt/volume ===
> >>> 
> >>> https://gist.github.com/86198201c79e927b46bd
> >>> 
> >>> 1.6G of RAM just for almost idle mount (we occasionally store Asterisk
> >>> recordings there). Triple OOM for 69 days of uptime.
> >>> 
> >>> Any thoughts?
> >>> 
> >>> On середа, 13 січня 2016 р. 16:26:59 EET Soumya Koduri wrote:
>  kill -USR1
> >>> 
> >>> ___
> >>> Gluster-devel mailing list
> >>> Gluster-devel@gluster.org
> >>> http://www.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Xavier Hernandez]

If this message appears way before the volume is unmounted, can you try 
to start the volume manually using this command and repeat the tests ?


glusterfs --fopen-keep-cache=off --volfile-server= 
--volfile-id=/ 


This will prevent invalidation requests to be sent to the kernel, so 
there shouldn't be any memory leak even if the worker thread exits 
prematurely.


If that solves the problem, we could try to determine the cause of the 
premature exit and solve it.


Xavi


On 20/01/16 10:08, Oleksandr Natalenko wrote:

Yes, there are couple of messages like this in my logs too (I guess one
message per each remount):

===
[2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
glusterfs-fuse: kernel notifier loop terminated
===

On середа, 20 січня 2016 р. 09:51:23 EET Xavier Hernandez wrote:

I'm seeing a similar problem with 3.7.6.

This latest statedump contains a lot of gf_fuse_mt_invalidate_node_t
objects in fuse. Looking at the code I see they are used to send
invalidations to kernel fuse, however this is done in a separate thread
that writes a log message when it exits. On the system I'm seeing the
memory leak, I can see that message in the log files:

[2016-01-18 23:04:55.384873] I [fuse-bridge.c:3875:notify_kernel_loop]
0-glusterfs-fuse: kernel notifier loop terminated

But the volume is still working at this moment, so any future inode
invalidations will leak memory because it was this thread that should
release it.

Can you check if you also see this message in the mount log ?

It seems that this thread terminates if write returns any error
different than ENOENT. I'm not sure if there could be any other error
that can cause this.

Xavi

On 20/01/16 00:13, Oleksandr Natalenko wrote:

Here is another RAM usage stats and statedump of GlusterFS mount
approaching to just another OOM:

===
root 32495  1.4 88.3 4943868 1697316 ? Ssl  Jan13 129:18
/usr/sbin/
glusterfs --volfile-server=server.example.com --volfile-id=volume
/mnt/volume ===

https://gist.github.com/86198201c79e927b46bd

1.6G of RAM just for almost idle mount (we occasionally store Asterisk
recordings there). Triple OOM for 69 days of uptime.

Any thoughts?

On середа, 13 січня 2016 р. 16:26:59 EET Soumya Koduri wrote:

kill -USR1


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel




___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

I see extra GF_FREE (node); added with two patches:

===
$ git diff HEAD~2 | gist
https://gist.github.com/9524fa2054cc48278ea8
===

Is that intentionally? I guess I face double-free issue.

On четвер, 21 січня 2016 р. 17:29:53 EET Kaleb KEITHLEY wrote:
> On 01/20/2016 04:08 AM, Oleksandr Natalenko wrote:
> > Yes, there are couple of messages like this in my logs too (I guess one
> > message per each remount):
> > 
> > ===
> > [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
> > glusterfs-fuse: kernel notifier loop terminated
> > ===
> 
> Bug reports and fixes for master and release-3.7 branches are:
> 
> master)
>  https://bugzilla.redhat.com/show_bug.cgi?id=1288857
>  http://review.gluster.org/12886
> 
> release-3.7)
>  https://bugzilla.redhat.com/show_bug.cgi?id=1288922
>  http://review.gluster.org/12887
> 
> The release-3.7 fix will be in glusterfs-3.7.7 when it's released.
> 
> I think with even with the above fixes applied there are still some
> issues remaining. I have submitted additional/revised fixes on top of
> the above fixes at:
> 
>  master: http://review.gluster.org/13274
>  release-3.7: http://review.gluster.org/13275
> 
> I invite you to review the patches in gerrit (review.gluster.org).
> 
> Regards,
> 
> --
> 
> Kaleb


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

With the proposed patches I get the following assertion while copying files to 
GlusterFS volume:

===
glusterfs: mem-pool.c:305: __gf_free: Assertion `0xCAFEBABE == header->magic' 
failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe9ffb700 (LWP 12635)]
0x76f215f8 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0  0x76f215f8 in raise () from /usr/lib/libc.so.6
#1  0x76f22a7a in abort () from /usr/lib/libc.so.6
#2  0x76f1a417 in __assert_fail_base () from /usr/lib/libc.so.6
#3  0x76f1a4c2 in __assert_fail () from /usr/lib/libc.so.6
#4  0x77b6046b in __gf_free (free_ptr=0x7fffdc0b8f00) at mem-pool.c:
305
#5  0x75144eb9 in notify_kernel_loop (data=0x63df90) at fuse-bridge.c:
3893
#6  0x772994a4 in start_thread () from /usr/lib/libpthread.so.0
#7  0x76fd713d in clone () from /usr/lib/libc.so.6
===

On четвер, 21 січня 2016 р. 17:29:53 EET Kaleb KEITHLEY wrote:
> On 01/20/2016 04:08 AM, Oleksandr Natalenko wrote:
> > Yes, there are couple of messages like this in my logs too (I guess one
> > message per each remount):
> > 
> > ===
> > [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
> > glusterfs-fuse: kernel notifier loop terminated
> > ===
> 
> Bug reports and fixes for master and release-3.7 branches are:
> 
> master)
>  https://bugzilla.redhat.com/show_bug.cgi?id=1288857
>  http://review.gluster.org/12886
> 
> release-3.7)
>  https://bugzilla.redhat.com/show_bug.cgi?id=1288922
>  http://review.gluster.org/12887
> 
> The release-3.7 fix will be in glusterfs-3.7.7 when it's released.
> 
> I think with even with the above fixes applied there are still some
> issues remaining. I have submitted additional/revised fixes on top of
> the above fixes at:
> 
>  master: http://review.gluster.org/13274
>  release-3.7: http://review.gluster.org/13275
> 
> I invite you to review the patches in gerrit (review.gluster.org).
> 
> Regards,
> 
> --
> 
> Kaleb


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Kaleb KEITHLEY]

On 01/20/2016 04:08 AM, Oleksandr Natalenko wrote:
> Yes, there are couple of messages like this in my logs too (I guess one 
> message per each remount):
> 
> ===
> [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
> glusterfs-fuse: kernel notifier loop terminated
> ===
>

Bug reports and fixes for master and release-3.7 branches are:

master)
 https://bugzilla.redhat.com/show_bug.cgi?id=1288857
 http://review.gluster.org/12886

release-3.7)
 https://bugzilla.redhat.com/show_bug.cgi?id=1288922
 http://review.gluster.org/12887

The release-3.7 fix will be in glusterfs-3.7.7 when it's released.

I think with even with the above fixes applied there are still some
issues remaining. I have submitted additional/revised fixes on top of
the above fixes at:

 master: http://review.gluster.org/13274
 release-3.7: http://review.gluster.org/13275

I invite you to review the patches in gerrit (review.gluster.org).

Regards,

--

Kaleb

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Kaleb KEITHLEY]

On 01/21/2016 06:59 PM, Oleksandr Natalenko wrote:
> I see extra GF_FREE (node); added with two patches:
> 
> ===
> $ git diff HEAD~2 | gist
> https://gist.github.com/9524fa2054cc48278ea8
> ===
> 
> Is that intentionally? I guess I face double-free issue.
> 

I presume you're referring to the release-3.7 branch.

Yup, bad edit. Long day. That's why we review. ;-)

Please try the latest.

Thanks,

--

Kaleb

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Xavier Hernandez]

I'm seeing a similar problem with 3.7.6.

This latest statedump contains a lot of gf_fuse_mt_invalidate_node_t 
objects in fuse. Looking at the code I see they are used to send 
invalidations to kernel fuse, however this is done in a separate thread 
that writes a log message when it exits. On the system I'm seeing the 
memory leak, I can see that message in the log files:


[2016-01-18 23:04:55.384873] I [fuse-bridge.c:3875:notify_kernel_loop] 
0-glusterfs-fuse: kernel notifier loop terminated


But the volume is still working at this moment, so any future inode 
invalidations will leak memory because it was this thread that should 
release it.


Can you check if you also see this message in the mount log ?

It seems that this thread terminates if write returns any error 
different than ENOENT. I'm not sure if there could be any other error 
that can cause this.


Xavi

On 20/01/16 00:13, Oleksandr Natalenko wrote:

Here is another RAM usage stats and statedump of GlusterFS mount approaching
to just another OOM:

===
root 32495  1.4 88.3 4943868 1697316 ? Ssl  Jan13 129:18 /usr/sbin/
glusterfs --volfile-server=server.example.com --volfile-id=volume /mnt/volume
===

https://gist.github.com/86198201c79e927b46bd

1.6G of RAM just for almost idle mount (we occasionally store Asterisk
recordings there). Triple OOM for 69 days of uptime.

Any thoughts?

On середа, 13 січня 2016 р. 16:26:59 EET Soumya Koduri wrote:

kill -USR1



___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Yes, there are couple of messages like this in my logs too (I guess one 
message per each remount):

===
[2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
glusterfs-fuse: kernel notifier loop terminated
===

On середа, 20 січня 2016 р. 09:51:23 EET Xavier Hernandez wrote:
> I'm seeing a similar problem with 3.7.6.
> 
> This latest statedump contains a lot of gf_fuse_mt_invalidate_node_t
> objects in fuse. Looking at the code I see they are used to send
> invalidations to kernel fuse, however this is done in a separate thread
> that writes a log message when it exits. On the system I'm seeing the
> memory leak, I can see that message in the log files:
> 
> [2016-01-18 23:04:55.384873] I [fuse-bridge.c:3875:notify_kernel_loop]
> 0-glusterfs-fuse: kernel notifier loop terminated
> 
> But the volume is still working at this moment, so any future inode
> invalidations will leak memory because it was this thread that should
> release it.
> 
> Can you check if you also see this message in the mount log ?
> 
> It seems that this thread terminates if write returns any error
> different than ENOENT. I'm not sure if there could be any other error
> that can cause this.
> 
> Xavi
> 
> On 20/01/16 00:13, Oleksandr Natalenko wrote:
> > Here is another RAM usage stats and statedump of GlusterFS mount
> > approaching to just another OOM:
> > 
> > ===
> > root 32495  1.4 88.3 4943868 1697316 ? Ssl  Jan13 129:18
> > /usr/sbin/
> > glusterfs --volfile-server=server.example.com --volfile-id=volume
> > /mnt/volume ===
> > 
> > https://gist.github.com/86198201c79e927b46bd
> > 
> > 1.6G of RAM just for almost idle mount (we occasionally store Asterisk
> > recordings there). Triple OOM for 69 days of uptime.
> > 
> > Any thoughts?
> > 
> > On середа, 13 січня 2016 р. 16:26:59 EET Soumya Koduri wrote:
> >> kill -USR1
> > 
> > ___
> > Gluster-devel mailing list
> > Gluster-devel@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Here is another RAM usage stats and statedump of GlusterFS mount approaching 
to just another OOM:

===
root 32495  1.4 88.3 4943868 1697316 ? Ssl  Jan13 129:18 /usr/sbin/
glusterfs --volfile-server=server.example.com --volfile-id=volume /mnt/volume
===

https://gist.github.com/86198201c79e927b46bd

1.6G of RAM just for almost idle mount (we occasionally store Asterisk 
recordings there). Triple OOM for 69 days of uptime.

Any thoughts?

On середа, 13 січня 2016 р. 16:26:59 EET Soumya Koduri wrote:
> kill -USR1


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

And another statedump of FUSE mount client consuming more than 7 GiB of RAM:

https://gist.github.com/136d7c49193c798b3ade

DHT-related leak?

On середа, 13 січня 2016 р. 16:26:59 EET Soumya Koduri wrote:
> On 01/13/2016 04:08 PM, Soumya Koduri wrote:
> > On 01/12/2016 12:46 PM, Oleksandr Natalenko wrote:
> >> Just in case, here is Valgrind output on FUSE client with 3.7.6 +
> >> API-related patches we discussed before:
> >> 
> >> https://gist.github.com/cd6605ca19734c1496a4
> > 
> > Thanks for sharing the results. I made changes to fix one leak reported
> > there wrt ' client_cbk_cache_invalidation' -
> > 
> >  - http://review.gluster.org/#/c/13232/
> > 
> > The other inode* related memory reported as lost is mainly (maybe)
> > because fuse client process doesn't cleanup its memory (doesn't use
> > fini()) while exiting the process.  Hence majority of those allocations
> > are listed as lost. But most of the inodes should have got purged when
> > we drop vfs cache. Did you do drop vfs cache before exiting the process?
> > 
> > I shall add some log statements and check that part
> 
> Also please take statedump of the fuse mount process (after dropping vfs
> cache) when you see high memory usage by issuing the following command -
>   'kill -USR1 '
> 
> The statedump will be copied to 'glusterdump..dump.tim
> estamp` file in /var/run/gluster or /usr/local/var/run/gluster.
> Please refer to [1] for more information.
> 
> Thanks,
> Soumya
> [1] http://review.gluster.org/#/c/8288/1/doc/debugging/statedump.md
> 
> > Thanks,
> > Soumya
> > 
> >> 12.01.2016 08:24, Soumya Koduri написав:
> >>> For fuse client, I tried vfs drop_caches as suggested by Vijay in an
> >>> earlier mail. Though all the inodes get purged, I still doesn't see
> >>> much difference in the memory footprint drop. Need to investigate what
> >>> else is consuming so much memory here.
> > 
> > ___
> > Gluster-users mailing list
> > gluster-us...@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

On 01/13/2016 04:08 PM, Soumya Koduri wrote:



On 01/12/2016 12:46 PM, Oleksandr Natalenko wrote:

Just in case, here is Valgrind output on FUSE client with 3.7.6 +
API-related patches we discussed before:

https://gist.github.com/cd6605ca19734c1496a4



Thanks for sharing the results. I made changes to fix one leak reported
there wrt ' client_cbk_cache_invalidation' -
 - http://review.gluster.org/#/c/13232/

The other inode* related memory reported as lost is mainly (maybe)
because fuse client process doesn't cleanup its memory (doesn't use
fini()) while exiting the process.  Hence majority of those allocations
are listed as lost. But most of the inodes should have got purged when
we drop vfs cache. Did you do drop vfs cache before exiting the process?

I shall add some log statements and check that part


Also please take statedump of the fuse mount process (after dropping vfs 
cache) when you see high memory usage by issuing the following command -

'kill -USR1 '

The statedump will be copied to 'glusterdump..dump.tim
estamp` file in /var/run/gluster or /usr/local/var/run/gluster.
Please refer to [1] for more information.

Thanks,
Soumya
[1] http://review.gluster.org/#/c/8288/1/doc/debugging/statedump.md



Thanks,
Soumya


12.01.2016 08:24, Soumya Koduri написав:

For fuse client, I tried vfs drop_caches as suggested by Vijay in an
earlier mail. Though all the inodes get purged, I still doesn't see
much difference in the memory footprint drop. Need to investigate what
else is consuming so much memory here.

___
Gluster-users mailing list
gluster-us...@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

On 01/12/2016 12:17 PM, Mathieu Chateau wrote:

I tried like suggested:

echo 3 > /proc/sys/vm/drop_caches
sync


It lower a bit usage:

before:

Images intégrées 2

after:

Images intégrées 1



Thanks Mathieu. There is a drop in memory usage after dropping vfs cache 
but doesn't seem significant. Not sure at this point what else may be 
consuming most of the memory.
Maybe after dropping the vfs cache, you could as well take valgrind 
results and see if there are huge chunks reported as lost from inode_new*.


I shall too look into it further and update.

-Soumya



Cordialement,

Mathieu CHATEAU
http://www.lotp.fr


2016-01-12 7:34 GMT+01:00 Mathieu Chateau >:

Hello,

I also experience high memory usage on my gluster clients. Sample :
Images intégrées 1

Can I help in testing/debugging ?



Cordialement,
Mathieu CHATEAU
http://www.lotp.fr

2016-01-12 7:24 GMT+01:00 Soumya Koduri >:



On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote:

Brief test shows that Ganesha stopped leaking and crashing,
so it seems
to be good for me.

Thanks for checking.

Nevertheless, back to my original question: what about FUSE
client? It
is still leaking despite all the fixes applied. Should it be
considered
another issue?


For fuse client, I tried vfs drop_caches as suggested by Vijay
in an earlier mail. Though all the inodes get purged, I still
doesn't see much difference in the memory footprint drop. Need
to investigate what else is consuming so much memory here.

Thanks,
Soumya



11.01.2016 12:26, Soumya Koduri написав:

I have made changes to fix the lookup leak in a
different way (as
discussed with Pranith) and uploaded them in the latest
patch set #4
 - http://review.gluster.org/#/c/13096/

Please check if it resolves the mem leak and hopefully
doesn't result
in any assertion :)

Thanks,
Soumya

On 01/08/2016 05:04 PM, Soumya Koduri wrote:

I could reproduce while testing deep directories
with in the mount
point. I root caus'ed the issue & had discussion
with Pranith to
understand the purpose and recommended way of taking
nlookup on inodes.

I shall make changes to my existing fix and post the
patch soon.
Thanks for your patience!

-Soumya

On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote:

OK, I've patched GlusterFS v3.7.6 with 43570a01
and 5cffb56b (the most
recent
revisions) and NFS-Ganesha v2.3.0 with 8685abfc
(most recent revision
too).

On traversing GlusterFS volume with many files
in one folder via NFS
mount I
get an assertion:

===
ganesha.nfsd: inode.c:716: __inode_forget:
Assertion `inode->nlookup >=
nlookup' failed.
===

I used GDB on NFS-Ganesha process to get
appropriate stacktraces:

1. short stacktrace of failed thread:

https://gist.github.com/7f63bb99c530d26ded18

2. full stacktrace of failed thread:

https://gist.github.com/d9bc7bc8f6a0bbff9e86

3. short stacktrace of all threads:

https://gist.github.com/f31da7725306854c719f

4. full stacktrace of all threads:

https://gist.github.com/65cbc562b01211ea5612

GlusterFS volume configuration:

https://gist.github.com/30f0129d16e25d4a5a52

ganesha.conf:

https://gist.github.com/9b5e59b8d6d8cb84c85d

How I mount NFS share:

===
mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o

defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100
===

On четвер, 7 січня 2016 р. 12:06:42 EET Soumya
Koduri wrote:

Entries_HWMark = 500;

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Mathieu Chateau]

Hello,

I also experience high memory usage on my gluster clients. Sample :
[image: Images intégrées 1]

Can I help in testing/debugging ?



Cordialement,
Mathieu CHATEAU
http://www.lotp.fr

2016-01-12 7:24 GMT+01:00 Soumya Koduri :

>
>
> On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote:
>
>> Brief test shows that Ganesha stopped leaking and crashing, so it seems
>> to be good for me.
>>
>> Thanks for checking.
>
> Nevertheless, back to my original question: what about FUSE client? It
>> is still leaking despite all the fixes applied. Should it be considered
>> another issue?
>>
>
> For fuse client, I tried vfs drop_caches as suggested by Vijay in an
> earlier mail. Though all the inodes get purged, I still doesn't see much
> difference in the memory footprint drop. Need to investigate what else is
> consuming so much memory here.
>
> Thanks,
> Soumya
>
>
>
>> 11.01.2016 12:26, Soumya Koduri написав:
>>
>>> I have made changes to fix the lookup leak in a different way (as
>>> discussed with Pranith) and uploaded them in the latest patch set #4
>>> - http://review.gluster.org/#/c/13096/
>>>
>>> Please check if it resolves the mem leak and hopefully doesn't result
>>> in any assertion :)
>>>
>>> Thanks,
>>> Soumya
>>>
>>> On 01/08/2016 05:04 PM, Soumya Koduri wrote:
>>>
 I could reproduce while testing deep directories with in the mount
 point. I root caus'ed the issue & had discussion with Pranith to
 understand the purpose and recommended way of taking nlookup on inodes.

 I shall make changes to my existing fix and post the patch soon.
 Thanks for your patience!

 -Soumya

 On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote:

> OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the most
> recent
> revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision
> too).
>
> On traversing GlusterFS volume with many files in one folder via NFS
> mount I
> get an assertion:
>
> ===
> ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >=
> nlookup' failed.
> ===
>
> I used GDB on NFS-Ganesha process to get appropriate stacktraces:
>
> 1. short stacktrace of failed thread:
>
> https://gist.github.com/7f63bb99c530d26ded18
>
> 2. full stacktrace of failed thread:
>
> https://gist.github.com/d9bc7bc8f6a0bbff9e86
>
> 3. short stacktrace of all threads:
>
> https://gist.github.com/f31da7725306854c719f
>
> 4. full stacktrace of all threads:
>
> https://gist.github.com/65cbc562b01211ea5612
>
> GlusterFS volume configuration:
>
> https://gist.github.com/30f0129d16e25d4a5a52
>
> ganesha.conf:
>
> https://gist.github.com/9b5e59b8d6d8cb84c85d
>
> How I mount NFS share:
>
> ===
> mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o
> defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100
> ===
>
> On четвер, 7 січня 2016 р. 12:06:42 EET Soumya Koduri wrote:
>
>> Entries_HWMark = 500;
>>
>
>
> ___
 Gluster-users mailing list
 gluster-us...@gluster.org
 http://www.gluster.org/mailman/listinfo/gluster-users

>>> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Mathieu Chateau]

I tried like suggested:

echo 3 > /proc/sys/vm/drop_caches
sync


It lower a bit usage:

before:

[image: Images intégrées 2]

after:

[image: Images intégrées 1]


Cordialement,

Mathieu CHATEAU
http://www.lotp.fr


2016-01-12 7:34 GMT+01:00 Mathieu Chateau :

> Hello,
>
> I also experience high memory usage on my gluster clients. Sample :
> [image: Images intégrées 1]
>
> Can I help in testing/debugging ?
>
>
>
> Cordialement,
> Mathieu CHATEAU
> http://www.lotp.fr
>
> 2016-01-12 7:24 GMT+01:00 Soumya Koduri :
>
>>
>>
>> On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote:
>>
>>> Brief test shows that Ganesha stopped leaking and crashing, so it seems
>>> to be good for me.
>>>
>>> Thanks for checking.
>>
>> Nevertheless, back to my original question: what about FUSE client? It
>>> is still leaking despite all the fixes applied. Should it be considered
>>> another issue?
>>>
>>
>> For fuse client, I tried vfs drop_caches as suggested by Vijay in an
>> earlier mail. Though all the inodes get purged, I still doesn't see much
>> difference in the memory footprint drop. Need to investigate what else is
>> consuming so much memory here.
>>
>> Thanks,
>> Soumya
>>
>>
>>
>>> 11.01.2016 12:26, Soumya Koduri написав:
>>>
 I have made changes to fix the lookup leak in a different way (as
 discussed with Pranith) and uploaded them in the latest patch set #4
 - http://review.gluster.org/#/c/13096/

 Please check if it resolves the mem leak and hopefully doesn't result
 in any assertion :)

 Thanks,
 Soumya

 On 01/08/2016 05:04 PM, Soumya Koduri wrote:

> I could reproduce while testing deep directories with in the mount
> point. I root caus'ed the issue & had discussion with Pranith to
> understand the purpose and recommended way of taking nlookup on inodes.
>
> I shall make changes to my existing fix and post the patch soon.
> Thanks for your patience!
>
> -Soumya
>
> On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote:
>
>> OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the most
>> recent
>> revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision
>> too).
>>
>> On traversing GlusterFS volume with many files in one folder via NFS
>> mount I
>> get an assertion:
>>
>> ===
>> ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup
>> >=
>> nlookup' failed.
>> ===
>>
>> I used GDB on NFS-Ganesha process to get appropriate stacktraces:
>>
>> 1. short stacktrace of failed thread:
>>
>> https://gist.github.com/7f63bb99c530d26ded18
>>
>> 2. full stacktrace of failed thread:
>>
>> https://gist.github.com/d9bc7bc8f6a0bbff9e86
>>
>> 3. short stacktrace of all threads:
>>
>> https://gist.github.com/f31da7725306854c719f
>>
>> 4. full stacktrace of all threads:
>>
>> https://gist.github.com/65cbc562b01211ea5612
>>
>> GlusterFS volume configuration:
>>
>> https://gist.github.com/30f0129d16e25d4a5a52
>>
>> ganesha.conf:
>>
>> https://gist.github.com/9b5e59b8d6d8cb84c85d
>>
>> How I mount NFS share:
>>
>> ===
>> mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o
>> defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100
>> ===
>>
>> On четвер, 7 січня 2016 р. 12:06:42 EET Soumya Koduri wrote:
>>
>>> Entries_HWMark = 500;
>>>
>>
>>
>> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
 ___
>> Gluster-users mailing list
>> gluster-us...@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

I have made changes to fix the lookup leak in a different way (as 
discussed with Pranith) and uploaded them in the latest patch set #4

- http://review.gluster.org/#/c/13096/

Please check if it resolves the mem leak and hopefully doesn't result in 
any assertion :)


Thanks,
Soumya

On 01/08/2016 05:04 PM, Soumya Koduri wrote:

I could reproduce while testing deep directories with in the mount
point. I root caus'ed the issue & had discussion with Pranith to
understand the purpose and recommended way of taking nlookup on inodes.

I shall make changes to my existing fix and post the patch soon.
Thanks for your patience!

-Soumya

On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote:

OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the most
recent
revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision
too).

On traversing GlusterFS volume with many files in one folder via NFS
mount I
get an assertion:

===
ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >=
nlookup' failed.
===

I used GDB on NFS-Ganesha process to get appropriate stacktraces:

1. short stacktrace of failed thread:

https://gist.github.com/7f63bb99c530d26ded18

2. full stacktrace of failed thread:

https://gist.github.com/d9bc7bc8f6a0bbff9e86

3. short stacktrace of all threads:

https://gist.github.com/f31da7725306854c719f

4. full stacktrace of all threads:

https://gist.github.com/65cbc562b01211ea5612

GlusterFS volume configuration:

https://gist.github.com/30f0129d16e25d4a5a52

ganesha.conf:

https://gist.github.com/9b5e59b8d6d8cb84c85d

How I mount NFS share:

===
mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o
defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100
===

On четвер, 7 січня 2016 р. 12:06:42 EET Soumya Koduri wrote:

Entries_HWMark = 500;




___
Gluster-users mailing list
gluster-us...@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote:

Brief test shows that Ganesha stopped leaking and crashing, so it seems
to be good for me.


Thanks for checking.


Nevertheless, back to my original question: what about FUSE client? It
is still leaking despite all the fixes applied. Should it be considered
another issue?


For fuse client, I tried vfs drop_caches as suggested by Vijay in an 
earlier mail. Though all the inodes get purged, I still doesn't see much 
difference in the memory footprint drop. Need to investigate what else 
is consuming so much memory here.


Thanks,
Soumya



11.01.2016 12:26, Soumya Koduri написав:

I have made changes to fix the lookup leak in a different way (as
discussed with Pranith) and uploaded them in the latest patch set #4
- http://review.gluster.org/#/c/13096/

Please check if it resolves the mem leak and hopefully doesn't result
in any assertion :)

Thanks,
Soumya

On 01/08/2016 05:04 PM, Soumya Koduri wrote:

I could reproduce while testing deep directories with in the mount
point. I root caus'ed the issue & had discussion with Pranith to
understand the purpose and recommended way of taking nlookup on inodes.

I shall make changes to my existing fix and post the patch soon.
Thanks for your patience!

-Soumya

On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote:

OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the most
recent
revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision
too).

On traversing GlusterFS volume with many files in one folder via NFS
mount I
get an assertion:

===
ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >=
nlookup' failed.
===

I used GDB on NFS-Ganesha process to get appropriate stacktraces:

1. short stacktrace of failed thread:

https://gist.github.com/7f63bb99c530d26ded18

2. full stacktrace of failed thread:

https://gist.github.com/d9bc7bc8f6a0bbff9e86

3. short stacktrace of all threads:

https://gist.github.com/f31da7725306854c719f

4. full stacktrace of all threads:

https://gist.github.com/65cbc562b01211ea5612

GlusterFS volume configuration:

https://gist.github.com/30f0129d16e25d4a5a52

ganesha.conf:

https://gist.github.com/9b5e59b8d6d8cb84c85d

How I mount NFS share:

===
mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o
defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100
===

On четвер, 7 січня 2016 р. 12:06:42 EET Soumya Koduri wrote:

Entries_HWMark = 500;




___
Gluster-users mailing list
gluster-us...@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Just in case, here is Valgrind output on FUSE client with 3.7.6 + 
API-related patches we discussed before:


https://gist.github.com/cd6605ca19734c1496a4

12.01.2016 08:24, Soumya Koduri написав:

For fuse client, I tried vfs drop_caches as suggested by Vijay in an
earlier mail. Though all the inodes get purged, I still doesn't see
much difference in the memory footprint drop. Need to investigate what
else is consuming so much memory here.

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Brief test shows that Ganesha stopped leaking and crashing, so it seems 
to be good for me.


Nevertheless, back to my original question: what about FUSE client? It 
is still leaking despite all the fixes applied. Should it be considered 
another issue?


11.01.2016 12:26, Soumya Koduri написав:

I have made changes to fix the lookup leak in a different way (as
discussed with Pranith) and uploaded them in the latest patch set #4
- http://review.gluster.org/#/c/13096/

Please check if it resolves the mem leak and hopefully doesn't result
in any assertion :)

Thanks,
Soumya

On 01/08/2016 05:04 PM, Soumya Koduri wrote:

I could reproduce while testing deep directories with in the mount
point. I root caus'ed the issue & had discussion with Pranith to
understand the purpose and recommended way of taking nlookup on 
inodes.


I shall make changes to my existing fix and post the patch soon.
Thanks for your patience!

-Soumya

On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote:
OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the 
most

recent
revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision
too).

On traversing GlusterFS volume with many files in one folder via NFS
mount I
get an assertion:

===
ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup 
>=

nlookup' failed.
===

I used GDB on NFS-Ganesha process to get appropriate stacktraces:

1. short stacktrace of failed thread:

https://gist.github.com/7f63bb99c530d26ded18

2. full stacktrace of failed thread:

https://gist.github.com/d9bc7bc8f6a0bbff9e86

3. short stacktrace of all threads:

https://gist.github.com/f31da7725306854c719f

4. full stacktrace of all threads:

https://gist.github.com/65cbc562b01211ea5612

GlusterFS volume configuration:

https://gist.github.com/30f0129d16e25d4a5a52

ganesha.conf:

https://gist.github.com/9b5e59b8d6d8cb84c85d

How I mount NFS share:

===
mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o
defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100
===

On четвер, 7 січня 2016 р. 12:06:42 EET Soumya Koduri wrote:

Entries_HWMark = 500;




___
Gluster-users mailing list
gluster-us...@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

I could reproduce while testing deep directories with in the mount 
point. I root caus'ed the issue & had discussion with Pranith to 
understand the purpose and recommended way of taking nlookup on inodes.


I shall make changes to my existing fix and post the patch soon.
Thanks for your patience!

-Soumya

On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote:

OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the most recent
revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision too).

On traversing GlusterFS volume with many files in one folder via NFS mount I
get an assertion:

===
ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >=
nlookup' failed.
===

I used GDB on NFS-Ganesha process to get appropriate stacktraces:

1. short stacktrace of failed thread:

https://gist.github.com/7f63bb99c530d26ded18

2. full stacktrace of failed thread:

https://gist.github.com/d9bc7bc8f6a0bbff9e86

3. short stacktrace of all threads:

https://gist.github.com/f31da7725306854c719f

4. full stacktrace of all threads:

https://gist.github.com/65cbc562b01211ea5612

GlusterFS volume configuration:

https://gist.github.com/30f0129d16e25d4a5a52

ganesha.conf:

https://gist.github.com/9b5e59b8d6d8cb84c85d

How I mount NFS share:

===
mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o
defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100
===

On четвер, 7 січня 2016 р. 12:06:42 EET Soumya Koduri wrote:

Entries_HWMark = 500;




___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

OK, here is valgrind log of patched Ganesha (I took recent version of 
your patchset, 8685abfc6d) with Entries_HWMARK set to 500.


https://gist.github.com/5397c152a259b9600af0

See no huge runtime leaks now. However, I've repeated this test with 
another volume in replica and got the following Ganesha error:


===
ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >= 
nlookup' failed.

===

06.01.2016 08:40, Soumya Koduri написав:

On 01/06/2016 03:53 AM, Oleksandr Natalenko wrote:
OK, I've repeated the same traversing test with patched GlusterFS API, 
and

here is new Valgrind log:

https://gist.github.com/17ecb16a11c9aed957f5


Fuse mount doesn't use gfapi helper. Does your above GlusterFS API
application call glfs_fini() during exit? glfs_fini() is responsible
for freeing the memory consumed by gfAPI applications.

Could you repeat the test with nfs-ganesha (which for sure calls
glfs_fini() and purges inodes if exceeds its inode cache limit) if
possible.

Thanks,
Soumya


Still leaks.

On вівторок, 5 січня 2016 р. 22:52:25 EET Soumya Koduri wrote:

On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote:

Unfortunately, both patches didn't make any difference for me.

I've patched 3.7.6 with both patches, recompiled and installed 
patched
GlusterFS package on client side and mounted volume with ~2M of 
files.

The I performed usual tree traverse with simple "find".

Memory RES value went from ~130M at the moment of mounting to ~1.5G
after traversing the volume for ~40 mins. Valgrind log still shows 
lots

of leaks. Here it is:

https://gist.github.com/56906ca6e657c4ffa4a1


Looks like you had done fuse mount. The patches which I have pasted
below apply to gfapi/nfs-ganesha applications.

Also, to resolve the nfs-ganesha issue which I had mentioned below  
(in
case if Entries_HWMARK option gets changed), I have posted below fix 
-

https://review.gerrithub.io/#/c/258687

Thanks,
Soumya


Ideas?

05.01.2016 12:31, Soumya Koduri написав:
I tried to debug the inode* related leaks and seen some 
improvements

after applying the below patches when ran the same test (but will
smaller load). Could you please apply those patches & confirm the
same?

a) http://review.gluster.org/13125

This will fix the inodes & their ctx related leaks during unexport 
and
the program exit. Please check the valgrind output after applying 
the

patch. It should not list any inodes related memory as lost.

b) http://review.gluster.org/13096

The reason the change in Entries_HWMARK (in your earlier mail) dint
have much effect is that the inode_nlookup count doesn't become 
zero
for those handles/inodes being closed by ganesha. Hence those 
inodes

shall get added to inode lru list instead of purge list which shall
get forcefully purged only when the number of gfapi inode table
entries reaches its limit (which is 137012).

This patch fixes those 'nlookup' counts. Please apply this patch 
and
reduce 'Entries_HWMARK' to much lower value and check if it 
decreases

the in-memory being consumed by ganesha process while being active.

CACHEINODE {

 Entries_HWMark = 500;

}


Note: I see an issue with nfs-ganesha during exit when the option
'Entries_HWMARK' gets changed. This is not related to any of the 
above

patches (or rather Gluster) and I am currently debugging it.

Thanks,
Soumya

On 12/25/2015 11:34 PM, Oleksandr Natalenko wrote:

1. test with Cache_Size = 256 and Entries_HWMark = 4096

Before find . -type f:

root  3120  0.6 11.0 879120 208408 ?   Ssl  17:39   0:00
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf 
-N

NIV_EVENT

After:

root  3120 11.4 24.3 1170076 458168 ?  Ssl  17:39  13:39
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf 
-N

NIV_EVENT

~250M leak.

2. test with default values (after ganesha restart)

Before:

root 24937  1.3 10.4 875016 197808 ?   Ssl  19:39   0:00
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf 
-N

NIV_EVENT

After:

root 24937  3.5 18.9 1022544 356340 ?  Ssl  19:39   0:40
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf 
-N

NIV_EVENT

~159M leak.

No reasonable correlation detected. Second test was finished much
faster than
first (I guess, server-side GlusterFS cache or server kernel page
cache is the
cause).

There are ~1.8M files on this test volume.

On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
Another addition: it seems to be GlusterFS API library memory 
leak
because NFS-Ganesha also consumes huge amount of memory while 
doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is 
memory

usage:

===
root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?


nfs-ganesha also has cache 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

On 01/06/2016 01:58 PM, Oleksandr Natalenko wrote:

OK, here is valgrind log of patched Ganesha (I took recent version of
your patchset, 8685abfc6d) with Entries_HWMARK set to 500.

https://gist.github.com/5397c152a259b9600af0

See no huge runtime leaks now.


Glad to hear this :)

However, I've repeated this test with

another volume in replica and got the following Ganesha error:

===
ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >=
nlookup' failed.
===


I repeated the tests on replica volume as well. But haven't hit any 
assert. Could you confirm if you have taken the latest gluster patch set 
#3 ?

 - http://review.gluster.org/#/c/13096/3

If you are hitting the issue even then, please provide the core if possible.

Thanks,
Soumya



06.01.2016 08:40, Soumya Koduri написав:

On 01/06/2016 03:53 AM, Oleksandr Natalenko wrote:

OK, I've repeated the same traversing test with patched GlusterFS
API, and
here is new Valgrind log:

https://gist.github.com/17ecb16a11c9aed957f5


Fuse mount doesn't use gfapi helper. Does your above GlusterFS API
application call glfs_fini() during exit? glfs_fini() is responsible
for freeing the memory consumed by gfAPI applications.

Could you repeat the test with nfs-ganesha (which for sure calls
glfs_fini() and purges inodes if exceeds its inode cache limit) if
possible.

Thanks,
Soumya


Still leaks.

On вівторок, 5 січня 2016 р. 22:52:25 EET Soumya Koduri wrote:

On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote:

Unfortunately, both patches didn't make any difference for me.

I've patched 3.7.6 with both patches, recompiled and installed patched
GlusterFS package on client side and mounted volume with ~2M of files.
The I performed usual tree traverse with simple "find".

Memory RES value went from ~130M at the moment of mounting to ~1.5G
after traversing the volume for ~40 mins. Valgrind log still shows
lots
of leaks. Here it is:

https://gist.github.com/56906ca6e657c4ffa4a1


Looks like you had done fuse mount. The patches which I have pasted
below apply to gfapi/nfs-ganesha applications.

Also, to resolve the nfs-ganesha issue which I had mentioned below (in
case if Entries_HWMARK option gets changed), I have posted below fix -
https://review.gerrithub.io/#/c/258687

Thanks,
Soumya


Ideas?

05.01.2016 12:31, Soumya Koduri написав:

I tried to debug the inode* related leaks and seen some improvements
after applying the below patches when ran the same test (but will
smaller load). Could you please apply those patches & confirm the
same?

a) http://review.gluster.org/13125

This will fix the inodes & their ctx related leaks during unexport
and
the program exit. Please check the valgrind output after applying the
patch. It should not list any inodes related memory as lost.

b) http://review.gluster.org/13096

The reason the change in Entries_HWMARK (in your earlier mail) dint
have much effect is that the inode_nlookup count doesn't become zero
for those handles/inodes being closed by ganesha. Hence those inodes
shall get added to inode lru list instead of purge list which shall
get forcefully purged only when the number of gfapi inode table
entries reaches its limit (which is 137012).

This patch fixes those 'nlookup' counts. Please apply this patch and
reduce 'Entries_HWMARK' to much lower value and check if it decreases
the in-memory being consumed by ganesha process while being active.

CACHEINODE {

 Entries_HWMark = 500;

}


Note: I see an issue with nfs-ganesha during exit when the option
'Entries_HWMARK' gets changed. This is not related to any of the
above
patches (or rather Gluster) and I am currently debugging it.

Thanks,
Soumya

On 12/25/2015 11:34 PM, Oleksandr Natalenko wrote:

1. test with Cache_Size = 256 and Entries_HWMark = 4096

Before find . -type f:

root  3120  0.6 11.0 879120 208408 ?   Ssl  17:39   0:00
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

After:

root  3120 11.4 24.3 1170076 458168 ?  Ssl  17:39  13:39
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

~250M leak.

2. test with default values (after ganesha restart)

Before:

root 24937  1.3 10.4 875016 197808 ?   Ssl  19:39   0:00
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

After:

root 24937  3.5 18.9 1022544 356340 ?  Ssl  19:39   0:40
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

~159M leak.

No reasonable correlation detected. Second test was finished much
faster than
first (I guess, server-side GlusterFS cache or server kernel page
cache is the
cause).

There are ~1.8M files on this test volume.

On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:

Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

I tried to debug the inode* related leaks and seen some improvements 
after applying the below patches when ran the same test (but will 
smaller load). Could you please apply those patches & confirm the same?


a) http://review.gluster.org/13125

This will fix the inodes & their ctx related leaks during unexport and 
the program exit. Please check the valgrind output after applying the 
patch. It should not list any inodes related memory as lost.


b) http://review.gluster.org/13096

The reason the change in Entries_HWMARK (in your earlier mail) dint have 
much effect is that the inode_nlookup count doesn't become zero for 
those handles/inodes being closed by ganesha. Hence those inodes shall 
get added to inode lru list instead of purge list which shall get 
forcefully purged only when the number of gfapi inode table entries 
reaches its limit (which is 137012).


This patch fixes those 'nlookup' counts. Please apply this patch and 
reduce 'Entries_HWMARK' to much lower value and check if it decreases 
the in-memory being consumed by ganesha process while being active.


CACHEINODE {
Entries_HWMark = 500;
}


Note: I see an issue with nfs-ganesha during exit when the option 
'Entries_HWMARK' gets changed. This is not related to any of the above 
patches (or rather Gluster) and I am currently debugging it.


Thanks,
Soumya


On 12/25/2015 11:34 PM, Oleksandr Natalenko wrote:

1. test with Cache_Size = 256 and Entries_HWMark = 4096

Before find . -type f:

root  3120  0.6 11.0 879120 208408 ?   Ssl  17:39   0:00 /usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT

After:

root  3120 11.4 24.3 1170076 458168 ?  Ssl  17:39  13:39 /usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT

~250M leak.

2. test with default values (after ganesha restart)

Before:

root 24937  1.3 10.4 875016 197808 ?   Ssl  19:39   0:00 /usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT

After:

root 24937  3.5 18.9 1022544 356340 ?  Ssl  19:39   0:40 /usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT

~159M leak.

No reasonable correlation detected. Second test was finished much faster than
first (I guess, server-side GlusterFS cache or server kernel page cache is the
cause).

There are ~1.8M files on this test volume.

On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:

Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
usage:

===
root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?


nfs-ganesha also has cache layer which can scale to millions of entries
depending on the number of files/directories being looked upon. However
there are parameters to tune it. So either try stat with few entries or
add below block in nfs-ganesha.conf file, set low limits and check the
difference. That may help us narrow down how much memory actually
consumed by core nfs-ganesha and gfAPI.

CACHEINODE {
Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache size
Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max no.
of entries in the cache.
}

Thanks,
Soumya


24.12.2015 16:32, Oleksandr Natalenko написав:

Still actual issue for 3.7.6. Any suggestions?

24.09.2015 10:14, Oleksandr Natalenko написав:

In our GlusterFS deployment we've encountered something like memory
leak in GlusterFS FUSE client.

We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
maildir format). Here is inode stats for both bricks and mountpoint:

===
Brick 1 (Server 1):

Filesystem InodesIUsed

  IFree IUse% Mounted on

/dev/mapper/vg_vd1_misc-lv08_mail   578768144 10954918

  5678132262% /bricks/r6sdLV08_vd1_mail

Brick 2 (Server 2):

Filesystem InodesIUsed

  IFree IUse% Mounted on

/dev/mapper/vg_vd0_misc-lv07_mail   578767984 10954913

  5678130712% /bricks/r6sdLV07_vd0_mail

Mountpoint (Server 3):

Filesystem  InodesIUsed  IFree
IUse% Mounted on
glusterfs.xxx:mail   578767760 10954915  567812845
2% /var/spool/mail/virtual
===

glusterfs.xxx domain has two A records for both Server 1 and Server 2.

Here is volume info:

===
Volume Name: mail
Type: Replicate
Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Brick2: 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Correct, I used FUSE mount. Shouldn't gfapi be used by FUSE mount helper (/
usr/bin/glusterfs)?

On вівторок, 5 січня 2016 р. 22:52:25 EET Soumya Koduri wrote:
> On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote:
> > Unfortunately, both patches didn't make any difference for me.
> > 
> > I've patched 3.7.6 with both patches, recompiled and installed patched
> > GlusterFS package on client side and mounted volume with ~2M of files.
> > The I performed usual tree traverse with simple "find".
> > 
> > Memory RES value went from ~130M at the moment of mounting to ~1.5G
> > after traversing the volume for ~40 mins. Valgrind log still shows lots
> > of leaks. Here it is:
> > 
> > https://gist.github.com/56906ca6e657c4ffa4a1
> 
> Looks like you had done fuse mount. The patches which I have pasted
> below apply to gfapi/nfs-ganesha applications.
> 
> Also, to resolve the nfs-ganesha issue which I had mentioned below  (in
> case if Entries_HWMARK option gets changed), I have posted below fix -
>   https://review.gerrithub.io/#/c/258687
> 
> Thanks,
> Soumya
> 
> > Ideas?
> > 
> > 05.01.2016 12:31, Soumya Koduri написав:
> >> I tried to debug the inode* related leaks and seen some improvements
> >> after applying the below patches when ran the same test (but will
> >> smaller load). Could you please apply those patches & confirm the
> >> same?
> >> 
> >> a) http://review.gluster.org/13125
> >> 
> >> This will fix the inodes & their ctx related leaks during unexport and
> >> the program exit. Please check the valgrind output after applying the
> >> patch. It should not list any inodes related memory as lost.
> >> 
> >> b) http://review.gluster.org/13096
> >> 
> >> The reason the change in Entries_HWMARK (in your earlier mail) dint
> >> have much effect is that the inode_nlookup count doesn't become zero
> >> for those handles/inodes being closed by ganesha. Hence those inodes
> >> shall get added to inode lru list instead of purge list which shall
> >> get forcefully purged only when the number of gfapi inode table
> >> entries reaches its limit (which is 137012).
> >> 
> >> This patch fixes those 'nlookup' counts. Please apply this patch and
> >> reduce 'Entries_HWMARK' to much lower value and check if it decreases
> >> the in-memory being consumed by ganesha process while being active.
> >> 
> >> CACHEINODE {
> >> 
> >> Entries_HWMark = 500;
> >> 
> >> }
> >> 
> >> 
> >> Note: I see an issue with nfs-ganesha during exit when the option
> >> 'Entries_HWMARK' gets changed. This is not related to any of the above
> >> patches (or rather Gluster) and I am currently debugging it.
> >> 
> >> Thanks,
> >> Soumya
> >> 
> >> On 12/25/2015 11:34 PM, Oleksandr Natalenko wrote:
> >>> 1. test with Cache_Size = 256 and Entries_HWMark = 4096
> >>> 
> >>> Before find . -type f:
> >>> 
> >>> root  3120  0.6 11.0 879120 208408 ?   Ssl  17:39   0:00
> >>> /usr/bin/
> >>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
> >>> NIV_EVENT
> >>> 
> >>> After:
> >>> 
> >>> root  3120 11.4 24.3 1170076 458168 ?  Ssl  17:39  13:39
> >>> /usr/bin/
> >>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
> >>> NIV_EVENT
> >>> 
> >>> ~250M leak.
> >>> 
> >>> 2. test with default values (after ganesha restart)
> >>> 
> >>> Before:
> >>> 
> >>> root 24937  1.3 10.4 875016 197808 ?   Ssl  19:39   0:00
> >>> /usr/bin/
> >>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
> >>> NIV_EVENT
> >>> 
> >>> After:
> >>> 
> >>> root 24937  3.5 18.9 1022544 356340 ?  Ssl  19:39   0:40
> >>> /usr/bin/
> >>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
> >>> NIV_EVENT
> >>> 
> >>> ~159M leak.
> >>> 
> >>> No reasonable correlation detected. Second test was finished much
> >>> faster than
> >>> first (I guess, server-side GlusterFS cache or server kernel page
> >>> cache is the
> >>> cause).
> >>> 
> >>> There are ~1.8M files on this test volume.
> >>> 
> >>> On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:
>  On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
> > Another addition: it seems to be GlusterFS API library memory leak
> > because NFS-Ganesha also consumes huge amount of memory while doing
> > ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
> > usage:
> > 
> > ===
> > root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
> > /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
> > /etc/ganesha/ganesha.conf -N NIV_EVENT
> > ===
> > 
> > 1.4G is too much for simple stat() :(.
> > 
> > Ideas?
>  
>  nfs-ganesha also has cache layer which can scale to millions of entries
>  depending on the number of files/directories being looked upon. However
>  there are parameters to tune it. So either try stat with few entries or
>  add below block in nfs-ganesha.conf file, set low limits and check the
>  

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote:

Unfortunately, both patches didn't make any difference for me.

I've patched 3.7.6 with both patches, recompiled and installed patched
GlusterFS package on client side and mounted volume with ~2M of files.
The I performed usual tree traverse with simple "find".

Memory RES value went from ~130M at the moment of mounting to ~1.5G
after traversing the volume for ~40 mins. Valgrind log still shows lots
of leaks. Here it is:

https://gist.github.com/56906ca6e657c4ffa4a1


Looks like you had done fuse mount. The patches which I have pasted 
below apply to gfapi/nfs-ganesha applications.


Also, to resolve the nfs-ganesha issue which I had mentioned below  (in 
case if Entries_HWMARK option gets changed), I have posted below fix -

https://review.gerrithub.io/#/c/258687

Thanks,
Soumya



Ideas?

05.01.2016 12:31, Soumya Koduri написав:

I tried to debug the inode* related leaks and seen some improvements
after applying the below patches when ran the same test (but will
smaller load). Could you please apply those patches & confirm the
same?

a) http://review.gluster.org/13125

This will fix the inodes & their ctx related leaks during unexport and
the program exit. Please check the valgrind output after applying the
patch. It should not list any inodes related memory as lost.

b) http://review.gluster.org/13096

The reason the change in Entries_HWMARK (in your earlier mail) dint
have much effect is that the inode_nlookup count doesn't become zero
for those handles/inodes being closed by ganesha. Hence those inodes
shall get added to inode lru list instead of purge list which shall
get forcefully purged only when the number of gfapi inode table
entries reaches its limit (which is 137012).

This patch fixes those 'nlookup' counts. Please apply this patch and
reduce 'Entries_HWMARK' to much lower value and check if it decreases
the in-memory being consumed by ganesha process while being active.

CACHEINODE {
Entries_HWMark = 500;
}


Note: I see an issue with nfs-ganesha during exit when the option
'Entries_HWMARK' gets changed. This is not related to any of the above
patches (or rather Gluster) and I am currently debugging it.

Thanks,
Soumya


On 12/25/2015 11:34 PM, Oleksandr Natalenko wrote:

1. test with Cache_Size = 256 and Entries_HWMark = 4096

Before find . -type f:

root  3120  0.6 11.0 879120 208408 ?   Ssl  17:39   0:00
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

After:

root  3120 11.4 24.3 1170076 458168 ?  Ssl  17:39  13:39
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

~250M leak.

2. test with default values (after ganesha restart)

Before:

root 24937  1.3 10.4 875016 197808 ?   Ssl  19:39   0:00
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

After:

root 24937  3.5 18.9 1022544 356340 ?  Ssl  19:39   0:40
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

~159M leak.

No reasonable correlation detected. Second test was finished much
faster than
first (I guess, server-side GlusterFS cache or server kernel page
cache is the
cause).

There are ~1.8M files on this test volume.

On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:

Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
usage:

===
root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?


nfs-ganesha also has cache layer which can scale to millions of entries
depending on the number of files/directories being looked upon. However
there are parameters to tune it. So either try stat with few entries or
add below block in nfs-ganesha.conf file, set low limits and check the
difference. That may help us narrow down how much memory actually
consumed by core nfs-ganesha and gfAPI.

CACHEINODE {
Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); #
cache size
Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10);
#Max no.
of entries in the cache.
}

Thanks,
Soumya


24.12.2015 16:32, Oleksandr Natalenko написав:

Still actual issue for 3.7.6. Any suggestions?

24.09.2015 10:14, Oleksandr Natalenko написав:

In our GlusterFS deployment we've encountered something like memory
leak in GlusterFS FUSE client.

We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
maildir format). Here is inode stats for both bricks and mountpoint:

===
Brick 1 (Server 1):

Filesystem Inodes IUsed

  IFree IUse% Mounted on

/dev/mapper/vg_vd1_misc-lv08_mail 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

On 01/06/2016 03:53 AM, Oleksandr Natalenko wrote:

OK, I've repeated the same traversing test with patched GlusterFS API, and
here is new Valgrind log:

https://gist.github.com/17ecb16a11c9aed957f5

Fuse mount doesn't use gfapi helper. Does your above GlusterFS API 
application call glfs_fini() during exit? glfs_fini() is responsible for 
freeing the memory consumed by gfAPI applications.


Could you repeat the test with nfs-ganesha (which for sure calls 
glfs_fini() and purges inodes if exceeds its inode cache limit) if possible.


Thanks,
Soumya


Still leaks.

On вівторок, 5 січня 2016 р. 22:52:25 EET Soumya Koduri wrote:

On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote:

Unfortunately, both patches didn't make any difference for me.

I've patched 3.7.6 with both patches, recompiled and installed patched
GlusterFS package on client side and mounted volume with ~2M of files.
The I performed usual tree traverse with simple "find".

Memory RES value went from ~130M at the moment of mounting to ~1.5G
after traversing the volume for ~40 mins. Valgrind log still shows lots
of leaks. Here it is:

https://gist.github.com/56906ca6e657c4ffa4a1


Looks like you had done fuse mount. The patches which I have pasted
below apply to gfapi/nfs-ganesha applications.

Also, to resolve the nfs-ganesha issue which I had mentioned below  (in
case if Entries_HWMARK option gets changed), I have posted below fix -
https://review.gerrithub.io/#/c/258687

Thanks,
Soumya


Ideas?

05.01.2016 12:31, Soumya Koduri написав:

I tried to debug the inode* related leaks and seen some improvements
after applying the below patches when ran the same test (but will
smaller load). Could you please apply those patches & confirm the
same?

a) http://review.gluster.org/13125

This will fix the inodes & their ctx related leaks during unexport and
the program exit. Please check the valgrind output after applying the
patch. It should not list any inodes related memory as lost.

b) http://review.gluster.org/13096

The reason the change in Entries_HWMARK (in your earlier mail) dint
have much effect is that the inode_nlookup count doesn't become zero
for those handles/inodes being closed by ganesha. Hence those inodes
shall get added to inode lru list instead of purge list which shall
get forcefully purged only when the number of gfapi inode table
entries reaches its limit (which is 137012).

This patch fixes those 'nlookup' counts. Please apply this patch and
reduce 'Entries_HWMARK' to much lower value and check if it decreases
the in-memory being consumed by ganesha process while being active.

CACHEINODE {

 Entries_HWMark = 500;

}


Note: I see an issue with nfs-ganesha during exit when the option
'Entries_HWMARK' gets changed. This is not related to any of the above
patches (or rather Gluster) and I am currently debugging it.

Thanks,
Soumya

On 12/25/2015 11:34 PM, Oleksandr Natalenko wrote:

1. test with Cache_Size = 256 and Entries_HWMark = 4096

Before find . -type f:

root  3120  0.6 11.0 879120 208408 ?   Ssl  17:39   0:00
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

After:

root  3120 11.4 24.3 1170076 458168 ?  Ssl  17:39  13:39
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

~250M leak.

2. test with default values (after ganesha restart)

Before:

root 24937  1.3 10.4 875016 197808 ?   Ssl  19:39   0:00
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

After:

root 24937  3.5 18.9 1022544 356340 ?  Ssl  19:39   0:40
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
NIV_EVENT

~159M leak.

No reasonable correlation detected. Second test was finished much
faster than
first (I guess, server-side GlusterFS cache or server kernel page
cache is the
cause).

There are ~1.8M files on this test volume.

On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:

Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
usage:

===
root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?


nfs-ganesha also has cache layer which can scale to millions of entries
depending on the number of files/directories being looked upon. However
there are parameters to tune it. So either try stat with few entries or
add below block in nfs-ganesha.conf file, set low limits and check the
difference. That may help us narrow down how much memory actually
consumed by core nfs-ganesha and gfAPI.

CACHEINODE {

 Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); #

cache size

 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

OK, I've repeated the same traversing test with patched GlusterFS API, and 
here is new Valgrind log:

https://gist.github.com/17ecb16a11c9aed957f5

Still leaks.

On вівторок, 5 січня 2016 р. 22:52:25 EET Soumya Koduri wrote:
> On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote:
> > Unfortunately, both patches didn't make any difference for me.
> > 
> > I've patched 3.7.6 with both patches, recompiled and installed patched
> > GlusterFS package on client side and mounted volume with ~2M of files.
> > The I performed usual tree traverse with simple "find".
> > 
> > Memory RES value went from ~130M at the moment of mounting to ~1.5G
> > after traversing the volume for ~40 mins. Valgrind log still shows lots
> > of leaks. Here it is:
> > 
> > https://gist.github.com/56906ca6e657c4ffa4a1
> 
> Looks like you had done fuse mount. The patches which I have pasted
> below apply to gfapi/nfs-ganesha applications.
> 
> Also, to resolve the nfs-ganesha issue which I had mentioned below  (in
> case if Entries_HWMARK option gets changed), I have posted below fix -
>   https://review.gerrithub.io/#/c/258687
> 
> Thanks,
> Soumya
> 
> > Ideas?
> > 
> > 05.01.2016 12:31, Soumya Koduri написав:
> >> I tried to debug the inode* related leaks and seen some improvements
> >> after applying the below patches when ran the same test (but will
> >> smaller load). Could you please apply those patches & confirm the
> >> same?
> >> 
> >> a) http://review.gluster.org/13125
> >> 
> >> This will fix the inodes & their ctx related leaks during unexport and
> >> the program exit. Please check the valgrind output after applying the
> >> patch. It should not list any inodes related memory as lost.
> >> 
> >> b) http://review.gluster.org/13096
> >> 
> >> The reason the change in Entries_HWMARK (in your earlier mail) dint
> >> have much effect is that the inode_nlookup count doesn't become zero
> >> for those handles/inodes being closed by ganesha. Hence those inodes
> >> shall get added to inode lru list instead of purge list which shall
> >> get forcefully purged only when the number of gfapi inode table
> >> entries reaches its limit (which is 137012).
> >> 
> >> This patch fixes those 'nlookup' counts. Please apply this patch and
> >> reduce 'Entries_HWMARK' to much lower value and check if it decreases
> >> the in-memory being consumed by ganesha process while being active.
> >> 
> >> CACHEINODE {
> >> 
> >> Entries_HWMark = 500;
> >> 
> >> }
> >> 
> >> 
> >> Note: I see an issue with nfs-ganesha during exit when the option
> >> 'Entries_HWMARK' gets changed. This is not related to any of the above
> >> patches (or rather Gluster) and I am currently debugging it.
> >> 
> >> Thanks,
> >> Soumya
> >> 
> >> On 12/25/2015 11:34 PM, Oleksandr Natalenko wrote:
> >>> 1. test with Cache_Size = 256 and Entries_HWMark = 4096
> >>> 
> >>> Before find . -type f:
> >>> 
> >>> root  3120  0.6 11.0 879120 208408 ?   Ssl  17:39   0:00
> >>> /usr/bin/
> >>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
> >>> NIV_EVENT
> >>> 
> >>> After:
> >>> 
> >>> root  3120 11.4 24.3 1170076 458168 ?  Ssl  17:39  13:39
> >>> /usr/bin/
> >>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
> >>> NIV_EVENT
> >>> 
> >>> ~250M leak.
> >>> 
> >>> 2. test with default values (after ganesha restart)
> >>> 
> >>> Before:
> >>> 
> >>> root 24937  1.3 10.4 875016 197808 ?   Ssl  19:39   0:00
> >>> /usr/bin/
> >>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
> >>> NIV_EVENT
> >>> 
> >>> After:
> >>> 
> >>> root 24937  3.5 18.9 1022544 356340 ?  Ssl  19:39   0:40
> >>> /usr/bin/
> >>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
> >>> NIV_EVENT
> >>> 
> >>> ~159M leak.
> >>> 
> >>> No reasonable correlation detected. Second test was finished much
> >>> faster than
> >>> first (I guess, server-side GlusterFS cache or server kernel page
> >>> cache is the
> >>> cause).
> >>> 
> >>> There are ~1.8M files on this test volume.
> >>> 
> >>> On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:
>  On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
> > Another addition: it seems to be GlusterFS API library memory leak
> > because NFS-Ganesha also consumes huge amount of memory while doing
> > ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
> > usage:
> > 
> > ===
> > root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
> > /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
> > /etc/ganesha/ganesha.conf -N NIV_EVENT
> > ===
> > 
> > 1.4G is too much for simple stat() :(.
> > 
> > Ideas?
>  
>  nfs-ganesha also has cache layer which can scale to millions of entries
>  depending on the number of files/directories being looked upon. However
>  there are parameters to tune it. So either try stat with few entries or
>  add below block in 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Unfortunately, both patches didn't make any difference for me.

I've patched 3.7.6 with both patches, recompiled and installed patched 
GlusterFS package on client side and mounted volume with ~2M of files. 
The I performed usual tree traverse with simple "find".


Memory RES value went from ~130M at the moment of mounting to ~1.5G 
after traversing the volume for ~40 mins. Valgrind log still shows lots 
of leaks. Here it is:


https://gist.github.com/56906ca6e657c4ffa4a1

Ideas?

05.01.2016 12:31, Soumya Koduri написав:

I tried to debug the inode* related leaks and seen some improvements
after applying the below patches when ran the same test (but will
smaller load). Could you please apply those patches & confirm the
same?

a) http://review.gluster.org/13125

This will fix the inodes & their ctx related leaks during unexport and
the program exit. Please check the valgrind output after applying the
patch. It should not list any inodes related memory as lost.

b) http://review.gluster.org/13096

The reason the change in Entries_HWMARK (in your earlier mail) dint
have much effect is that the inode_nlookup count doesn't become zero
for those handles/inodes being closed by ganesha. Hence those inodes
shall get added to inode lru list instead of purge list which shall
get forcefully purged only when the number of gfapi inode table
entries reaches its limit (which is 137012).

This patch fixes those 'nlookup' counts. Please apply this patch and
reduce 'Entries_HWMARK' to much lower value and check if it decreases
the in-memory being consumed by ganesha process while being active.

CACHEINODE {
Entries_HWMark = 500;
}


Note: I see an issue with nfs-ganesha during exit when the option
'Entries_HWMARK' gets changed. This is not related to any of the above
patches (or rather Gluster) and I am currently debugging it.

Thanks,
Soumya


On 12/25/2015 11:34 PM, Oleksandr Natalenko wrote:

1. test with Cache_Size = 256 and Entries_HWMark = 4096

Before find . -type f:

root  3120  0.6 11.0 879120 208408 ?   Ssl  17:39   0:00 
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N 
NIV_EVENT


After:

root  3120 11.4 24.3 1170076 458168 ?  Ssl  17:39  13:39 
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N 
NIV_EVENT


~250M leak.

2. test with default values (after ganesha restart)

Before:

root 24937  1.3 10.4 875016 197808 ?   Ssl  19:39   0:00 
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N 
NIV_EVENT


After:

root 24937  3.5 18.9 1022544 356340 ?  Ssl  19:39   0:40 
/usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N 
NIV_EVENT


~159M leak.

No reasonable correlation detected. Second test was finished much 
faster than
first (I guess, server-side GlusterFS cache or server kernel page 
cache is the

cause).

There are ~1.8M files on this test volume.

On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:

Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is 
memory

usage:

===
root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?


nfs-ganesha also has cache layer which can scale to millions of 
entries
depending on the number of files/directories being looked upon. 
However
there are parameters to tune it. So either try stat with few entries 
or
add below block in nfs-ganesha.conf file, set low limits and check 
the

difference. That may help us narrow down how much memory actually
consumed by core nfs-ganesha and gfAPI.

CACHEINODE {
	Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache 
size
	Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max 
no.

of entries in the cache.
}

Thanks,
Soumya


24.12.2015 16:32, Oleksandr Natalenko написав:

Still actual issue for 3.7.6. Any suggestions?

24.09.2015 10:14, Oleksandr Natalenko написав:
In our GlusterFS deployment we've encountered something like 
memory

leak in GlusterFS FUSE client.

We use replicated (×2) GlusterFS volume to store mail 
(exim+dovecot,
maildir format). Here is inode stats for both bricks and 
mountpoint:


===
Brick 1 (Server 1):

Filesystem Inodes
IUsed


  IFree IUse% Mounted on

/dev/mapper/vg_vd1_misc-lv08_mail   578768144 
10954918


  5678132262% /bricks/r6sdLV08_vd1_mail

Brick 2 (Server 2):

Filesystem Inodes
IUsed


  IFree IUse% Mounted on

/dev/mapper/vg_vd0_misc-lv07_mail   578767984 
10954913


  5678130712% /bricks/r6sdLV07_vd0_mail

Mountpoint (Server 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Another Valgrind run.

I did the following:

===
valgrind --leak-check=full --show-leak-kinds=all --log-
file="valgrind_fuse.log" /usr/bin/glusterfs -N --volfile-
server=some.server.com --volfile-id=somevolume /mnt/volume
===

then cd to /mnt/volume and find . -type f. After traversing some part of 
hierarchy I've stopped find and did umount /mnt/volume. Here is 
valgrind_fuse.log file:

https://gist.github.com/7e2679e1e72e48f75a2b

On четвер, 31 грудня 2015 р. 14:09:03 EET Soumya Koduri wrote:
> On 12/28/2015 02:32 PM, Soumya Koduri wrote:
> > - Original Message -
> > 
> >> From: "Pranith Kumar Karampuri" <pkara...@redhat.com>
> >> To: "Oleksandr Natalenko" <oleksa...@natalenko.name>, "Soumya Koduri"
> >> <skod...@redhat.com> Cc: gluster-us...@gluster.org,
> >> gluster-devel@gluster.org
> >> Sent: Monday, December 28, 2015 9:32:07 AM
> >> Subject: Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS
> >> FUSE client>> 
> >> On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote:
> >>> Also, here is valgrind output with our custom tool, that does GlusterFS
> >>> volume
> >>> traversing (with simple stats) just like find tool. In this case
> >>> NFS-Ganesha
> >>> is not used.
> >>> 
> >>> https://gist.github.com/e4602a50d3c98f7a2766
> >> 
> >> hi Oleksandr,
> >> 
> >> I went through the code. Both NFS Ganesha and the custom tool use
> >> 
> >> gfapi and the leak is stemming from that. I am not very familiar with
> >> this part of code but there seems to be one inode_unref() that is
> >> missing in failure path of resolution. Not sure if that is corresponding
> >> to the leaks.
> >> 
> >> Soumya,
> >> 
> >>  Could this be the issue? review.gluster.org seems to be down. So
> >> 
> >> couldn't send the patch. Please ping me on IRC.
> >> diff --git a/api/src/glfs-resolve.c b/api/src/glfs-resolve.c
> >> index b5efcba..52b538b 100644
> >> --- a/api/src/glfs-resolve.c
> >> +++ b/api/src/glfs-resolve.c
> >> @@ -467,9 +467,11 @@ priv_glfs_resolve_at (struct glfs *fs, xlator_t
> >> *subvol, inode_t *at,
> >> 
> >>   }
> >>   
> >>   }
> >> 
> >> -   if (parent && next_component)
> >> +   if (parent && next_component) {
> >> +   inode_unref (parent);
> >> +   parent = NULL;
> >> 
> >>   /* resolution failed mid-way */
> >>   goto out;
> >> 
> >> +}
> >> 
> >>   /* At this point, all components up to the last parent
> >>   directory
> >>   
> >>  have been resolved successfully (@parent). Resolution of
> >> 
> >> basename
> > 
> > yes. This could be one of the reasons. There are few leaks with respect to
> > inode references in gfAPI. See below.
> > 
> > 
> > On GlusterFS side, looks like majority of the leaks are related to inodes
> > and their contexts. Possible reasons which I can think of are:
> > 
> > 1) When there is a graph switch, old inode table and their entries are not
> > purged (this is a known issue). There was an effort put to fix this
> > issue. But I think it had other side-effects and hence not been applied.
> > Maybe we should revive those changes again.
> > 
> > 2) With regard to above, old entries can be purged in case if any request
> > comes with the reference to old inode (as part of 'glfs_resolve_inode'),
> > provided their reference counts are properly decremented. But this is not
> > happening at the moment in gfapi.
> > 
> > 3) Applications should hold and release their reference as needed and
> > required. There are certain fixes needed in this area as well (including
> > the fix provided by Pranith above).> 
> >  From code-inspection, have made changes to fix few leaks of case (2) &
> >  (3) with respect to gfAPI.>  
> > http://review.gluster.org/#/c/13096 (yet to test the changes)
> > 
> > I haven't yet narrowed down any suspects pertaining to only NFS-Ganesha.
> > Will re-check and update.
> I tried similar tests but with smaller set of files. I could see the
> inode_ctx leak even without graph switches involved. I suspect that
> could be because valgrind checks for memory leaks during the exit of the
> program. We call '

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Vijay Bellur]

On 01/03/2016 09:23 AM, Oleksandr Natalenko wrote:

Another Valgrind run.

I did the following:

===
valgrind --leak-check=full --show-leak-kinds=all --log-
file="valgrind_fuse.log" /usr/bin/glusterfs -N --volfile-
server=some.server.com --volfile-id=somevolume /mnt/volume
===

then cd to /mnt/volume and find . -type f. After traversing some part of
hierarchy I've stopped find and did umount /mnt/volume. Here is
valgrind_fuse.log file:

https://gist.github.com/7e2679e1e72e48f75a2b



Can you please try the same by dropping caches before umount?

echo 3 > /proc/sys/vm/drop_caches

Gluster relies on vfs sending forgets and releases to clean up the 
inodes and the contexts in the inodes maintained by various translators.


Thanks,
Vijay

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Here is another Valgrind log of similar scenario but with drop_caches before 
umount:

https://gist.github.com/06997ecc8c7bce83aec1

Also, I've tried to drop caches on production VM with GluserFS volume mounted 
and memleaking for several weeks with absolutely no effect:

===
root   945  0.1 48.2 1273900 739244 ?  Ssl   2015  58:54 /usr/sbin/
glusterfs --volfile-server=server.example.com --volfile-id=volume /mnt/volume
===

The numbers above stayed the same before drop as well as 5 mins after the 
drop.

On неділя, 3 січня 2016 р. 13:35:51 EET Vijay Bellur wrote:
> /proc/sys/vm/drop_caches


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

- Original Message -
> From: "Pranith Kumar Karampuri" <pkara...@redhat.com>
> To: "Oleksandr Natalenko" <oleksa...@natalenko.name>, "Soumya Koduri" 
> <skod...@redhat.com>
> Cc: gluster-us...@gluster.org, gluster-devel@gluster.org
> Sent: Monday, December 28, 2015 9:32:07 AM
> Subject: Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE 
> client
> 
> 
> 
> On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote:
> > Also, here is valgrind output with our custom tool, that does GlusterFS
> > volume
> > traversing (with simple stats) just like find tool. In this case
> > NFS-Ganesha
> > is not used.
> >
> > https://gist.github.com/e4602a50d3c98f7a2766
> hi Oleksandr,
>I went through the code. Both NFS Ganesha and the custom tool use
> gfapi and the leak is stemming from that. I am not very familiar with
> this part of code but there seems to be one inode_unref() that is
> missing in failure path of resolution. Not sure if that is corresponding
> to the leaks.
> 
> Soumya,
> Could this be the issue? review.gluster.org seems to be down. So
> couldn't send the patch. Please ping me on IRC.
> diff --git a/api/src/glfs-resolve.c b/api/src/glfs-resolve.c
> index b5efcba..52b538b 100644
> --- a/api/src/glfs-resolve.c
> +++ b/api/src/glfs-resolve.c
> @@ -467,9 +467,11 @@ priv_glfs_resolve_at (struct glfs *fs, xlator_t
> *subvol, inode_t *at,
>  }
>  }
> 
> -   if (parent && next_component)
> +   if (parent && next_component) {
> +   inode_unref (parent);
> +   parent = NULL;
>  /* resolution failed mid-way */
>  goto out;
> +}
> 
>  /* At this point, all components up to the last parent directory
> have been resolved successfully (@parent). Resolution of
> basename
> 
yes. This could be one of the reasons. There are few leaks with respect to 
inode references in gfAPI. See below.


On GlusterFS side, looks like majority of the leaks are related to inodes and 
their contexts. Possible reasons which I can think of are:

1) When there is a graph switch, old inode table and their entries are not 
purged (this is a known issue). There was an effort put to fix this issue. But 
I think it had other side-effects and hence not been applied. Maybe we should 
revive those changes again.

2) With regard to above, old entries can be purged in case if any request comes 
with the reference to old inode (as part of 'glfs_resolve_inode'), provided 
their reference counts are properly decremented. But this is not happening at 
the moment in gfapi.

3) Applications should hold and release their reference as needed and required. 
There are certain fixes needed in this area as well (including the fix provided 
by Pranith above).

From code-inspection, have made changes to fix few leaks of case (2) & (3) with 
respect to gfAPI.
http://review.gluster.org/#/c/13096 (yet to test the changes)

I haven't yet narrowed down any suspects pertaining to only NFS-Ganesha. Will 
re-check and update.

Thanks,
Soumya


> Pranith
> >
> > One may see GlusterFS-related leaks here as well.
> >
> > On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:
> >> On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
> >>> Another addition: it seems to be GlusterFS API library memory leak
> >>> because NFS-Ganesha also consumes huge amount of memory while doing
> >>> ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
> >>> usage:
> >>>
> >>> ===
> >>> root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
> >>> /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
> >>> /etc/ganesha/ganesha.conf -N NIV_EVENT
> >>> ===
> >>>
> >>> 1.4G is too much for simple stat() :(.
> >>>
> >>> Ideas?
> >> nfs-ganesha also has cache layer which can scale to millions of entries
> >> depending on the number of files/directories being looked upon. However
> >> there are parameters to tune it. So either try stat with few entries or
> >> add below block in nfs-ganesha.conf file, set low limits and check the
> >> difference. That may help us narrow down how much memory actually
> >> consumed by core nfs-ganesha and gfAPI.
> >>
> >> CACHEINODE {
> >>Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache size
> >>Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max no.
> >> of entries in the cache.
> >>

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Pranith Kumar Karampuri]

On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote:

Also, here is valgrind output with our custom tool, that does GlusterFS volume
traversing (with simple stats) just like find tool. In this case NFS-Ganesha
is not used.

https://gist.github.com/e4602a50d3c98f7a2766

hi Oleksandr,
  I went through the code. Both NFS Ganesha and the custom tool use 
gfapi and the leak is stemming from that. I am not very familiar with 
this part of code but there seems to be one inode_unref() that is 
missing in failure path of resolution. Not sure if that is corresponding 
to the leaks.


Soumya,
   Could this be the issue? review.gluster.org seems to be down. So 
couldn't send the patch. Please ping me on IRC.

diff --git a/api/src/glfs-resolve.c b/api/src/glfs-resolve.c
index b5efcba..52b538b 100644
--- a/api/src/glfs-resolve.c
+++ b/api/src/glfs-resolve.c
@@ -467,9 +467,11 @@ priv_glfs_resolve_at (struct glfs *fs, xlator_t 
*subvol, inode_t *at,

}
}

-   if (parent && next_component)
+   if (parent && next_component) {
+   inode_unref (parent);
+   parent = NULL;
/* resolution failed mid-way */
goto out;
+}

/* At this point, all components up to the last parent directory
   have been resolved successfully (@parent). Resolution of 
basename


Pranith


One may see GlusterFS-related leaks here as well.

On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:

Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
usage:

===
root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?

nfs-ganesha also has cache layer which can scale to millions of entries
depending on the number of files/directories being looked upon. However
there are parameters to tune it. So either try stat with few entries or
add below block in nfs-ganesha.conf file, set low limits and check the
difference. That may help us narrow down how much memory actually
consumed by core nfs-ganesha and gfAPI.

CACHEINODE {
Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache size
Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max no.
of entries in the cache.
}

Thanks,
Soumya


24.12.2015 16:32, Oleksandr Natalenko написав:

Still actual issue for 3.7.6. Any suggestions?

24.09.2015 10:14, Oleksandr Natalenko написав:

In our GlusterFS deployment we've encountered something like memory
leak in GlusterFS FUSE client.

We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
maildir format). Here is inode stats for both bricks and mountpoint:

===
Brick 1 (Server 1):

Filesystem InodesIUsed

  IFree IUse% Mounted on

/dev/mapper/vg_vd1_misc-lv08_mail   578768144 10954918

  5678132262% /bricks/r6sdLV08_vd1_mail

Brick 2 (Server 2):

Filesystem InodesIUsed

  IFree IUse% Mounted on

/dev/mapper/vg_vd0_misc-lv07_mail   578767984 10954913

  5678130712% /bricks/r6sdLV07_vd0_mail

Mountpoint (Server 3):

Filesystem  InodesIUsed  IFree
IUse% Mounted on
glusterfs.xxx:mail   578767760 10954915  567812845
2% /var/spool/mail/virtual
===

glusterfs.xxx domain has two A records for both Server 1 and Server 2.

Here is volume info:

===
Volume Name: mail
Type: Replicate
Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
Options Reconfigured:
nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
features.cache-invalidation-timeout: 10
performance.stat-prefetch: off
performance.quick-read: on
performance.read-ahead: off
performance.flush-behind: on
performance.write-behind: on
performance.io-thread-count: 4
performance.cache-max-file-size: 1048576
performance.cache-size: 67108864
performance.readdir-ahead: off
===

Soon enough after mounting and exim/dovecot start, glusterfs client
process begins to consume huge amount of RAM:

===
user@server3 ~$ ps aux | grep glusterfs | grep mail
root 28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03 4310:05
/usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
--volfile-server=glusterfs.xxx --volfile-id=mail
/var/spool/mail/virtual
===

That is, ~15 GiB of RAM.

Also we've tried to use mountpoint withing separate KVM VM with 2 or 3
GiB of RAM, and soon after starting mail daemons got OOM killer for
glusterfs client process.

Mounting same share via 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

Thanks for sharing the results. Shall look at the leaks and update.

-Soumya

On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote:

Also, here is valgrind output with our custom tool, that does GlusterFS volume
traversing (with simple stats) just like find tool. In this case NFS-Ganesha
is not used.

https://gist.github.com/e4602a50d3c98f7a2766

One may see GlusterFS-related leaks here as well.

On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:

Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
usage:

===
root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?


nfs-ganesha also has cache layer which can scale to millions of entries
depending on the number of files/directories being looked upon. However
there are parameters to tune it. So either try stat with few entries or
add below block in nfs-ganesha.conf file, set low limits and check the
difference. That may help us narrow down how much memory actually
consumed by core nfs-ganesha and gfAPI.

CACHEINODE {
Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache size
Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max no.
of entries in the cache.
}

Thanks,
Soumya


24.12.2015 16:32, Oleksandr Natalenko написав:

Still actual issue for 3.7.6. Any suggestions?

24.09.2015 10:14, Oleksandr Natalenko написав:

In our GlusterFS deployment we've encountered something like memory
leak in GlusterFS FUSE client.

We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
maildir format). Here is inode stats for both bricks and mountpoint:

===
Brick 1 (Server 1):

Filesystem InodesIUsed

  IFree IUse% Mounted on

/dev/mapper/vg_vd1_misc-lv08_mail   578768144 10954918

  5678132262% /bricks/r6sdLV08_vd1_mail

Brick 2 (Server 2):

Filesystem InodesIUsed

  IFree IUse% Mounted on

/dev/mapper/vg_vd0_misc-lv07_mail   578767984 10954913

  5678130712% /bricks/r6sdLV07_vd0_mail

Mountpoint (Server 3):

Filesystem  InodesIUsed  IFree
IUse% Mounted on
glusterfs.xxx:mail   578767760 10954915  567812845
2% /var/spool/mail/virtual
===

glusterfs.xxx domain has two A records for both Server 1 and Server 2.

Here is volume info:

===
Volume Name: mail
Type: Replicate
Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
Options Reconfigured:
nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
features.cache-invalidation-timeout: 10
performance.stat-prefetch: off
performance.quick-read: on
performance.read-ahead: off
performance.flush-behind: on
performance.write-behind: on
performance.io-thread-count: 4
performance.cache-max-file-size: 1048576
performance.cache-size: 67108864
performance.readdir-ahead: off
===

Soon enough after mounting and exim/dovecot start, glusterfs client
process begins to consume huge amount of RAM:

===
user@server3 ~$ ps aux | grep glusterfs | grep mail
root 28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03 4310:05
/usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
--volfile-server=glusterfs.xxx --volfile-id=mail
/var/spool/mail/virtual
===

That is, ~15 GiB of RAM.

Also we've tried to use mountpoint withing separate KVM VM with 2 or 3
GiB of RAM, and soon after starting mail daemons got OOM killer for
glusterfs client process.

Mounting same share via NFS works just fine. Also, we have much less
iowait and loadavg on client side with NFS.

Also, we've tried to change IO threads count and cache size in order
to limit memory usage with no luck. As you can see, total cache size
is 4×64==256 MiB (compare to 15 GiB).

Enabling-disabling stat-prefetch, read-ahead and readdir-ahead didn't
help as well.

Here are volume memory stats:

===
Memory status for volume : mail
--
Brick : server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Mallinfo

Arena: 36859904
Ordblks  : 10357
Smblks   : 519
Hblks: 21
Hblkhd   : 30515200
Usmblks  : 0
Fsmblks  : 53440
Uordblks : 18604144
Fordblks : 18255760
Keepcost : 114112

Mempool Stats
-
NameHotCount ColdCount PaddedSizeof
AllocCount MaxAlloc   Misses Max-StdAlloc
 - 
--   
mail-server:fd_t   0  1024  108
30773120  137

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

1. test with Cache_Size = 256 and Entries_HWMark = 4096

Before find . -type f:

root  3120  0.6 11.0 879120 208408 ?   Ssl  17:39   0:00 /usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT

After:

root  3120 11.4 24.3 1170076 458168 ?  Ssl  17:39  13:39 /usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT

~250M leak.

2. test with default values (after ganesha restart)

Before:

root 24937  1.3 10.4 875016 197808 ?   Ssl  19:39   0:00 /usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT

After:

root 24937  3.5 18.9 1022544 356340 ?  Ssl  19:39   0:40 /usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT

~159M leak.

No reasonable correlation detected. Second test was finished much faster than 
first (I guess, server-side GlusterFS cache or server kernel page cache is the 
cause).

There are ~1.8M files on this test volume.

On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:
> On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
> > Another addition: it seems to be GlusterFS API library memory leak
> > because NFS-Ganesha also consumes huge amount of memory while doing
> > ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
> > usage:
> > 
> > ===
> > root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
> > /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
> > /etc/ganesha/ganesha.conf -N NIV_EVENT
> > ===
> > 
> > 1.4G is too much for simple stat() :(.
> > 
> > Ideas?
> 
> nfs-ganesha also has cache layer which can scale to millions of entries
> depending on the number of files/directories being looked upon. However
> there are parameters to tune it. So either try stat with few entries or
> add below block in nfs-ganesha.conf file, set low limits and check the
> difference. That may help us narrow down how much memory actually
> consumed by core nfs-ganesha and gfAPI.
> 
> CACHEINODE {
>   Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache size
>   Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max no.
> of entries in the cache.
> }
> 
> Thanks,
> Soumya
> 
> > 24.12.2015 16:32, Oleksandr Natalenko написав:
> >> Still actual issue for 3.7.6. Any suggestions?
> >> 
> >> 24.09.2015 10:14, Oleksandr Natalenko написав:
> >>> In our GlusterFS deployment we've encountered something like memory
> >>> leak in GlusterFS FUSE client.
> >>> 
> >>> We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
> >>> maildir format). Here is inode stats for both bricks and mountpoint:
> >>> 
> >>> ===
> >>> Brick 1 (Server 1):
> >>> 
> >>> Filesystem InodesIUsed
> >>> 
> >>>  IFree IUse% Mounted on
> >>> 
> >>> /dev/mapper/vg_vd1_misc-lv08_mail   578768144 10954918
> >>> 
> >>>  5678132262% /bricks/r6sdLV08_vd1_mail
> >>> 
> >>> Brick 2 (Server 2):
> >>> 
> >>> Filesystem InodesIUsed
> >>> 
> >>>  IFree IUse% Mounted on
> >>> 
> >>> /dev/mapper/vg_vd0_misc-lv07_mail   578767984 10954913
> >>> 
> >>>  5678130712% /bricks/r6sdLV07_vd0_mail
> >>> 
> >>> Mountpoint (Server 3):
> >>> 
> >>> Filesystem  InodesIUsed  IFree
> >>> IUse% Mounted on
> >>> glusterfs.xxx:mail   578767760 10954915  567812845
> >>> 2% /var/spool/mail/virtual
> >>> ===
> >>> 
> >>> glusterfs.xxx domain has two A records for both Server 1 and Server 2.
> >>> 
> >>> Here is volume info:
> >>> 
> >>> ===
> >>> Volume Name: mail
> >>> Type: Replicate
> >>> Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
> >>> Status: Started
> >>> Number of Bricks: 1 x 2 = 2
> >>> Transport-type: tcp
> >>> Bricks:
> >>> Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
> >>> Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
> >>> Options Reconfigured:
> >>> nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
> >>> features.cache-invalidation-timeout: 10
> >>> performance.stat-prefetch: off
> >>> performance.quick-read: on
> >>> performance.read-ahead: off
> >>> performance.flush-behind: on
> >>> performance.write-behind: on
> >>> performance.io-thread-count: 4
> >>> performance.cache-max-file-size: 1048576
> >>> performance.cache-size: 67108864
> >>> performance.readdir-ahead: off
> >>> ===
> >>> 
> >>> Soon enough after mounting and exim/dovecot start, glusterfs client
> >>> process begins to consume huge amount of RAM:
> >>> 
> >>> ===
> >>> user@server3 ~$ ps aux | grep glusterfs | grep mail
> >>> root 28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03 4310:05
> >>> /usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
> >>> --volfile-server=glusterfs.xxx --volfile-id=mail
> >>> /var/spool/mail/virtual
> >>> ===
> >>> 
> >>> That is, ~15 GiB of RAM.
> >>> 
> >>> Also we've tried to use mountpoint withing 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

On 12/25/2015 08:56 PM, Oleksandr Natalenko wrote:

What units Cache_Size is measured in? Bytes?

Its actually (Cache_Size * sizeof_ptr) bytes. If possible, could you 
please run ganesha process under valgrind? Will help in detecting leaks.


Thanks,
Soumya


25.12.2015 16:58, Soumya Koduri написав:

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:

Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
usage:

===
root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?

nfs-ganesha also has cache layer which can scale to millions of
entries depending on the number of files/directories being looked
upon. However there are parameters to tune it. So either try stat with
few entries or add below block in nfs-ganesha.conf file, set low
limits and check the difference. That may help us narrow down how much
memory actually consumed by core nfs-ganesha and gfAPI.

CACHEINODE {
Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache
size
Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max
no. of entries in the cache.
}

Thanks,
Soumya



24.12.2015 16:32, Oleksandr Natalenko написав:

Still actual issue for 3.7.6. Any suggestions?

24.09.2015 10:14, Oleksandr Natalenko написав:

In our GlusterFS deployment we've encountered something like memory
leak in GlusterFS FUSE client.

We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
maildir format). Here is inode stats for both bricks and mountpoint:

===
Brick 1 (Server 1):

Filesystem Inodes IUsed
 IFree IUse% Mounted on
/dev/mapper/vg_vd1_misc-lv08_mail   578768144 10954918
 5678132262% /bricks/r6sdLV08_vd1_mail

Brick 2 (Server 2):

Filesystem Inodes IUsed
 IFree IUse% Mounted on
/dev/mapper/vg_vd0_misc-lv07_mail   578767984 10954913
 5678130712% /bricks/r6sdLV07_vd0_mail

Mountpoint (Server 3):

Filesystem  InodesIUsed  IFree
IUse% Mounted on
glusterfs.xxx:mail   578767760 10954915  567812845
2% /var/spool/mail/virtual
===

glusterfs.xxx domain has two A records for both Server 1 and Server 2.

Here is volume info:

===
Volume Name: mail
Type: Replicate
Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
Options Reconfigured:
nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
features.cache-invalidation-timeout: 10
performance.stat-prefetch: off
performance.quick-read: on
performance.read-ahead: off
performance.flush-behind: on
performance.write-behind: on
performance.io-thread-count: 4
performance.cache-max-file-size: 1048576
performance.cache-size: 67108864
performance.readdir-ahead: off
===

Soon enough after mounting and exim/dovecot start, glusterfs client
process begins to consume huge amount of RAM:

===
user@server3 ~$ ps aux | grep glusterfs | grep mail
root 28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03 4310:05
/usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
--volfile-server=glusterfs.xxx --volfile-id=mail
/var/spool/mail/virtual
===

That is, ~15 GiB of RAM.

Also we've tried to use mountpoint withing separate KVM VM with 2 or 3
GiB of RAM, and soon after starting mail daemons got OOM killer for
glusterfs client process.

Mounting same share via NFS works just fine. Also, we have much less
iowait and loadavg on client side with NFS.

Also, we've tried to change IO threads count and cache size in order
to limit memory usage with no luck. As you can see, total cache size
is 4×64==256 MiB (compare to 15 GiB).

Enabling-disabling stat-prefetch, read-ahead and readdir-ahead didn't
help as well.

Here are volume memory stats:

===
Memory status for volume : mail
--
Brick : server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Mallinfo

Arena: 36859904
Ordblks  : 10357
Smblks   : 519
Hblks: 21
Hblkhd   : 30515200
Usmblks  : 0
Fsmblks  : 53440
Uordblks : 18604144
Fordblks : 18255760
Keepcost : 114112

Mempool Stats
-
NameHotCount ColdCount PaddedSizeof
AllocCount MaxAlloc   Misses Max-StdAlloc
 - 
--   
mail-server:fd_t   0  1024  108
30773120  13700
mail-server:dentry_t   16110   274   84
23567614816384  1106499 1152
mail-server:inode_t1636321 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

What units Cache_Size is measured in? Bytes?

25.12.2015 16:58, Soumya Koduri написав:

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:

Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
usage:

===
root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?

nfs-ganesha also has cache layer which can scale to millions of
entries depending on the number of files/directories being looked
upon. However there are parameters to tune it. So either try stat with
few entries or add below block in nfs-ganesha.conf file, set low
limits and check the difference. That may help us narrow down how much
memory actually consumed by core nfs-ganesha and gfAPI.

CACHEINODE {
Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache size
Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max
no. of entries in the cache.
}

Thanks,
Soumya



24.12.2015 16:32, Oleksandr Natalenko написав:

Still actual issue for 3.7.6. Any suggestions?

24.09.2015 10:14, Oleksandr Natalenko написав:

In our GlusterFS deployment we've encountered something like memory
leak in GlusterFS FUSE client.

We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
maildir format). Here is inode stats for both bricks and mountpoint:

===
Brick 1 (Server 1):

Filesystem Inodes
IUsed

 IFree IUse% Mounted on
/dev/mapper/vg_vd1_misc-lv08_mail   578768144 
10954918

 5678132262% /bricks/r6sdLV08_vd1_mail

Brick 2 (Server 2):

Filesystem Inodes
IUsed

 IFree IUse% Mounted on
/dev/mapper/vg_vd0_misc-lv07_mail   578767984 
10954913

 5678130712% /bricks/r6sdLV07_vd0_mail

Mountpoint (Server 3):

Filesystem  InodesIUsed  IFree
IUse% Mounted on
glusterfs.xxx:mail   578767760 10954915  567812845
2% /var/spool/mail/virtual
===

glusterfs.xxx domain has two A records for both Server 1 and Server 
2.


Here is volume info:

===
Volume Name: mail
Type: Replicate
Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
Options Reconfigured:
nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
features.cache-invalidation-timeout: 10
performance.stat-prefetch: off
performance.quick-read: on
performance.read-ahead: off
performance.flush-behind: on
performance.write-behind: on
performance.io-thread-count: 4
performance.cache-max-file-size: 1048576
performance.cache-size: 67108864
performance.readdir-ahead: off
===

Soon enough after mounting and exim/dovecot start, glusterfs client
process begins to consume huge amount of RAM:

===
user@server3 ~$ ps aux | grep glusterfs | grep mail
root 28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03 4310:05
/usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
--volfile-server=glusterfs.xxx --volfile-id=mail
/var/spool/mail/virtual
===

That is, ~15 GiB of RAM.

Also we've tried to use mountpoint withing separate KVM VM with 2 or 
3

GiB of RAM, and soon after starting mail daemons got OOM killer for
glusterfs client process.

Mounting same share via NFS works just fine. Also, we have much less
iowait and loadavg on client side with NFS.

Also, we've tried to change IO threads count and cache size in order
to limit memory usage with no luck. As you can see, total cache size
is 4×64==256 MiB (compare to 15 GiB).

Enabling-disabling stat-prefetch, read-ahead and readdir-ahead 
didn't

help as well.

Here are volume memory stats:

===
Memory status for volume : mail
--
Brick : server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Mallinfo

Arena: 36859904
Ordblks  : 10357
Smblks   : 519
Hblks: 21
Hblkhd   : 30515200
Usmblks  : 0
Fsmblks  : 53440
Uordblks : 18604144
Fordblks : 18255760
Keepcost : 114112

Mempool Stats
-
NameHotCount ColdCount PaddedSizeof
AllocCount MaxAlloc   Misses Max-StdAlloc
 - 
--   
mail-server:fd_t   0  1024  108
30773120  13700
mail-server:dentry_t   16110   274   84
23567614816384  1106499 1152
mail-server:inode_t1636321  156
23721687616384  1876651 1169
mail-trash:fd_t0  1024  108
  0000
mail-trash:dentry_t0   

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Soumya Koduri]

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:

Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
usage:

===
root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?
nfs-ganesha also has cache layer which can scale to millions of entries 
depending on the number of files/directories being looked upon. However 
there are parameters to tune it. So either try stat with few entries or 
add below block in nfs-ganesha.conf file, set low limits and check the 
difference. That may help us narrow down how much memory actually 
consumed by core nfs-ganesha and gfAPI.


CACHEINODE {
Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache size
	Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max no. 
of entries in the cache.

}

Thanks,
Soumya



24.12.2015 16:32, Oleksandr Natalenko написав:

Still actual issue for 3.7.6. Any suggestions?

24.09.2015 10:14, Oleksandr Natalenko написав:

In our GlusterFS deployment we've encountered something like memory
leak in GlusterFS FUSE client.

We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
maildir format). Here is inode stats for both bricks and mountpoint:

===
Brick 1 (Server 1):

Filesystem InodesIUsed
 IFree IUse% Mounted on
/dev/mapper/vg_vd1_misc-lv08_mail   578768144 10954918
 5678132262% /bricks/r6sdLV08_vd1_mail

Brick 2 (Server 2):

Filesystem InodesIUsed
 IFree IUse% Mounted on
/dev/mapper/vg_vd0_misc-lv07_mail   578767984 10954913
 5678130712% /bricks/r6sdLV07_vd0_mail

Mountpoint (Server 3):

Filesystem  InodesIUsed  IFree
IUse% Mounted on
glusterfs.xxx:mail   578767760 10954915  567812845
2% /var/spool/mail/virtual
===

glusterfs.xxx domain has two A records for both Server 1 and Server 2.

Here is volume info:

===
Volume Name: mail
Type: Replicate
Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
Options Reconfigured:
nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
features.cache-invalidation-timeout: 10
performance.stat-prefetch: off
performance.quick-read: on
performance.read-ahead: off
performance.flush-behind: on
performance.write-behind: on
performance.io-thread-count: 4
performance.cache-max-file-size: 1048576
performance.cache-size: 67108864
performance.readdir-ahead: off
===

Soon enough after mounting and exim/dovecot start, glusterfs client
process begins to consume huge amount of RAM:

===
user@server3 ~$ ps aux | grep glusterfs | grep mail
root 28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03 4310:05
/usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
--volfile-server=glusterfs.xxx --volfile-id=mail
/var/spool/mail/virtual
===

That is, ~15 GiB of RAM.

Also we've tried to use mountpoint withing separate KVM VM with 2 or 3
GiB of RAM, and soon after starting mail daemons got OOM killer for
glusterfs client process.

Mounting same share via NFS works just fine. Also, we have much less
iowait and loadavg on client side with NFS.

Also, we've tried to change IO threads count and cache size in order
to limit memory usage with no luck. As you can see, total cache size
is 4×64==256 MiB (compare to 15 GiB).

Enabling-disabling stat-prefetch, read-ahead and readdir-ahead didn't
help as well.

Here are volume memory stats:

===
Memory status for volume : mail
--
Brick : server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Mallinfo

Arena: 36859904
Ordblks  : 10357
Smblks   : 519
Hblks: 21
Hblkhd   : 30515200
Usmblks  : 0
Fsmblks  : 53440
Uordblks : 18604144
Fordblks : 18255760
Keepcost : 114112

Mempool Stats
-
NameHotCount ColdCount PaddedSizeof
AllocCount MaxAlloc   Misses Max-StdAlloc
 - 
--   
mail-server:fd_t   0  1024  108
30773120  13700
mail-server:dentry_t   16110   274   84
23567614816384  1106499 1152
mail-server:inode_t1636321  156
23721687616384  1876651 1169
mail-trash:fd_t0  1024  108
  0000
mail-trash:dentry_t0 32768   84
  0000
mail-trash:inode_t 4

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Also, here is valgrind output with our custom tool, that does GlusterFS volume 
traversing (with simple stats) just like find tool. In this case NFS-Ganesha 
is not used.

https://gist.github.com/e4602a50d3c98f7a2766

One may see GlusterFS-related leaks here as well.

On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:
> On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
> > Another addition: it seems to be GlusterFS API library memory leak
> > because NFS-Ganesha also consumes huge amount of memory while doing
> > ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
> > usage:
> > 
> > ===
> > root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
> > /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
> > /etc/ganesha/ganesha.conf -N NIV_EVENT
> > ===
> > 
> > 1.4G is too much for simple stat() :(.
> > 
> > Ideas?
> 
> nfs-ganesha also has cache layer which can scale to millions of entries
> depending on the number of files/directories being looked upon. However
> there are parameters to tune it. So either try stat with few entries or
> add below block in nfs-ganesha.conf file, set low limits and check the
> difference. That may help us narrow down how much memory actually
> consumed by core nfs-ganesha and gfAPI.
> 
> CACHEINODE {
>   Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache size
>   Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max no.
> of entries in the cache.
> }
> 
> Thanks,
> Soumya
> 
> > 24.12.2015 16:32, Oleksandr Natalenko написав:
> >> Still actual issue for 3.7.6. Any suggestions?
> >> 
> >> 24.09.2015 10:14, Oleksandr Natalenko написав:
> >>> In our GlusterFS deployment we've encountered something like memory
> >>> leak in GlusterFS FUSE client.
> >>> 
> >>> We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
> >>> maildir format). Here is inode stats for both bricks and mountpoint:
> >>> 
> >>> ===
> >>> Brick 1 (Server 1):
> >>> 
> >>> Filesystem InodesIUsed
> >>> 
> >>>  IFree IUse% Mounted on
> >>> 
> >>> /dev/mapper/vg_vd1_misc-lv08_mail   578768144 10954918
> >>> 
> >>>  5678132262% /bricks/r6sdLV08_vd1_mail
> >>> 
> >>> Brick 2 (Server 2):
> >>> 
> >>> Filesystem InodesIUsed
> >>> 
> >>>  IFree IUse% Mounted on
> >>> 
> >>> /dev/mapper/vg_vd0_misc-lv07_mail   578767984 10954913
> >>> 
> >>>  5678130712% /bricks/r6sdLV07_vd0_mail
> >>> 
> >>> Mountpoint (Server 3):
> >>> 
> >>> Filesystem  InodesIUsed  IFree
> >>> IUse% Mounted on
> >>> glusterfs.xxx:mail   578767760 10954915  567812845
> >>> 2% /var/spool/mail/virtual
> >>> ===
> >>> 
> >>> glusterfs.xxx domain has two A records for both Server 1 and Server 2.
> >>> 
> >>> Here is volume info:
> >>> 
> >>> ===
> >>> Volume Name: mail
> >>> Type: Replicate
> >>> Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
> >>> Status: Started
> >>> Number of Bricks: 1 x 2 = 2
> >>> Transport-type: tcp
> >>> Bricks:
> >>> Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
> >>> Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
> >>> Options Reconfigured:
> >>> nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
> >>> features.cache-invalidation-timeout: 10
> >>> performance.stat-prefetch: off
> >>> performance.quick-read: on
> >>> performance.read-ahead: off
> >>> performance.flush-behind: on
> >>> performance.write-behind: on
> >>> performance.io-thread-count: 4
> >>> performance.cache-max-file-size: 1048576
> >>> performance.cache-size: 67108864
> >>> performance.readdir-ahead: off
> >>> ===
> >>> 
> >>> Soon enough after mounting and exim/dovecot start, glusterfs client
> >>> process begins to consume huge amount of RAM:
> >>> 
> >>> ===
> >>> user@server3 ~$ ps aux | grep glusterfs | grep mail
> >>> root 28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03 4310:05
> >>> /usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
> >>> --volfile-server=glusterfs.xxx --volfile-id=mail
> >>> /var/spool/mail/virtual
> >>> ===
> >>> 
> >>> That is, ~15 GiB of RAM.
> >>> 
> >>> Also we've tried to use mountpoint withing separate KVM VM with 2 or 3
> >>> GiB of RAM, and soon after starting mail daemons got OOM killer for
> >>> glusterfs client process.
> >>> 
> >>> Mounting same share via NFS works just fine. Also, we have much less
> >>> iowait and loadavg on client side with NFS.
> >>> 
> >>> Also, we've tried to change IO threads count and cache size in order
> >>> to limit memory usage with no luck. As you can see, total cache size
> >>> is 4×64==256 MiB (compare to 15 GiB).
> >>> 
> >>> Enabling-disabling stat-prefetch, read-ahead and readdir-ahead didn't
> >>> help as well.
> >>> 
> >>> Here are volume memory stats:
> >>> 
> >>> ===
> >>> Memory status for volume : mail
> >>> --
> >>> Brick : 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

OK, I've rebuild GlusterFS v3.7.6 with debug enabled as well as NFS-Ganesha 
with debug enabled as well (and libc allocator).

Here is my test steps:

1. launch nfs-ganesha:

valgrind --leak-check=full --show-leak-kinds=all --log-file="valgrind.log" /
opt/nfs-ganesha/bin/ganesha.nfsd -F -L ./ganesha.log -f ./ganesha.conf -N 
NIV_EVENT

2. mount NFS share:

mount -t nfs4 127.0.0.1:/share share -o 
defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100

3. cd to share and run find . for some time

4. CTRL+C find, unmount share.

5. CTRL+C NFS-Ganesha.

Here is full valgrind output:

https://gist.github.com/eebd9f94ababd8130d49

One may see the probability of massive leaks at the end of valgrind output 
related to both GlusterFS and NFS-Ganesha code.

On пʼятниця, 25 грудня 2015 р. 23:29:07 EET Soumya Koduri wrote:
> On 12/25/2015 08:56 PM, Oleksandr Natalenko wrote:
> > What units Cache_Size is measured in? Bytes?
> 
> Its actually (Cache_Size * sizeof_ptr) bytes. If possible, could you
> please run ganesha process under valgrind? Will help in detecting leaks.
> 
> Thanks,
> Soumya
> 
> > 25.12.2015 16:58, Soumya Koduri написав:
> >> On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
> >>> Another addition: it seems to be GlusterFS API library memory leak
> >>> because NFS-Ganesha also consumes huge amount of memory while doing
> >>> ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
> >>> usage:
> >>> 
> >>> ===
> >>> root  5416 34.2 78.5 2047176 1480552 ? Ssl  12:02 117:54
> >>> /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
> >>> /etc/ganesha/ganesha.conf -N NIV_EVENT
> >>> ===
> >>> 
> >>> 1.4G is too much for simple stat() :(.
> >>> 
> >>> Ideas?
> >> 
> >> nfs-ganesha also has cache layer which can scale to millions of
> >> entries depending on the number of files/directories being looked
> >> upon. However there are parameters to tune it. So either try stat with
> >> few entries or add below block in nfs-ganesha.conf file, set low
> >> limits and check the difference. That may help us narrow down how much
> >> memory actually consumed by core nfs-ganesha and gfAPI.
> >> 
> >> CACHEINODE {
> >> 
> >> Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache
> >> 
> >> size
> >> 
> >> Entries_HWMark(uint32, range 1 to UINT32_MAX, default 10); #Max
> >> 
> >> no. of entries in the cache.
> >> }
> >> 
> >> Thanks,
> >> Soumya
> >> 
> >>> 24.12.2015 16:32, Oleksandr Natalenko написав:
>  Still actual issue for 3.7.6. Any suggestions?
>  
>  24.09.2015 10:14, Oleksandr Natalenko написав:
> > In our GlusterFS deployment we've encountered something like memory
> > leak in GlusterFS FUSE client.
> > 
> > We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
> > maildir format). Here is inode stats for both bricks and mountpoint:
> > 
> > ===
> > Brick 1 (Server 1):
> > 
> > Filesystem Inodes IUsed
> > 
> >  IFree IUse% Mounted on
> > 
> > /dev/mapper/vg_vd1_misc-lv08_mail   578768144 10954918
> > 
> >  5678132262% /bricks/r6sdLV08_vd1_mail
> > 
> > Brick 2 (Server 2):
> > 
> > Filesystem Inodes IUsed
> > 
> >  IFree IUse% Mounted on
> > 
> > /dev/mapper/vg_vd0_misc-lv07_mail   578767984 10954913
> > 
> >  5678130712% /bricks/r6sdLV07_vd0_mail
> > 
> > Mountpoint (Server 3):
> > 
> > Filesystem  InodesIUsed  IFree
> > IUse% Mounted on
> > glusterfs.xxx:mail   578767760 10954915  567812845
> > 2% /var/spool/mail/virtual
> > ===
> > 
> > glusterfs.xxx domain has two A records for both Server 1 and Server 2.
> > 
> > Here is volume info:
> > 
> > ===
> > Volume Name: mail
> > Type: Replicate
> > Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
> > Status: Started
> > Number of Bricks: 1 x 2 = 2
> > Transport-type: tcp
> > Bricks:
> > Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
> > Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
> > Options Reconfigured:
> > nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
> > features.cache-invalidation-timeout: 10
> > performance.stat-prefetch: off
> > performance.quick-read: on
> > performance.read-ahead: off
> > performance.flush-behind: on
> > performance.write-behind: on
> > performance.io-thread-count: 4
> > performance.cache-max-file-size: 1048576
> > performance.cache-size: 67108864
> > performance.readdir-ahead: off
> > ===
> > 
> > Soon enough after mounting and exim/dovecot start, glusterfs client
> > process begins to consume huge amount of RAM:
> > 
> > ===
> > user@server3 ~$ ps aux | grep glusterfs | grep mail
> > 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

Here are two consecutive statedumps of brick in question memory usage 
[1] [2]. glusterfs client process went from ~630 MB to ~1350 MB of 
memory usage in less than one hour.


Volume options:

===
cluster.lookup-optimize: on
cluster.readdir-optimize: on
client.event-threads: 4
network.inode-lru-limit: 4096
server.event-threads: 8
performance.client-io-threads: on
storage.linux-aio: on
performance.write-behind-window-size: 4194304
performance.stat-prefetch: on
performance.quick-read: on
performance.read-ahead: on
performance.flush-behind: on
performance.write-behind: on
performance.io-thread-count: 4
performance.cache-max-file-size: 1048576
performance.cache-size: 33554432
performance.readdir-ahead: on
===

I observe such a behavior on similar volumes where millions of files are 
stored. The volume in question holds ~11M of small files (mail storage).


So, memleak persists. Had to switch to NFS temporarily :(.

Any idea?

[1] https://gist.github.com/46697b70ffe193fa797e
[2] https://gist.github.com/3a968ca909bfdeb31cca

28.09.2015 14:31, Raghavendra Bhat написав:

Hi Oleksandr,

You are right. The description should have said it as the limit on the
number of inodes in the lru list of the inode cache. I have sent a
patch for that.
http://review.gluster.org/#/c/12242/ [3]

Regards,
Raghavendra Bhat

On Thu, Sep 24, 2015 at 1:44 PM, Oleksandr Natalenko
 wrote:


I've checked statedump of volume in question and haven't found lots
of iobuf as mentioned in that bugreport.

However, I've noticed that there are lots of LRU records like this:

===
[conn.1.bound_xl./bricks/r6sdLV07_vd0_mail/mail.lru.1]
gfid=c4b29310-a19d-451b-8dd1-b3ac2d86b595
nlookup=1
fd-count=0
ref=0
ia_type=1
===

In fact, there are 16383 of them. I've checked "gluster volume set
help" in order to find something LRU-related and have found this:

===
Option: network.inode-lru-limit
Default Value: 16384
Description: Specifies the maximum megabytes of memory to be used in
the inode cache.
===

Is there error in description stating "maximum megabytes of memory"?
Shouldn't it mean "maximum amount of LRU records"? If no, is that
true, that inode cache could grow up to 16 GiB for client, and one
must lower network.inode-lru-limit value?

Another thought: we've enabled write-behind, and the default
write-behind-window-size value is 1 MiB. So, one may conclude that
with lots of small files written, write-behind buffer could grow up
to inode-lru-limit×write-behind-window-size=16 GiB? Who could
explain that to me?

24.09.2015 10:42, Gabi C write:


oh, my bad...
coulb be this one?

https://bugzilla.redhat.com/show_bug.cgi?id=1126831 [1] [2]
Anyway, on ovirt+gluster w I experienced similar behavior...


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel [2]




Links:
--
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1126831
[2] http://www.gluster.org/mailman/listinfo/gluster-devel
[3] http://review.gluster.org/#/c/12242/

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Raghavendra Bhat]

Hi Oleksandr,

You are right. The description should have said it as the limit on the
number of inodes in the lru list of the inode cache. I have sent a patch
for that.
http://review.gluster.org/#/c/12242/

Regards,
Raghavendra Bhat


On Thu, Sep 24, 2015 at 1:44 PM, Oleksandr Natalenko <
oleksa...@natalenko.name> wrote:

> I've checked statedump of volume in question and haven't found lots of
> iobuf as mentioned in that bugreport.
>
> However, I've noticed that there are lots of LRU records like this:
>
> ===
> [conn.1.bound_xl./bricks/r6sdLV07_vd0_mail/mail.lru.1]
> gfid=c4b29310-a19d-451b-8dd1-b3ac2d86b595
> nlookup=1
> fd-count=0
> ref=0
> ia_type=1
> ===
>
> In fact, there are 16383 of them. I've checked "gluster volume set help"
> in order to find something LRU-related and have found this:
>
> ===
> Option: network.inode-lru-limit
> Default Value: 16384
> Description: Specifies the maximum megabytes of memory to be used in the
> inode cache.
> ===
>
> Is there error in description stating "maximum megabytes of memory"?
> Shouldn't it mean "maximum amount of LRU records"? If no, is that true,
> that inode cache could grow up to 16 GiB for client, and one must lower
> network.inode-lru-limit value?
>
> Another thought: we've enabled write-behind, and the default
> write-behind-window-size value is 1 MiB. So, one may conclude that with
> lots of small files written, write-behind buffer could grow up to
> inode-lru-limit×write-behind-window-size=16 GiB? Who could explain that to
> me?
>
> 24.09.2015 10:42, Gabi C write:
>
>> oh, my bad...
>> coulb be this one?
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1126831 [2]
>> Anyway, on ovirt+gluster w I experienced similar behavior...
>>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Gabi C]

google vdsm memory leak..it's been discussed on list last year and earlier
this one...

On Thu, Sep 24, 2015 at 10:14 AM, Oleksandr Natalenko <
oleksa...@natalenko.name> wrote:

> In our GlusterFS deployment we've encountered something like memory leak
> in GlusterFS FUSE client.
>
> We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
> maildir format). Here is inode stats for both bricks and mountpoint:
>
> ===
> Brick 1 (Server 1):
>
> Filesystem InodesIUsed
>   IFree IUse% Mounted on
> /dev/mapper/vg_vd1_misc-lv08_mail   578768144 10954918
> 5678132262% /bricks/r6sdLV08_vd1_mail
>
> Brick 2 (Server 2):
>
> Filesystem InodesIUsed
>   IFree IUse% Mounted on
> /dev/mapper/vg_vd0_misc-lv07_mail   578767984 10954913
> 5678130712% /bricks/r6sdLV07_vd0_mail
>
> Mountpoint (Server 3):
>
> Filesystem  InodesIUsed  IFree IUse%
> Mounted on
> glusterfs.xxx:mail   578767760 10954915  5678128452%
> /var/spool/mail/virtual
> ===
>
> glusterfs.xxx domain has two A records for both Server 1 and Server 2.
>
> Here is volume info:
>
> ===
> Volume Name: mail
> Type: Replicate
> Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
> Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
> Options Reconfigured:
> nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
> features.cache-invalidation-timeout: 10
> performance.stat-prefetch: off
> performance.quick-read: on
> performance.read-ahead: off
> performance.flush-behind: on
> performance.write-behind: on
> performance.io-thread-count: 4
> performance.cache-max-file-size: 1048576
> performance.cache-size: 67108864
> performance.readdir-ahead: off
> ===
>
> Soon enough after mounting and exim/dovecot start, glusterfs client
> process begins to consume huge amount of RAM:
>
> ===
> user@server3 ~$ ps aux | grep glusterfs | grep mail
> root 28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03 4310:05
> /usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
> --volfile-server=glusterfs.xxx --volfile-id=mail /var/spool/mail/virtual
> ===
>
> That is, ~15 GiB of RAM.
>
> Also we've tried to use mountpoint withing separate KVM VM with 2 or 3 GiB
> of RAM, and soon after starting mail daemons got OOM killer for glusterfs
> client process.
>
> Mounting same share via NFS works just fine. Also, we have much less
> iowait and loadavg on client side with NFS.
>
> Also, we've tried to change IO threads count and cache size in order to
> limit memory usage with no luck. As you can see, total cache size is
> 4×64==256 MiB (compare to 15 GiB).
>
> Enabling-disabling stat-prefetch, read-ahead and readdir-ahead didn't help
> as well.
>
> Here are volume memory stats:
>
> ===
> Memory status for volume : mail
> --
> Brick : server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
> Mallinfo
> 
> Arena: 36859904
> Ordblks  : 10357
> Smblks   : 519
> Hblks: 21
> Hblkhd   : 30515200
> Usmblks  : 0
> Fsmblks  : 53440
> Uordblks : 18604144
> Fordblks : 18255760
> Keepcost : 114112
>
> Mempool Stats
> -
> NameHotCount ColdCount PaddedSizeof AllocCount
> MaxAlloc   Misses Max-StdAlloc
>  -  --
>   
> mail-server:fd_t   0  1024  108
>  30773120  13700
> mail-server:dentry_t   16110   274   84
> 23567614816384  1106499 1152
> mail-server:inode_t1636321  156
> 23721687616384  1876651 1169
> mail-trash:fd_t0  1024  108
> 0000
> mail-trash:dentry_t0 32768   84
> 0000
> mail-trash:inode_t 4 32764  156
> 4400
> mail-trash:trash_local_t   064 8628
> 0000
> mail-changetimerecorder:gf_ctr_local_t 06416540
>   0000
> mail-changelog:rpcsvc_request_t 0 8 2828
> 0000
> mail-changelog:changelog_local_t 064  116
> 0000
> mail-bitrot-stub:br_stub_local_t 0   512   84
> 79204400
> mail-locks:pl_local_t  032  148
> 6812757400
> mail-upcall:upcall_local_t 0   512  108
> 0000
> mail-marker:marker_local_t 0   128

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

I've checked statedump of volume in question and haven't found lots of 
iobuf as mentioned in that bugreport.


However, I've noticed that there are lots of LRU records like this:

===
[conn.1.bound_xl./bricks/r6sdLV07_vd0_mail/mail.lru.1]
gfid=c4b29310-a19d-451b-8dd1-b3ac2d86b595
nlookup=1
fd-count=0
ref=0
ia_type=1
===

In fact, there are 16383 of them. I've checked "gluster volume set help" 
in order to find something LRU-related and have found this:


===
Option: network.inode-lru-limit
Default Value: 16384
Description: Specifies the maximum megabytes of memory to be used in the 
inode cache.

===

Is there error in description stating "maximum megabytes of memory"? 
Shouldn't it mean "maximum amount of LRU records"? If no, is that true, 
that inode cache could grow up to 16 GiB for client, and one must lower 
network.inode-lru-limit value?


Another thought: we've enabled write-behind, and the default 
write-behind-window-size value is 1 MiB. So, one may conclude that with 
lots of small files written, write-behind buffer could grow up to 
inode-lru-limit×write-behind-window-size=16 GiB? Who could explain that 
to me?


24.09.2015 10:42, Gabi C write:

oh, my bad...
coulb be this one?

https://bugzilla.redhat.com/show_bug.cgi?id=1126831 [2]
Anyway, on ovirt+gluster w I experienced similar behavior...

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

We use bare GlusterFS installation with no oVirt involved.

24.09.2015 10:29, Gabi C wrote:

google vdsm memory leak..it's been discussed on list last year and
earlier this one...

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel