Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Glomski, Patrick
Hello, Pranith. The typical behavior is that the %cpu on a glusterfsd process jumps to number of processor cores available (800% or 1200%, depending on the pair of nodes involved) and the load average on the machine goes very high (~20). The volume's heal statistics output shows that it is

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Pranith Kumar Karampuri
On 01/21/2016 08:25 PM, Glomski, Patrick wrote: Hello, Pranith. The typical behavior is that the %cpu on a glusterfsd process jumps to number of processor cores available (800% or 1200%, depending on the pair of nodes involved) and the load average on the machine goes very high (~20). The

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Glomski, Patrick
I should mention that the problem is not currently occurring and there are no heals (output appended). By restarting the gluster services, we can stop the crawl, which lowers the load for a while. Subsequent crawls seem to finish properly. For what it's worth, files/folders that show up in the

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Pranith Kumar Karampuri
On 01/21/2016 09:26 PM, Glomski, Patrick wrote: I should mention that the problem is not currently occurring and there are no heals (output appended). By restarting the gluster services, we can stop the crawl, which lowers the load for a while. Subsequent crawls seem to finish properly. For

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Pranith Kumar Karampuri
On 01/22/2016 12:51 AM, Glomski, Patrick wrote: Pranith, could this kind of behavior be self-inflicted by us deleting files directly from the bricks? We have done that in the past to clean up an issues where gluster wouldn't allow us to delete from the mount. Not sure. I haven't seen many

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Glomski, Patrick
Unfortunately, all samba mounts to the gluster volume through the gfapi vfs plugin have been disabled for the last 6 hours or so and frequency of %cpu spikes is increased. We had switched to sharing a fuse mount through samba, but I just disabled that as well. There are no samba shares of this

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Pranith Kumar Karampuri
On 01/22/2016 07:19 AM, Pranith Kumar Karampuri wrote: On 01/22/2016 07:13 AM, Glomski, Patrick wrote: We use the samba glusterfs virtual filesystem (the current version provided on download.gluster.org ), but no windows clients connecting directly. Hmm.. Is

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Pranith Kumar Karampuri
On 01/22/2016 07:13 AM, Glomski, Patrick wrote: We use the samba glusterfs virtual filesystem (the current version provided on download.gluster.org ), but no windows clients connecting directly. Hmm.. Is there a way to disable using this and check if the CPU%

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Glomski, Patrick
Samba version is 4.1.17 that you guys maintain at download.gluster.org. The vfs plugin comes packaged with it. http://download.gluster.org/pub/gluster/glusterfs/samba/EPEL.repo/epel-6/x86_64/ # smbd --version Version 4.1.17 # rpm -qa | grep samba-vfs-glusterfs

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Pranith Kumar Karampuri
Do you have any windows clients? I see a lot of getxattr calls for "glusterfs.get_real_filename" which lead to full readdirs of the directories on the brick. Pranith On 01/22/2016 12:51 AM, Glomski, Patrick wrote: Pranith, could this kind of behavior be self-inflicted by us deleting files

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Glomski, Patrick
We use the samba glusterfs virtual filesystem (the current version provided on download.gluster.org), but no windows clients connecting directly. On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri < pkara...@redhat.com> wrote: > Do you have any windows clients? I see a lot of getxattr

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Pranith Kumar Karampuri
On 01/22/2016 07:25 AM, Glomski, Patrick wrote: Unfortunately, all samba mounts to the gluster volume through the gfapi vfs plugin have been disabled for the last 6 hours or so and frequency of %cpu spikes is increased. We had switched to sharing a fuse mount through samba, but I just

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Raghavendra Talur
On Jan 22, 2016 7:27 AM, "Pranith Kumar Karampuri" wrote: > > > > On 01/22/2016 07:19 AM, Pranith Kumar Karampuri wrote: >> >> >> >> On 01/22/2016 07:13 AM, Glomski, Patrick wrote: >>> >>> We use the samba glusterfs virtual filesystem (the current version provided on

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Pranith Kumar Karampuri
On 01/22/2016 07:25 AM, Glomski, Patrick wrote: Unfortunately, all samba mounts to the gluster volume through the gfapi vfs plugin have been disabled for the last 6 hours or so and frequency of %cpu spikes is increased. We had switched to sharing a fuse mount through samba, but I just

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Poornima Gurusiddaiah
bin...@corvidtec.com>, "Poornima Gurusiddaiah" <pguru...@redhat.com> > Sent: Friday, January 22, 2016 8:37:50 AM > Subject: Re: [Gluster-devel] [Gluster-users] heal hanging > On Jan 22, 2016 7:27 AM, "Pranith Kumar Karampuri" < pkara...@redhat.com > >

Re: [Gluster-devel] [Gluster-users] heal hanging

2016-01-21 Thread Glomski, Patrick
Pranith, could this kind of behavior be self-inflicted by us deleting files directly from the bricks? We have done that in the past to clean up an issues where gluster wouldn't allow us to delete from the mount. If so, is it feasible to clean them up by running a search on the .glusterfs