Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

I perform the tests using 1) rsync (massive copy of millions of files); 2) 
find (simple tree traversing).

To check if memory leak happens, I use find tool. I've performed two 
traversing (w/ and w/o fopen-keep-cache=off) with remount between them, but I 
didn't encounter "kernel notifier loop terminated" message during both 
traversing as well as before unmounting volume. Nevertheless, memory still 
leaks (at least up to 3 GiB in each case), so I believe invalidation requests 
are not the case.

I've also checked logs for the volume where I do rsync, and the message 
"kernel notifier loop terminated" happens somewhere in the middle of rsyncing, 
not before unmounting. But the memory starts leaking on rsync start as well, 
not just after "kernel notifier loop terminated" message. So, I believe, 
"kernel notifier loop terminated" is not the case again.

Also, I've tried to implement quick and dirty GlusterFS FUSE client using API 
(see https://github.com/pfactum/xglfs), and with latest patches from this 
thread (http://review.gluster.org/#/c/13096/, http://review.gluster.org/#/c/
13125/ and http://review.gluster.org/#/c/13232/) my FUSE client does not leak 
on tree traversing. So, I believe, this should be related to GlusterFS FUSE 
implementation.

How could I debug memory leak better?

On четвер, 21 січня 2016 р. 10:32:32 EET Xavier Hernandez wrote:
> If this message appears way before the volume is unmounted, can you try
> to start the volume manually using this command and repeat the tests ?
> 
> glusterfs --fopen-keep-cache=off --volfile-server=
> --volfile-id=/ 
> 
> This will prevent invalidation requests to be sent to the kernel, so
> there shouldn't be any memory leak even if the worker thread exits
> prematurely.
> 
> If that solves the problem, we could try to determine the cause of the
> premature exit and solve it.
> 
> Xavi
> 
> On 20/01/16 10:08, Oleksandr Natalenko wrote:
> > Yes, there are couple of messages like this in my logs too (I guess one
> > message per each remount):
> > 
> > ===
> > [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
> > glusterfs-fuse: kernel notifier loop terminated
> > ===
> > 
> > On середа, 20 січня 2016 р. 09:51:23 EET Xavier Hernandez wrote:
> >> I'm seeing a similar problem with 3.7.6.
> >> 
> >> This latest statedump contains a lot of gf_fuse_mt_invalidate_node_t
> >> objects in fuse. Looking at the code I see they are used to send
> >> invalidations to kernel fuse, however this is done in a separate thread
> >> that writes a log message when it exits. On the system I'm seeing the
> >> memory leak, I can see that message in the log files:
> >> 
> >> [2016-01-18 23:04:55.384873] I [fuse-bridge.c:3875:notify_kernel_loop]
> >> 0-glusterfs-fuse: kernel notifier loop terminated
> >> 
> >> But the volume is still working at this moment, so any future inode
> >> invalidations will leak memory because it was this thread that should
> >> release it.
> >> 
> >> Can you check if you also see this message in the mount log ?
> >> 
> >> It seems that this thread terminates if write returns any error
> >> different than ENOENT. I'm not sure if there could be any other error
> >> that can cause this.
> >> 
> >> Xavi
> >> 
> >> On 20/01/16 00:13, Oleksandr Natalenko wrote:
> >>> Here is another RAM usage stats and statedump of GlusterFS mount
> >>> approaching to just another OOM:
> >>> 
> >>> ===
> >>> root 32495  1.4 88.3 4943868 1697316 ? Ssl  Jan13 129:18
> >>> /usr/sbin/
> >>> glusterfs --volfile-server=server.example.com --volfile-id=volume
> >>> /mnt/volume ===
> >>> 
> >>> https://gist.github.com/86198201c79e927b46bd
> >>> 
> >>> 1.6G of RAM just for almost idle mount (we occasionally store Asterisk
> >>> recordings there). Triple OOM for 69 days of uptime.
> >>> 
> >>> Any thoughts?
> >>> 
> >>> On середа, 13 січня 2016 р. 16:26:59 EET Soumya Koduri wrote:
>  kill -USR1
> >>> 
> >>> ___
> >>> Gluster-devel mailing list
> >>> Gluster-devel@gluster.org
> >>> http://www.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Xavier Hernandez]

If this message appears way before the volume is unmounted, can you try 
to start the volume manually using this command and repeat the tests ?


glusterfs --fopen-keep-cache=off --volfile-server= 
--volfile-id=/ 


This will prevent invalidation requests to be sent to the kernel, so 
there shouldn't be any memory leak even if the worker thread exits 
prematurely.


If that solves the problem, we could try to determine the cause of the 
premature exit and solve it.


Xavi


On 20/01/16 10:08, Oleksandr Natalenko wrote:

Yes, there are couple of messages like this in my logs too (I guess one
message per each remount):

===
[2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
glusterfs-fuse: kernel notifier loop terminated
===

On середа, 20 січня 2016 р. 09:51:23 EET Xavier Hernandez wrote:

I'm seeing a similar problem with 3.7.6.

This latest statedump contains a lot of gf_fuse_mt_invalidate_node_t
objects in fuse. Looking at the code I see they are used to send
invalidations to kernel fuse, however this is done in a separate thread
that writes a log message when it exits. On the system I'm seeing the
memory leak, I can see that message in the log files:

[2016-01-18 23:04:55.384873] I [fuse-bridge.c:3875:notify_kernel_loop]
0-glusterfs-fuse: kernel notifier loop terminated

But the volume is still working at this moment, so any future inode
invalidations will leak memory because it was this thread that should
release it.

Can you check if you also see this message in the mount log ?

It seems that this thread terminates if write returns any error
different than ENOENT. I'm not sure if there could be any other error
that can cause this.

Xavi

On 20/01/16 00:13, Oleksandr Natalenko wrote:

Here is another RAM usage stats and statedump of GlusterFS mount
approaching to just another OOM:

===
root 32495  1.4 88.3 4943868 1697316 ? Ssl  Jan13 129:18
/usr/sbin/
glusterfs --volfile-server=server.example.com --volfile-id=volume
/mnt/volume ===

https://gist.github.com/86198201c79e927b46bd

1.6G of RAM just for almost idle mount (we occasionally store Asterisk
recordings there). Triple OOM for 69 days of uptime.

Any thoughts?

On середа, 13 січня 2016 р. 16:26:59 EET Soumya Koduri wrote:

kill -USR1


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel




___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] GlusterFS FUSE client hangs on rsyncing lots of file

[Raghavendra G]

On Thu, Jan 21, 2016 at 10:49 AM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On 01/18/2016 02:28 PM, Oleksandr Natalenko wrote:
>
>> XFS. Server side works OK, I'm able to mount volume again. Brick is 30%
>> full.
>>
>
> Oleksandr,
>   Will it be possible to get the statedump of the client, bricks
> output next time it happens?
>
> https://github.com/gluster/glusterfs/blob/master/doc/debugging/statedump.md#how-to-generate-statedump


We also need to dump inode information. To do that you've to add "all=yes"
to /var/run/gluster/glusterdump.options before you issue commands to get
statedump.


>
>
> Pranith
>
>
>
>> On понеділок, 18 січня 2016 р. 15:07:18 EET baul jianguo wrote:
>>
>>> What is your brick file system? and the glusterfsd process and all
>>> thread status?
>>> I met same issue when client app such as rsync stay in D status,and
>>> the brick process and relate thread also be in the D status.
>>> And the brick dev disk util is 100% .
>>>
>>> On Sun, Jan 17, 2016 at 6:13 AM, Oleksandr Natalenko
>>>
>>>  wrote:
>>>
 Wrong assumption, rsync hung again.

 On субота, 16 січня 2016 р. 22:53:04 EET Oleksandr Natalenko wrote:

> One possible reason:
>
> cluster.lookup-optimize: on
> cluster.readdir-optimize: on
>
> I've disabled both optimizations, and at least as of now rsync still
> does
> its job with no issues. I would like to find out what option causes
> such
> a
> behavior and why. Will test more.
>
> On пʼятниця, 15 січня 2016 р. 16:09:51 EET Oleksandr Natalenko wrote:
>
>> Another observation: if rsyncing is resumed after hang, rsync itself
>> hangs a lot faster because it does stat of already copied files. So,
>> the
>> reason may be not writing itself, but massive stat on GlusterFS volume
>> as well.
>>
>> 15.01.2016 09:40, Oleksandr Natalenko написав:
>>
>>> While doing rsync over millions of files from ordinary partition to
>>> GlusterFS volume, just after approx. first 2 million rsync hang
>>> happens, and the following info appears in dmesg:
>>>
>>> ===
>>> [17075038.924481] INFO: task rsync:10310 blocked for more than 120
>>> seconds.
>>> [17075038.931948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [17075038.940748] rsync   D 88207fc13680 0 10310
>>> 10309 0x0080
>>> [17075038.940752]  8809c578be18 0086 8809c578bfd8
>>> 00013680
>>> [17075038.940756]  8809c578bfd8 00013680 880310cbe660
>>> 881159d16a30
>>> [17075038.940759]  881e3aa25800 8809c578be48 881159d16b10
>>> 88087d553980
>>> [17075038.940762] Call Trace:
>>> [17075038.940770]  [] schedule+0x29/0x70
>>> [17075038.940797]  []
>>> __fuse_request_send+0x13d/0x2c0
>>> [fuse]
>>> [17075038.940801]  [] ?
>>> fuse_get_req_nofail_nopages+0xc0/0x1e0 [fuse]
>>> [17075038.940805]  [] ? wake_up_bit+0x30/0x30
>>> [17075038.940809]  [] fuse_request_send+0x12/0x20
>>> [fuse]
>>> [17075038.940813]  [] fuse_flush+0xff/0x150 [fuse]
>>> [17075038.940817]  [] filp_close+0x34/0x80
>>> [17075038.940821]  [] __close_fd+0x78/0xa0
>>> [17075038.940824]  [] SyS_close+0x23/0x50
>>> [17075038.940828]  []
>>> system_call_fastpath+0x16/0x1b
>>> ===
>>>
>>> rsync blocks in D state, and to kill it, I have to do umount --lazy
>>> on
>>> GlusterFS mountpoint, and then kill corresponding client glusterfs
>>> process. Then rsync exits.
>>>
>>> Here is GlusterFS volume info:
>>>
>>> ===
>>> Volume Name: asterisk_records
>>> Type: Distributed-Replicate
>>> Volume ID: dc1fe561-fa3a-4f2e-8330-ec7e52c75ba4
>>> Status: Started
>>> Number of Bricks: 3 x 2 = 6
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1:
>>> server1:/bricks/10_megaraid_0_3_9_x_0_4_3_hdd_r1_nolvm_hdd_storage_01
>>> /as
>>> te
>>> risk/records Brick2:
>>> server2:/bricks/10_megaraid_8_5_14_x_8_6_16_hdd_r1_nolvm_hdd_storage_
>>> 01/
>>> as
>>> terisk/records Brick3:
>>> server1:/bricks/11_megaraid_0_5_4_x_0_6_5_hdd_r1_nolvm_hdd_storage_02
>>> /as
>>> te
>>> risk/records Brick4:
>>> server2:/bricks/11_megaraid_8_7_15_x_8_8_20_hdd_r1_nolvm_hdd_storage_
>>> 02/
>>> as
>>> terisk/records Brick5:
>>> server1:/bricks/12_megaraid_0_7_6_x_0_13_14_hdd_r1_nolvm_hdd_storage_
>>> 03/
>>> as
>>> terisk/records Brick6:
>>> server2:/bricks/12_megaraid_8_9_19_x_8_13_24_hdd_r1_nolvm_hdd_storage
>>> _03
>>> /a
>>> sterisk/records Options Reconfigured:
>>> cluster.lookup-optimize: on
>>> cluster.readdir-optimize: on
>>> client.event-threads: 2
>>> network.inode-lru-limit: 4096
>>> server.event-threads: 4
>>> 

Re: [Gluster-devel] [Gluster-users] heal hanging

[Glomski, Patrick]

Hello, Pranith. The typical behavior is that the %cpu on a glusterfsd
process jumps to number of processor cores available (800% or 1200%,
depending on the pair of nodes involved) and the load average on the
machine goes very high (~20). The volume's heal statistics output shows
that it is crawling one of the bricks and trying to heal, but this crawl
hangs and never seems to finish.

The number of files in the xattrop directory varies over time, so I ran a
wc -l as you requested periodically for some time and then started
including a datestamped list of the files that were in the xattrops
directory on each brick to see which were persistent. All bricks had files
in the xattrop folder, so all results are attached.

Please let me know if there is anything else I can provide.

Patrick


On Thu, Jan 21, 2016 at 12:01 AM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

> hey,
>Which process is consuming so much cpu? I went through the logs you
> gave me. I see that the following files are in gfid mismatch state:
>
> <066e4525-8f8b-43aa-b7a1-86bbcecc68b9/safebrowsing-backup>,
> <1d48754b-b38c-403d-94e2-0f5c41d5f885/recovery.bak>,
> ,
>
> Could you give me the output of "ls /indices/xattrop | wc -l"
> output on all the bricks which are acting this way? This will tell us the
> number of pending self-heals on the system.
>
> Pranith
>
>
> On 01/20/2016 09:26 PM, David Robinson wrote:
>
> resending with parsed logs...
>
>
>
>
>
> I am having issues with 3.6.6 where the load will spike up to 800% for one
> of the glusterfsd processes and the users can no longer access the system.
> If I reboot the node, the heal will finish normally after a few minutes and
> the system will be responsive, but a few hours later the issue will start
> again.  It look like it is hanging in a heal and spinning up the load on
> one of the bricks.  The heal gets stuck and says it is crawling and never
> returns.  After a few minutes of the heal saying it is crawling, the load
> spikes up and the mounts become unresponsive.
>
> Any suggestions on how to fix this?  It has us stopped cold as the user
> can no longer access the systems when the load spikes... Logs attached.
>
> System setup info is:
>
> [root@gfs01a ~]# gluster volume info homegfs
>
> Volume Name: homegfs
> Type: Distributed-Replicate
> Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071
> Status: Started
> Number of Bricks: 4 x 2 = 8
> Transport-type: tcp
> Bricks:
> Brick1: gfsib01a.corvidtec.com:/data/brick01a/homegfs
> Brick2: gfsib01b.corvidtec.com:/data/brick01b/homegfs
> Brick3: gfsib01a.corvidtec.com:/data/brick02a/homegfs
> Brick4: gfsib01b.corvidtec.com:/data/brick02b/homegfs
> Brick5: gfsib02a.corvidtec.com:/data/brick01a/homegfs
> Brick6: gfsib02b.corvidtec.com:/data/brick01b/homegfs
> Brick7: gfsib02a.corvidtec.com:/data/brick02a/homegfs
> Brick8: gfsib02b.corvidtec.com:/data/brick02b/homegfs
> Options Reconfigured:
> performance.io-thread-count: 32
> performance.cache-size: 128MB
> performance.write-behind-window-size: 128MB
> server.allow-insecure: on
> network.ping-timeout: 42
> storage.owner-gid: 100
> geo-replication.indexing: off
> geo-replication.ignore-pid-check: on
> changelog.changelog: off
> changelog.fsync-interval: 3
> changelog.rollover-time: 15
> server.manage-gids: on
> diagnostics.client-log-level: WARNING
>
> [root@gfs01a ~]# rpm -qa | grep gluster
> gluster-nagios-common-0.1.1-0.el6.noarch
> glusterfs-fuse-3.6.6-1.el6.x86_64
> glusterfs-debuginfo-3.6.6-1.el6.x86_64
> glusterfs-libs-3.6.6-1.el6.x86_64
> glusterfs-geo-replication-3.6.6-1.el6.x86_64
> glusterfs-api-3.6.6-1.el6.x86_64
> glusterfs-devel-3.6.6-1.el6.x86_64
> glusterfs-api-devel-3.6.6-1.el6.x86_64
> glusterfs-3.6.6-1.el6.x86_64
> glusterfs-cli-3.6.6-1.el6.x86_64
> glusterfs-rdma-3.6.6-1.el6.x86_64
> samba-vfs-glusterfs-4.1.11-2.el6.x86_64
> glusterfs-server-3.6.6-1.el6.x86_64
> glusterfs-extra-xlators-3.6.6-1.el6.x86_64
>
>
>
>
>
> ___
> Gluster-devel mailing 
> listGluster-devel@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-devel
>
>
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>


gfs01a_data_brick01a_homegfs
Description: Binary data


gfs01a_data_brick02a_homegfs
Description: Binary data


gfs01b_data_brick01b_homegfs
Description: Binary data


gfs01b_data_brick02b_homegfs
Description: Binary data


gfs02a_data_brick01a_homegfs
Description: Binary data


gfs02a_data_brick02a_homegfs
Description: Binary data


gfs02b_data_brick01b_homegfs
Description: Binary data


gfs02b_data_brick02b_homegfs
Description: Binary data
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] heal hanging

[Pranith Kumar Karampuri]

On 01/21/2016 08:25 PM, Glomski, Patrick wrote:
Hello, Pranith. The typical behavior is that the %cpu on a glusterfsd 
process jumps to number of processor cores available (800% or 1200%, 
depending on the pair of nodes involved) and the load average on the 
machine goes very high (~20). The volume's heal statistics output 
shows that it is crawling one of the bricks and trying to heal, but 
this crawl hangs and never seems to finish.


The number of files in the xattrop directory varies over time, so I 
ran a wc -l as you requested periodically for some time and then 
started including a datestamped list of the files that were in the 
xattrops directory on each brick to see which were persistent. All 
bricks had files in the xattrop folder, so all results are attached.
Thanks this info is helpful. I don't see a lot of files. Could you give 
output of "gluster volume heal  info"? Is there any directory 
in there which is LARGE?


Pranith


Please let me know if there is anything else I can provide.

Patrick


On Thu, Jan 21, 2016 at 12:01 AM, Pranith Kumar Karampuri 
> wrote:


hey,
   Which process is consuming so much cpu? I went through the
logs you gave me. I see that the following files are in gfid
mismatch state:

<066e4525-8f8b-43aa-b7a1-86bbcecc68b9/safebrowsing-backup>,
<1d48754b-b38c-403d-94e2-0f5c41d5f885/recovery.bak>,
,

Could you give me the output of "ls /indices/xattrop |
wc -l" output on all the bricks which are acting this way? This
will tell us the number of pending self-heals on the system.

Pranith


On 01/20/2016 09:26 PM, David Robinson wrote:

resending with parsed logs...

I am having issues with 3.6.6 where the load will spike up to
800% for one of the glusterfsd processes and the users can no
longer access the system.  If I reboot the node, the heal will
finish normally after a few minutes and the system will be
responsive, but a few hours later the issue will start again. 
It look like it is hanging in a heal and spinning up the load

on one of the bricks.  The heal gets stuck and says it is
crawling and never returns.  After a few minutes of the heal
saying it is crawling, the load spikes up and the mounts become
unresponsive.
Any suggestions on how to fix this?  It has us stopped cold as
the user can no longer access the systems when the load
spikes... Logs attached.
System setup info is:
[root@gfs01a ~]# gluster volume info homegfs

Volume Name: homegfs
Type: Distributed-Replicate
Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: gfsib01a.corvidtec.com:/data/brick01a/homegfs
Brick2: gfsib01b.corvidtec.com:/data/brick01b/homegfs
Brick3: gfsib01a.corvidtec.com:/data/brick02a/homegfs
Brick4: gfsib01b.corvidtec.com:/data/brick02b/homegfs
Brick5: gfsib02a.corvidtec.com:/data/brick01a/homegfs
Brick6: gfsib02b.corvidtec.com:/data/brick01b/homegfs
Brick7: gfsib02a.corvidtec.com:/data/brick02a/homegfs
Brick8: gfsib02b.corvidtec.com:/data/brick02b/homegfs
Options Reconfigured:
performance.io-thread-count: 32
performance.cache-size: 128MB
performance.write-behind-window-size: 128MB
server.allow-insecure: on
network.ping-timeout: 42
storage.owner-gid: 100
geo-replication.indexing: off
geo-replication.ignore-pid-check: on
changelog.changelog: off
changelog.fsync-interval: 3
changelog.rollover-time: 15
server.manage-gids: on
diagnostics.client-log-level: WARNING
[root@gfs01a ~]# rpm -qa | grep gluster
gluster-nagios-common-0.1.1-0.el6.noarch
glusterfs-fuse-3.6.6-1.el6.x86_64
glusterfs-debuginfo-3.6.6-1.el6.x86_64
glusterfs-libs-3.6.6-1.el6.x86_64
glusterfs-geo-replication-3.6.6-1.el6.x86_64
glusterfs-api-3.6.6-1.el6.x86_64
glusterfs-devel-3.6.6-1.el6.x86_64
glusterfs-api-devel-3.6.6-1.el6.x86_64
glusterfs-3.6.6-1.el6.x86_64
glusterfs-cli-3.6.6-1.el6.x86_64
glusterfs-rdma-3.6.6-1.el6.x86_64
samba-vfs-glusterfs-4.1.11-2.el6.x86_64
glusterfs-server-3.6.6-1.el6.x86_64
glusterfs-extra-xlators-3.6.6-1.el6.x86_64



___
Gluster-devel mailing list
Gluster-devel@gluster.org  
http://www.gluster.org/mailman/listinfo/gluster-devel



___
Gluster-users mailing list
gluster-us...@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users




___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Netbsd regressions are failing because of connection problems?

[Michael Scherer]

Le jeudi 21 janvier 2016 à 06:05 +0530, Pranith Kumar Karampuri a
écrit :
> /origin/*
> ERROR: Error cloning remote repo 'origin'
> hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch 
> --tags --progress git://review.gluster.org/glusterfs.git 
> +refs/heads/*:refs/remotes/origin/*" returned status code 128:
> stdout:
> stderr: fatal: unable to connect to review.gluster.org:
> review.gluster.org[0: 184.107.76.10]: errno=Connection refused

SO I found nothing in gerrit nor netbsd. ANd not the DNS, since it
managed to resolve stuff fine.

I suspect the problem was on gerrit, nor on netbsd. Did it happened
again ?
-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS




signature.asc
Description: This is a digitally signed message part
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] heal hanging

[Glomski, Patrick]

I should mention that the problem is not currently occurring and there are
no heals (output appended). By restarting the gluster services, we can stop
the crawl, which lowers the load for a while. Subsequent crawls seem to
finish properly. For what it's worth, files/folders that show up in the
'volume info' output during a hung crawl don't seem to be anything out of
the ordinary.

Over the past four days, the typical time before the problem recurs after
suppressing it in this manner is an hour. Last night when we reached out to
you was the last time it happened and the load has been low since (a
relief).  David believes that recursively listing the files (ls -alR or
similar) from a client mount can force the issue to happen, but obviously
I'd rather not unless we have some precise thing we're looking for. Let me
know if you'd like me to attempt to drive the system unstable like that and
what I should look for. As it's a production system, I'd rather not leave
it in this state for long.

[root@gfs01a xattrop]# gluster volume heal homegfs info
Brick gfs01a.corvidtec.com:/data/brick01a/homegfs/
Number of entries: 0

Brick gfs01b.corvidtec.com:/data/brick01b/homegfs/
Number of entries: 0

Brick gfs01a.corvidtec.com:/data/brick02a/homegfs/
Number of entries: 0

Brick gfs01b.corvidtec.com:/data/brick02b/homegfs/
Number of entries: 0

Brick gfs02a.corvidtec.com:/data/brick01a/homegfs/
Number of entries: 0

Brick gfs02b.corvidtec.com:/data/brick01b/homegfs/
Number of entries: 0

Brick gfs02a.corvidtec.com:/data/brick02a/homegfs/
Number of entries: 0

Brick gfs02b.corvidtec.com:/data/brick02b/homegfs/
Number of entries: 0




On Thu, Jan 21, 2016 at 10:40 AM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On 01/21/2016 08:25 PM, Glomski, Patrick wrote:
>
> Hello, Pranith. The typical behavior is that the %cpu on a glusterfsd
> process jumps to number of processor cores available (800% or 1200%,
> depending on the pair of nodes involved) and the load average on the
> machine goes very high (~20). The volume's heal statistics output shows
> that it is crawling one of the bricks and trying to heal, but this crawl
> hangs and never seems to finish.
>
>
> The number of files in the xattrop directory varies over time, so I ran a
> wc -l as you requested periodically for some time and then started
> including a datestamped list of the files that were in the xattrops
> directory on each brick to see which were persistent. All bricks had files
> in the xattrop folder, so all results are attached.
>
> Thanks this info is helpful. I don't see a lot of files. Could you give
> output of "gluster volume heal  info"? Is there any directory in
> there which is LARGE?
>
> Pranith
>
>
> Please let me know if there is anything else I can provide.
>
> Patrick
>
>
> On Thu, Jan 21, 2016 at 12:01 AM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>> hey,
>>Which process is consuming so much cpu? I went through the logs
>> you gave me. I see that the following files are in gfid mismatch state:
>>
>> <066e4525-8f8b-43aa-b7a1-86bbcecc68b9/safebrowsing-backup>,
>> <1d48754b-b38c-403d-94e2-0f5c41d5f885/recovery.bak>,
>> ,
>>
>> Could you give me the output of "ls /indices/xattrop | wc -l"
>> output on all the bricks which are acting this way? This will tell us the
>> number of pending self-heals on the system.
>>
>> Pranith
>>
>>
>> On 01/20/2016 09:26 PM, David Robinson wrote:
>>
>> resending with parsed logs...
>>
>>
>>
>>
>>
>> I am having issues with 3.6.6 where the load will spike up to 800% for
>> one of the glusterfsd processes and the users can no longer access the
>> system.  If I reboot the node, the heal will finish normally after a few
>> minutes and the system will be responsive, but a few hours later the issue
>> will start again.  It look like it is hanging in a heal and spinning up the
>> load on one of the bricks.  The heal gets stuck and says it is crawling and
>> never returns.  After a few minutes of the heal saying it is crawling, the
>> load spikes up and the mounts become unresponsive.
>>
>> Any suggestions on how to fix this?  It has us stopped cold as the user
>> can no longer access the systems when the load spikes... Logs attached.
>>
>> System setup info is:
>>
>> [root@gfs01a ~]# gluster volume info homegfs
>>
>> Volume Name: homegfs
>> Type: Distributed-Replicate
>> Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071
>> Status: Started
>> Number of Bricks: 4 x 2 = 8
>> Transport-type: tcp
>> Bricks:
>> Brick1: gfsib01a.corvidtec.com:/data/brick01a/homegfs
>> Brick2: gfsib01b.corvidtec.com:/data/brick01b/homegfs
>> Brick3: gfsib01a.corvidtec.com:/data/brick02a/homegfs
>> Brick4: gfsib01b.corvidtec.com:/data/brick02b/homegfs
>> Brick5: gfsib02a.corvidtec.com:/data/brick01a/homegfs
>> Brick6: gfsib02b.corvidtec.com:/data/brick01b/homegfs
>> Brick7: gfsib02a.corvidtec.com:/data/brick02a/homegfs
>> Brick8: gfsib02b.corvidtec.com:/data/brick02b/homegfs
>> Options Reconfigured:
>> 

Re: [Gluster-devel] [Gluster-users] heal hanging

[Pranith Kumar Karampuri]

On 01/21/2016 09:26 PM, Glomski, Patrick wrote:
I should mention that the problem is not currently occurring and there 
are no heals (output appended). By restarting the gluster services, we 
can stop the crawl, which lowers the load for a while. Subsequent 
crawls seem to finish properly. For what it's worth, files/folders 
that show up in the 'volume info' output during a hung crawl don't 
seem to be anything out of the ordinary.


Over the past four days, the typical time before the problem recurs 
after suppressing it in this manner is an hour. Last night when we 
reached out to you was the last time it happened and the load has been 
low since (a relief).  David believes that recursively listing the 
files (ls -alR or similar) from a client mount can force the issue to 
happen, but obviously I'd rather not unless we have some precise thing 
we're looking for. Let me know if you'd like me to attempt to drive 
the system unstable like that and what I should look for. As it's a 
production system, I'd rather not leave it in this state for long.


Will it be possible to send glustershd, mount logs of the past 4 days? I 
would like to see if this is because of directory self-heal going wild 
(Ravi is working on throttling feature for 3.8, which will allow to put 
breaks on self-heal traffic)


Pranith


[root@gfs01a xattrop]# gluster volume heal homegfs info
Brick gfs01a.corvidtec.com:/data/brick01a/homegfs/
Number of entries: 0

Brick gfs01b.corvidtec.com:/data/brick01b/homegfs/
Number of entries: 0

Brick gfs01a.corvidtec.com:/data/brick02a/homegfs/
Number of entries: 0

Brick gfs01b.corvidtec.com:/data/brick02b/homegfs/
Number of entries: 0

Brick gfs02a.corvidtec.com:/data/brick01a/homegfs/
Number of entries: 0

Brick gfs02b.corvidtec.com:/data/brick01b/homegfs/
Number of entries: 0

Brick gfs02a.corvidtec.com:/data/brick02a/homegfs/
Number of entries: 0

Brick gfs02b.corvidtec.com:/data/brick02b/homegfs/
Number of entries: 0




On Thu, Jan 21, 2016 at 10:40 AM, Pranith Kumar Karampuri 
> wrote:




On 01/21/2016 08:25 PM, Glomski, Patrick wrote:

Hello, Pranith. The typical behavior is that the %cpu on a
glusterfsd process jumps to number of processor cores available
(800% or 1200%, depending on the pair of nodes involved) and the
load average on the machine goes very high (~20). The volume's
heal statistics output shows that it is crawling one of the
bricks and trying to heal, but this crawl hangs and never seems
to finish.

The number of files in the xattrop directory varies over time, so
I ran a wc -l as you requested periodically for some time and
then started including a datestamped list of the files that were
in the xattrops directory on each brick to see which were
persistent. All bricks had files in the xattrop folder, so all
results are attached.

Thanks this info is helpful. I don't see a lot of files. Could you
give output of "gluster volume heal  info"? Is there any
directory in there which is LARGE?

Pranith



Please let me know if there is anything else I can provide.

Patrick


On Thu, Jan 21, 2016 at 12:01 AM, Pranith Kumar Karampuri
> wrote:

hey,
   Which process is consuming so much cpu? I went through
the logs you gave me. I see that the following files are in
gfid mismatch state:

<066e4525-8f8b-43aa-b7a1-86bbcecc68b9/safebrowsing-backup>,
<1d48754b-b38c-403d-94e2-0f5c41d5f885/recovery.bak>,
,

Could you give me the output of "ls
/indices/xattrop | wc -l" output on all the
bricks which are acting this way? This will tell us the
number of pending self-heals on the system.

Pranith


On 01/20/2016 09:26 PM, David Robinson wrote:

resending with parsed logs...

I am having issues with 3.6.6 where the load will spike up
to 800% for one of the glusterfsd processes and the users
can no longer access the system.  If I reboot the node,
the heal will finish normally after a few minutes and the
system will be responsive, but a few hours later the issue
will start again.  It look like it is hanging in a heal
and spinning up the load on one of the bricks.  The heal
gets stuck and says it is crawling and never returns.
After a few minutes of the heal saying it is crawling, the
load spikes up and the mounts become unresponsive.
Any suggestions on how to fix this?  It has us stopped
cold as the user can no longer access the systems when the
load spikes... Logs attached.
System setup info is:
[root@gfs01a ~]# gluster volume info homegfs

Volume Name: homegfs
Type: Distributed-Replicate
Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071
Status: 

Re: [Gluster-devel] Netbsd regressions are failing because of connection problems?

[Emmanuel Dreyfus]

On Thu, Jan 21, 2016 at 04:49:28PM +0100, Michael Scherer wrote:
> > review.gluster.org[0: 184.107.76.10]: errno=Connection refused
> 
> SO I found nothing in gerrit nor netbsd. ANd not the DNS, since it
> managed to resolve stuff fine.
> 
> I suspect the problem was on gerrit, nor on netbsd. Did it happened
> again ?

I could imagine problems with exhausted system resources, but it would
not produce a "Connection refused".

-- 
Emmanuel Dreyfus
m...@netbsd.org
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Closing bugs filed against the mainline/devel version earlier

[Niels de Vos]

Hi Naga,

a while back you closed a bunch of bugs that roughly matched this
criteria:

 - select mainline bug to potentially close
 - is there a cloned bug to a stable release (backport)
 - is the backport marked as CLOSED/CURRENTRELEASE
 -> close the mainline bug as CLOSED/NEXTRELEASE

 (from 
http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/12760/focus=12888 
)

Most of this looks fine, except that we can not use the "cloned" field.
There are bugs that get filed against different versions, and then get
marked as "depends on" or "blocks". Filing bugs through cloning is only
one option, and misses the other bugs. We really would need to use those
other fields instead.

If you are interested in this, you could help us by defining the policy
to close mainline bugs. The current policy is documented here:

  
http://gluster.readthedocs.org/en/latest/Contributors-Guide/Bug-report-Life-Cycle/

Could you send a pull request with an updated text?

Also, can you share the script or bugzilla search query that you used to
identify the bugs that you closed? It is something that we would need to
run after each release is made. A note should get added to the release
process:

  
http://gluster.readthedocs.org/en/latest/Contributors-Guide/GlusterFS-Release-process/

A pull request with the script that you used can be sent to our
release-tools repository on GitHub:

  https://github.com/gluster/release-tools

We have some developers that are (slowly) working on automating the bug
status when patches get posted, merged and releases are tagged. Once
they have some advancement in their project, we can integrate your
close-mainline-bugs script too.

Many thanks,
Niels


signature.asc
Description: PGP signature
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

I see extra GF_FREE (node); added with two patches:

===
$ git diff HEAD~2 | gist
https://gist.github.com/9524fa2054cc48278ea8
===

Is that intentionally? I guess I face double-free issue.

On четвер, 21 січня 2016 р. 17:29:53 EET Kaleb KEITHLEY wrote:
> On 01/20/2016 04:08 AM, Oleksandr Natalenko wrote:
> > Yes, there are couple of messages like this in my logs too (I guess one
> > message per each remount):
> > 
> > ===
> > [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
> > glusterfs-fuse: kernel notifier loop terminated
> > ===
> 
> Bug reports and fixes for master and release-3.7 branches are:
> 
> master)
>  https://bugzilla.redhat.com/show_bug.cgi?id=1288857
>  http://review.gluster.org/12886
> 
> release-3.7)
>  https://bugzilla.redhat.com/show_bug.cgi?id=1288922
>  http://review.gluster.org/12887
> 
> The release-3.7 fix will be in glusterfs-3.7.7 when it's released.
> 
> I think with even with the above fixes applied there are still some
> issues remaining. I have submitted additional/revised fixes on top of
> the above fixes at:
> 
>  master: http://review.gluster.org/13274
>  release-3.7: http://review.gluster.org/13275
> 
> I invite you to review the patches in gerrit (review.gluster.org).
> 
> Regards,
> 
> --
> 
> Kaleb


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Netbsd regressions are failing because of connection problems?

[Michael Scherer]

Le jeudi 21 janvier 2016 à 16:07 +, Emmanuel Dreyfus a écrit :
> On Thu, Jan 21, 2016 at 04:49:28PM +0100, Michael Scherer wrote:
> > > review.gluster.org[0: 184.107.76.10]: errno=Connection refused
> > 
> > SO I found nothing in gerrit nor netbsd. ANd not the DNS, since it
> > managed to resolve stuff fine.
> > 
> > I suspect the problem was on gerrit, nor on netbsd. Did it happened
> > again ?
> 
> I could imagine problems with exhausted system resources, but it would
> not produce a "Connection refused".

Depend, if they exhausted FD or something ? I am not a java specialist.

Could also just be too long to answer due to the load, but it was not
loaded :/

-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS




signature.asc
Description: This is a digitally signed message part
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Oleksandr Natalenko]

With the proposed patches I get the following assertion while copying files to 
GlusterFS volume:

===
glusterfs: mem-pool.c:305: __gf_free: Assertion `0xCAFEBABE == header->magic' 
failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe9ffb700 (LWP 12635)]
0x76f215f8 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0  0x76f215f8 in raise () from /usr/lib/libc.so.6
#1  0x76f22a7a in abort () from /usr/lib/libc.so.6
#2  0x76f1a417 in __assert_fail_base () from /usr/lib/libc.so.6
#3  0x76f1a4c2 in __assert_fail () from /usr/lib/libc.so.6
#4  0x77b6046b in __gf_free (free_ptr=0x7fffdc0b8f00) at mem-pool.c:
305
#5  0x75144eb9 in notify_kernel_loop (data=0x63df90) at fuse-bridge.c:
3893
#6  0x772994a4 in start_thread () from /usr/lib/libpthread.so.0
#7  0x76fd713d in clone () from /usr/lib/libc.so.6
===

On четвер, 21 січня 2016 р. 17:29:53 EET Kaleb KEITHLEY wrote:
> On 01/20/2016 04:08 AM, Oleksandr Natalenko wrote:
> > Yes, there are couple of messages like this in my logs too (I guess one
> > message per each remount):
> > 
> > ===
> > [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
> > glusterfs-fuse: kernel notifier loop terminated
> > ===
> 
> Bug reports and fixes for master and release-3.7 branches are:
> 
> master)
>  https://bugzilla.redhat.com/show_bug.cgi?id=1288857
>  http://review.gluster.org/12886
> 
> release-3.7)
>  https://bugzilla.redhat.com/show_bug.cgi?id=1288922
>  http://review.gluster.org/12887
> 
> The release-3.7 fix will be in glusterfs-3.7.7 when it's released.
> 
> I think with even with the above fixes applied there are still some
> issues remaining. I have submitted additional/revised fixes on top of
> the above fixes at:
> 
>  master: http://review.gluster.org/13274
>  release-3.7: http://review.gluster.org/13275
> 
> I invite you to review the patches in gerrit (review.gluster.org).
> 
> Regards,
> 
> --
> 
> Kaleb


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Kaleb KEITHLEY]

On 01/20/2016 04:08 AM, Oleksandr Natalenko wrote:
> Yes, there are couple of messages like this in my logs too (I guess one 
> message per each remount):
> 
> ===
> [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
> glusterfs-fuse: kernel notifier loop terminated
> ===
>

Bug reports and fixes for master and release-3.7 branches are:

master)
 https://bugzilla.redhat.com/show_bug.cgi?id=1288857
 http://review.gluster.org/12886

release-3.7)
 https://bugzilla.redhat.com/show_bug.cgi?id=1288922
 http://review.gluster.org/12887

The release-3.7 fix will be in glusterfs-3.7.7 when it's released.

I think with even with the above fixes applied there are still some
issues remaining. I have submitted additional/revised fixes on top of
the above fixes at:

 master: http://review.gluster.org/13274
 release-3.7: http://review.gluster.org/13275

I invite you to review the patches in gerrit (review.gluster.org).

Regards,

--

Kaleb

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] heal hanging

[Pranith Kumar Karampuri]

On 01/22/2016 12:51 AM, Glomski, Patrick wrote:
Pranith, could this kind of behavior be self-inflicted by us deleting 
files directly from the bricks? We have done that in the past to clean 
up an issues where gluster wouldn't allow us to delete from the mount.


Not sure. I haven't seen many people do this.


If so, is it feasible to clean them up by running a search on the 
.glusterfs directories directly and removing files with a reference 
count of 1 that are non-zero size (or directly checking the xattrs to 
be sure that it's not a DHT link).


find /data/brick01a/homegfs/.glusterfs -type f -not -empty -links -2 
-exec rm -f "{}" \;


Is there anything I'm inherently missing with that approach that will 
further corrupt the system?


You should make sure to only remove the files that are of the form 
.gluterfs/ab/cd/abcd


I am yet to go over the logs you provided. Will let you know.

Pranith



On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick 
> 
wrote:


Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl has
been running on one of the bricks on gfs02b for 25 min or so and
users cannot access the volume.

I re-listed the xattrop directories as well as a 'top' entry and
heal statistics. Then I restarted the gluster services on gfs02a.

=== top ===
PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEM TIME+ COMMAND
 8969 root  20   0 2815m 204m 3588 S 1181.0  0.6 591:06.93
glusterfsd

=== xattrop ===
/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-41f19453-91e4-437c-afa9-3b25614de210
xattrop-9b815879-2f4d-402b-867c-a6d65087788c

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-70131855-3cfb-49af-abce-9d23f57fb393
xattrop-dfb77848-a39d-4417-a725-9beca75d78c6

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
e6e47ed9-309b-42a7-8c44-28c29b9a20f8
xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0

/data/brick02b/homegfs/.glusterfs/indices/xattrop:
xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413

/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-7e20fdb1-5224-4b9a-be06-568708526d70

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
8034bc06-92cd-4fa5-8aaf-09039e79d2c8
c9ce22ed-6d8b-471b-a111-b39e57f0b512
94fa1d60-45ad-4341-b69c-315936b51e8d
xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7

/data/brick02b/homegfs/.glusterfs/indices/xattrop:
xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d


=== heal stats ===

homegfs [b0-gfsib01a] : Starting time of crawl   : Thu Jan 21
12:36:45 2016
homegfs [b0-gfsib01a] : Ending time of crawl : Thu Jan 21
12:36:45 2016
homegfs [b0-gfsib01a] : Type of crawl: INDEX
homegfs [b0-gfsib01a] : No. of entries healed: 0
homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
homegfs [b0-gfsib01a] : No. of heal failed entries   : 0

homegfs [b1-gfsib01b] : Starting time of crawl   : Thu Jan 21
12:36:19 2016
homegfs [b1-gfsib01b] : Ending time of crawl : Thu Jan 21
12:36:19 2016
homegfs [b1-gfsib01b] : Type of crawl: INDEX
homegfs [b1-gfsib01b] : No. of entries healed: 0
homegfs [b1-gfsib01b] : No. of entries in split-brain: 0
homegfs [b1-gfsib01b] : No. of heal failed entries   : 1

homegfs [b2-gfsib01a] : Starting time of crawl   : Thu Jan 21
12:36:48 2016
homegfs [b2-gfsib01a] : Ending time of crawl : Thu Jan 21
12:36:48 2016
homegfs [b2-gfsib01a] : Type of crawl: INDEX
homegfs [b2-gfsib01a] : No. of entries healed: 0
homegfs [b2-gfsib01a] : No. of entries in split-brain: 0
homegfs [b2-gfsib01a] : No. of heal failed entries   : 0

homegfs [b3-gfsib01b] : Starting time of crawl   : Thu Jan 21
12:36:47 2016
homegfs [b3-gfsib01b] : Ending time of crawl : Thu Jan 21
12:36:47 2016
homegfs [b3-gfsib01b] : Type of crawl: INDEX
homegfs [b3-gfsib01b] : No. of entries healed: 0
homegfs [b3-gfsib01b] : No. of entries in split-brain: 0
homegfs [b3-gfsib01b] : No. of heal failed entries   : 0

homegfs [b4-gfsib02a] : Starting time of crawl   : Thu Jan 21
12:36:06 2016
homegfs [b4-gfsib02a] : Ending time of crawl : Thu Jan 21
12:36:06 2016
homegfs [b4-gfsib02a] : Type of crawl: INDEX
homegfs [b4-gfsib02a] : No. of entries healed: 0
homegfs [b4-gfsib02a] : No. of entries in split-brain: 0
homegfs [b4-gfsib02a] : No. of heal failed entries   : 0

homegfs [b5-gfsib02b] : 

[Gluster-devel] 答复: Re: Gluster AFR volume write performance has been seriously affected by GLUSTERFS_WRITE_IS_APPEND in afr_writev

[li . ping288]

Hi Pranith, it is appreciated for your reply.

Pranith Kumar Karampuri  写于 2016/01/20 18:51:19:

> 发件人:  Pranith Kumar Karampuri 
> 收件人:  li.ping...@zte.com.cn, gluster-devel@gluster.org, 
> 日期:  2016/01/20 18:51
> 主题: Re: [Gluster-devel] Gluster AFR volume write performance has 
> been seriously affected by GLUSTERFS_WRITE_IS_APPEND in afr_writev
> 
> Sorry for the delay in response.

> On 01/15/2016 02:34 PM, li.ping...@zte.com.cn wrote:
> GLUSTERFS_WRITE_IS_APPEND Setting in afr_writev function at 
> glusterfs client end makes the posix_writev in the server end  deal 
> IO write fops from parallel  to serial in consequence. 
> 
> i.e.  multiple io-worker threads carrying out IO write fops are 
> blocked in posix_writev to execute final write fop pwrite/pwritev in
> __posix_writev function ONE AFTER ANOTHER. 
> 
> For example: 
> 
> thread1: iot_worker -> ...  -> posix_writev()   | 
> thread2: iot_worker -> ...  -> posix_writev()   | 
> thread3: iot_worker -> ...  -> posix_writev()   -> __posix_writev() 
> thread4: iot_worker -> ...  -> posix_writev()   | 
> 
> there are 4 iot_worker thread doing the 128KB IO write fops as 
> above, but only one can execute __posix_writev function and the 
> others have to wait. 
> 
> however, if the afr volume is configured on with storage.linux-aio 
> which is off in default,  the iot_worker will use posix_aio_writev 
> instead of posix_writev to write data. 
> the posix_aio_writev function won't be affected by 
> GLUSTERFS_WRITE_IS_APPEND, and the AFR volume write performance goes up. 

> I think this is a bug :-(.

Yeah, I agree with you. I suppose the GLUSTERFS_WRITE_IS_APPEND is a 
misuse in afr_writev.
I checked the original intent of GLUSTERS_WRITE_IS_APPEND change at review 
website:
http://review.gluster.org/#/c/5501/ 

The initial purpose seems to avoid an unnecessary fsync() in 
afr_changelog_post_op_safe function if the writing data position 
was currently at the end of the file, detected by 
(preop.ia_size == offset || (fd->flags & O_APPEND)) in posix_writev.

In comparison with the afr write performance loss, I think 
it costs too much. 

I suggest to make the GLUSTERS_WRITE_IS_APPEND setting configurable
just as ensure-durability in afr.

> 
> So, my question is whether  AFR volume could work fine with 
> storage.linux-aio configuration which bypass the 
> GLUSTERFS_WRITE_IS_APPEND setting in afr_writev, 
> and why glusterfs keeps posix_aio_writev different from posix_writev ? 
> 
> Any replies to clear my confusion would be grateful, and thanks in 
advance.
> What is the workload you have? multiple writers on same file workloads?

I test the afr gluster volume by fio like this:
fio --filename=/mnt/afr/20G.dat --direct=1 --rw=write --bs=128k --size=20G 
--numjobs=8 
--runtime=60 --group_reporting --name=afr_test  --iodepth=1 
--ioengine=libaio

The Glusterfs BRICKS are two IBM X3550 M3. 

The local disk direct write performance of 128KB IO req block size is 
about 18MB/s 
in single thread and 80MB/s in 8 multi-threads.

If the GLUSTERS_WRITE_IS_APPEND is configed, the afr gluster volume write 
performance is 18MB/s
as the single thread, and if not, the performance is nearby 
75MB/s.(network bandwith is enough)

> 
> Pranith
> 
> 
> 
> ZTE Information Security Notice: The information contained in this 
> mail (and any attachment transmitted herewith) is privileged and 
> confidential and is intended for the exclusive use of the addressee
> (s).  If you are not an intended recipient, any disclosure, 
> reproduction, distribution or other dissemination or use of the 
> information contained is strictly prohibited.  If you have received 
> this mail in error, please delete it and notify us immediately.
> 

> 
> 

> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel

ZTE Information Security Notice: The information contained in this mail (and 
any attachment transmitted herewith) is privileged and confidential and is 
intended for the exclusive use of the addressee(s).  If you are not an intended 
recipient, any disclosure, reproduction, distribution or other dissemination or 
use of the information contained is strictly prohibited.  If you have received 
this mail in error, please delete it and notify us immediately.
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] heal hanging

[Glomski, Patrick]

Unfortunately, all samba mounts to the gluster volume through the gfapi vfs
plugin have been disabled for the last 6 hours or so and frequency of %cpu
spikes is increased. We had switched to sharing a fuse mount through samba,
but I just disabled that as well. There are no samba shares of this volume
now. The spikes now happen every thirty minutes or so. We've resorted to
just rebooting the machine with high load for the present.

On Thu, Jan 21, 2016 at 8:49 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On 01/22/2016 07:13 AM, Glomski, Patrick wrote:
>
> We use the samba glusterfs virtual filesystem (the current version
> provided on download.gluster.org), but no windows clients connecting
> directly.
>
>
> Hmm.. Is there a way to disable using this and check if the CPU% still
> increases? What getxattr of "glusterfs.get_real_filename " does is
> to scan the entire directory looking for strcasecmp(,
> ). If anything matches then it will return the
> . But the problem is the scan is costly. So I wonder if
> this is the reason for the CPU spikes.
>
> Pranith
>
>
> On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>> Do you have any windows clients? I see a lot of getxattr calls for
>> "glusterfs.get_real_filename" which lead to full readdirs of the
>> directories on the brick.
>>
>> Pranith
>>
>> On 01/22/2016 12:51 AM, Glomski, Patrick wrote:
>>
>> Pranith, could this kind of behavior be self-inflicted by us deleting
>> files directly from the bricks? We have done that in the past to clean up
>> an issues where gluster wouldn't allow us to delete from the mount.
>>
>> If so, is it feasible to clean them up by running a search on the
>> .glusterfs directories directly and removing files with a reference count
>> of 1 that are non-zero size (or directly checking the xattrs to be sure
>> that it's not a DHT link).
>>
>> find /data/brick01a/homegfs/.glusterfs -type f -not -empty -links -2
>> -exec rm -f "{}" \;
>>
>> Is there anything I'm inherently missing with that approach that will
>> further corrupt the system?
>>
>>
>> On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick <
>> patrick.glom...@corvidtec.com> wrote:
>>
>>> Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl has been
>>> running on one of the bricks on gfs02b for 25 min or so and users cannot
>>> access the volume.
>>>
>>> I re-listed the xattrop directories as well as a 'top' entry and heal
>>> statistics. Then I restarted the gluster services on gfs02a.
>>>
>>> === top ===
>>> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
>>> COMMAND
>>>  8969 root  20   0 2815m 204m 3588 S 1181.0  0.6 591:06.93
>>> glusterfsd
>>>
>>> === xattrop ===
>>> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-41f19453-91e4-437c-afa9-3b25614de210
>>> xattrop-9b815879-2f4d-402b-867c-a6d65087788c
>>>
>>> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-70131855-3cfb-49af-abce-9d23f57fb393
>>> xattrop-dfb77848-a39d-4417-a725-9beca75d78c6
>>>
>>> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>>> e6e47ed9-309b-42a7-8c44-28c29b9a20f8
>>> xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
>>> xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
>>> xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0
>>>
>>> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
>>> xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413
>>>
>>> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531
>>>
>>> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-7e20fdb1-5224-4b9a-be06-568708526d70
>>>
>>> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>>> 8034bc06-92cd-4fa5-8aaf-09039e79d2c8
>>> c9ce22ed-6d8b-471b-a111-b39e57f0b512
>>> 94fa1d60-45ad-4341-b69c-315936b51e8d
>>> xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7
>>>
>>> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>>> xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d
>>>
>>>
>>> === heal stats ===
>>>
>>> homegfs [b0-gfsib01a] : Starting time of crawl   : Thu Jan 21
>>> 12:36:45 2016
>>> homegfs [b0-gfsib01a] : Ending time of crawl : Thu Jan 21
>>> 12:36:45 2016
>>> homegfs [b0-gfsib01a] : Type of crawl: INDEX
>>> homegfs [b0-gfsib01a] : No. of entries healed: 0
>>> homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
>>> homegfs [b0-gfsib01a] : No. of heal failed entries   : 0
>>>
>>> homegfs [b1-gfsib01b] : Starting time of crawl   : Thu Jan 21
>>> 12:36:19 2016
>>> homegfs [b1-gfsib01b] : Ending time of crawl : Thu Jan 21
>>> 12:36:19 2016
>>> homegfs [b1-gfsib01b] : Type of crawl: INDEX
>>> homegfs [b1-gfsib01b] : No. of entries healed: 0
>>> homegfs [b1-gfsib01b] : No. of entries in split-brain: 0
>>> homegfs [b1-gfsib01b] : No. of heal failed entries   : 1
>>>
>>> homegfs [b2-gfsib01a] : Starting time of 

Re: [Gluster-devel] [Gluster-users] heal hanging

[Pranith Kumar Karampuri]

On 01/22/2016 07:19 AM, Pranith Kumar Karampuri wrote:



On 01/22/2016 07:13 AM, Glomski, Patrick wrote:
We use the samba glusterfs virtual filesystem (the current version 
provided on download.gluster.org ), but 
no windows clients connecting directly.


Hmm.. Is there a way to disable using this and check if the CPU% still 
increases? What getxattr of "glusterfs.get_real_filename " 
does is to scan the entire directory looking for strcasecmp(, 
). If anything matches then it will return the 
. But the problem is the scan is costly. So I wonder 
if this is the reason for the CPU spikes.

+Raghavendra Talur, +Poornima

Raghavendra, Poornima,
When are these getxattrs triggered? Did you guys see any 
brick CPU spikes before? I initially thought it could be because of big 
directory heals. But this is happening even when no self-heals are 
required. So I had to move away from that theory.


Pranith


Pranith


On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri 
> wrote:


Do you have any windows clients? I see a lot of getxattr calls
for "glusterfs.get_real_filename" which lead to full readdirs of
the directories on the brick.

Pranith

On 01/22/2016 12:51 AM, Glomski, Patrick wrote:

Pranith, could this kind of behavior be self-inflicted by us
deleting files directly from the bricks? We have done that in
the past to clean up an issues where gluster wouldn't allow us
to delete from the mount.

If so, is it feasible to clean them up by running a search on
the .glusterfs directories directly and removing files with a
reference count of 1 that are non-zero size (or directly
checking the xattrs to be sure that it's not a DHT link).

find /data/brick01a/homegfs/.glusterfs -type f -not -empty
-links -2 -exec rm -f "{}" \;

Is there anything I'm inherently missing with that approach that
will further corrupt the system?


On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick
> wrote:

Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl
has been running on one of the bricks on gfs02b for 25 min
or so and users cannot access the volume.

I re-listed the xattrop directories as well as a 'top' entry
and heal statistics. Then I restarted the gluster services
on gfs02a.

=== top ===
PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
COMMAND
 8969 root  20   0 2815m 204m 3588 S 1181.0  0.6
591:06.93 glusterfsd

=== xattrop ===
/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-41f19453-91e4-437c-afa9-3b25614de210
xattrop-9b815879-2f4d-402b-867c-a6d65087788c

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-70131855-3cfb-49af-abce-9d23f57fb393
xattrop-dfb77848-a39d-4417-a725-9beca75d78c6

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
e6e47ed9-309b-42a7-8c44-28c29b9a20f8
xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0

/data/brick02b/homegfs/.glusterfs/indices/xattrop:
xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413

/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-7e20fdb1-5224-4b9a-be06-568708526d70

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
8034bc06-92cd-4fa5-8aaf-09039e79d2c8
c9ce22ed-6d8b-471b-a111-b39e57f0b512
94fa1d60-45ad-4341-b69c-315936b51e8d
xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7

/data/brick02b/homegfs/.glusterfs/indices/xattrop:
xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d


=== heal stats ===

homegfs [b0-gfsib01a] : Starting time of crawl   : Thu
Jan 21 12:36:45 2016
homegfs [b0-gfsib01a] : Ending time of crawl : Thu
Jan 21 12:36:45 2016
homegfs [b0-gfsib01a] : Type of crawl: INDEX
homegfs [b0-gfsib01a] : No. of entries healed: 0
homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
homegfs [b0-gfsib01a] : No. of heal failed entries   : 0

homegfs [b1-gfsib01b] : Starting time of crawl   : Thu
Jan 21 12:36:19 2016
homegfs [b1-gfsib01b] : Ending time of crawl : Thu
Jan 21 12:36:19 2016
homegfs [b1-gfsib01b] : Type of crawl: INDEX
homegfs [b1-gfsib01b] : No. of entries healed: 0
homegfs [b1-gfsib01b] : No. of entries in 

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

[Kaleb KEITHLEY]

On 01/21/2016 06:59 PM, Oleksandr Natalenko wrote:
> I see extra GF_FREE (node); added with two patches:
> 
> ===
> $ git diff HEAD~2 | gist
> https://gist.github.com/9524fa2054cc48278ea8
> ===
> 
> Is that intentionally? I guess I face double-free issue.
> 

I presume you're referring to the release-3.7 branch.

Yup, bad edit. Long day. That's why we review. ;-)

Please try the latest.

Thanks,

--

Kaleb

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] heal hanging

[Pranith Kumar Karampuri]

On 01/22/2016 07:13 AM, Glomski, Patrick wrote:
We use the samba glusterfs virtual filesystem (the current version 
provided on download.gluster.org ), but 
no windows clients connecting directly.


Hmm.. Is there a way to disable using this and check if the CPU% still 
increases? What getxattr of "glusterfs.get_real_filename " does 
is to scan the entire directory looking for strcasecmp(, 
). If anything matches then it will return the 
. But the problem is the scan is costly. So I wonder 
if this is the reason for the CPU spikes.


Pranith


On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri 
> wrote:


Do you have any windows clients? I see a lot of getxattr calls for
"glusterfs.get_real_filename" which lead to full readdirs of the
directories on the brick.

Pranith

On 01/22/2016 12:51 AM, Glomski, Patrick wrote:

Pranith, could this kind of behavior be self-inflicted by us
deleting files directly from the bricks? We have done that in the
past to clean up an issues where gluster wouldn't allow us to
delete from the mount.

If so, is it feasible to clean them up by running a search on the
.glusterfs directories directly and removing files with a
reference count of 1 that are non-zero size (or directly checking
the xattrs to be sure that it's not a DHT link).

find /data/brick01a/homegfs/.glusterfs -type f -not -empty -links
-2 -exec rm -f "{}" \;

Is there anything I'm inherently missing with that approach that
will further corrupt the system?


On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick
> wrote:

Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl
has been running on one of the bricks on gfs02b for 25 min or
so and users cannot access the volume.

I re-listed the xattrop directories as well as a 'top' entry
and heal statistics. Then I restarted the gluster services on
gfs02a.

=== top ===
PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
COMMAND
 8969 root  20   0 2815m 204m 3588 S 1181.0  0.6
591:06.93 glusterfsd

=== xattrop ===
/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-41f19453-91e4-437c-afa9-3b25614de210
xattrop-9b815879-2f4d-402b-867c-a6d65087788c

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-70131855-3cfb-49af-abce-9d23f57fb393
xattrop-dfb77848-a39d-4417-a725-9beca75d78c6

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
e6e47ed9-309b-42a7-8c44-28c29b9a20f8
xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0

/data/brick02b/homegfs/.glusterfs/indices/xattrop:
xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413

/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-7e20fdb1-5224-4b9a-be06-568708526d70

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
8034bc06-92cd-4fa5-8aaf-09039e79d2c8
c9ce22ed-6d8b-471b-a111-b39e57f0b512
94fa1d60-45ad-4341-b69c-315936b51e8d
xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7

/data/brick02b/homegfs/.glusterfs/indices/xattrop:
xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d


=== heal stats ===

homegfs [b0-gfsib01a] : Starting time of crawl   : Thu
Jan 21 12:36:45 2016
homegfs [b0-gfsib01a] : Ending time of crawl : Thu
Jan 21 12:36:45 2016
homegfs [b0-gfsib01a] : Type of crawl: INDEX
homegfs [b0-gfsib01a] : No. of entries healed: 0
homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
homegfs [b0-gfsib01a] : No. of heal failed entries   : 0

homegfs [b1-gfsib01b] : Starting time of crawl   : Thu
Jan 21 12:36:19 2016
homegfs [b1-gfsib01b] : Ending time of crawl : Thu
Jan 21 12:36:19 2016
homegfs [b1-gfsib01b] : Type of crawl: INDEX
homegfs [b1-gfsib01b] : No. of entries healed: 0
homegfs [b1-gfsib01b] : No. of entries in split-brain: 0
homegfs [b1-gfsib01b] : No. of heal failed entries   : 1

homegfs [b2-gfsib01a] : Starting time of crawl   : Thu
Jan 21 12:36:48 2016
homegfs [b2-gfsib01a] : Ending time of crawl : Thu
Jan 21 12:36:48 2016
homegfs [b2-gfsib01a] : Type of crawl: INDEX
homegfs [b2-gfsib01a] : No. of entries healed: 0

Re: [Gluster-devel] [Gluster-users] heal hanging

[Glomski, Patrick]

Samba version is 4.1.17 that you guys maintain at download.gluster.org. The
vfs plugin comes packaged with it.

http://download.gluster.org/pub/gluster/glusterfs/samba/EPEL.repo/epel-6/x86_64/

# smbd --version
Version 4.1.17

# rpm -qa | grep samba-vfs-glusterfs
samba-vfs-glusterfs-4.1.17-4.el6rhs.x86_64

Let us know if there's anything else we can provide,

Patrick


On Thu, Jan 21, 2016 at 10:07 PM, Raghavendra Talur 
wrote:

>
> On Jan 22, 2016 7:27 AM, "Pranith Kumar Karampuri" 
> wrote:
> >
> >
> >
> > On 01/22/2016 07:19 AM, Pranith Kumar Karampuri wrote:
> >>
> >>
> >>
> >> On 01/22/2016 07:13 AM, Glomski, Patrick wrote:
> >>>
> >>> We use the samba glusterfs virtual filesystem (the current version
> provided on download.gluster.org), but no windows clients connecting
> directly.
> >>
> >>
> >> Hmm.. Is there a way to disable using this and check if the CPU% still
> increases? What getxattr of "glusterfs.get_real_filename " does is
> to scan the entire directory looking for strcasecmp(,
> ). If anything matches then it will return the
> . But the problem is the scan is costly. So I wonder if
> this is the reason for the CPU spikes.
> >
> > +Raghavendra Talur, +Poornima
> >
> > Raghavendra, Poornima,
> > When are these getxattrs triggered? Did you guys see any
> brick CPU spikes before? I initially thought it could be because of big
> directory heals. But this is happening even when no self-heals are
> required. So I had to move away from that theory.
>
> These getxattrs are triggered when a SMB client performs a path based
> operation. It is necessary then that some client was connected.
>
> The last fix to go in that code for 3.6 was
> http://review.gluster.org/#/c/10403/.
>
> I am not able to determine which release of 3.6 it made into. Will update.
>
> Also we would need version of Samba installed. Including the vfs plugin
> package.
>
> There is a for loop of strcmp involved here which does take a lot of CPU.
> It should be for short bursts though and is expected and harmless.
>
> >
> > Pranith
> >
> >>
> >> Pranith
> >>>
> >>>
> >>> On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
> 
>  Do you have any windows clients? I see a lot of getxattr calls for
> "glusterfs.get_real_filename" which lead to full readdirs of the
> directories on the brick.
> 
>  Pranith
> 
>  On 01/22/2016 12:51 AM, Glomski, Patrick wrote:
> >
> > Pranith, could this kind of behavior be self-inflicted by us
> deleting files directly from the bricks? We have done that in the past to
> clean up an issues where gluster wouldn't allow us to delete from the mount.
> >
> > If so, is it feasible to clean them up by running a search on the
> .glusterfs directories directly and removing files with a reference count
> of 1 that are non-zero size (or directly checking the xattrs to be sure
> that it's not a DHT link).
> >
> > find /data/brick01a/homegfs/.glusterfs -type f -not -empty -links -2
> -exec rm -f "{}" \;
> >
> > Is there anything I'm inherently missing with that approach that
> will further corrupt the system?
> >
> >
> > On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick <
> patrick.glom...@corvidtec.com> wrote:
> >>
> >> Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl has
> been running on one of the bricks on gfs02b for 25 min or so and users
> cannot access the volume.
> >>
> >> I re-listed the xattrop directories as well as a 'top' entry and
> heal statistics. Then I restarted the gluster services on gfs02a.
> >>
> >> === top ===
> >> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
> COMMAND
> >>  8969 root  20   0 2815m 204m 3588 S 1181.0  0.6 591:06.93
> glusterfsd
> >>
> >> === xattrop ===
> >> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
> >> xattrop-41f19453-91e4-437c-afa9-3b25614de210
> xattrop-9b815879-2f4d-402b-867c-a6d65087788c
> >>
> >> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
> >> xattrop-70131855-3cfb-49af-abce-9d23f57fb393
> xattrop-dfb77848-a39d-4417-a725-9beca75d78c6
> >>
> >> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
> >> e6e47ed9-309b-42a7-8c44-28c29b9a20f8
> xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
> >> xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
> xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0
> >>
> >> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
> >> xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
> xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413
> >>
> >> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
> >> xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531
> >>
> >> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
> >> xattrop-7e20fdb1-5224-4b9a-be06-568708526d70
> >>
> >> 

Re: [Gluster-devel] [Gluster-users] heal hanging

[Pranith Kumar Karampuri]

Do you have any windows clients? I see a lot of getxattr calls for 
"glusterfs.get_real_filename" which lead to full readdirs of the 
directories on the brick.


Pranith

On 01/22/2016 12:51 AM, Glomski, Patrick wrote:
Pranith, could this kind of behavior be self-inflicted by us deleting 
files directly from the bricks? We have done that in the past to clean 
up an issues where gluster wouldn't allow us to delete from the mount.


If so, is it feasible to clean them up by running a search on the 
.glusterfs directories directly and removing files with a reference 
count of 1 that are non-zero size (or directly checking the xattrs to 
be sure that it's not a DHT link).


find /data/brick01a/homegfs/.glusterfs -type f -not -empty -links -2 
-exec rm -f "{}" \;


Is there anything I'm inherently missing with that approach that will 
further corrupt the system?



On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick 
> 
wrote:


Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl has
been running on one of the bricks on gfs02b for 25 min or so and
users cannot access the volume.

I re-listed the xattrop directories as well as a 'top' entry and
heal statistics. Then I restarted the gluster services on gfs02a.

=== top ===
PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEM TIME+ COMMAND
 8969 root  20   0 2815m 204m 3588 S 1181.0  0.6 591:06.93
glusterfsd

=== xattrop ===
/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-41f19453-91e4-437c-afa9-3b25614de210
xattrop-9b815879-2f4d-402b-867c-a6d65087788c

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-70131855-3cfb-49af-abce-9d23f57fb393
xattrop-dfb77848-a39d-4417-a725-9beca75d78c6

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
e6e47ed9-309b-42a7-8c44-28c29b9a20f8
xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0

/data/brick02b/homegfs/.glusterfs/indices/xattrop:
xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413

/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-7e20fdb1-5224-4b9a-be06-568708526d70

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
8034bc06-92cd-4fa5-8aaf-09039e79d2c8
c9ce22ed-6d8b-471b-a111-b39e57f0b512
94fa1d60-45ad-4341-b69c-315936b51e8d
xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7

/data/brick02b/homegfs/.glusterfs/indices/xattrop:
xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d


=== heal stats ===

homegfs [b0-gfsib01a] : Starting time of crawl   : Thu Jan 21
12:36:45 2016
homegfs [b0-gfsib01a] : Ending time of crawl : Thu Jan 21
12:36:45 2016
homegfs [b0-gfsib01a] : Type of crawl: INDEX
homegfs [b0-gfsib01a] : No. of entries healed: 0
homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
homegfs [b0-gfsib01a] : No. of heal failed entries   : 0

homegfs [b1-gfsib01b] : Starting time of crawl   : Thu Jan 21
12:36:19 2016
homegfs [b1-gfsib01b] : Ending time of crawl : Thu Jan 21
12:36:19 2016
homegfs [b1-gfsib01b] : Type of crawl: INDEX
homegfs [b1-gfsib01b] : No. of entries healed: 0
homegfs [b1-gfsib01b] : No. of entries in split-brain: 0
homegfs [b1-gfsib01b] : No. of heal failed entries   : 1

homegfs [b2-gfsib01a] : Starting time of crawl   : Thu Jan 21
12:36:48 2016
homegfs [b2-gfsib01a] : Ending time of crawl : Thu Jan 21
12:36:48 2016
homegfs [b2-gfsib01a] : Type of crawl: INDEX
homegfs [b2-gfsib01a] : No. of entries healed: 0
homegfs [b2-gfsib01a] : No. of entries in split-brain: 0
homegfs [b2-gfsib01a] : No. of heal failed entries   : 0

homegfs [b3-gfsib01b] : Starting time of crawl   : Thu Jan 21
12:36:47 2016
homegfs [b3-gfsib01b] : Ending time of crawl : Thu Jan 21
12:36:47 2016
homegfs [b3-gfsib01b] : Type of crawl: INDEX
homegfs [b3-gfsib01b] : No. of entries healed: 0
homegfs [b3-gfsib01b] : No. of entries in split-brain: 0
homegfs [b3-gfsib01b] : No. of heal failed entries   : 0

homegfs [b4-gfsib02a] : Starting time of crawl   : Thu Jan 21
12:36:06 2016
homegfs [b4-gfsib02a] : Ending time of crawl : Thu Jan 21
12:36:06 2016
homegfs [b4-gfsib02a] : Type of crawl: INDEX
homegfs [b4-gfsib02a] : No. of entries healed: 0
homegfs [b4-gfsib02a] : No. of entries in split-brain: 0
homegfs [b4-gfsib02a] : No. of heal failed entries   : 0

homegfs [b5-gfsib02b] : Starting time of crawl   : Thu Jan 21

Re: [Gluster-devel] [Gluster-users] heal hanging

[Glomski, Patrick]

We use the samba glusterfs virtual filesystem (the current version provided
on download.gluster.org), but no windows clients connecting directly.

On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

> Do you have any windows clients? I see a lot of getxattr calls for
> "glusterfs.get_real_filename" which lead to full readdirs of the
> directories on the brick.
>
> Pranith
>
> On 01/22/2016 12:51 AM, Glomski, Patrick wrote:
>
> Pranith, could this kind of behavior be self-inflicted by us deleting
> files directly from the bricks? We have done that in the past to clean up
> an issues where gluster wouldn't allow us to delete from the mount.
>
> If so, is it feasible to clean them up by running a search on the
> .glusterfs directories directly and removing files with a reference count
> of 1 that are non-zero size (or directly checking the xattrs to be sure
> that it's not a DHT link).
>
> find /data/brick01a/homegfs/.glusterfs -type f -not -empty -links -2 -exec
> rm -f "{}" \;
>
> Is there anything I'm inherently missing with that approach that will
> further corrupt the system?
>
>
> On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick <
> patrick.glom...@corvidtec.com> wrote:
>
>> Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl has been
>> running on one of the bricks on gfs02b for 25 min or so and users cannot
>> access the volume.
>>
>> I re-listed the xattrop directories as well as a 'top' entry and heal
>> statistics. Then I restarted the gluster services on gfs02a.
>>
>> === top ===
>> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
>> COMMAND
>>  8969 root  20   0 2815m 204m 3588 S 1181.0  0.6 591:06.93
>> glusterfsd
>>
>> === xattrop ===
>> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>> xattrop-41f19453-91e4-437c-afa9-3b25614de210
>> xattrop-9b815879-2f4d-402b-867c-a6d65087788c
>>
>> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>> xattrop-70131855-3cfb-49af-abce-9d23f57fb393
>> xattrop-dfb77848-a39d-4417-a725-9beca75d78c6
>>
>> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>> e6e47ed9-309b-42a7-8c44-28c29b9a20f8
>> xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
>> xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
>> xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0
>>
>> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>> xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
>> xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413
>>
>> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>> xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531
>>
>> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>> xattrop-7e20fdb1-5224-4b9a-be06-568708526d70
>>
>> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>> 8034bc06-92cd-4fa5-8aaf-09039e79d2c8  c9ce22ed-6d8b-471b-a111-b39e57f0b512
>> 94fa1d60-45ad-4341-b69c-315936b51e8d
>> xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7
>>
>> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>> xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d
>>
>>
>> === heal stats ===
>>
>> homegfs [b0-gfsib01a] : Starting time of crawl   : Thu Jan 21
>> 12:36:45 2016
>> homegfs [b0-gfsib01a] : Ending time of crawl : Thu Jan 21
>> 12:36:45 2016
>> homegfs [b0-gfsib01a] : Type of crawl: INDEX
>> homegfs [b0-gfsib01a] : No. of entries healed: 0
>> homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
>> homegfs [b0-gfsib01a] : No. of heal failed entries   : 0
>>
>> homegfs [b1-gfsib01b] : Starting time of crawl   : Thu Jan 21
>> 12:36:19 2016
>> homegfs [b1-gfsib01b] : Ending time of crawl : Thu Jan 21
>> 12:36:19 2016
>> homegfs [b1-gfsib01b] : Type of crawl: INDEX
>> homegfs [b1-gfsib01b] : No. of entries healed: 0
>> homegfs [b1-gfsib01b] : No. of entries in split-brain: 0
>> homegfs [b1-gfsib01b] : No. of heal failed entries   : 1
>>
>> homegfs [b2-gfsib01a] : Starting time of crawl   : Thu Jan 21
>> 12:36:48 2016
>> homegfs [b2-gfsib01a] : Ending time of crawl : Thu Jan 21
>> 12:36:48 2016
>> homegfs [b2-gfsib01a] : Type of crawl: INDEX
>> homegfs [b2-gfsib01a] : No. of entries healed: 0
>> homegfs [b2-gfsib01a] : No. of entries in split-brain: 0
>> homegfs [b2-gfsib01a] : No. of heal failed entries   : 0
>>
>> homegfs [b3-gfsib01b] : Starting time of crawl   : Thu Jan 21
>> 12:36:47 2016
>> homegfs [b3-gfsib01b] : Ending time of crawl : Thu Jan 21
>> 12:36:47 2016
>> homegfs [b3-gfsib01b] : Type of crawl: INDEX
>> homegfs [b3-gfsib01b] : No. of entries healed: 0
>> homegfs [b3-gfsib01b] : No. of entries in split-brain: 0
>> homegfs [b3-gfsib01b] : No. of heal failed entries   : 0
>>
>> homegfs [b4-gfsib02a] : Starting time of crawl   : Thu Jan 21
>> 12:36:06 2016
>> homegfs [b4-gfsib02a] : Ending time of crawl : Thu Jan 21
>> 12:36:06 2016
>> homegfs [b4-gfsib02a] : Type of crawl: INDEX
>> homegfs [b4-gfsib02a] : No. of entries healed: 0
>> homegfs 

Re: [Gluster-devel] [Gluster-users] heal hanging

[Pranith Kumar Karampuri]

On 01/22/2016 07:25 AM, Glomski, Patrick wrote:
Unfortunately, all samba mounts to the gluster volume through the 
gfapi vfs plugin have been disabled for the last 6 hours or so and 
frequency of %cpu spikes is increased. We had switched to sharing a 
fuse mount through samba, but I just disabled that as well. There are 
no samba shares of this volume now. The spikes now happen every thirty 
minutes or so. We've resorted to just rebooting the machine with high 
load for the present.


Next time this CPU spike happens, could you collect samples of gstack 
 every second for 10-20 seconds? That helps in finding the 
heavily hit function calls.


Something like "for i in {1..20}; do gstack  > 
sample-$i.txt; done"


Pranith


On Thu, Jan 21, 2016 at 8:49 PM, Pranith Kumar Karampuri 
> wrote:




On 01/22/2016 07:13 AM, Glomski, Patrick wrote:

We use the samba glusterfs virtual filesystem (the current
version provided on download.gluster.org
), but no windows clients connecting
directly.


Hmm.. Is there a way to disable using this and check if the CPU%
still increases? What getxattr of "glusterfs.get_real_filename
" does is to scan the entire directory looking for
strcasecmp(, ). If anything matches
then it will return the . But the problem is the
scan is costly. So I wonder if this is the reason for the CPU spikes.

Pranith



On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri
> wrote:

Do you have any windows clients? I see a lot of getxattr
calls for "glusterfs.get_real_filename" which lead to full
readdirs of the directories on the brick.

Pranith

On 01/22/2016 12:51 AM, Glomski, Patrick wrote:

Pranith, could this kind of behavior be self-inflicted by us
deleting files directly from the bricks? We have done that
in the past to clean up an issues where gluster wouldn't
allow us to delete from the mount.

If so, is it feasible to clean them up by running a search
on the .glusterfs directories directly and removing files
with a reference count of 1 that are non-zero size (or
directly checking the xattrs to be sure that it's not a DHT
link).

find /data/brick01a/homegfs/.glusterfs -type f -not -empty
-links -2 -exec rm -f "{}" \;

Is there anything I'm inherently missing with that approach
that will further corrupt the system?


On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick
> wrote:

Load spiked again: ~1200%cpu on gfs02a for glusterfsd.
Crawl has been running on one of the bricks on gfs02b
for 25 min or so and users cannot access the volume.

I re-listed the xattrop directories as well as a 'top'
entry and heal statistics. Then I restarted the gluster
services on gfs02a.

=== top ===
PID USER  PR  NI VIRT  RES  SHR S %CPU %MEMTIME+
COMMAND
 8969 root  20   0 2815m 204m 3588 S 1181.0 0.6
591:06.93 glusterfsd

=== xattrop ===
/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-41f19453-91e4-437c-afa9-3b25614de210
xattrop-9b815879-2f4d-402b-867c-a6d65087788c

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-70131855-3cfb-49af-abce-9d23f57fb393
xattrop-dfb77848-a39d-4417-a725-9beca75d78c6

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
e6e47ed9-309b-42a7-8c44-28c29b9a20f8
xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0

/data/brick02b/homegfs/.glusterfs/indices/xattrop:
xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413

/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-7e20fdb1-5224-4b9a-be06-568708526d70

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
8034bc06-92cd-4fa5-8aaf-09039e79d2c8
c9ce22ed-6d8b-471b-a111-b39e57f0b512
94fa1d60-45ad-4341-b69c-315936b51e8d
xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7

/data/brick02b/homegfs/.glusterfs/indices/xattrop:
xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d


=== heal stats ===

homegfs [b0-gfsib01a] : Starting time of 

Re: [Gluster-devel] [Gluster-users] heal hanging

[Raghavendra Talur]

On Jan 22, 2016 7:27 AM, "Pranith Kumar Karampuri" 
wrote:
>
>
>
> On 01/22/2016 07:19 AM, Pranith Kumar Karampuri wrote:
>>
>>
>>
>> On 01/22/2016 07:13 AM, Glomski, Patrick wrote:
>>>
>>> We use the samba glusterfs virtual filesystem (the current version
provided on download.gluster.org), but no windows clients connecting
directly.
>>
>>
>> Hmm.. Is there a way to disable using this and check if the CPU% still
increases? What getxattr of "glusterfs.get_real_filename " does is
to scan the entire directory looking for strcasecmp(,
). If anything matches then it will return the
. But the problem is the scan is costly. So I wonder if
this is the reason for the CPU spikes.
>
> +Raghavendra Talur, +Poornima
>
> Raghavendra, Poornima,
> When are these getxattrs triggered? Did you guys see any
brick CPU spikes before? I initially thought it could be because of big
directory heals. But this is happening even when no self-heals are
required. So I had to move away from that theory.

These getxattrs are triggered when a SMB client performs a path based
operation. It is necessary then that some client was connected.

The last fix to go in that code for 3.6 was
http://review.gluster.org/#/c/10403/.

I am not able to determine which release of 3.6 it made into. Will update.

Also we would need version of Samba installed. Including the vfs plugin
package.

There is a for loop of strcmp involved here which does take a lot of CPU.
It should be for short bursts though and is expected and harmless.

>
> Pranith
>
>>
>> Pranith
>>>
>>>
>>> On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

 Do you have any windows clients? I see a lot of getxattr calls for
"glusterfs.get_real_filename" which lead to full readdirs of the
directories on the brick.

 Pranith

 On 01/22/2016 12:51 AM, Glomski, Patrick wrote:
>
> Pranith, could this kind of behavior be self-inflicted by us deleting
files directly from the bricks? We have done that in the past to clean up
an issues where gluster wouldn't allow us to delete from the mount.
>
> If so, is it feasible to clean them up by running a search on the
.glusterfs directories directly and removing files with a reference count
of 1 that are non-zero size (or directly checking the xattrs to be sure
that it's not a DHT link).
>
> find /data/brick01a/homegfs/.glusterfs -type f -not -empty -links -2
-exec rm -f "{}" \;
>
> Is there anything I'm inherently missing with that approach that will
further corrupt the system?
>
>
> On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick <
patrick.glom...@corvidtec.com> wrote:
>>
>> Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl has
been running on one of the bricks on gfs02b for 25 min or so and users
cannot access the volume.
>>
>> I re-listed the xattrop directories as well as a 'top' entry and
heal statistics. Then I restarted the gluster services on gfs02a.
>>
>> === top ===
>> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
COMMAND
>>  8969 root  20   0 2815m 204m 3588 S 1181.0  0.6 591:06.93
glusterfsd
>>
>> === xattrop ===
>> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>> xattrop-41f19453-91e4-437c-afa9-3b25614de210
xattrop-9b815879-2f4d-402b-867c-a6d65087788c
>>
>> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>> xattrop-70131855-3cfb-49af-abce-9d23f57fb393
xattrop-dfb77848-a39d-4417-a725-9beca75d78c6
>>
>> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>> e6e47ed9-309b-42a7-8c44-28c29b9a20f8
xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
>> xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0
>>
>> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>> xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413
>>
>> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
>> xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531
>>
>> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
>> xattrop-7e20fdb1-5224-4b9a-be06-568708526d70
>>
>> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
>> 8034bc06-92cd-4fa5-8aaf-09039e79d2c8
c9ce22ed-6d8b-471b-a111-b39e57f0b512
>> 94fa1d60-45ad-4341-b69c-315936b51e8d
xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7
>>
>> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
>> xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d
>>
>>
>> === heal stats ===
>>
>> homegfs [b0-gfsib01a] : Starting time of crawl   : Thu Jan 21
12:36:45 2016
>> homegfs [b0-gfsib01a] : Ending time of crawl : Thu Jan 21
12:36:45 2016
>> homegfs [b0-gfsib01a] : Type of crawl: INDEX
>> homegfs [b0-gfsib01a] : No. of entries healed: 0
>> 

Re: [Gluster-devel] Netbsd regressions are failing because of connection problems?

[Emmanuel Dreyfus]

Michael Scherer  wrote:

> Depend, if they exhausted FD or something ? I am not a java specialist.

It is not the same errno, AFAIK.
 
> Could also just be too long to answer due to the load, but it was not
> loaded :/

High loads give timeouts. I may be wrong, but I beleive connection
refused is really when it gets a TCP RST.


-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
m...@netbsd.org
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] heal hanging

[Pranith Kumar Karampuri]

On 01/22/2016 07:25 AM, Glomski, Patrick wrote:
Unfortunately, all samba mounts to the gluster volume through the 
gfapi vfs plugin have been disabled for the last 6 hours or so and 
frequency of %cpu spikes is increased. We had switched to sharing a 
fuse mount through samba, but I just disabled that as well. There are 
no samba shares of this volume now. The spikes now happen every thirty 
minutes or so. We've resorted to just rebooting the machine with high 
load for the present.


Could you see if the logs of following type are not at all coming?
[2016-01-21 15:13:00.005736] E 
[server-rpc-fops.c:768:server_getxattr_cbk] 0-homegfs-server: 110: 
GETXATTR /wks_backup (40e582d6-b0c7-4099-ba88-9168a3c

32ca6) (glusterfs.get_real_filename:desktop.ini) ==> (Permission denied)

These are operations that failed. Operations that succeed are the ones 
that will scan the directory. But I don't have a way to find them other 
than using tcpdumps.


At the moment I have 2 theories:
1) these get_real_filename calls
2) [2016-01-21 16:10:38.017828] E [server-helpers.c:46:gid_resolve] 
0-gid-cache: getpwuid_r(494) failed

"

Yessir they are.  Normally, sssd would look to the local cache file in 
/var/lib/sss/db/ first, to get any group or userid information, then go 
out to the domain controller.  I put the options that we are using on 
our GFS volumes below…  Thanks for your help.


We had been running sssd with sssd_nss and sssd_be sub-processes on 
these systems for a long time, under the GFS 3.5.2 code, and not run 
into the problem that David described with the high cpu usage on sssd_nss.


*"
*That was Tom Young's email 1.5 years back when we debugged it. But the 
process which was consuming lot of cpu is sssd_nss. So I am not sure if 
it is same issue. Let us debug to see '1)' doesn't happen. The gstack 
traces I asked for should also help.


Pranith


On Thu, Jan 21, 2016 at 8:49 PM, Pranith Kumar Karampuri 
> wrote:




On 01/22/2016 07:13 AM, Glomski, Patrick wrote:

We use the samba glusterfs virtual filesystem (the current
version provided on download.gluster.org
), but no windows clients connecting
directly.


Hmm.. Is there a way to disable using this and check if the CPU%
still increases? What getxattr of "glusterfs.get_real_filename
" does is to scan the entire directory looking for
strcasecmp(, ). If anything matches
then it will return the . But the problem is the
scan is costly. So I wonder if this is the reason for the CPU spikes.

Pranith



On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri
> wrote:

Do you have any windows clients? I see a lot of getxattr
calls for "glusterfs.get_real_filename" which lead to full
readdirs of the directories on the brick.

Pranith

On 01/22/2016 12:51 AM, Glomski, Patrick wrote:

Pranith, could this kind of behavior be self-inflicted by us
deleting files directly from the bricks? We have done that
in the past to clean up an issues where gluster wouldn't
allow us to delete from the mount.

If so, is it feasible to clean them up by running a search
on the .glusterfs directories directly and removing files
with a reference count of 1 that are non-zero size (or
directly checking the xattrs to be sure that it's not a DHT
link).

find /data/brick01a/homegfs/.glusterfs -type f -not -empty
-links -2 -exec rm -f "{}" \;

Is there anything I'm inherently missing with that approach
that will further corrupt the system?


On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick
> wrote:

Load spiked again: ~1200%cpu on gfs02a for glusterfsd.
Crawl has been running on one of the bricks on gfs02b
for 25 min or so and users cannot access the volume.

I re-listed the xattrop directories as well as a 'top'
entry and heal statistics. Then I restarted the gluster
services on gfs02a.

=== top ===
PID USER  PR  NI VIRT  RES  SHR S %CPU %MEMTIME+
COMMAND
 8969 root  20   0 2815m 204m 3588 S 1181.0 0.6
591:06.93 glusterfsd

=== xattrop ===
/data/brick01a/homegfs/.glusterfs/indices/xattrop:
xattrop-41f19453-91e4-437c-afa9-3b25614de210
xattrop-9b815879-2f4d-402b-867c-a6d65087788c

/data/brick02a/homegfs/.glusterfs/indices/xattrop:
xattrop-70131855-3cfb-49af-abce-9d23f57fb393
xattrop-dfb77848-a39d-4417-a725-9beca75d78c6

/data/brick01b/homegfs/.glusterfs/indices/xattrop:
   

Re: [Gluster-devel] Jenkins accounts for all devs.

[Ravishankar N]

On 01/14/2016 12:16 PM, Kaushal M wrote:

On Thu, Jan 14, 2016 at 10:33 AM, Raghavendra Talur  wrote:


On Thu, Jan 14, 2016 at 10:32 AM, Ravishankar N 
wrote:

On 01/08/2016 12:03 PM, Raghavendra Talur wrote:

P.S: Stop using the "universal" jenkins account to trigger jenkins build
if you are not a maintainer.
If you are a maintainer and don't have your own jenkins account then get
one soon!


I would request for a jenkins account for non-maintainers too, at least
for the devs who are actively contributing code (as opposed to random
one-off commits from persons). That way, if the regression failure is
*definitely* not in my patch (or) is a spurious failure (or) is something
that I need to take a netbsd slave offline to debug etc.,  I don't have to
be blocked on the Maintainer. Since the accounts are anyway tied to an
individual, it should be easy to spot if someone habitually re-trigger
regressions without any initial debugging.


+1

We'd like to give everyone accounts. But the way we're providing
accounts now gives admin accounts to all. This is not very secure.

This was one of the reasons misc setup freeipa.gluster.org, to provide
controlled accounts for all. But it hasn't been used yet. We would
need to integrate jenkins and the slaves with freeipa, which would
give everyone easy access.


Hi Michael,
Do you think it is possible to have this integration soon so that all 
contributors can re-trigger/initiate builds by themselves?

Regards,
Ravi

-Ravi



___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel



___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Jenkins accounts for all devs.

[Mohammed Rafi K C]

On 01/22/2016 11:31 AM, Ravishankar N wrote:
> On 01/14/2016 12:16 PM, Kaushal M wrote:
>> On Thu, Jan 14, 2016 at 10:33 AM, Raghavendra Talur
>>  wrote:
>>>
>>> On Thu, Jan 14, 2016 at 10:32 AM, Ravishankar N
>>> 
>>> wrote:
 On 01/08/2016 12:03 PM, Raghavendra Talur wrote:
> P.S: Stop using the "universal" jenkins account to trigger jenkins
> build
> if you are not a maintainer.
> If you are a maintainer and don't have your own jenkins account
> then get
> one soon!
>
 I would request for a jenkins account for non-maintainers too, at
 least
 for the devs who are actively contributing code (as opposed to random
 one-off commits from persons). That way, if the regression failure is
 *definitely* not in my patch (or) is a spurious failure (or) is
 something
 that I need to take a netbsd slave offline to debug etc.,  I don't
 have to
 be blocked on the Maintainer. Since the accounts are anyway tied to an
 individual, it should be easy to spot if someone habitually re-trigger
 regressions without any initial debugging.

>>> +1

+2 ;)

In the last couple of days, It has become painful procedure  to get one
patch merged upstream. Having an account for everyone will definitely
help here to some extend.

Regards
Rafi KC




>> We'd like to give everyone accounts. But the way we're providing
>> accounts now gives admin accounts to all. This is not very secure.
>>
>> This was one of the reasons misc setup freeipa.gluster.org, to provide
>> controlled accounts for all. But it hasn't been used yet. We would
>> need to integrate jenkins and the slaves with freeipa, which would
>> give everyone easy access.
>
> Hi Michael,
> Do you think it is possible to have this integration soon so that all
> contributors can re-trigger/initiate builds by themselves?
> Regards,
> Ravi
 -Ravi
>>>
>>>
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] heal hanging

[Poornima Gurusiddaiah]

Hi, 

As mentioned glusterfs.get_real_filename getxattr is called when we need to 
check if a file(case insensitive) exists in a directory. 

You could run the following command to get to know the perf details. 
I guess you need to have debuginfo installed for this to work. 

Record perf: 
$perf record -a -g -p  -o perf-glusterfsd.data 

Generate the stat: 
perf report -g -i perf-glusterfsd.data 

Attaching perf might slow down the process further, hence recommended to use it 
in test setup. 

Regards, 
Poornima 

- Original Message -

> From: "Raghavendra Talur" 
> To: "Pranith Kumar Karampuri" 
> Cc: "Gluster Devel" , "Patrick Glomski"
> , gluster-us...@gluster.org, "David Robinson"
> , "Poornima Gurusiddaiah" 
> Sent: Friday, January 22, 2016 8:37:50 AM
> Subject: Re: [Gluster-devel] [Gluster-users] heal hanging

> On Jan 22, 2016 7:27 AM, "Pranith Kumar Karampuri" < pkara...@redhat.com >
> wrote:
> >
> >
> >
> > On 01/22/2016 07:19 AM, Pranith Kumar Karampuri wrote:
> >>
> >>
> >>
> >> On 01/22/2016 07:13 AM, Glomski, Patrick wrote:
> >>>
> >>> We use the samba glusterfs virtual filesystem (the current version
> >>> provided on download.gluster.org ), but no windows clients connecting
> >>> directly.
> >>
> >>
> >> Hmm.. Is there a way to disable using this and check if the CPU% still
> >> increases? What getxattr of "glusterfs.get_real_filename " does
> >> is to scan the entire directory looking for strcasecmp(,
> >> ). If anything matches then it will return the
> >> . But the problem is the scan is costly. So I wonder if
> >> this is the reason for the CPU spikes.
> >
> > +Raghavendra Talur, +Poornima
> >
> > Raghavendra, Poornima,
> > When are these getxattrs triggered? Did you guys see any brick CPU spikes
> > before? I initially thought it could be because of big directory heals.
> > But this is happening even when no self-heals are required. So I had to
> > move away from that theory.

> These getxattrs are triggered when a SMB client performs a path based
> operation. It is necessary then that some client was connected.

> The last fix to go in that code for 3.6 was
> http://review.gluster.org/#/c/10403/ .

> I am not able to determine which release of 3.6 it made into. Will update.

> Also we would need version of Samba installed. Including the vfs plugin
> package.

> There is a for loop of strcmp involved here which does take a lot of CPU. It
> should be for short bursts though and is expected and harmless.

> >
> > Pranith
> >
> >>
> >> Pranith
> >>>
> >>>
> >>> On Thu, Jan 21, 2016 at 8:37 PM, Pranith Kumar Karampuri <
> >>> pkara...@redhat.com > wrote:
> 
>  Do you have any windows clients? I see a lot of getxattr calls for
>  "glusterfs.get_real_filename" which lead to full readdirs of the
>  directories on the brick.
> 
>  Pranith
> 
>  On 01/22/2016 12:51 AM, Glomski, Patrick wrote:
> >
> > Pranith, could this kind of behavior be self-inflicted by us deleting
> > files directly from the bricks? We have done that in the past to clean
> > up an issues where gluster wouldn't allow us to delete from the mount.
> >
> > If so, is it feasible to clean them up by running a search on the
> > .glusterfs directories directly and removing files with a reference
> > count of 1 that are non-zero size (or directly checking the xattrs to
> > be sure that it's not a DHT link).
> >
> > find /data/brick01a/homegfs/.glusterfs -type f -not -empty -links -2
> > -exec rm -f "{}" \;
> >
> > Is there anything I'm inherently missing with that approach that will
> > further corrupt the system?
> >
> >
> > On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick <
> > patrick.glom...@corvidtec.com > wrote:
> >>
> >> Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl has been
> >> running on one of the bricks on gfs02b for 25 min or so and users
> >> cannot access the volume.
> >>
> >> I re-listed the xattrop directories as well as a 'top' entry and heal
> >> statistics. Then I restarted the gluster services on gfs02a.
> >>
> >> === top ===
> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >> 8969 root 20 0 2815m 204m 3588 S 1181.0 0.6 591:06.93 glusterfsd
> >>
> >> === xattrop ===
> >> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
> >> xattrop-41f19453-91e4-437c-afa9-3b25614de210
> >> xattrop-9b815879-2f4d-402b-867c-a6d65087788c
> >>
> >> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
> >> xattrop-70131855-3cfb-49af-abce-9d23f57fb393
> >> xattrop-dfb77848-a39d-4417-a725-9beca75d78c6
> >>
> >> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
> >> 

Re: [Gluster-devel] [Gluster-users] heal hanging

[Glomski, Patrick]

Pranith, could this kind of behavior be self-inflicted by us deleting files
directly from the bricks? We have done that in the past to clean up an
issues where gluster wouldn't allow us to delete from the mount.

If so, is it feasible to clean them up by running a search on the
.glusterfs directories directly and removing files with a reference count
of 1 that are non-zero size (or directly checking the xattrs to be sure
that it's not a DHT link).

find /data/brick01a/homegfs/.glusterfs -type f -not -empty -links -2 -exec
rm -f "{}" \;

Is there anything I'm inherently missing with that approach that will
further corrupt the system?


On Thu, Jan 21, 2016 at 1:02 PM, Glomski, Patrick <
patrick.glom...@corvidtec.com> wrote:

> Load spiked again: ~1200%cpu on gfs02a for glusterfsd. Crawl has been
> running on one of the bricks on gfs02b for 25 min or so and users cannot
> access the volume.
>
> I re-listed the xattrop directories as well as a 'top' entry and heal
> statistics. Then I restarted the gluster services on gfs02a.
>
> === top ===
> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
> COMMAND
>  8969 root  20   0 2815m 204m 3588 S 1181.0  0.6 591:06.93
> glusterfsd
>
> === xattrop ===
> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
> xattrop-41f19453-91e4-437c-afa9-3b25614de210
> xattrop-9b815879-2f4d-402b-867c-a6d65087788c
>
> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
> xattrop-70131855-3cfb-49af-abce-9d23f57fb393
> xattrop-dfb77848-a39d-4417-a725-9beca75d78c6
>
> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
> e6e47ed9-309b-42a7-8c44-28c29b9a20f8
> xattrop-5c797a64-bde7-4eac-b4fc-0befc632e125
> xattrop-38ec65a1-00b5-4544-8a6c-bf0f531a1934
> xattrop-ef0980ad-f074-4163-979f-16d5ef85b0a0
>
> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
> xattrop-7402438d-0ee7-4fcf-b9bb-b561236f99bc
> xattrop-8ffbf5f7-ace3-497d-944e-93ac85241413
>
> /data/brick01a/homegfs/.glusterfs/indices/xattrop:
> xattrop-0115acd0-caae-4dfd-b3b4-7cc42a0ff531
>
> /data/brick02a/homegfs/.glusterfs/indices/xattrop:
> xattrop-7e20fdb1-5224-4b9a-be06-568708526d70
>
> /data/brick01b/homegfs/.glusterfs/indices/xattrop:
> 8034bc06-92cd-4fa5-8aaf-09039e79d2c8  c9ce22ed-6d8b-471b-a111-b39e57f0b512
> 94fa1d60-45ad-4341-b69c-315936b51e8d
> xattrop-9c04623a-64ce-4f66-8b23-dbaba49119c7
>
> /data/brick02b/homegfs/.glusterfs/indices/xattrop:
> xattrop-b8c8f024-d038-49a2-9a53-c54ead09111d
>
>
> === heal stats ===
>
> homegfs [b0-gfsib01a] : Starting time of crawl   : Thu Jan 21 12:36:45
> 2016
> homegfs [b0-gfsib01a] : Ending time of crawl : Thu Jan 21 12:36:45
> 2016
> homegfs [b0-gfsib01a] : Type of crawl: INDEX
> homegfs [b0-gfsib01a] : No. of entries healed: 0
> homegfs [b0-gfsib01a] : No. of entries in split-brain: 0
> homegfs [b0-gfsib01a] : No. of heal failed entries   : 0
>
> homegfs [b1-gfsib01b] : Starting time of crawl   : Thu Jan 21 12:36:19
> 2016
> homegfs [b1-gfsib01b] : Ending time of crawl : Thu Jan 21 12:36:19
> 2016
> homegfs [b1-gfsib01b] : Type of crawl: INDEX
> homegfs [b1-gfsib01b] : No. of entries healed: 0
> homegfs [b1-gfsib01b] : No. of entries in split-brain: 0
> homegfs [b1-gfsib01b] : No. of heal failed entries   : 1
>
> homegfs [b2-gfsib01a] : Starting time of crawl   : Thu Jan 21 12:36:48
> 2016
> homegfs [b2-gfsib01a] : Ending time of crawl : Thu Jan 21 12:36:48
> 2016
> homegfs [b2-gfsib01a] : Type of crawl: INDEX
> homegfs [b2-gfsib01a] : No. of entries healed: 0
> homegfs [b2-gfsib01a] : No. of entries in split-brain: 0
> homegfs [b2-gfsib01a] : No. of heal failed entries   : 0
>
> homegfs [b3-gfsib01b] : Starting time of crawl   : Thu Jan 21 12:36:47
> 2016
> homegfs [b3-gfsib01b] : Ending time of crawl : Thu Jan 21 12:36:47
> 2016
> homegfs [b3-gfsib01b] : Type of crawl: INDEX
> homegfs [b3-gfsib01b] : No. of entries healed: 0
> homegfs [b3-gfsib01b] : No. of entries in split-brain: 0
> homegfs [b3-gfsib01b] : No. of heal failed entries   : 0
>
> homegfs [b4-gfsib02a] : Starting time of crawl   : Thu Jan 21 12:36:06
> 2016
> homegfs [b4-gfsib02a] : Ending time of crawl : Thu Jan 21 12:36:06
> 2016
> homegfs [b4-gfsib02a] : Type of crawl: INDEX
> homegfs [b4-gfsib02a] : No. of entries healed: 0
> homegfs [b4-gfsib02a] : No. of entries in split-brain: 0
> homegfs [b4-gfsib02a] : No. of heal failed entries   : 0
>
> homegfs [b5-gfsib02b] : Starting time of crawl   : Thu Jan 21 12:13:40
> 2016
> homegfs [b5-gfsib02b] :*** Crawl is in
> progress ***
> homegfs [b5-gfsib02b] : Type of crawl: INDEX
> homegfs [b5-gfsib02b] : No. of entries healed: 0
> homegfs [b5-gfsib02b] : No. of entries in split-brain: 0
> homegfs [b5-gfsib02b] : No. of heal failed entries   : 0
>
> homegfs [b6-gfsib02a] : Starting time of crawl   : Thu Jan 21 12:36:58
>