A while back we found an issue with PHP around file locks.
https://bugs.php.net/bug.php?id=53076
This would leave hanging processes that we could only kill by forcibly
remounting the filesystems.
Its not really a PHP bug but they are not willing to fix the code to
deal with Glusters idiosyncrasies.
On 08/09/2018 09:10 AM, Nithya Balachandran wrote:
Hi,
Please provide the following:
1. gluster volume info
2. statedump of the fuse process when it hangs
Thanks,
Nithya
On 9 August 2018 at 18:24, mabi <[email protected]
<mailto:[email protected]>> wrote:
Hello,
I recently upgraded my GlusterFS replica 2+1 (aribter) to version
3.12.12 and now I see a weird behaviour on my client (using FUSE
mount) where I have processes (PHP 5.6 FPM) trying to access a
specific directory and then the process blocks. I can't kill the
process either, not even with kill -9. I need to reboot the
machine in order to get rid of these blocked processes.
This directory has one particularity compared to the other
directories it is that it has reached it's quota soft-limit as you
can see here in the output of gluster volume quota list:
Path Hard-limit Soft-limit
Used Available Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/directory 100.0GB 80%(80.0GB)
90.5GB 9.5GB Yes No
That does not mean that it is the quota's fault but it might be a
hint where to start looking for... And by the way can someone
explain me what the soft-limit does? or does it not do anything
special?
Here is an the linux stack of a blocking process on that directory
which happened with a simple "ls -la":
[Thu Aug 9 14:21:07 2018] INFO: task ls:2272 blocked for more
than 120 seconds.
[Thu Aug 9 14:21:07 2018] Not tainted 3.16.0-4-amd64 #1
[Thu Aug 9 14:21:07 2018] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Aug 9 14:21:07 2018] ls D ffff88017ef93200
0 2272 2268 0x00000004
[Thu Aug 9 14:21:07 2018] ffff88017653f490 0000000000000286
0000000000013200 ffff880174d7bfd8
[Thu Aug 9 14:21:07 2018] 0000000000013200 ffff88017653f490
ffff8800eeb3d5f0 ffff8800fefac800
[Thu Aug 9 14:21:07 2018] ffff880174d7bbe0 ffff8800eeb3d6d0
ffff8800fefac800 ffff8800ffe1e1c0
[Thu Aug 9 14:21:07 2018] Call Trace:
[Thu Aug 9 14:21:07 2018] [<ffffffffa00dc50d>] ?
__fuse_request_send+0xbd/0x270 [fuse]
[Thu Aug 9 14:21:07 2018] [<ffffffff810abce0>] ?
prepare_to_wait_event+0xf0/0xf0
[Thu Aug 9 14:21:07 2018] [<ffffffffa00e0791>] ?
fuse_dentry_revalidate+0x181/0x300 [fuse]
[Thu Aug 9 14:21:07 2018] [<ffffffff811b944e>] ?
lookup_fast+0x25e/0x2b0
[Thu Aug 9 14:21:07 2018] [<ffffffff811bacc5>] ?
path_lookupat+0x155/0x780
[Thu Aug 9 14:21:07 2018] [<ffffffff81195715>] ?
kmem_cache_alloc+0x75/0x480
[Thu Aug 9 14:21:07 2018] [<ffffffffa00dfca9>] ?
fuse_getxattr+0xe9/0x150 [fuse]
[Thu Aug 9 14:21:07 2018] [<ffffffff811bb316>] ?
filename_lookup+0x26/0xc0
[Thu Aug 9 14:21:07 2018] [<ffffffff811bf594>] ?
user_path_at_empty+0x54/0x90
[Thu Aug 9 14:21:07 2018] [<ffffffff81193e08>] ?
kmem_cache_free+0xd8/0x210
[Thu Aug 9 14:21:07 2018] [<ffffffff811bf59f>] ?
user_path_at_empty+0x5f/0x90
[Thu Aug 9 14:21:07 2018] [<ffffffff811b3d46>] ?
vfs_fstatat+0x46/0x90
[Thu Aug 9 14:21:07 2018] [<ffffffff811b421d>] ?
SYSC_newlstat+0x1d/0x40
[Thu Aug 9 14:21:07 2018] [<ffffffff811d34b8>] ?
SyS_lgetxattr+0x58/0x80
[Thu Aug 9 14:21:07 2018] [<ffffffff81525d0d>] ?
system_call_fast_compare_end+0x10/0x15
My 3 gluster nodes are all Debian 9 and my client Debian 8.
Let me know if you need more information.
Best regards,
Mabi
_______________________________________________
Gluster-users mailing list
[email protected] <mailto:[email protected]>
https://lists.gluster.org/mailman/listinfo/gluster-users
<https://lists.gluster.org/mailman/listinfo/gluster-users>
_______________________________________________
Gluster-users mailing list
[email protected]
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Alvin Starr || land: (905)513-7688
Netvel Inc. || Cell: (416)806-0133
[email protected] ||
_______________________________________________
Gluster-users mailing list
[email protected]
https://lists.gluster.org/mailman/listinfo/gluster-users
