Hello all,
We have a distributed volume running in 7 hosts and 28 bricks.
We've been experiencing some strange behaviors over the time, some of them are
solved when the gluster services are restarted, but other problems persist.
In the case below, and this one happens many times, some people access the
gluster filesystem via Samba.
They have a mapped drive in their windows machine.
Most of the time things work as expected, but a recurrent issue is that
sometimes when directories are created, they get a "permission denied" message.
Apparently, the directories are created.
This sometimes happens also when they are renaming directories. And other times
they get a message telling them that the files/directories do not exist.
There are some logs below.
[2024-06-18 14:02:52.931157 +0000] W [MSGID: 109037]
[dht-common.c:4711:dht_getxattr_get_real_filename_cbk] 0-tank-dht: Failed to
get real filename. [No such file or directory]
[2024-06-18 14:02:52.934395 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2097:client4_0_create_cbk] 0-tank-client-5: remote
operation failed. [{path=/dir59/BallData.mat}, {errno=2}, {error=No such file
or directory}]
[2024-06-18 14:04:52.221379 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2097:client4_0_create_cbk] 0-tank-client-5: remote
operation failed. [{path=/dir59/BallData.mat}, {errno=2}, {error=No such file
or directory}]
Here we have multiple attempts to rename a file
[2024-06-18 14:05:54.992761 +0000] I [MSGID: 109064]
[dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat
(ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4)
=> /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>)
[2024-06-18 14:05:55.527004 +0000] I [MSGID: 109064]
[dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat
(ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4)
=> /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>)
The message "I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht:
renaming /dir59/BallData1.mat (ae1ddf78-5e3d-470c-8a4c-cedca17829b8)
(hash=tank-client-4/cache=tank-client-4) => /dir59/BallData.mat ((null))
(hash=tank-client-5/cache=<nul>) " repeated 2 times between [2024-06-18
14:05:55.527004 +0000] and [2024-06-18 14:06:11.390068 +0000]
[2024-06-18 14:06:13.417710 +0000] I [MSGID: 109064]
[dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat
(ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4)
=> /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>)
There are many messages in the logs like the ones below
[2024-06-16 12:28:07.720713 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib64/glusterfs/9.4/xlator/performance/md-cache.so(+0x11683)
[0x7f998d68f683] -->/lib64/libglusterfs.so.0(dict_get_iatt+0x40)
[0x7f999507bd10] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x85)
[0x7f99950799b5] ) 0-dict: dict OR key (virt-gf-prestat) is NULL [Invalid
argument]
[2024-06-16 12:28:29.099386 +0000] W [MSGID: 114061]
[client-common.c:3327:client_pre_readdirp_v2] 0-tank-client-0: remote_fd is -1.
EBADFD [{gfid=8def2ad4-72fd-49ed-af12-4b1f9878b2a4}, {errno=77}, {error=File
descriptor in bad state}]
[2024-06-16 12:37:56.662296 +0000] I [MSGID: 109061]
[dht-layout.c:641:dht_layout_normalize] 0-tank-dht: Found anomalies
[{path=/dir59}, {gfid=3e58f9ed-8565-4ce2-bbbb-77b18b841929}, {holes=3},
{overlaps=0}]
[2024-06-16 12:37:56.663670 +0000] W [MSGID: 109005]
[dht-selfheal.c:1064:dht_selfheal_dir_mkdir_cbk] 0-tank-dht: Healing of path
failed [{path=/dir59}, {gfid=3e58f9ed-8565-4ce2-bbbb-77b18b841929}, {errno=2},
{error=No such file or directory}]
[2024-06-16 12:37:56.672373 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib64/glusterfs/9.4/xlator/cluster/distribute.so(+0x516c6)
[0x7f998e3426c6] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3b)
[0x7f999507a5bb] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x85)
[0x7f99950799b5] ) 0-dict: dict OR key (trusted.glusterfs.dht.commithash) is
NULL [Invalid argument]
Did any of you been through this before? The "permission denied" happens
randomly.
About the renaming of files/directories, sometimes I think that the renames are
being done, but the windows client obtains an error and is not informed of the
success of the operation, and then, when they try again to rename, the original
filename does not exist anymore, and they get "File not Found". But I'm not
sure of this.
Thanks.
Carlo Rodrigues
Cloud & Infra Engineer
[email protected]
T +351 21 319 92 00
[Logo Description automatically generated]
Claranet Portugal
www.claranet.pt<http://www.claranet.pt/>
Todas as informações contidas nesta mensagem eletrónica da Claranet estão
abrangidas pelo disclaimer disponível em
https://www.claranet.pt/email-disclaimer
All the information contained within this electronic message from Claranet is
covered by the disclaimer at https://www.claranet.pt/email-disclaimer-ENG
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
[email protected]
https://lists.gluster.org/mailman/listinfo/gluster-users
