All,
I have a gluster filesystem (glusterfs-4.0.2-1, Type:
Distributed-Replicate, Number of Bricks: 5 x 3 = 15)
I have one directory that is used for slurm statefiles, which seems to
get out of sync fairly often. There are particular files that end up
never healing.
Since the files are ephemeral, I'm ok with losing them (for now).
Following some advice, I deleted UUID files that were in
/GLUSTER/brick1/.glusterfs/indices/xattrop/
This makes gluster volume heal GDATA statistics heal-count show no
issues, however the issue is still there. Even though nothing is showing
up with gluster volume heal GDATA info, there are some files/directories
that, if I try to access them at all, I get "Transport endpoint is not
connected"
There is even a directory, which is empty but if I try to 'rmdir' it, I
get "rmdir: failed to remove ‘/DATA/slurmstate.old/slurm/’: Software
caused connection abort" and the mount goes bad. I have to umount/mount
it to get it back.
There is a bit of info in the log file that has to do with the crash
which is attached.
How do I clean this up? And what is the 'proper' way to handle when you
have a file that will not heal even in a 3-way replicate?
Brian Andrus
[2018-06-27 14:16:00.075738] I [MSGID: 114046]
[client-handshake.c:1176:client_setvolume_cbk] 0-GDATA-client-12: Connected to
GDATA-client-12, attached to remote volume '/GLUSTER/brick1'.
[2018-06-27 14:16:00.075755] I [MSGID: 108005]
[afr-common.c:5081:__afr_handle_child_up_event] 0-GDATA-replicate-4: Subvolume
'GDATA-client-12' came back up; going online.
[2018-06-27 14:16:00.076274] W [rpc-clnt.c:1739:rpc_clnt_submit]
0-GDATA-client-14: error returned while attempting to connect to host:(null),
port:0
[2018-06-27 14:16:00.076468] W [rpc-clnt.c:1739:rpc_clnt_submit]
0-GDATA-client-14: error returned while attempting to connect to host:(null),
port:0
[2018-06-27 14:16:00.076582] I [rpc-clnt.c:2071:rpc_clnt_reconfig]
0-GDATA-client-14: changing port to 49152 (from 0)
[2018-06-27 14:16:00.076772] W [rpc-clnt.c:1739:rpc_clnt_submit]
0-GDATA-client-13: error returned while attempting to connect to host:(null),
port:0
[2018-06-27 14:16:00.076922] W [rpc-clnt.c:1739:rpc_clnt_submit]
0-GDATA-client-13: error returned while attempting to connect to host:(null),
port:0
[2018-06-27 14:16:00.077407] I [MSGID: 114046]
[client-handshake.c:1176:client_setvolume_cbk] 0-GDATA-client-13: Connected to
GDATA-client-13, attached to remote volume '/GLUSTER/brick1'.
[2018-06-27 14:16:00.077422] I [MSGID: 108002] [afr-common.c:5378:afr_notify]
0-GDATA-replicate-4: Client-quorum is met
[2018-06-27 14:16:00.079479] W [rpc-clnt.c:1739:rpc_clnt_submit]
0-GDATA-client-14: error returned while attempting to connect to host:(null),
port:0
[2018-06-27 14:16:00.079723] W [rpc-clnt.c:1739:rpc_clnt_submit]
0-GDATA-client-14: error returned while attempting to connect to host:(null),
port:0
[2018-06-27 14:16:00.080249] I [MSGID: 114046]
[client-handshake.c:1176:client_setvolume_cbk] 0-GDATA-client-14: Connected to
GDATA-client-14, attached to remote volume '/GLUSTER/brick1'.
[2018-06-27 14:16:00.081176] I [fuse-bridge.c:4234:fuse_init] 0-glusterfs-fuse:
FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2018-06-27 14:16:00.081196] I [fuse-bridge.c:4864:fuse_graph_sync] 0-fuse:
switched to graph 0
[2018-06-27 14:16:00.088870] I [MSGID: 109005]
[dht-selfheal.c:2328:dht_selfheal_directory] 0-GDATA-dht: Directory selfheal
failed: Unable to form layout for directory /
[2018-06-27 14:16:03.675890] W [MSGID: 108027]
[afr-common.c:2255:afr_attempt_readsubvol_set] 0-GDATA-replicate-1: no read
subvols for /slurmstate.old/slurm
[2018-06-27 14:16:03.675921] I [MSGID: 109063]
[dht-layout.c:693:dht_layout_normalize] 0-GDATA-dht: Found anomalies in
/slurmstate.old/slurm (gfid = 00000000-0000-0000-0000-000000000000). Holes=1
overlaps=0
[2018-06-27 14:16:03.675936] W [MSGID: 109005]
[dht-selfheal.c:2303:dht_selfheal_directory] 0-GDATA-dht: Directory selfheal
failed: 1 subvolumes down.Not fixing. path = /slurmstate.old/slurm, gfid =
8ed6a9e9-2820-40bd-8d9d-77b7f79c774
8
[2018-06-27 14:16:03.679061] I [MSGID: 108026]
[afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-GDATA-replicate-2:
performing entry selfheal on 8ed6a9e9-2820-40bd-8d9d-77b7f79c7748
[2018-06-27 14:16:03.681899] W [MSGID: 108015]
[afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-GDATA-replicate-2:
expunging file 8ed6a9e9-2820-40bd-8d9d-77b7f79c7748/heartbeat
(00000000-0000-0000-0000-000000000000) on GDATA-cli
ent-6
[2018-06-27 14:16:03.683080] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: /slurmstate.old/slurm/qos_usage
(848b3d5e-3492-4343-a1b2-a86cc975b3c2) [No data available
]
[2018-06-27 14:16:03.683624] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data
available]
[2018-06-27 14:16:03.684098] I [MSGID: 108026]
[afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-GDATA-replicate-1:
performing entry selfheal on 8ed6a9e9-2820-40bd-8d9d-77b7f79c7748
[2018-06-27 14:16:03.684982] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: /slurmstate.old/slurm/job_state.old
(05eb1a4e-6b6e-4e6f-a13b-a016431f0e1f) [No data avail
able]
[2018-06-27 14:16:03.686319] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: /slurmstate.old/slurm/resv_state
(6658195b-9a22-4663-89ab-c7e3f4fc5258) [No data availabl
e]
[2018-06-27 14:16:03.690118] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data
available]
[2018-06-27 14:16:03.690283] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data
available]
[2018-06-27 14:16:03.690807] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: /slurmstate.old/slurm/node_state
(91e1ee64-154e-4aff-be2d-fcd2ac616dbe) [No data availabl
e]
[2018-06-27 14:16:03.692947] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: /slurmstate.old/slurm/node_state.old
(d1a4ce4f-fc99-418e-ac0e-78a191795275) [No data avai
lable]
[2018-06-27 14:16:03.693357] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data
available]
The message "W [MSGID: 114031] [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk]
0-GDATA-client-4: remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [No data available]" repeated 2 times
between [2018-06-27 1
4:16:03.693357] and [2018-06-27 14:16:03.694530]
[2018-06-27 14:16:03.694560] W [MSGID: 109048]
[dht-common.c:9736:dht_rmdir_cached_lookup_cbk] 0-GDATA-dht:
/slurmstate.old/slurm/qos_usage not found on cached subvol GDATA-replicate-2
[No data available]
[2018-06-27 14:16:03.695685] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data
available]
[2018-06-27 14:16:03.697547] W [MSGID: 108015]
[afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-GDATA-replicate-1:
expunging file 8ed6a9e9-2820-40bd-8d9d-77b7f79c7748/node_state.old
(00000000-0000-0000-0000-000000000000) on GDATA-client-4
[2018-06-27 14:16:03.697770] W [MSGID: 108015]
[afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-GDATA-replicate-1:
expunging file 8ed6a9e9-2820-40bd-8d9d-77b7f79c7748/node_state
(00000000-0000-0000-0000-000000000000) on GDATA-client-4
[2018-06-27 14:16:03.697954] W [MSGID: 108015]
[afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-GDATA-replicate-1:
expunging file 8ed6a9e9-2820-40bd-8d9d-77b7f79c7748/resv_state
(00000000-0000-0000-0000-000000000000) on GDATA-client-4
[2018-06-27 14:16:03.698160] W [MSGID: 108015]
[afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-GDATA-replicate-1:
expunging file 8ed6a9e9-2820-40bd-8d9d-77b7f79c7748/job_state.old
(00000000-0000-0000-0000-000000000000) on GDATA-client-4
[2018-06-27 14:16:03.698950] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data
available]
[2018-06-27 14:16:03.699007] W [MSGID: 108027]
[afr-common.c:2255:afr_attempt_readsubvol_set] 0-GDATA-replicate-1: no read
subvols for /slurmstate.old/slurm/node_state.old
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(1) op(RMDIR)
frame : type(1) op(RMDIR)
frame : type(1) op(RMDIR)
frame : type(1) op(RMDIR)
frame : type(1) op(RMDIR)
frame : type(1) op(RMDIR)
frame : type(1) op(RMDIR)
frame : type(1) op(RMDIR)
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2018-06-27 14:16:03
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 4.0.2
[2018-06-27 14:16:03.699346] W [MSGID: 114031]
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-GDATA-client-4: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data
available]
/lib64/libglusterfs.so.0(+0x244b0)[0x7f3db028c4b0]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f3db02963e4]
/lib64/libc.so.6(+0x362f0)[0x7f3dae8f22f0]
/usr/lib64/glusterfs/4.0.2/xlator/cluster/distribute.so(+0x5d840)[0x7f3da2214840]
/usr/lib64/glusterfs/4.0.2/xlator/cluster/replicate.so(+0x69c90)[0x7f3da24cfc90]
/usr/lib64/glusterfs/4.0.2/xlator/cluster/replicate.so(+0x6aaaa)[0x7f3da24d0aaa]
/usr/lib64/glusterfs/4.0.2/xlator/cluster/replicate.so(+0x6ae43)[0x7f3da24d0e43]
/lib64/libglusterfs.so.0(+0x60840)[0x7f3db02c8840]
/lib64/libc.so.6(+0x48030)[0x7f3dae904030]
---------
[2018-06-27 14:16:03.699540] W [MSGID: 108027]
[afr-common.c:2255:afr_attempt_readsubvol_set] 0-GDATA-replicate-1: no read
subvols for /slurmstate.old/slurm/node_state
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
