Hello Rafi, a few days ago I had the scenario described below. Here are the log entries. As you can see glusterd was stopped at 15:22:48. The same time FUSE of shared_storage was stopped. It seems to be nearly the same error message. You can also see that I restarted glusterd manually but FUSE of shared_storage was not restarted
*#-> /var/log/glusterfs/glusterd.log[2018-11-21 15:22:37.406032] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:21:45.377310] and [2018-11-21 15:22:39.382300][2018-11-21 15:22:48.623951] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7559) [0x7fb1e5f11559] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xfd) [0x561c3626340d] -->/usr/sbin/glusterd(cleanup_and_exit+0x5a) [0x561c362631fa] ) 0-: received signum (15), shutting down[2018-11-21 15:22:53.491120] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 4.1.5 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)[2018-11-21 15:22:53.494326] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536[2018-11-21 15:22:53.494372] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory[2018-11-21 15:22:53.494393] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory[2018-11-21 15:22:53.496564] I [socket.c:4477:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED[2018-11-21 15:22:53.496598] I [socket.c:4480:socket_init] 0-socket.management: SSL support for glusterd is ENABLED[2018-11-21 15:22:53.496607] I [socket.c:4497:socket_init] 0-socket.management: using private polling thread[2018-11-21 15:22:53.496615] I [socket.c:4507:socket_init] 0-socket.management: using certificate depth 1[2018-11-21 15:22:53.496891] E [socket.c:4554:socket_init] 0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:53.499933] W [MSGID: 103071] [rdma.c:4629:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device][2018-11-21 15:22:53.499970] W [MSGID: 103055] [rdma.c:4938:init] 0-rdma.management: Failed to initialize IB Device[2018-11-21 15:22:53.499983] W [rpc-transport.c:351:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed[2018-11-21 15:22:53.500148] W [rpcsvc.c:1781:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed[2018-11-21 15:22:53.500174] E [MSGID: 106244] [glusterd.c:1764:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport[2018-11-21 15:22:53.500361] I [socket.c:4480:socket_init] 0-socket.management: SSL support for glusterd is ENABLED[2018-11-21 15:22:53.500373] I [socket.c:4507:socket_init] 0-socket.management: using certificate depth 1[2018-11-21 15:22:53.500615] E [socket.c:4554:socket_init] 0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:55.353929] I [MSGID: 106513] [glusterd-store.c:2240:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 40100[2018-11-21 15:22:55.423433] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 7e7fd30a-94c9-4c21-9c4b-3817508ef121[2018-11-21 15:22:55.429214] I [MSGID: 106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0[2018-11-21 15:22:55.429317] I [MSGID: 106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0[2018-11-21 15:22:55.429379] W [MSGID: 106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout[2018-11-21 15:22:55.429443] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600[2018-11-21 15:22:55.429623] I [socket.c:4477:socket_init] 0-management: SSL support on the I/O path is ENABLED[2018-11-21 15:22:55.429635] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED[2018-11-21 15:22:55.429643] I [socket.c:4497:socket_init] 0-management: using private polling thread[2018-11-21 15:22:55.429650] I [socket.c:4507:socket_init] 0-management: using certificate depth 1[2018-11-21 15:22:55.429914] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:55.431676] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600[2018-11-21 15:22:55.431846] I [socket.c:4477:socket_init] 0-management: SSL support on the I/O path is ENABLED[2018-11-21 15:22:55.431858] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED[2018-11-21 15:22:55.431866] I [socket.c:4497:socket_init] 0-management: using private polling thread[2018-11-21 15:22:55.431873] I [socket.c:4507:socket_init] 0-management: using certificate depth 1[2018-11-21 15:22:55.432136] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabledFinal graph:+------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option transport.socket.ssl-enabled on 4: option rpc-auth.auth-glusterfs on 5: option rpc-auth.auth-unix on 6: option rpc-auth.auth-null on 7: option rpc-auth-allow-insecure on 8: option transport.listen-backlog 10 9: option event-threads 1 10: option ping-timeout 0 11: option transport.socket.read-fail-log off 12: option transport.socket.keepalive-interval 2 13: option transport.socket.keepalive-time 10 14: option transport-type rdma 15: option working-directory /var/lib/glusterd 16: end-volume 17: +------------------------------------------------------------------------------+[2018-11-21 15:22:55.431634] W [MSGID: 106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout[2018-11-21 15:22:55.433711] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1[2018-11-21 15:22:58.975934] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick2/glusterbrick on port 49153[2018-11-21 15:22:58.979840] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick1/glusterbrick on port 49152[2018-11-21 15:22:59.462323] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b, host: fs-apollo-c1-n1, port: 0[2018-11-21 15:22:59.463676] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:22:59.463728] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2018-11-21 15:22:59.464948] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600[2018-11-21 15:22:59.465116] I [socket.c:4480:socket_init] 0-nfs: SSL support for glusterd is ENABLED[2018-11-21 15:22:59.465130] I [socket.c:4507:socket_init] 0-nfs: using certificate depth 1[2018-11-21 15:22:59.465387] E [socket.c:4554:socket_init] 0-nfs: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:59.466154] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped[2018-11-21 15:22:59.466195] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is stopped[2018-11-21 15:22:59.467179] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600[2018-11-21 15:22:59.467309] I [socket.c:4480:socket_init] 0-glustershd: SSL support for glusterd is ENABLED[2018-11-21 15:22:59.467321] I [socket.c:4507:socket_init] 0-glustershd: using certificate depth 1[2018-11-21 15:22:59.467553] E [socket.c:4554:socket_init] 0-glustershd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:59.473619] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 4143[2018-11-21 15:22:59.478144] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 127.0.0.1:49149 <http://127.0.0.1:49149> failed (Success)[2018-11-21 15:23:00.473882] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd service is stopped[2018-11-21 15:23:00.474011] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting glustershd service[2018-11-21 15:23:01.478838] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600[2018-11-21 15:23:01.479039] I [socket.c:4480:socket_init] 0-quotad: SSL support for glusterd is ENABLED[2018-11-21 15:23:01.479054] I [socket.c:4507:socket_init] 0-quotad: using certificate depth 1[2018-11-21 15:23:01.479336] E [socket.c:4554:socket_init] 0-quotad: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:01.480664] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped[2018-11-21 15:23:01.480711] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: quotad service is stopped[2018-11-21 15:23:01.480777] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600[2018-11-21 15:23:01.480902] I [socket.c:4480:socket_init] 0-bitd: SSL support for glusterd is ENABLED[2018-11-21 15:23:01.480912] I [socket.c:4507:socket_init] 0-bitd: using certificate depth 1[2018-11-21 15:23:01.481145] E [socket.c:4554:socket_init] 0-bitd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:01.486112] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping bitd daemon running in pid: 4343[2018-11-21 15:23:02.486437] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: bitd service is stopped[2018-11-21 15:23:02.486531] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting bitd service[2018-11-21 15:23:03.492955] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600[2018-11-21 15:23:03.493131] I [socket.c:4480:socket_init] 0-scrub: SSL support for glusterd is ENABLED[2018-11-21 15:23:03.493147] I [socket.c:4507:socket_init] 0-scrub: using certificate depth 1[2018-11-21 15:23:03.493416] E [socket.c:4554:socket_init] 0-scrub: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:03.498042] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping scrub daemon running in pid: 4354[2018-11-21 15:23:04.498341] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: scrub service is stopped[2018-11-21 15:23:04.498426] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting scrub service[2018-11-21 15:23:05.503972] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /gluster/brick1/glusterbrick[2018-11-21 15:23:05.504033] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick1/glusterbrick on port 49152[2018-11-21 15:23:05.504103] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600[2018-11-21 15:23:05.504229] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED[2018-11-21 15:23:05.504241] I [socket.c:4507:socket_init] 0-management: using certificate depth 1[2018-11-21 15:23:05.504505] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.510177] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /gluster/brick2/glusterbrick[2018-11-21 15:23:05.510210] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick2/glusterbrick on port 49153[2018-11-21 15:23:05.510259] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600[2018-11-21 15:23:05.510368] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED[2018-11-21 15:23:05.510380] I [socket.c:4507:socket_init] 0-management: using certificate depth 1[2018-11-21 15:23:05.510635] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.516357] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /var/lib/glusterd/ss_brick[2018-11-21 15:23:05.516387] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49154[2018-11-21 15:23:05.516433] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600[2018-11-21 15:23:05.516538] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED[2018-11-21 15:23:05.516550] I [socket.c:4507:socket_init] 0-management: using certificate depth 1[2018-11-21 15:23:05.516795] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.522223] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600[2018-11-21 15:23:05.522301] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED[2018-11-21 15:23:05.522308] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1[2018-11-21 15:23:05.522709] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.523245] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600[2018-11-21 15:23:05.523364] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED[2018-11-21 15:23:05.523379] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1[2018-11-21 15:23:05.523544] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.523993] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600[2018-11-21 15:23:05.524564] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED[2018-11-21 15:23:05.524585] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1[2018-11-21 15:23:05.524804] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.525379] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600[2018-11-21 15:23:05.525451] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED[2018-11-21 15:23:05.525458] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1[2018-11-21 15:23:05.525649] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.526277] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600[2018-11-21 15:23:05.526348] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED[2018-11-21 15:23:05.526355] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1[2018-11-21 15:23:05.526507] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.527193] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600[2018-11-21 15:23:05.527265] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED[2018-11-21 15:23:05.527274] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1[2018-11-21 15:23:05.527437] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.529501] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7, host: fs-apollo-c1-n3, port: 0[2018-11-21 15:23:05.530466] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:05.531325] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2018-11-21 15:23:05.531934] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:23:05.532036] I [MSGID: 106163] [glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 40100[2018-11-21 15:23:05.532812] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49154[2018-11-21 15:23:05.533478] I [MSGID: 106163] [glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 40100[2018-11-21 15:23:05.534519] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:05.535549] I [MSGID: 106490] [glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:23:05.538671] I [MSGID: 106493] [glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to fs-apollo-c1-n1 (0), ret: 0, op_ret: 0[2018-11-21 15:23:05.541086] I [MSGID: 106490] [glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:05.543763] I [MSGID: 106493] [glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to fs-apollo-c1-n3 (0), ret: 0, op_ret: 0[2018-11-21 15:23:05.546026] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:23:05.547097] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:23:05.547154] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2018-11-21 15:23:05.547765] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:05.548774] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2018-11-21 15:23:05.548958] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:31.411509] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:23:33.372957] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 3 times between [2018-11-21 15:23:33.372957] and [2018-11-21 15:23:33.376516][2018-11-21 15:24:25.407528] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:24:27.370775] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 3 times between [2018-11-21 15:24:27.370775] and [2018-11-21 15:24:27.373796][2018-11-21 15:25:19.407885] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:25:21.371482] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2018-11-21 15:26:13.415390] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:25:21.371482] and [2018-11-21 15:26:15.378777][2018-11-21 15:27:07.403685] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:27:09.371804] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2018-11-21 15:28:02.728185] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output error)[2018-11-21 15:28:01.415055] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:27:09.371804] and [2018-11-21 15:28:03.370301][2018-11-21 15:28:55.414118] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:28:57.375508] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol reqThe message "I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2018-11-21 15:28:55.414118] and [2018-11-21 15:30:43.419479]The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:28:57.375508] and [2018-11-21 15:30:45.383263][2018-11-21 15:31:37.415788] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:31:39.375304] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2018-11-21 15:32:31.415510] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:31:39.375304] and [2018-11-21 15:32:33.375024][2018-11-21 15:33:25.407192] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:33:27.379564] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2018-11-21 15:34:20.746832] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output error)[2018-11-21 15:34:19.411130] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:33:27.379564] and [2018-11-21 15:34:21.368809][2018-11-21 15:35:13.415187] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:35:15.371647] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2018-11-21 15:36:00.958833] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Locking failed on fs-apollo-c1-n3. Please check log file for details.[2018-11-21 15:36:00.960197] W [glusterd-locks.c:622:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43) [0x7f206876cc43] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x30c73) [0x7f206875dc73] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdb5d3) [0x7f20688085d3] ) 0-management: Lock for archive1 held by 7e7fd30a-94c9-4c21-9c4b-3817508ef121[2018-11-21 15:36:00.960234] E [MSGID: 106118] [glusterd-op-sm.c:4173:glusterd_op_ac_lock] 0-management: Unable to acquire lock for archive1[2018-11-21 15:36:00.960311] E [MSGID: 106376] [glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler returned: -1[2018-11-21 15:36:00.961494] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Locking failed on fs-apollo-c1-n1. Please check log file for details.[2018-11-21 15:36:00.961603] E [MSGID: 106150] [glusterd-syncop.c:1957:gd_sync_task_begin] 0-management: Locking Peers Failed.[2018-11-21 15:36:00.962117] W [glusterd-locks.c:856:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43) [0x7f206876cc43] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3099f) [0x7f206875d99f] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdba74) [0x7f2068808a74] ) 0-management: Lock owner mismatch. Lock for vol archive1 held by 7e7fd30a-94c9-4c21-9c4b-3817508ef121[2018-11-21 15:36:00.962144] E [MSGID: 106117] [glusterd-op-sm.c:4236:glusterd_op_ac_unlock] 0-management: Unable to release lock for archive1[2018-11-21 15:36:00.962222] E [MSGID: 106376] [glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler returned: 1[2018-11-21 15:36:00.962406] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on fs-apollo-c1-n1. Please check log file for details.[2018-11-21 15:36:00.962973] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on fs-apollo-c1-n3. Please check log file for details.[2018-11-21 15:36:00.963033] E [MSGID: 106151] [glusterd-syncop.c:1640:gd_unlock_op_phase] 0-management: Failed to unlock on some peer(s)[2018-11-21 15:36:00.991734] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting to connect to host:(null), port:0[2018-11-21 15:36:01.186043] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting to connect to host:(null), port:0[2018-11-21 15:36:07.416984] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:35:15.371647] and [2018-11-21 15:36:09.383236][2018-11-21 15:37:01.410979] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:37:03.383408] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol reqThe message "I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2018-11-21 15:37:01.410979] and [2018-11-21 15:38:49.410710]The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:37:03.383408] and [2018-11-21 15:38:51.378524][2018-11-21 15:39:43.421483] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:39:44.786383] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output error)[2018-11-21 15:39:45.373093] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2018-11-21 15:40:37.412592] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:39:45.373093] and [2018-11-21 15:40:39.378904][2018-11-21 15:41:31.415866] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:41:33.374993] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2018-11-21 15:42:25.415585] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:41:33.374993] and [2018-11-21 15:42:27.379013][2018-11-21 15:43:19.411237] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:43:21.376718] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2018-11-21 15:44:13.425206] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:43:21.376718] and [2018-11-21 15:44:15.375796][2018-11-21 15:45:07.424344] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:45:09.382299] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2018-11-21 15:46:02.769673] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output error)[2018-11-21 15:46:01.422742] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list reqThe message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:45:09.382299] and [2018-11-21 15:46:03.370463][2018-11-21 15:46:55.426992] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2018-11-21 15:46:57.379488] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req#-> /var/log/glusterfs/run-gluster-shared_storage.log[2018-11-21 15:22:48.623969] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7559) [0x7f9e25569559] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x561fd19c840d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x5a) [0x561fd19c81fa] ) 0-: received signum (15), shutting down* *[2018-11-21 15:22:48.624042] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/run/gluster/shared_storage'.[2018-11-21 15:22:48.624185] E [fuse-bridge.c:212:check_and_dump_fuse_W] 0-glusterfs-fuse: writing to fuse device failed: Bad file descriptor[2018-11-21 15:22:48.624246] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /run/gluster/shared_storage[2018-11-21 15:22:48.624679] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/run/gluster/shared_storage'.#-> /var/log/glusterfs/bricks/var-lib-glusterd-ss_brick.log[2018-11-21 15:22:48.626860] E [socket.c:2792:socket_poller] 0-tcp.gluster_shared_storage-server: socket_poller 10.10.12.62:49151 <http://10.10.12.62:49151> failed (Success)[2018-11-21 15:22:48.627048] I [MSGID: 115036] [server.c:483:server_rpc_notify] 0-gluster_shared_storage-server: disconnecting connection from CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21 15:22:48.627214] I [MSGID: 101055] [client_t.c:444:gf_client_unref] 0-gluster_shared_storage-server: Shutting down connection CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21 15:22:48.632303] W [socket.c:599:__socket_rwv] 0-glusterfs: readv on 10.10.12.62:24007 <http://10.10.12.62:24007> failed (Success)[2018-11-21 15:22:48.632531] I [glusterfsd-mgmt.c:2348:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: fs-apollo-c1-n2[2018-11-21 15:22:48.632558] I [glusterfsd-mgmt.c:2369:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers[2018-11-21 15:22:59.478134] I [MSGID: 115036] [server.c:483:server_rpc_notify] 0-gluster_shared_storage-server: disconnecting connection from CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21 15:22:59.478287] I [MSGID: 101055] [client_t.c:444:gf_client_unref] 0-gluster_shared_storage-server: Shutting down connection CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21 15:23:05.529596] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing[2018-11-21 15:23:05.625165] I [addr.c:55:compare_addr_and_update] 0-/var/lib/glusterd/ss_brick: allowed = "*", received addr = "10.10.12.62"[2018-11-21 15:23:05.625203] I [login.c:45:gf_auth] 0-auth/login: connecting user name: fs-apollo-c1-n2[2018-11-21 15:23:05.625238] I [login.c:111:gf_auth] 0-auth/login: allowed user names: *[2018-11-21 15:23:05.625254] I [MSGID: 115029] [server-handshake.c:763:server_setvolume] 0-gluster_shared_storage-server: accepted client from CTX_ID:1c203d59-02da-46ec-b836-371ebb47e3dc-GRAPH_ID:0-PID:20295-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0 (version: 4.1.5)[2018-11-21 15:36:01.009586] I [glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed[2018-11-21 15:36:01.017077] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing[2018-11-21 15:36:01.206445] I [glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed[2018-11-21 15:36:01.212266] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing* Am Fr., 9. Nov. 2018 um 11:53 Uhr schrieb David Spisla <[email protected]>: > Sure, I will do that the next time shared_storage is unmount after > stopping glusterd > > Am Fr., 9. Nov. 2018 um 10:23 Uhr schrieb Rafi Kavungal Chundattu Parambil > <[email protected]>: > >> Hi David Spisla, >> >> gluster_shared_storage works just like yet another volume(There are some >> special handling related to authentication). So bringing down a glusterd >> node shouldn't have lead to unmount of the volume. >> >> Can you please share the mount log, brick logs at time when you stopped >> glusterd. Also make sure your shared storage bricks up and running. >> >> Regards >> Rafi KC >> >> ----- Original Message ----- >> From: "David Spisla" <[email protected]> >> To: [email protected] >> Sent: Tuesday, November 6, 2018 3:25:18 PM >> Subject: [Gluster-users] Shared Storage is unmount after stopping glusterd >> >> Dear Gluster Community, >> I have a v4.1.5 4-node-cluster with SLES15 machines. The last time I >> observe that after stopping glusterd with >> $ sudo systemctl stop glusterd >> >> sometimes also the FUSE mount of the shared_storage on the same node is >> unmount but shared_storage ist still in the volume list. I mean really >> "sometimes" because it is not reproducible in a deterministic way and I am >> not sure if there is really a dependency between stopping glusterd and >> shared_storage. >> >> What is the expected behaviour if stopping glusterd? In my opinion all >> volumes included shared_storage should still have FUSE mounts. >> >> Regards >> David Spisla >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> [email protected] >> https://lists.gluster.org/mailman/listinfo/gluster-users >> >
_______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
