Of course. Please find attached. Hope they can shed some light on this.
Thanks, Seva 28.04.2017, 12:41, "Mohammed Rafi K C" <[email protected]>: > Can you share the glusterd logs from the three nodes ? > > Rafi KC > > On 04/28/2017 02:34 PM, Seva Gluschenko wrote: >> Dear Community, >> >> I call for your wisdom, as it appears that googling for keywords doesn't >> help much. >> >> I have a glusterfs volume with replica count 2, and I tried to perform the >> online upgrade procedure described in the docs >> (http://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.10/). It >> all went almost fine when I'd done with the first replica, the only problem >> was the self-heal procedure that refused to complete until I commented out >> all IPv6 entries in the /etc/hosts. >> >> So far, being sure that it all should work on the 2nd replica pretty the >> same as it was on the 1st one, I had proceeded with the upgrade on the >> replica 2. All of a sudden, it told me that it doesn't see the first replica >> at all. The state before upgrade was: >> >> sst2# gluster volume status >> Status of volume: gv0 >> Gluster process TCP Port RDMA Port Online Pid >> >> ------------------------------------------------------------------------------ >> Brick sst0:/var/glusterfs 49152 0 Y 3482 >> Brick sst2:/var/glusterfs 49152 0 Y 29863 >> NFS Server on localhost 2049 0 Y 25175 >> Self-heal Daemon on localhost N/A N/A Y 25283 >> NFS Server on sst0 N/A N/A N N/A >> Self-heal Daemon on sst0 N/A N/A Y 4827 >> NFS Server on sst1 N/A N/A N N/A >> Self-heal Daemon on sst1 N/A N/A Y 15009 >> >> Task Status of Volume gv0 >> >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> sst2# gluster peer status >> Number of Peers: 2 >> >> Hostname: sst0 >> Uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc >> State: Peer in Cluster (Connected) >> >> Hostname: sst1 >> Uuid: 5a2198de-f536-4328-a278-7f746f276e35 >> State: Sent and Received peer request (Connected) >> >> sst2# gluster volume heal gv0 info >> Brick sst0:/var/glusterfs >> Number of entries: 0 >> >> Brick sst2:/var/glusterfs >> Number of entries: 0 >> >> After upgrade, it looked like this: >> >> sst2# gluster volume status >> Status of volume: gv0 >> Gluster process TCP Port RDMA Port Online Pid >> >> ------------------------------------------------------------------------------ >> Brick sst2:/var/glusterfs N/A N/A N N/A >> NFS Server on localhost N/A N/A N N/A >> NFS Server on localhost N/A N/A N N/A >> >> Task Status of Volume gv0 >> >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> sst2# gluster peer status >> Number of Peers: 2 >> >> Hostname: sst1 >> Uuid: 5a2198de-f536-4328-a278-7f746f276e35 >> State: Sent and Received peer request (Connected) >> >> Hostname: sst0 >> Uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc >> State: Peer Rejected (Connected) >> >> My biggest fault probably, at that point I googled and found this article >> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Resolving%20Peer%20Rejected/ >> -- and followed its advice, removing at sst2 all the /var/lib/glusterd >> contents except the glusterd.info file. As the result, the node, >> predictably, lost all information about the volume. >> >> sst2# gluster volume status >> No volumes present >> >> sst2# gluster peer status >> Number of Peers: 2 >> >> Hostname: sst0 >> Uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc >> State: Accepted peer request (Connected) >> >> Hostname: sst1 >> Uuid: 5a2198de-f536-4328-a278-7f746f276e35 >> State: Accepted peer request (Connected) >> >> Okay, I thought, this is might be a high time to re-add the brick. Not that >> easy, Jack: >> >> sst0# gluster volume add-brick gv0 replica 2 'sst2:/var/glusterfs' >> volume add-brick: failed: Operation failed >> >> The reason appeared to be natural: sst0 still knows that there was the >> replica on sst2. What should I do then? At this point, I tried to recover >> the volume information on sst2 by putting it offline and copying all the >> volume info from the sst0. Of course it wasn't enough to just copy as is, I >> modified /var/lib/glusterd/vols/gv0/sst*\:-var-glusterfs, setting >> listen-port=0 for the remote brick (sst0) and listen-port=49152 for the >> local brick (sst2). It didn't help much, unfortunately. The final state I've >> reached is as follows: >> >> sst2# gluster peer status >> Number of Peers: 2 >> >> Hostname: sst1 >> Uuid: 5a2198de-f536-4328-a278-7f746f276e35 >> State: Sent and Received peer request (Connected) >> >> Hostname: sst0 >> Uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc >> State: Sent and Received peer request (Connected) >> >> sst2# gluster volume info >> >> Volume Name: gv0 >> Type: Replicate >> Volume ID: dd4996c0-04e6-4f9b-a04e-73279c4f112b >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: sst0:/var/glusterfs >> Brick2: sst2:/var/glusterfs >> Options Reconfigured: >> cluster.self-heal-daemon: enable >> performance.readdir-ahead: on >> storage.owner-uid: 1000 >> storage.owner-gid: 1000 >> >> sst2# gluster volume status >> Status of volume: gv0 >> Gluster process TCP Port RDMA Port Online Pid >> >> ------------------------------------------------------------------------------ >> Brick sst2:/var/glusterfs N/A N/A N N/A >> NFS Server on localhost N/A N/A N N/A >> NFS Server on localhost N/A N/A N N/A >> >> Task Status of Volume gv0 >> >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> Meanwhile, on sst0: >> >> sst0# gluster volume info >> >> Volume Name: gv0 >> Type: Replicate >> Volume ID: dd4996c0-04e6-4f9b-a04e-73279c4f112b >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: sst0:/var/glusterfs >> Brick2: sst2:/var/glusterfs >> Options Reconfigured: >> storage.owner-gid: 1000 >> storage.owner-uid: 1000 >> performance.readdir-ahead: on >> cluster.self-heal-daemon: enable >> >> sst0 ~ # gluster volume status >> Status of volume: gv0 >> Gluster process TCP Port RDMA Port Online Pid >> >> ------------------------------------------------------------------------------ >> Brick sst0:/var/glusterfs 49152 0 Y 31263 >> NFS Server on localhost N/A N/A N N/A >> Self-heal Daemon on localhost N/A N/A Y 31254 >> >> Task Status of Volume gv0 >> >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> Any ideas how to bring the sst2 back to normal are appreciated. As a last >> resort solution, I can schedule the downtime, backup data, kill the volume >> and start all over, but I would like to know if there is a shorter path. >> Thank you very much in advance. >> >> -- >> Best Regards, >> >> Seva Gluschenko
[2017-04-28 07:45:46.949444] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:45:46.964849] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:45:46.964879] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:45:46.980085] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:45:46.980117] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:45:46.980126] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:45:46.980174] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:45:46.980331] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:45:51.805596] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:45:51.823515] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:45:51.956771] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:45:51.956944] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:45:51.956964] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:45:51.956991] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:45:51.957487] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 07:45:51.957481] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:45:51.979336] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 07:45:51.985947] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:45:51.987908] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 07:45:51.988045] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 1132736709, remote cksum = 59001066 on peer sst0.webkontrol.ru [2017-04-28 07:45:51.988138] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst0.webkontrol.ru (0), ret: 0, op_ret: -1 [2017-04-28 07:45:51.988957] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0.webkontrol.ru, port: 0 [2017-04-28 07:45:51.989872] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:47:46.876416] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:47:51.365167] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 The message "I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0" repeated 2 times between [2017-04-28 07:47:51.365167] and [2017-04-28 07:48:38.114271] [2017-04-28 07:50:36.736022] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:50:53.415851] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:51:03.496294] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7ff1e94456ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55fc1d4f37b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55fc1d4f3624] ) 0-: received signum (15), shutting down [2017-04-28 07:51:45.565862] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:51:45.578720] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:51:45.578750] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:51:45.586456] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:51:45.586494] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:51:45.586502] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:51:45.586547] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:51:45.586559] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:51:50.633962] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:51:50.634079] I [MSGID: 106194] [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-04-28 07:51:50.634131] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory] Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 07:51:50.641966] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 07:51:50.708781] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:51:50.709243] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:51:50.711011] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:51:50.711738] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 07:51:50.712126] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst0.webkontrol.ru (24007), ret: 0, op_ret: -1 [2017-04-28 07:51:57.559160] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst0 24007 [2017-04-28 07:51:57.559628] I [MSGID: 106129] [glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: sst0 (24007) [2017-04-28 07:51:57.560342] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:51:57.560368] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:51:57.560840] I [MSGID: 106498] [glusterd-handler.c:3616:glusterd_friend_add] 0-management: connect returned 0 [2017-04-28 07:51:57.563115] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0 [2017-04-28 07:51:57.563142] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:51:57.564249] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0, port: 0 [2017-04-28 07:52:03.403817] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:53:10.797531] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:53:15.247410] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst1 24007 [2017-04-28 07:53:15.248012] I [MSGID: 106129] [glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: sst1 (24007) [2017-04-28 07:53:15.248718] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:53:15.248740] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:53:15.249100] I [MSGID: 106498] [glusterd-handler.c:3616:glusterd_friend_add] 0-management: connect returned 0 [2017-04-28 07:53:15.264998] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1 [2017-04-28 07:53:15.265049] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:53:15.268831] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0 [2017-04-28 07:53:18.406851] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:58:57.729300] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fcecfbcf6ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x558bdb9147b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x558bdb914624] ) 0-: received signum (15), shutting down [2017-04-28 08:09:27.689903] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 08:09:27.702947] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 08:09:27.702975] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 08:09:27.706577] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 08:09:27.706614] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 08:09:27.706623] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 08:09:27.706668] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 08:09:27.706680] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 08:09:32.555268] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 08:09:32.658765] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 08:09:32.658928] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 08:09:32.658968] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 08:09:32.658994] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 08:09:32.659625] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 08:09:32.659619] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 08:09:32.659990] I [MSGID: 106194] [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-04-28 08:09:32.660357] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 08:09:32.683981] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 08:09:32.693205] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:09:32.706749] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:09:32.720846] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 08:09:32.720986] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst0.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 08:09:32.730415] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0 [2017-04-28 08:09:32.730457] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 08:09:32.733649] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 08:09:32.743630] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1 [2017-04-28 08:09:32.744075] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 08:09:39.918968] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:10:36.874787] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 08:19:42.780315] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x5 sent = 2017-04-28 08:09:32.730694. timeout = 600 for sst0:24007 [2017-04-28 08:19:42.780375] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x5 sent = 2017-04-28 08:09:32.752906. timeout = 600 for sst1:24007 [2017-04-28 08:49:13.066613] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f8e90e8c6ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x558a399827b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x558a39982624] ) 0-: received signum (15), shutting down [2017-04-28 08:53:09.957876] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 08:53:09.981856] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 08:53:09.982048] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 08:53:09.989872] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 08:53:09.989909] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 08:53:09.989919] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 08:53:09.989967] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 08:53:09.989980] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 08:53:14.863321] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 08:53:14.922604] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 08:53:14.922769] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 08:53:14.922809] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 08:53:14.922836] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 08:53:14.923680] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 08:53:14.923673] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 08:53:14.924038] I [MSGID: 106194] [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-04-28 08:53:14.924394] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 08:53:14.939987] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 08:53:14.942194] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:53:14.953057] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:53:21.072426] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:53:35.596254] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 08:57:50.550439] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 08:58:33.322742] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-04-28 08:58:33.323176] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-04-28 08:59:12.333937] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 07:28:03.196004] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:28:03.203614] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:28:03.203654] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:28:03.208918] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:28:03.208954] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:28:03.208968] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:28:03.209036] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:28:03.209056] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:28:04.693741] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:28:04.694694] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:28:04.696166] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:28:04.696294] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:28:04.696307] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:28:04.696327] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:28:04.696652] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 07:28:04.696648] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:28:04.697145] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 07:28:04.740957] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:28:04.742308] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2 [2017-04-28 07:28:04.742322] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:28:04.777048] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:28:04.777150] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 07:28:04.822564] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:28:05.551442] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0, port: 0 [2017-04-28 07:28:05.556904] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:28:05.560526] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 07:28:05.560596] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 4224731823, remote cksum = 1131418324 on peer sst0 [2017-04-28 07:28:05.560663] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst0 (0), ret: 0, op_ret: -1 [2017-04-28 07:28:39.446867] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst0 24007 [2017-04-28 07:28:41.333950] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:28:46.326386] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:28:56.471370] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:32:45.753176] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f2653de26ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x557fad3967b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x557fad396624] ) 0-: received signum (15), shutting down [2017-04-28 07:34:36.198249] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:34:36.205642] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:34:36.205682] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:34:36.211006] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:34:36.211041] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:34:36.211054] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:34:36.211119] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:34:36.211139] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:34:37.795217] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:34:37.797039] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:34:37.799291] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:34:37.799333] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:34:37.799355] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:34:37.800437] E [MSGID: 106187] [glusterd-store.c:4559:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore [2017-04-28 07:34:37.800479] E [MSGID: 101019] [xlator.c:503:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2017-04-28 07:34:37.800496] E [MSGID: 101066] [graph.c:325:glusterfs_graph_init] 0-management: initializing translator failed [2017-04-28 07:34:37.800516] E [MSGID: 101176] [graph.c:681:glusterfs_graph_activate] 0-graph: init failed [2017-04-28 07:34:37.800962] W [glusterfsd.c:1329:cleanup_and_exit] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xe5) [0x563af9820185] -->/usr/sbin/glusterd(glusterfs_process_volfp+0x1b7) [0x563af9820047] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x563af981f624] ) 0-: received signum (1), shutting down [2017-04-28 07:36:16.994152] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:36:17.002052] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:36:17.002096] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:36:17.008031] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:36:17.008072] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:36:17.008088] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:36:17.008160] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:36:17.008181] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:36:18.537736] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:36:18.537844] I [MSGID: 106194] [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-04-28 07:36:18.537898] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory] Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 07:36:18.538873] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 07:36:18.545097] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:36:18.545144] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:36:18.548869] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 07:36:18.549154] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst0.webkontrol.ru (24007), ret: 0, op_ret: -1 [2017-04-28 07:36:19.871627] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:36:25.519090] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst0 24007 [2017-04-28 07:36:25.519575] I [MSGID: 106129] [glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: sst0 (24007) [2017-04-28 07:36:25.520506] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:36:25.520540] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:36:25.521090] I [MSGID: 106498] [glusterd-handler.c:3616:glusterd_friend_add] 0-management: connect returned 0 [2017-04-28 07:36:25.536539] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0 [2017-04-28 07:36:25.536568] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:36:25.540030] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0, port: 0 [2017-04-28 07:36:27.783553] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:36:27.784256] I [MSGID: 106129] [glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: sst2 (24007) [2017-04-28 07:36:27.785385] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:36:27.785419] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:36:27.785850] I [MSGID: 106498] [glusterd-handler.c:3616:glusterd_friend_add] 0-management: connect returned 0 [2017-04-28 07:36:27.801417] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2 [2017-04-28 07:36:27.801448] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:37:50.228006] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:38:04.539534] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:39:30.570346] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:39:36.600198] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:39:40.535823] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:41:40.679683] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:44:36.991808] W [socket.c:593:__socket_rwv] 0-management: readv on sst2:24007 failed (No data available) [2017-04-28 07:44:36.991882] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Probe Sent to Peer>, has disconnected from glusterd. [2017-04-28 07:44:36.992313] E [rpc-clnt.c:365:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb0a3cc35fb] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7fb0a3a90eb1] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb0a3a90fce] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7fb0a3a92654] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7fb0a3a93138] ))))) 0-management: forced unwinding frame type(Peer mgmt) op(--(2)) called at 2017-04-28 07:36:27.801497 (xid=0x5) [2017-04-28 07:44:36.992398] I [socket.c:3550:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1) [2017-04-28 07:44:36.992416] E [rpcsvc.c:1348:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 1) to rpc-transport (socket.management) [2017-04-28 07:44:36.992450] E [MSGID: 106430] [glusterd-utils.c:497:glusterd_submit_reply] 0-glusterd: Reply submission failed [2017-04-28 07:44:48.057303] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 07:45:51.993206] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:51:03.529476] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Probe Sent to Peer>, has disconnected from glusterd. [2017-04-28 07:51:15.129289] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 07:53:15.259543] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:53:15.263540] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:53:15.263668] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 07:53:15.266626] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:53:15.267540] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-28 07:58:57.733476] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Sent and Received peer request>, has disconnected from glusterd. [2017-04-28 07:59:09.177314] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 08:09:32.718239] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:09:32.727764] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 08:09:32.727897] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 08:09:32.754625] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 08:49:13.070794] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Sent and Received peer request>, has disconnected from glusterd. [2017-04-28 08:49:24.601282] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 08:53:14.954821] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-27 13:53:11.966640] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-27 13:53:11.969733] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-27 13:53:11.969748] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-27 13:53:11.972015] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-27 13:53:11.972030] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-27 13:53:11.972036] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-27 13:53:11.972065] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-27 13:53:11.972074] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-27 13:53:13.118479] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-27 13:53:13.148442] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-27 13:53:13.150180] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 13:53:13.150312] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 13:53:13.150326] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 13:53:13.150345] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 13:53:13.150800] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-27 13:53:13.150796] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 13:53:13.152013] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-27 13:53:13.687393] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 13:53:13.688892] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 13:53:13.689174] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 13:53:13.693990] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 13:53:13.694097] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst1.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-27 13:53:13.694872] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-27 13:53:13.696982] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 13:53:16.147785] E [socket.c:2310:socket_connect_finish] 0-management: connection to 2a01:4f8:b15:1023::2:24007 failed (No route to host) [2017-04-27 13:53:16.147844] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Peer in Cluster>, has disconnected from glusterd. [2017-04-27 13:53:16.148656] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0 [2017-04-27 13:53:16.150042] E [socket.c:2310:socket_connect_finish] 0-management: connection to 2a01:4f8:b0a:101b::2:24007 failed (No route to host) [2017-04-27 13:53:16.150088] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst1> (<5a2198de-f536-4328-a278-7f746f276e35>), in state <Peer in Cluster>, has disconnected from glusterd. [2017-04-27 13:53:16.151683] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 13:53:16.151725] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 13:53:16.985462] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0 [2017-04-27 13:53:16.987764] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 13:53:16.987812] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 13:53:16.987901] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 13:53:16.988211] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2017-04-27 13:53:16.988437] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2017-04-27 13:53:16.988479] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped [2017-04-27 13:53:16.991101] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2017-04-27 13:53:16.992191] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2017-04-27 13:53:16.992233] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped [2017-04-27 13:53:16.992275] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service [2017-04-27 13:53:17.994617] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2017-04-27 13:53:17.994907] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-04-27 13:53:17.994947] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-04-27 13:53:17.995007] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2017-04-27 13:53:17.995239] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-27 13:53:17.995270] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-27 13:53:17.995324] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2017-04-27 13:53:17.995543] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-27 13:53:17.995572] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-27 13:53:17.997781] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 13:53:17.998064] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-04-27 13:53:17.998531] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0 [2017-04-27 13:53:17.999620] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 13:53:17.999712] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 13:53:18.001367] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-04-27 13:53:18.002771] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick sst0:/var/glusterfs has disconnected from glusterd. [2017-04-27 13:53:18.002993] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 13:53:22.012073] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/glusterfs on port 49152 [2017-04-27 13:53:44.017756] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 14:41:27.972234] I [MSGID: 106533] [glusterd-volume-ops.c:916:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gv0 [2017-04-27 14:41:34.404273] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 14:44:27.822099] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 14:54:08.030062] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fcf739d46ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55bb4de5f7b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55bb4de5f624] ) 0-: received signum (15), shutting down [2017-04-27 14:54:29.766745] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-27 14:54:29.770355] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-27 14:54:29.770374] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-27 14:54:29.772949] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-27 14:54:29.772967] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-27 14:54:29.772974] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-27 14:54:29.773006] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-27 14:54:29.773016] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-27 14:54:30.980463] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-27 14:54:31.015698] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-27 14:54:31.017131] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 14:54:31.017267] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 14:54:31.017281] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 14:54:31.017309] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 14:54:31.017651] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-27 14:54:31.017646] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 14:54:31.018602] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-27 14:54:31.024673] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 14:54:31.028369] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 14:54:31.028858] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0 [2017-04-27 14:54:31.029630] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2017-04-27 14:54:31.029715] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2017-04-27 14:54:31.029731] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped [2017-04-27 14:54:31.030598] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2017-04-27 14:54:31.030981] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2017-04-27 14:54:31.030998] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped [2017-04-27 14:54:31.031015] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service [2017-04-27 14:54:32.032199] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2017-04-27 14:54:32.032504] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-04-27 14:54:32.032543] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-04-27 14:54:32.032603] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2017-04-27 14:54:32.032840] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-27 14:54:32.032870] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-27 14:54:32.032923] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2017-04-27 14:54:32.033141] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-27 14:54:32.033170] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-27 14:54:32.035494] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 14:54:32.035825] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-04-27 14:54:32.038639] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 14:54:32.040678] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-04-27 14:54:32.041627] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick sst0:/var/glusterfs has disconnected from glusterd. [2017-04-27 14:54:32.041914] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0 [2017-04-27 14:54:32.042781] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 14:54:32.042828] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 14:54:32.045661] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 14:54:32.086515] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0 [2017-04-27 14:54:32.094695] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 14:54:32.094753] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 14:54:32.096411] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 14:54:32.096616] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 14:54:32.097496] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-27 14:54:32.099173] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 14:54:36.049986] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/glusterfs on port 49153 [2017-04-27 14:59:37.402947] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 15:00:50.647128] I [MSGID: 106533] [glusterd-volume-ops.c:916:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gv0 [2017-04-27 15:15:09.894711] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fc8e95e16ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x56485ac0e7b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x56485ac0e624] ) 0-: received signum (15), shutting down [2017-04-27 15:16:51.130160] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-27 15:16:51.134590] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-27 15:16:51.134613] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-27 15:16:51.137835] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-27 15:16:51.137858] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-27 15:16:51.137868] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-27 15:16:51.137910] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-27 15:16:51.137929] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-27 15:16:52.334108] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-27 15:16:52.364194] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-27 15:16:52.365653] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 15:16:52.365782] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 15:16:52.365794] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 15:16:52.365813] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 15:16:52.366121] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-27 15:16:52.366117] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 15:16:52.366995] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-27 15:16:53.247237] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0 [2017-04-27 15:16:53.248077] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2017-04-27 15:16:53.248177] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2017-04-27 15:16:53.248195] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped [2017-04-27 15:16:53.249126] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2017-04-27 15:16:53.249534] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2017-04-27 15:16:53.249551] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped [2017-04-27 15:16:53.249569] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service [2017-04-27 15:16:54.251325] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2017-04-27 15:16:54.251621] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-04-27 15:16:54.251690] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-04-27 15:16:54.251751] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2017-04-27 15:16:54.251984] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-27 15:16:54.252016] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-27 15:16:54.252084] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2017-04-27 15:16:54.252304] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-27 15:16:54.252334] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-27 15:16:54.254557] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 15:16:54.254859] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-04-27 15:16:54.255157] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 15:16:54.255227] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 15:16:54.255367] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 15:16:54.255459] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0 [2017-04-27 15:16:54.257751] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-04-27 15:16:54.258679] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick sst0:/var/glusterfs has disconnected from glusterd. [2017-04-27 15:16:54.259058] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 15:16:54.259195] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 15:16:54.259253] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 15:16:54.259604] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 15:16:54.260863] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 15:16:54.261834] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-27 15:16:54.263351] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 15:16:54.264200] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 15:16:54.267379] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 15:16:54.268157] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0 [2017-04-27 15:16:54.272611] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 15:16:58.067807] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 15:16:58.267770] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/glusterfs on port 49152 [2017-04-27 15:17:08.145868] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 15:17:11.346625] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:07:08.939198] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:07:22.465221] I [MSGID: 106533] [glusterd-volume-ops.c:916:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gv0 [2017-04-28 07:07:23.219676] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-04-28 07:07:23.219740] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-04-28 07:07:23.219839] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-28 07:07:23.219867] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-28 07:07:23.219959] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-28 07:07:23.219983] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-28 07:07:23.255153] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd [2017-04-28 07:07:23.272213] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd [2017-04-28 07:26:57.666373] W [socket.c:593:__socket_rwv] 0-management: readv on sst1:24007 failed (No data available) [2017-04-28 07:26:57.666447] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst1> (<5a2198de-f536-4328-a278-7f746f276e35>), in state <Peer in Cluster>, has disconnected from glusterd. [2017-04-28 07:26:57.666602] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 07:26:57.666630] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 07:27:09.534588] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst1:24007 failed (Connection refused) [2017-04-28 07:28:04.734953] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:28:04.739439] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:28:05.548023] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 1131418324, remote cksum = 4224731823 on peer sst1 [2017-04-28 07:28:05.548159] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: -1 [2017-04-28 07:28:05.560095] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0 [2017-04-28 07:31:35.775788] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-04-28 07:31:35.775843] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-04-28 07:31:35.775940] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-28 07:31:35.775966] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-28 07:31:35.776055] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-28 07:31:35.776077] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-28 07:31:35.790917] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd [2017-04-28 07:31:35.808099] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd [2017-04-28 07:32:19.702837] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:32:45.754974] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst1> (<5a2198de-f536-4328-a278-7f746f276e35>), in state <Peer Rejected>, has disconnected from glusterd. [2017-04-28 07:32:45.755166] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 07:32:45.755197] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 07:32:57.566578] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst1:24007 failed (Connection refused) [2017-04-28 07:36:18.548429] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 24007 [2017-04-28 07:36:25.529048] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:36:25.532889] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:36:25.533005] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst1.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 07:36:25.535830] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:36:25.536507] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0 [2017-04-28 07:37:34.559221] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:39:52.354497] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:44:36.988131] W [socket.c:593:__socket_rwv] 0-management: readv on sst2:24007 failed (No data available) [2017-04-28 07:44:36.988232] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Peer in Cluster>, has disconnected from glusterd. [2017-04-28 07:44:36.988390] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 07:44:36.988420] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 07:44:48.647919] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 07:45:51.983723] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:45:51.985981] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:45:51.986141] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 59001066, remote cksum = 1132736709 on peer sst2 [2017-04-28 07:45:51.986269] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: -1 [2017-04-28 07:45:51.987609] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0 [2017-04-28 07:48:15.716415] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:48:28.863531] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:51:03.525725] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Peer Rejected>, has disconnected from glusterd. [2017-04-28 07:51:03.525969] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 07:51:03.526019] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 07:51:13.688993] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 07:51:50.710054] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 24007 [2017-04-28 07:51:57.559226] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:51:57.560345] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:51:57.560460] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 07:51:57.561004] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:51:57.561841] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-28 07:52:31.170428] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:52:56.783783] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:53:42.149638] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:55:03.447810] I [MSGID: 106482] [glusterd-brick-ops.c:447:__glusterd_handle_add_brick] 0-management: Received add brick req [2017-04-28 07:55:03.447852] I [MSGID: 106578] [glusterd-brick-ops.c:499:__glusterd_handle_add_brick] 0-management: replica-count is 2 [2017-04-28 07:55:03.447865] E [MSGID: 106291] [glusterd-brick-ops.c:626:__glusterd_handle_add_brick] 0-management: [2017-04-28 07:55:31.925551] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:56:56.035196] I [MSGID: 106482] [glusterd-brick-ops.c:447:__glusterd_handle_add_brick] 0-management: Received add brick req [2017-04-28 07:56:56.035247] I [MSGID: 106578] [glusterd-brick-ops.c:499:__glusterd_handle_add_brick] 0-management: replica-count is 2 [2017-04-28 07:56:56.035262] E [MSGID: 106291] [glusterd-brick-ops.c:626:__glusterd_handle_add_brick] 0-management: [2017-04-28 07:58:57.730694] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Accepted peer request>, has disconnected from glusterd. [2017-04-28 07:58:57.730885] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 07:58:57.730916] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 07:59:07.732465] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 08:09:32.693558] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:09:32.718118] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 08:09:32.718248] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 08:09:32.721359] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2 [2017-04-28 08:09:32.721401] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 08:09:32.731133] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 08:09:53.596619] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:10:53.987182] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 08:15:03.737152] E [MSGID: 106062] [glusterd-utils.c:9338:glusterd_max_opversion_use_rsp_dict] 0-management: Maximum supported op-version not set in destination dictionary [2017-04-28 08:15:26.819219] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:15:54.288580] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:19:24.708030] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:19:38.912746] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x2f sent = 2017-04-28 08:09:32.721691. timeout = 600 for sst2:24007 [2017-04-28 08:49:13.071592] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Sent and Received peer request>, has disconnected from glusterd. [2017-04-28 08:49:13.071837] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 08:49:13.071868] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 08:49:24.046693] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 08:53:14.944791] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:59:59.771701] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-04-28 09:00:35.795605] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:59:59.772167] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-04-28 09:01:11.179203] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
_______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
