[Gluster-users] Gluster peer status disconnected on node where glusterd has been started first
Hello, I am testing SSL/TLS support in gluster. I have gluster replica set with 2 node: # gluster volume info Volume Name: repofiles Type: Replicate Volume ID: 4b0e2a74-f1ca-4fe7-8518-23919e1b5fa0 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: dist-int-master03:/storage/gluster_brick_repofiles Brick2: dist-int-master04:/storage/gluster_brick_repofiles Options Reconfigured: performance.readdir-ahead: on auth.ssl-allow: dist-master03,dist-master04 client.ssl: on server.ssl: on If umount all shares, stop all gluster process and try to start glusterd process I found strange behavior on node where I has started first glusterd. On gluster 3.7.3 it look like output of gluster peer status is disconnected. But the second node show gluster peer status connected. Restart glusterd service on node where I has start first glusterd have solved the problem. 1. # cat /etc/issue Ubuntu 14.04.2 LTS \n \l # dpkg -l | grep gluster ii glusterfs-client 3.7.3-ubuntu1~trusty1amd64clustered file-system (client package) ii glusterfs-common 3.7.3-ubuntu1~trusty1amd64GlusterFS common libraries and translator modules ii glusterfs-server 3.7.3-ubuntu1~trusty1amd64clustered file-system (server package) In log before restart glusterd: [2015-08-21 13:31:51.474562] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.3 (args: /usr/sbin /glusterd -p /var/run/glusterd.pid) [2015-08-21 13:31:51.42] I [MSGID: 106478] [glusterd.c:1346:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-08-21 13:31:51.477807] I [MSGID: 106479] [glusterd.c:1395:init] 0-management: Using /var/lib/glusterd as working directory [2015-08-21 13:31:51.480745] I [socket.c:3968:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED [2015-08-21 13:31:51.480775] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED [2015-08-21 13:31:51.480785] I [socket.c:3988:socket_init] 0-socket.management: using private polling thread [2015-08-21 13:31:51.481924] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2015-08-21 13:31:51.481948] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device [2015-08-21 13:31:51.481959] W [rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2015-08-21 13:31:51.482005] W [rpcsvc.c:1597:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-08-21 13:31:51.482021] E [MSGID: 106243] [glusterd.c:1619:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2015-08-21 13:31:51.482131] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED [2015-08-21 13:31:54.273846] I [MSGID: 106513] [glusterd-store.c:2036:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30703 [2015-08-21 13:31:54.506187] I [MSGID: 106498] [glusterd-handler.c:3570:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-08-21 13:31:54.506288] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-08-21 13:31:54.506416] I [socket.c:3968:socket_init] 0-management: SSL support on the I/O path is ENABLED [2015-08-21 13:31:54.506430] I [socket.c:3971:socket_init] 0-management: SSL support for glusterd is ENABLED [2015-08-21 13:31:54.506439] I [socket.c:3988:socket_init] 0-management: using private polling thread [2015-08-21 13:31:54.508248] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid argument [2015-08-21 13:31:54.508277] E [socket.c:3019:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2015-08-21 13:31:54.510377] E [socket.c:3075:socket_connect] 0-management: connection attempt on failed, (Connection refused) [2015-08-21 13:31:54.510826] I [MSGID: 106004] [glusterd-handler.c:5051:__glusterd_peer_rpc_notify] 0-management: Peer (<264cf70c-688c-4d67-b835-5ec7b7689894>), in state , has disconnected from glusterd. [2015-08-21 13:31:54.510856] E [socket.c:406:ssl_setup_connection] 0-management: SSL connect error [2015-08-21 13:31:54.510889] E [socket.c:2442:socket_poller] 0-management: client setup failed [2015-08-21 13:31:54.510885] E [MSGID: 106155] [glusterd-utils.c:199:glusterd_unlock] 0-management: Cluster lock not held! [2015-08-21 13:31:54.510936] I [MSGID: 106004] [glusterd-handler.c:5051:__glusterd_peer_rpc_notify] 0-management: Peer (<264cf70c-688c-4d67-b835-5ec7b7689894>), in state , has disconnected from glusterd. [2015-08-21 13:31:54.510960] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=588 max=0 total=0 [2015-08-21 13:31:54.510971] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=124 max=
Re: [Gluster-users] Gluster peer status disconnected
Hi, Thanks for reply. I would like to share why this happen, Actually I am doing this in the college lab, on single workstation,where I have installed VM and 5 nodes.One as client and 4 are in the server pool. College LAN has DHCP server. IPs generated at the time of cluster creation changed due to that reason. Once IP got changed, My gluster cluster got down. showing this above message on peer status command. I did changed all hosts configuration,and also stopped and deleted previous volumes created on that. and tried many times to check status of my server pool. I checked ping , confirmed that daemon is running. and also as per iptables flush command.But it was not working. Finally I created new nodes in virtual box and created new server pool. Its working now. Thanks.. Shyam D. On Mon, Apr 20, 2015 at 10:09 AM, Sweta Anandpara wrote: > Forgot to include your personal mail. > > > Forwarded Message Subject: Re: [Gluster-users] Gluster > peer status disconnected Date: Mon, 20 Apr 2015 10:07:32 +0530 From: Sweta > AnandparaTo: > gluster-users@gluster.org > > Hi Shyam, > > Were the peers in a connected state anytime *before*? > > If yes, then these are the things you can do to troubleshoot and get the > cluster back to normal (some of which Neils has already mentioned ): > > 1. Check if the nodes are reachable.. 'ping ' > 2. Check if the glusterd daemon is in an active state in all the nodes.. > 'service glusterd status' > 3. Firewall issue - flush IPtables on the node which is disconnected.. > 'iptables -F' > > Do let us know if this helps.. > > Regarding *how* you landed up in this situation, as Atin mentioned, it > would be better to be a little specific on the things that you did at your > end. > > Thanks, > Shweta > > On 04/19/2015 06:45 PM, Niels de Vos wrote: > > On Sun, Apr 19, 2015 at 03:04:15PM +0530, Shyam Deshmukh wrote: > > Hi, > > I am getting following status and my cluster is down due to the same. > Please help. > > gluster@gluster1:~$ sudo gluster peer status > Number of Peers: 3 > > Hostname: gluster3 > Uuid: c6f6574b-9779-4635-a7c2-06185a9ae973 > State: Peer in Cluster (Disconnected) > > Hostname: gluster4 > Uuid: c0e2ad61-da65-4e4c-a863-e9e8c4e84710 > State: Peer in Cluster (Disconnected) > > Hostname: gluster2 > Uuid: 45ac276d-9e65-4e32-8abe-51f8ba0241ce > State: Peer in Cluster (Disconnected) > gluster@gluster1:~$ > > You can probably start by verifying that the hostnames exactly as listed > above can get resolved to their correct IP-addresses. If that works > fine, check if you can connect to port 24007 on those systems. > > This might be a network issue, firewall or glusterd is not running > (anymore) on those systems. > > Sometimes it also helps to check on other systems, and se if the output > of "gluster peer status" is different there. > > Good luck! > > > > ___ > Gluster-users mailing > listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users > > > > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster peer status disconnected
Hi Shyam, Were the peers in a connected state anytime *before*? If yes, then these are the things you can do to troubleshoot and get the cluster back to normal (some of which Neils has already mentioned ): 1. Check if the nodes are reachable.. 'ping ' 2. Check if the glusterd daemon is in an active state in all the nodes.. 'service glusterd status' 3. Firewall issue - flush IPtables on the node which is disconnected.. 'iptables -F' Do let us know if this helps.. Regarding *how* you landed up in this situation, as Atin mentioned, it would be better to be a little specific on the things that you did at your end. Thanks, Shweta On 04/19/2015 06:45 PM, Niels de Vos wrote: On Sun, Apr 19, 2015 at 03:04:15PM +0530, Shyam Deshmukh wrote: Hi, I am getting following status and my cluster is down due to the same. Please help. gluster@gluster1:~$ sudo gluster peer status Number of Peers: 3 Hostname: gluster3 Uuid: c6f6574b-9779-4635-a7c2-06185a9ae973 State: Peer in Cluster (Disconnected) Hostname: gluster4 Uuid: c0e2ad61-da65-4e4c-a863-e9e8c4e84710 State: Peer in Cluster (Disconnected) Hostname: gluster2 Uuid: 45ac276d-9e65-4e32-8abe-51f8ba0241ce State: Peer in Cluster (Disconnected) gluster@gluster1:~$ You can probably start by verifying that the hostnames exactly as listed above can get resolved to their correct IP-addresses. If that works fine, check if you can connect to port 24007 on those systems. This might be a network issue, firewall or glusterd is not running (anymore) on those systems. Sometimes it also helps to check on other systems, and se if the output of "gluster peer status" is different there. Good luck! ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster peer status disconnected
On 04/19/2015 03:04 PM, Shyam Deshmukh wrote: > Hi, > > I am getting following status and my cluster is down due to the same. > Please help. > > gluster@gluster1:~$ sudo gluster peer status > Number of Peers: 3 > > Hostname: gluster3 > Uuid: c6f6574b-9779-4635-a7c2-06185a9ae973 > State: Peer in Cluster (Disconnected) > > Hostname: gluster4 > Uuid: c0e2ad61-da65-4e4c-a863-e9e8c4e84710 > State: Peer in Cluster (Disconnected) > > Hostname: gluster2 > Uuid: 45ac276d-9e65-4e32-8abe-51f8ba0241ce > State: Peer in Cluster (Disconnected) > gluster@gluster1:~$ Please provide the activities done before you saw this change. Nothing can be analyzed/concluded with this information. ~Atin > > > Thanks > Shyam > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > -- ~Atin ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster peer status disconnected
On Sun, Apr 19, 2015 at 03:04:15PM +0530, Shyam Deshmukh wrote: > Hi, > > I am getting following status and my cluster is down due to the same. > Please help. > > gluster@gluster1:~$ sudo gluster peer status > Number of Peers: 3 > > Hostname: gluster3 > Uuid: c6f6574b-9779-4635-a7c2-06185a9ae973 > State: Peer in Cluster (Disconnected) > > Hostname: gluster4 > Uuid: c0e2ad61-da65-4e4c-a863-e9e8c4e84710 > State: Peer in Cluster (Disconnected) > > Hostname: gluster2 > Uuid: 45ac276d-9e65-4e32-8abe-51f8ba0241ce > State: Peer in Cluster (Disconnected) > gluster@gluster1:~$ You can probably start by verifying that the hostnames exactly as listed above can get resolved to their correct IP-addresses. If that works fine, check if you can connect to port 24007 on those systems. This might be a network issue, firewall or glusterd is not running (anymore) on those systems. Sometimes it also helps to check on other systems, and se if the output of "gluster peer status" is different there. Good luck! pgptctxyNrm7X.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Gluster peer status disconnected
Hi, I am getting following status and my cluster is down due to the same. Please help. gluster@gluster1:~$ sudo gluster peer status Number of Peers: 3 Hostname: gluster3 Uuid: c6f6574b-9779-4635-a7c2-06185a9ae973 State: Peer in Cluster (Disconnected) Hostname: gluster4 Uuid: c0e2ad61-da65-4e4c-a863-e9e8c4e84710 State: Peer in Cluster (Disconnected) Hostname: gluster2 Uuid: 45ac276d-9e65-4e32-8abe-51f8ba0241ce State: Peer in Cluster (Disconnected) gluster@gluster1:~$ Thanks Shyam ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
