Hi, Recently, my company needed to change our hostnames used in the Gluster Pool.
In a first moment, we have two Gluster Nodes called storage1 and storage2. Our volumes used two bricks: storage1:/MYVOLYME and storage2:/MYVOLUME. We put the storage1 and storage2 IPs in the /etc/hosts file of our nodes and in our client servers. After some time, more client servers started to using Gluster and we discovered that using hostnames without domain (using /etc/hosts) in all client servers is a pain in the a$$ :(. So, we decided to change them to something like storage1.mydomain.com and storage2.mydomain.com. Remember that, at this point, we had already some volumes (with bricks): $ gluster volume info MYVOL [...] Brick1: storage1:/MYDIR Brick1: storage2:/MYDIR For simplicity, let's consider that we had two Gluster Nodes, each one with the following entries in /etc/hosts: 10.10.10.1 storage1 10.10.10.2 storage2 To implement the hostname changes, we've changed the etc hosts file to: 10.10.10.1 storage1 storage1.mydomain.com 10.10.10.2 storage2 storage2.mydomain.com And we've run in storage1: $ gluster peer probe storage2.mydomain.com peer probe: success Everything works well during some time, but the glusterd starts to fail after any reboot: $ service glusterfs-server status glusterfs-server start/running, process 14714 $ service glusterfs-server restart glusterfs-server stop/waiting glusterfs-server start/running, process 14860 $ service glusterfs-server status glusterfs-server stop/waiting To start the service again, it was necessary to rollback the hostname1 config to storage2 in /var/lib/glusterd/peers/OUR_UUID. After some try and error, we discovered that if we change the order of the entries in /etc/hosts and repeat the process, everything worked. It is, from: 10.10.10.1 storage1 storage1.mydomain.com 10.10.10.2 storage2 storage2.mydomain.com To: 10.10.10.1 storage1.mydomain.com storage1 10.10.10.2 storage2.mydomain.com storage2 And run: gluster peer probe storage2.mydomain.com service glusterfs-server restart So we've checked the Glusterd debug log and checked the GlusterFS source code and discovered that the big secret was the function glusterd_friend_find_by_hostname, in the file xlators/mgmt/glusterd/src/glusterd-utils.c. This function is called for each brick that isn't a local brick and does the following things: - It checks if the brick hostname is equal to some peer hostname; - If it's, this peer is our wanted friend; - If not, it gets the brick IP (resolves the hostname using the function getaddrinfo) and checks if the brick IP is equal to the peer hostname; - It is, we could run gluster peer probe 10.10.10.2. Once the brick IP (storage2 resolves to 10.10.10.2) would have equal to the peer "hostname" (10.10.10.2); - If it's, this peer is our wanted friend; - If not, gets the reverse of the brick IP (using the function getnameinfo) and checks if the brick reverse is equal to the peer hostname; - This is why changing the order of the entries in /etc/hosts worked as an workaround for us; - If not, returns and error (and Glusterd will fail). However, we think that comparing the brick IP (resolving the brick hostname) and the peer IP (resolving the peer hostname) would be a simpler and more comprehensive solution. Once both brick and peer will have difference hostnames, but the same IP, it would work. The solution could be: - It checks if the brick hostname is equal to some peer hostname; - If it's, this peer is our wanted friend; - If not, it gets both the brick IP (resolves the hostname using the function getaddrinfo) and the peer IP (resolves the peer hostname) and, for each IP pair, check if a brick IP is equal to a peer IP; - If it's, this peer is our wanted friend; - If not, returns and error (and Glusterd will fail). What do you think about it? -- *Rarylson Freitas* Computer Engineer
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel