I was having trouble setting up geo-replication, and I finally figured out why.
Gsyncd is trying to use the wrong (but valid) name for the slave server, and
it’s resolving to an address it can’t reach. It does this even though I tried
to setup the geo-replication to a specific IP address, and the initial create
succeeded.
The environment I’m working with has server1 & server3 with bricks for a
replication volume, and my slave server ‘spire’ is located remotely. Because
there are multiple paths between the systems, I wanted to force geo-replication
to occur with a specific slave address, so I tried to setup replication with an
ip based URL.
Setup/config of the geo-replication works, and appears to start, but is
permanently in a failure state:
[root@server1 geo-replication]# gluster volume geo-replication status
No active geo-replication sessions
[root@server1 geo-replication]# gluster volume geo-replication gvOvirt
10.78.4.1::geobk-Ovirt create push-pem
Creating geo-replication session between gvOvirt & 10.78.4.1::geobk-Ovirt has
been successful
[root@server1 geo-replication]# gluster volume geo-replication status
MASTER NODE MASTER VOL MASTER BRICK SLAVE
STATUS CHECKPOINT STATUS CRAWL STATUS
-------------------------------------------------------------------------------------------------------------------------------------------------------
server1.xxx.xxx.xxx gvOvirt /v0/ha-engine/gbOvirt
ssh://10.78.4.1::geobk-Ovirt Not Started N/A N/A
server3.xxx.xxx.xxx gvOvirt /v0/ha-engine/gbOvirt
ssh://10.78.4.1::geobk-Ovirt Not Started N/A N/A
[root@server1 geo-replication]# gluster volume geo-replication gvOvirt
10.78.4.1::geobk-Ovirt start
Starting geo-replication session between gvOvirt & 10.78.4.1::geobk-Ovirt has
been successful
[root@server1 geo-replication]# gluster volume geo-replication status
MASTER NODE MASTER VOL MASTER BRICK SLAVE
STATUS CHECKPOINT STATUS CRAWL STATUS
--------------------------------------------------------------------------------------------------------------------------------------------------
server1.xxx.xxx.xxx gvOvirt /v0/ha-engine/gbOvirt
ssh://10.78.4.1::geobk-Ovirt faulty N/A N/A
server3.xxx.xxx.xxx gvOvirt /v0/ha-engine/gbOvirt
ssh://10.78.4.1::geobk-Ovirt faulty N/A N/A
Tracking it down in the logs, it’s because it’s trying to connect to
“root@spire” instead of “[email protected]”, even though the setup seemed to use
10.78.4.1 just fine:
2014-09-23 14:13:41.595123] I [monitor(monitor):130:monitor] Monitor: starting
gsyncd worker
[2014-09-23 14:13:41.764898] I [gsyncd(/v0/ha-engine/gbOvirt):532:main_i]
<top>: syncing: gluster://localhost:gvOvirt ->
ssh://root@spire:gluster://localhost:geobk-Ovirt
[2014-09-23 14:13:53.40934] E
[syncdutils(/v0/ha-engine/gbOvirt):223:log_raise_exception] <top>: connection
to peer is broken
[2014-09-23 14:13:53.41244] E [resource(/v0/ha-engine/gbOvirt):204:errlog]
Popen: command "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-gaHXrp/e68410dcff276efa39a94dc5b33e0d8e.sock root@spire
/nonexistent/gsyncd --session-owner c9a371cd-8644-4706-a4b7-f12bc2c37ac6 -N
--listen --timeout 120 gluster://localhost:geobk-Ovirt" returned with 255,
saying:
[2014-09-23 14:13:53.41356] E [resource(/v0/ha-engine/gbOvirt):207:logerr]
Popen: ssh> ssh_exchange_identification: Connection closed by remote host
[2014-09-23 14:13:53.41569] I [syncdutils(/v0/ha-engine/gbOvirt):192:finalize]
<top>: exiting.
[2014-09-23 14:13:54.44226] I [monitor(monitor):157:monitor] Monitor:
worker(/v0/ha-engine/gbOvirt) died in startup phase
So where did it pick this up from? If anything, I think it’d try the full name
of the slave server, but it didn’t even try that. The slave’s hostname, btw, is
spire.yyy.yyy.xxx, not even in the same domain as the master.
I worked around this by setting up a new hostname resolution for ‘spire' and
tweaking the search path so it resolved things the way I wanted (to 10.78.4.1),
but that shouldn’t be necessary. Also seems like this might cause security
issues (well, ok, probably not since it’s rsyncing over ssh, but traffic could
definitely wind up where you didn’t expect it) because it’s not trying to
connect to what I explicitly told it to. And in this case, it ran afoul of some
sshd root login allowed rules, but could have easily been other firewall rules
too. And confusion because the geo-replication status report is misleading,
it’s trying to sync to an address that isn’t reflected there.
Seems like a bug to me, but figured I’d check here first.
-Darrell
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
