Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
emon/repce.py", line 110, in > > worker > > res = getattr(self.obj, rmeth)(*in_data[2:]) > > File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line > > 37, in init > > return Changes.cl_init() > > File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line > > 21, in __getattr__ > > from libgfchangelog import Changes as LChanges > > File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line > > 17, in > > class Changes(object): > > File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line > > 19, in Changes > > use_errno=True) > > File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__ > > self._handle = _dlopen(self._name, mode) > > OSError: libgfchangelog.so: cannot open shared object file: No such file or > > directory > > [2018-07-23 11:33:23.119609] E [repce(worker > > /urd-gds/gluster):206:__call__] RepceClient: call failed > > call=29589:140155686246208:1532345603.11method=init > > error=OSError > > [2018-07-23 11:33:23.119708] E [syncdutils(worker > > /urd-gds/gluster):330:log_raise_exception] : FAIL: > > Traceback (most recent call last): > > File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, in > > main > > func(args) > > File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72, in > > subcmd_worker > > local.service_loop(remote) > > File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1236, > > in service_loop > > changelog_agent.init() > > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in > > __call__ > > return self.ins(self.meth, *a) > > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in > > __call__ > > raise res > > OSError: libgfchangelog.so: cannot open shared object file: No such file or > > directory > > [2018-07-23 11:33:23.130100] I [repce(agent > > /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. > > [2018-07-23 11:33:24.104176] I [monitor(monitor):272:monitor] Monitor: > > worker died in startup phase brick=/urd-gds/gluster > > > > Thanks, Sunny!! > > > > Regards > > Marcus Pedersén > > > > > > Från: Sunny Kumar > > Skickat: den 23 juli 2018 12:53 > > Till: Marcus Pedersén > > Kopia: Kotresh Hiremath Ravishankar; gluster-users@gluster.org > > Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work > > > > Hi Marcus, > > > > On Mon, Jul 23, 2018 at 4:04 PM Marcus Pedersén > > wrote: > > > > > > Hi Sunny, > > > ldconfig -p /usr/local/lib | grep libgf > > > Output: > > > libgfxdr.so.0 (libc6,x86-64) => /lib64/libgfxdr.so.0 > > > libgfrpc.so.0 > > > (libc6,x86-64) => /lib64/libgfrpc.so.0 > > > libgfdb.so.0 (libc6,x86-64) => /lib64/libgfdb.so.0 > > > libgfchangelog.so.0 > > > (libc6,x86-64) => /lib64/libgfchangelog.so.0 > > > libgfapi.so.0 (libc6,x86-64) => /lib64/libgfapi.so.0 > > > > > > So that seems to be alright, right? > > > > > Yes, this seems wright can you share the gsyncd.log again > > > Best regards > > > Marcus > > > > > > > > > Marcus Pedersén > > > Systemadministrator > > > Interbull Centre > > > > > > Sent from my phone > > > > > > > > > Den 23 juli 2018 11:17 skrev Sunny Kumar : > > > > > > Hi Marcus, > > > > > > On Wed, Jul 18, 2018 at 4:08 PM Marcus Pedersén > > > wrote: > > > > > > > > Hi Kotresh, > > > > > > > > I ran: > > > > > > > > #ldconfig /usr/lib > > > can you do - > > > ldconfig /usr/local/lib > > > > > > > > > Output: > > > > > > > > > > > on all nodes in both clusters but I still get the same error. > > > > > > > > What to do? > > > > > > > > > > > > Output for: > > > > > > > > # ldconfig -p /u
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
s/python/syncdaemon/subcmds.py", line 72, in > > subcmd_worker > > local.service_loop(remote) > > File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1236, > > in service_loop > > changelog_agent.init() > > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in > > __call__ > > return self.ins(self.meth, *a) > > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in > > __call__ > > raise res > > OSError: libgfchangelog.so: cannot open shared object file: No such file or > > directory > > [2018-07-23 11:33:23.130100] I [repce(agent > > /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. > > [2018-07-23 11:33:24.104176] I [monitor(monitor):272:monitor] Monitor: > > worker died in startup phase brick=/urd-gds/gluster > > > > Thanks, Sunny!! > > > > Regards > > Marcus Pedersén > > > > > > Från: Sunny Kumar > > Skickat: den 23 juli 2018 12:53 > > Till: Marcus Pedersén > > Kopia: Kotresh Hiremath Ravishankar; gluster-users@gluster.org > > Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work > > > > Hi Marcus, > > > > On Mon, Jul 23, 2018 at 4:04 PM Marcus Pedersén > > wrote: > > > > > > Hi Sunny, > > > ldconfig -p /usr/local/lib | grep libgf > > > Output: > > > libgfxdr.so.0 (libc6,x86-64) => /lib64/libgfxdr.so.0 > > > libgfrpc.so.0 > > > (libc6,x86-64) => /lib64/libgfrpc.so.0 > > > libgfdb.so.0 (libc6,x86-64) => /lib64/libgfdb.so.0 > > > libgfchangelog.so.0 > > > (libc6,x86-64) => /lib64/libgfchangelog.so.0 > > > libgfapi.so.0 (libc6,x86-64) => /lib64/libgfapi.so.0 > > > > > > So that seems to be alright, right? > > > > > Yes, this seems wright can you share the gsyncd.log again > > > Best regards > > > Marcus > > > > > > > > > Marcus Pedersén > > > Systemadministrator > > > Interbull Centre > > > > > > Sent from my phone > > > > > > > > > Den 23 juli 2018 11:17 skrev Sunny Kumar : > > > > > > Hi Marcus, > > > > > > On Wed, Jul 18, 2018 at 4:08 PM Marcus Pedersén > > > wrote: > > > > > > > > Hi Kotresh, > > > > > > > > I ran: > > > > > > > > #ldconfig /usr/lib > > > can you do - > > > ldconfig /usr/local/lib > > > > > > > > > Output: > > > > > > > > > > > on all nodes in both clusters but I still get the same error. > > > > > > > > What to do? > > > > > > > > > > > > Output for: > > > > > > > > # ldconfig -p /usr/lib | grep libgf > > > > > > > > libgfxdr.so.0 (libc6,x86-64) => /lib64/libgfxdr.so.0 > > > > libgfrpc.so.0 (libc6,x86-64) => /lib64/libgfrpc.so.0 > > > > libgfdb.so.0 (libc6,x86-64) => /lib64/libgfdb.so.0 > > > > libgfchangelog.so.0 (libc6,x86-64) => /lib64/libgfchangelog.so.0 > > > > libgfapi.so.0 (libc6,x86-64) => /lib64/libgfapi.so.0 > > > > > > > > > > > > I read somewere that you could change some settings for geo-replication > > > > to speed up sync. > > > > > > > > I can not remember where I saw that and what config parameters. > > > > > > > > When geo-replication works I have 30TB on master cluster that has to be > > > > synced to slave nodes, > > > > > > > > and that will take a while before the slave nodes have catched up. > > > > > > > > > > > > Thanks and regards > > > > > > > > Marcus Pedersén > > > > > > > > > > > > Part of gsyncd.log: > > > > > > > > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, > > > > in __call__ > > > > raise res > > > > OSError: libgfchangelog.so: cannot open shared object file: No such > > > > file or directory > > > > [2018-07-18 10:23:52.305119] I [repce(agent &
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
epceServer: terminating on reaching EOF. > [2018-07-23 11:33:24.104176] I [monitor(monitor):272:monitor] Monitor: worker > died in startup phase brick=/urd-gds/gluster > > Thanks, Sunny!! > > Regards > Marcus Pedersén > > > Från: Sunny Kumar > Skickat: den 23 juli 2018 12:53 > Till: Marcus Pedersén > Kopia: Kotresh Hiremath Ravishankar; gluster-users@gluster.org > Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work > > Hi Marcus, > > On Mon, Jul 23, 2018 at 4:04 PM Marcus Pedersén > wrote: > > > > Hi Sunny, > > ldconfig -p /usr/local/lib | grep libgf > > Output: > > libgfxdr.so.0 (libc6,x86-64) => /lib64/libgfxdr.so.0 > > libgfrpc.so.0 > > (libc6,x86-64) => /lib64/libgfrpc.so.0 > > libgfdb.so.0 (libc6,x86-64) => /lib64/libgfdb.so.0 > > libgfchangelog.so.0 (libc6,x86-64) > > => /lib64/libgfchangelog.so.0 > > libgfapi.so.0 (libc6,x86-64) => /lib64/libgfapi.so.0 > > > > So that seems to be alright, right? > > > Yes, this seems wright can you share the gsyncd.log again > > Best regards > > Marcus > > > > > > Marcus Pedersén > > Systemadministrator > > Interbull Centre > > > > Sent from my phone > > > > > > Den 23 juli 2018 11:17 skrev Sunny Kumar : > > > > Hi Marcus, > > > > On Wed, Jul 18, 2018 at 4:08 PM Marcus Pedersén > > wrote: > > > > > > Hi Kotresh, > > > > > > I ran: > > > > > > #ldconfig /usr/lib > > can you do - > > ldconfig /usr/local/lib > > > > > > Output: > > > > > > > > on all nodes in both clusters but I still get the same error. > > > > > > What to do? > > > > > > > > > Output for: > > > > > > # ldconfig -p /usr/lib | grep libgf > > > > > > libgfxdr.so.0 (libc6,x86-64) => /lib64/libgfxdr.so.0 > > > libgfrpc.so.0 (libc6,x86-64) => /lib64/libgfrpc.so.0 > > > libgfdb.so.0 (libc6,x86-64) => /lib64/libgfdb.so.0 > > > libgfchangelog.so.0 (libc6,x86-64) => /lib64/libgfchangelog.so.0 > > > libgfapi.so.0 (libc6,x86-64) => /lib64/libgfapi.so.0 > > > > > > > > > I read somewere that you could change some settings for geo-replication > > > to speed up sync. > > > > > > I can not remember where I saw that and what config parameters. > > > > > > When geo-replication works I have 30TB on master cluster that has to be > > > synced to slave nodes, > > > > > > and that will take a while before the slave nodes have catched up. > > > > > > > > > Thanks and regards > > > > > > Marcus Pedersén > > > > > > > > > Part of gsyncd.log: > > > > > > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in > > > __call__ > > > raise res > > > OSError: libgfchangelog.so: cannot open shared object file: No such file > > > or directory > > > [2018-07-18 10:23:52.305119] I [repce(agent > > > /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching > > > EOF. > > > [2018-07-18 10:23:53.273298] I [monitor(monitor):272:monitor] Monitor: > > > worker died in startup phase brick=/urd-gds/gluster > > > [2018-07-18 10:24:03.294312] I [monitor(monitor):158:monitor] Monitor: > > > starting gsyncd worker brick=/urd-gds/gluster > > > slave_node=urd-gds-geo-000 > > > [2018-07-18 10:24:03.334563] I [gsyncd(agent /urd-gds/gluster):297:main] > > > : Using session config file > > > path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf > > > [2018-07-18 10:24:03.334702] I [gsyncd(worker /urd-gds/gluster):297:main] > > > : Using session config file > > > path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf > > > [2018-07-18 10:24:03.335380] I [changelogagent(agent > > > /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... > > > [2018-07-18 10:24:03.343605] I [resource(worker > > > /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection > > > between maste
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
Hi, Can you confirm the location for libgfchangelog.so by sharing output of following command - # find /usr/ -name libglusterfs.so - Sunny On Mon, Jul 23, 2018 at 5:12 PM Marcus Pedersén wrote: > > Hi Sunny, > Here comes a part of gsyncd.log (The same info is repeated over and over > again): > > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in > __call__ > raise res > OSError: libgfchangelog.so: cannot open shared object file: No such file or > directory > [2018-07-23 11:33:09.254915] I [repce(agent > /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. > [2018-07-23 11:33:10.225150] I [monitor(monitor):272:monitor] Monitor: worker > died in startup phase brick=/urd-gds/gluster > [2018-07-23 11:33:20.250036] I [monitor(monitor):158:monitor] Monitor: > starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 > [2018-07-23 11:33:20.326205] I [gsyncd(agent /urd-gds/gluster):297:main] > : Using session config file > path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf > [2018-07-23 11:33:20.326282] I [gsyncd(worker /urd-gds/gluster):297:main] > : Using session config file > path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf > [2018-07-23 11:33:20.327152] I [changelogagent(agent > /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... > [2018-07-23 11:33:20.335777] I [resource(worker > /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection > between master and slave... > [2018-07-23 11:33:22.11188] I [resource(worker > /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master and > slave established. duration=1.6752 > [2018-07-23 11:33:22.11744] I [resource(worker > /urd-gds/gluster):1067:connect] GLUSTER: Mounting gluster volume locally... > [2018-07-23 11:33:23.101602] I [resource(worker > /urd-gds/gluster):1090:connect] GLUSTER: Mounted gluster volume > duration=1.0894 > [2018-07-23 11:33:23.102168] I [subcmds(worker > /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. > Acknowledging back to monitor > [2018-07-23 11:33:23.119129] E [repce(agent /urd-gds/gluster):114:worker] > : call failed: > Traceback (most recent call last): > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 110, in > worker > res = getattr(self.obj, rmeth)(*in_data[2:]) > File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 37, > in init > return Changes.cl_init() > File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 21, > in __getattr__ > from libgfchangelog import Changes as LChanges > File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 17, > in > class Changes(object): > File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 19, > in Changes > use_errno=True) > File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__ > self._handle = _dlopen(self._name, mode) > OSError: libgfchangelog.so: cannot open shared object file: No such file or > directory > [2018-07-23 11:33:23.119609] E [repce(worker /urd-gds/gluster):206:__call__] > RepceClient: call failed call=29589:140155686246208:1532345603.11 > method=init error=OSError > [2018-07-23 11:33:23.119708] E [syncdutils(worker > /urd-gds/gluster):330:log_raise_exception] : FAIL: > Traceback (most recent call last): > File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, in main > func(args) > File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72, in > subcmd_worker > local.service_loop(remote) > File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1236, in > service_loop > changelog_agent.init() > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in > __call__ > return self.ins(self.meth, *a) > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in > __call__ > raise res > OSError: libgfchangelog.so: cannot open shared object file: No such file or > directory > [2018-07-23 11:33:23.130100] I [repce(agent > /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. > [2018-07-23 11:33:24.104176] I [monitor(monitor):272:monitor] Monitor: worker > died in startup phase brick=/urd-gds/gluster > > Thanks, Sunny!! > > Regards > Marcus Pedersén > > > Från: Sunny Kumar > Skickat: den 23
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
Hi Sunny, Here comes a part of gsyncd.log (The same info is repeated over and over again): File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in __call__ raise res OSError: libgfchangelog.so: cannot open shared object file: No such file or directory [2018-07-23 11:33:09.254915] I [repce(agent /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. [2018-07-23 11:33:10.225150] I [monitor(monitor):272:monitor] Monitor: worker died in startup phase brick=/urd-gds/gluster [2018-07-23 11:33:20.250036] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 [2018-07-23 11:33:20.326205] I [gsyncd(agent /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-07-23 11:33:20.326282] I [gsyncd(worker /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-07-23 11:33:20.327152] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... [2018-07-23 11:33:20.335777] I [resource(worker /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-07-23 11:33:22.11188] I [resource(worker /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master and slave established. duration=1.6752 [2018-07-23 11:33:22.11744] I [resource(worker /urd-gds/gluster):1067:connect] GLUSTER: Mounting gluster volume locally... [2018-07-23 11:33:23.101602] I [resource(worker /urd-gds/gluster):1090:connect] GLUSTER: Mounted gluster volume duration=1.0894 [2018-07-23 11:33:23.102168] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor [2018-07-23 11:33:23.119129] E [repce(agent /urd-gds/gluster):114:worker] : call failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 110, in worker res = getattr(self.obj, rmeth)(*in_data[2:]) File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 37, in init return Changes.cl_init() File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 21, in __getattr__ from libgfchangelog import Changes as LChanges File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 17, in class Changes(object): File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 19, in Changes use_errno=True) File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__ self._handle = _dlopen(self._name, mode) OSError: libgfchangelog.so: cannot open shared object file: No such file or directory [2018-07-23 11:33:23.119609] E [repce(worker /urd-gds/gluster):206:__call__] RepceClient: call failed call=29589:140155686246208:1532345603.11 method=init error=OSError [2018-07-23 11:33:23.119708] E [syncdutils(worker /urd-gds/gluster):330:log_raise_exception] : FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, in main func(args) File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72, in subcmd_worker local.service_loop(remote) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1236, in service_loop changelog_agent.init() File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in __call__ raise res OSError: libgfchangelog.so: cannot open shared object file: No such file or directory [2018-07-23 11:33:23.130100] I [repce(agent /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. [2018-07-23 11:33:24.104176] I [monitor(monitor):272:monitor] Monitor: worker died in startup phase brick=/urd-gds/gluster Thanks, Sunny!! Regards Marcus Pedersén Från: Sunny Kumar Skickat: den 23 juli 2018 12:53 Till: Marcus Pedersén Kopia: Kotresh Hiremath Ravishankar; gluster-users@gluster.org Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work Hi Marcus, On Mon, Jul 23, 2018 at 4:04 PM Marcus Pedersén wrote: > > Hi Sunny, > ldconfig -p /usr/local/lib | grep libgf > Output: > libgfxdr.so.0 (libc6,x86-64) => /lib64/libgfxdr.so.0 > libgfrpc.so.0 (libc6,x86-64) => > /lib64/libgfrpc.so.0 > libgfdb.so.0 (libc6,x86-64) => /lib64/libgfdb.so.0 > libgfchangelog.so.0 (libc6,x86-64) => > /li
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
ct file: No such file or > directory > [2018-07-18 10:24:05.984647] E [repce(worker /urd-gds/gluster):206:__call__] > RepceClient: call failed call=1146:139672481965888:1531909445.98 > method=init error=OSError > [2018-07-18 10:24:05.984747] E [syncdutils(worker > /urd-gds/gluster):330:log_raise_exception] : FAIL: > Traceback (most recent call last): > File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, in main > func(args) > File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72, in > subcmd_worker > local.service_loop(remote) > File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1236, in > service_loop > changelog_agent.init() > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in > __call__ > return self.ins(self.meth, *a) > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in > __call__ > raise res > OSError: libgfchangelog.so: cannot open shared object file: No such file or > directory I think then you will not see this. > [2018-07-18 10:24:05.994826] I [repce(agent > /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. > [2018-07-18 10:24:06.969984] I [monitor(monitor):272:monitor] Monitor: worker > died in startup phase brick=/urd-gds/gluster > > > > Från: Kotresh Hiremath Ravishankar > Skickat: den 18 juli 2018 06:05 > Till: Marcus Pedersén > Kopia: gluster-users@gluster.org > Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work > > Hi Marcus, > > Well there is nothing wrong in setting up a symlink for gluster binary > location, but > there is a geo-rep command to set it so that gsyncd will search there. > > To set on master > #gluster vol geo-rep config gluster-command-dir > > > To set on slave > #gluster vol geo-rep config slave-gluster-command-dir > > > Thanks, > Kotresh HR > > > On Wed, Jul 18, 2018 at 9:28 AM, Kotresh Hiremath Ravishankar > wrote: >> >> Hi Marcus, >> >> I am testing out 4.1 myself and I will have some update today. >> For this particular traceback, gsyncd is not able to find the library. >> Is it the rpm install? If so, gluster libraries would be in /usr/lib. >> Please run the cmd below. >> >> #ldconfig /usr/lib >> #ldconfig -p /usr/lib | grep libgf (This should list libgfchangelog.so) >> >> Geo-rep should be fixed automatically. >> >> Thanks, >> Kotresh HR >> >> On Wed, Jul 18, 2018 at 1:27 AM, Marcus Pedersén >> wrote: >>> >>> Hi again, >>> >>> I continue to do some testing, but now I have come to a stage where I need >>> help. >>> >>> >>> gsyncd.log was complaining about that /usr/local/sbin/gluster was missing >>> so I made a link. >>> >>> After that /usr/local/sbin/glusterfs was missing so I made a link there as >>> well. >>> >>> Both links were done on all slave nodes. >>> >>> >>> Now I have a new error that I can not resolve myself. >>> >>> It can not open libgfchangelog.so >>> >>> >>> Many thanks! >>> >>> Regards >>> >>> Marcus Pedersén >>> >>> >>> Part of gsyncd.log: >>> >>> OSError: libgfchangelog.so: cannot open shared object file: No such file or >>> directory >>> [2018-07-17 19:32:06.517106] I [repce(agent >>> /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. >>> [2018-07-17 19:32:07.479553] I [monitor(monitor):272:monitor] Monitor: >>> worker died in startup phase brick=/urd-gds/gluster >>> [2018-07-17 19:32:17.500709] I [monitor(monitor):158:monitor] Monitor: >>> starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 >>> [2018-07-17 19:32:17.541547] I [gsyncd(agent /urd-gds/gluster):297:main] >>> : Using session config file >>> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf >>> [2018-07-17 19:32:17.541959] I [gsyncd(worker /urd-gds/gluster):297:main] >>> : Using session config file >>> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf >>> [2018-07-17 19:32:17.542363] I [changelogagent(agent >>> /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... >>> [2018-07-17 19:32:17.550894] I [resource(worker >>> /urd-gds/glust
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
hangelog_agent.init() > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in > __call__ > return self.ins(self.meth, *a) > File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in > __call__ > raise res > OSError: libgfchangelog.so: cannot open shared object file: No such file or > directory I think then you will not see this. > [2018-07-18 10:24:05.994826] I [repce(agent > /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. > [2018-07-18 10:24:06.969984] I [monitor(monitor):272:monitor] Monitor: worker > died in startup phase brick=/urd-gds/gluster > > > > Från: Kotresh Hiremath Ravishankar > Skickat: den 18 juli 2018 06:05 > Till: Marcus Pedersén > Kopia: gluster-users@gluster.org > Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work > > Hi Marcus, > > Well there is nothing wrong in setting up a symlink for gluster binary > location, but > there is a geo-rep command to set it so that gsyncd will search there. > > To set on master > #gluster vol geo-rep config gluster-command-dir > > > To set on slave > #gluster vol geo-rep config slave-gluster-command-dir > > > Thanks, > Kotresh HR > > > On Wed, Jul 18, 2018 at 9:28 AM, Kotresh Hiremath Ravishankar > wrote: >> >> Hi Marcus, >> >> I am testing out 4.1 myself and I will have some update today. >> For this particular traceback, gsyncd is not able to find the library. >> Is it the rpm install? If so, gluster libraries would be in /usr/lib. >> Please run the cmd below. >> >> #ldconfig /usr/lib >> #ldconfig -p /usr/lib | grep libgf (This should list libgfchangelog.so) >> >> Geo-rep should be fixed automatically. >> >> Thanks, >> Kotresh HR >> >> On Wed, Jul 18, 2018 at 1:27 AM, Marcus Pedersén >> wrote: >>> >>> Hi again, >>> >>> I continue to do some testing, but now I have come to a stage where I need >>> help. >>> >>> >>> gsyncd.log was complaining about that /usr/local/sbin/gluster was missing >>> so I made a link. >>> >>> After that /usr/local/sbin/glusterfs was missing so I made a link there as >>> well. >>> >>> Both links were done on all slave nodes. >>> >>> >>> Now I have a new error that I can not resolve myself. >>> >>> It can not open libgfchangelog.so >>> >>> >>> Many thanks! >>> >>> Regards >>> >>> Marcus Pedersén >>> >>> >>> Part of gsyncd.log: >>> >>> OSError: libgfchangelog.so: cannot open shared object file: No such file or >>> directory >>> [2018-07-17 19:32:06.517106] I [repce(agent >>> /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. >>> [2018-07-17 19:32:07.479553] I [monitor(monitor):272:monitor] Monitor: >>> worker died in startup phase brick=/urd-gds/gluster >>> [2018-07-17 19:32:17.500709] I [monitor(monitor):158:monitor] Monitor: >>> starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 >>> [2018-07-17 19:32:17.541547] I [gsyncd(agent /urd-gds/gluster):297:main] >>> : Using session config file >>> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf >>> [2018-07-17 19:32:17.541959] I [gsyncd(worker /urd-gds/gluster):297:main] >>> : Using session config file >>> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf >>> [2018-07-17 19:32:17.542363] I [changelogagent(agent >>> /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... >>> [2018-07-17 19:32:17.550894] I [resource(worker >>> /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection >>> between master and slave... >>> [2018-07-17 19:32:19.166246] I [resource(worker >>> /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master >>> and slave established.duration=1.6151 >>> [2018-07-17 19:32:19.166806] I [resource(worker >>> /urd-gds/gluster):1067:connect] GLUSTER: Mounting gluster volume locally... >>> [2018-07-17 19:32:20.257344] I [resource(worker >>> /urd-gds/gluster):1090:connect] GLUSTER: Mounted gluster volume >>> duration=1.0901 >>> [2018-07-17 19:32:20.257921] I [subcmds(worker >>> /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. >
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
Hi again, I continue to do some testing, but now I have come to a stage where I need help. gsyncd.log was complaining about that /usr/local/sbin/gluster was missing so I made a link. After that /usr/local/sbin/glusterfs was missing so I made a link there as well. Both links were done on all slave nodes. Now I have a new error that I can not resolve myself. It can not open libgfchangelog.so Many thanks! Regards Marcus Pedersén Part of gsyncd.log: OSError: libgfchangelog.so: cannot open shared object file: No such file or directory [2018-07-17 19:32:06.517106] I [repce(agent /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. [2018-07-17 19:32:07.479553] I [monitor(monitor):272:monitor] Monitor: worker died in startup phase brick=/urd-gds/gluster [2018-07-17 19:32:17.500709] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 [2018-07-17 19:32:17.541547] I [gsyncd(agent /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-07-17 19:32:17.541959] I [gsyncd(worker /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-07-17 19:32:17.542363] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... [2018-07-17 19:32:17.550894] I [resource(worker /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-07-17 19:32:19.166246] I [resource(worker /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master and slave established.duration=1.6151 [2018-07-17 19:32:19.166806] I [resource(worker /urd-gds/gluster):1067:connect] GLUSTER: Mounting gluster volume locally... [2018-07-17 19:32:20.257344] I [resource(worker /urd-gds/gluster):1090:connect] GLUSTER: Mounted gluster volume duration=1.0901 [2018-07-17 19:32:20.257921] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor [2018-07-17 19:32:20.274647] E [repce(agent /urd-gds/gluster):114:worker] : call failed: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 110, in worker res = getattr(self.obj, rmeth)(*in_data[2:]) File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 37, in init return Changes.cl_init() File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 21, in __getattr__ from libgfchangelog import Changes as LChanges File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 17, in class Changes(object): File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 19, in Changes use_errno=True) File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__ self._handle = _dlopen(self._name, mode) OSError: libgfchangelog.so: cannot open shared object file: No such file or directory [2018-07-17 19:32:20.275093] E [repce(worker /urd-gds/gluster):206:__call__] RepceClient: call failed call=6078:139982918485824:1531855940.27 method=init error=OSError [2018-07-17 19:32:20.275192] E [syncdutils(worker /urd-gds/gluster):330:log_raise_exception] : FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, in main func(args) File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72, in subcmd_worker local.service_loop(remote) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1236, in service_loop changelog_agent.init() File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in __call__ raise res OSError: libgfchangelog.so: cannot open shared object file: No such file or directory [2018-07-17 19:32:20.286787] I [repce(agent /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. [2018-07-17 19:32:21.259891] I [monitor(monitor):272:monitor] Monitor: worker died in startup phase brick=/urd-gds/gluster Från: gluster-users-boun...@gluster.org för Marcus Pedersén Skickat: den 16 juli 2018 21:59 Till: khire...@redhat.com Kopia: gluster-users@gluster.org Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work Hi Kotresh, I have been testing for a bit and as you can see from the logs I sent before permission is denied for geouser on slave node on file: /var/log/glusterfs/cli.log I have turned selinux off and just for testing I changed permissions on /var/log/glusterfs/cli.log so geouser
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
;/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in __call__ raise res OSError: libgfchangelog.so: cannot open shared object file: No such file or directory [2018-07-17 19:32:20.286787] I [repce(agent /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. [2018-07-17 19:32:21.259891] I [monitor(monitor):272:monitor] Monitor: worker died in startup phase brick=/urd-gds/gluster Från: gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> mailto:gluster-users-boun...@gluster.org>> för Marcus Pedersén mailto:marcus.peder...@slu.se>> Skickat: den 16 juli 2018 21:59 Till: khire...@redhat.com<mailto:khire...@redhat.com> Kopia: gluster-users@gluster.org<mailto:gluster-users@gluster.org> Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work Hi Kotresh, I have been testing for a bit and as you can see from the logs I sent before permission is denied for geouser on slave node on file: /var/log/glusterfs/cli.log I have turned selinux off and just for testing I changed permissions on /var/log/glusterfs/cli.log so geouser can access it. Starting geo-replication after that gives response successful but all nodes get status Faulty. If I run: gluster-mountbroker status I get: +-+-+---+--+--+ | NODE| NODE STATUS | MOUNT ROOT| GROUP | USERS | +-+-+---+--+--+ | urd-gds-geo-001.hgen.slu.se<http://urd-gds-geo-001.hgen.slu.se> | UP | /var/mountbroker-root(OK) | geogroup(OK) | geouser(urd-gds-volume) | | urd-gds-geo-002 | UP | /var/mountbroker-root(OK) | geogroup(OK) | geouser(urd-gds-volume) | | localhost | UP | /var/mountbroker-root(OK) | geogroup(OK) | geouser(urd-gds-volume) | +-+-+---+--+--+ and that is all nodes on slave cluster, so mountbroker seems ok. gsyncd.log logs an error about /usr/local/sbin/gluster is missing. That is correct cos gluster is in /sbin/gluster and /urs/sbin/gluster Another error is that SSH between master and slave is broken, but now when I have changed permission on /var/log/glusterfs/cli.log I can run: ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 geouser@urd-gds-geo-001 gluster --xml --remote-host=localhost volume info urd-gds-volume as geouser and that works, which means that the ssh connection works. Is the permissions on /var/log/glusterfs/cli.log changed when geo-replication is setup? Is gluster supposed to be in /usr/local/sbin/gluster? Do I have any options or should I remove current geo-replication and create a new? How much do I need to clean up before creating a new geo-replication? In that case can I pause geo-replication, mount slave cluster on master cluster and run rsync , just to speed up transfer of files? Many thanks in advance! Marcus Pedersén Part from the gsyncd.log: [2018-07-16 19:34:56.26287] E [syncdutils(worker /urd-gds/gluster):749:errlog] Popen: command returned errorcmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replicatio\ n/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-WrbZ22/bf60c68f1a195dad59573a8dbaa309f2.sock geouser@urd-gds-geo-001 /nonexistent/gsyncd slave urd-gds-volume geouser@urd-gds-geo-001::urd-gds-volu\ me --master-node urd-gds-001 --master-node-id 912bebfd-1a7f-44dc-b0b7-f001a20d58cd --master-brick /urd-gds/gluster --local-node urd-gds-geo-000 --local-node-id 03075698-2bbf-43e4-a99a-65fe82f61794 --slave-timeo\ ut 120 --slave-log-level INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/local/sbin/ error=1 [2018-07-16 19:34:56.26583] E [syncdutils(worker /urd-gds/gluster):753:logerr] Popen: ssh> failure: execution of "/usr/local/sbin/gluster" failed with ENOENT (No such file or directory) [2018-07-16 19:34:56.33901] I [repce(agent /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. [2018-07-16 19:34:56.34307] I [monitor(monitor):262:monitor] Monitor: worker died before establishing connectionbrick=/urd-gds/gluster [2018-07-16 19:35:06.59412] I [monitor(monitor):158:monitor] Monitor: starting gsyncd workerbrick=/urd-gds/gluster slave_node=urd-gds-geo-000 [2018-07-16 19:35:06.99509] I [gsyncd(worker /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
led >> call=6078:139982918485824:1531855940.27 method=init error=OSError >> [2018-07-17 19:32:20.275192] E [syncdutils(worker >> /urd-gds/gluster):330:log_raise_exception] : FAIL: >> Traceback (most recent call last): >> File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, >> in main >> func(args) >> File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72, >> in subcmd_worker >> local.service_loop(remote) >> File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line >> 1236, in service_loop >> changelog_agent.init() >> File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in >> __call__ >> return self.ins(self.meth, *a) >> File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in >> __call__ >> raise res >> OSError: libgfchangelog.so: cannot open shared object file: No such file >> or directory >> [2018-07-17 19:32:20.286787] I [repce(agent >> /urd-gds/gluster):89:service_loop] >> RepceServer: terminating on reaching EOF. >> [2018-07-17 19:32:21.259891] I [monitor(monitor):272:monitor] Monitor: >> worker died in startup phase brick=/urd-gds/gluster >> >> >> >> -- >> *Från:* gluster-users-boun...@gluster.org > .org> för Marcus Pedersén >> *Skickat:* den 16 juli 2018 21:59 >> *Till:* khire...@redhat.com >> >> *Kopia:* gluster-users@gluster.org >> *Ämne:* Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not >> work >> >> >> Hi Kotresh, >> >> I have been testing for a bit and as you can see from the logs I sent >> before permission is denied for geouser on slave node on file: >> >> /var/log/glusterfs/cli.log >> >> I have turned selinux off and just for testing I changed permissions on >> /var/log/glusterfs/cli.log so geouser can access it. >> >> Starting geo-replication after that gives response successful but all >> nodes get status Faulty. >> >> >> If I run: gluster-mountbroker status >> >> I get: >> >> +-+-+--- >> +--+--+ >> | NODE| NODE STATUS | MOUNT ROOT >> |GROUP | USERS | >> +-+-+--- >> +--+--+ >> | urd-gds-geo-001.hgen.slu.se | UP | /var/mountbroker-root(OK) >> | geogroup(OK) | geouser(urd-gds-volume) | >> | urd-gds-geo-002 | UP | /var/mountbroker-root(OK) | >> geogroup(OK) | geouser(urd-gds-volume) | >> | localhost | UP | /var/mountbroker-root(OK) | >> geogroup(OK) | geouser(urd-gds-volume) | >> +-+-+--- >> +--+--+ >> >> >> and that is all nodes on slave cluster, so mountbroker seems ok. >> >> >> gsyncd.log logs an error about /usr/local/sbin/gluster is missing. >> >> That is correct cos gluster is in /sbin/gluster and /urs/sbin/gluster >> >> Another error is that SSH between master and slave is broken, >> >> but now when I have changed permission on /var/log/glusterfs/cli.log I >> can run: >> >> ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i >> /var/lib/glusterd/geo-replication/secret.pem -p 22 >> geouser@urd-gds-geo-001 gluster --xml --remote-host=localhost volume >> info urd-gds-volume >> >> as geouser and that works, which means that the ssh connection works. >> >> >> Is the permissions on /var/log/glusterfs/cli.log changed when >> geo-replication is setup? >> >> Is gluster supposed to be in /usr/local/sbin/gluster? >> >> >> Do I have any options or should I remove current geo-replication and >> create a new? >> >> How much do I need to clean up before creating a new geo-replication? >> >> In that case can I pause geo-replication, mount slave cluster on master >> cluster and run rsync , just to speed up transfer of files? >> >> >> Many thanks in advance! >> >> Marcus Pedersén >> >> >> Part from the gsyncd.log: >> >> [2018-07-16 19:34:56.26287] E [syncdutils(worker >> /urd-gds/gluster):749:errlog] Popen: command returned errorcmd=ssh >> -oPasswordAuthentication=no -oStrictHostK
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
hon/syncdaemon/repce.py", line 207, in > __call__ > raise res > OSError: libgfchangelog.so: cannot open shared object file: No such file > or directory > [2018-07-17 19:32:20.286787] I [repce(agent /urd-gds/gluster):89:service_loop] > RepceServer: terminating on reaching EOF. > [2018-07-17 19:32:21.259891] I [monitor(monitor):272:monitor] Monitor: > worker died in startup phase brick=/urd-gds/gluster > > > > -- > *Från:* gluster-users-boun...@gluster.org gluster.org> för Marcus Pedersén > *Skickat:* den 16 juli 2018 21:59 > *Till:* khire...@redhat.com > > *Kopia:* gluster-users@gluster.org > *Ämne:* Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work > > > Hi Kotresh, > > I have been testing for a bit and as you can see from the logs I sent > before permission is denied for geouser on slave node on file: > > /var/log/glusterfs/cli.log > > I have turned selinux off and just for testing I changed permissions on > /var/log/glusterfs/cli.log so geouser can access it. > > Starting geo-replication after that gives response successful but all > nodes get status Faulty. > > > If I run: gluster-mountbroker status > > I get: > > +-+-+--- > +--+--+ > | NODE| NODE STATUS | MOUNT ROOT > |GROUP | USERS | > +-+-+--- > +--+--+ > | urd-gds-geo-001.hgen.slu.se | UP | /var/mountbroker-root(OK) | > geogroup(OK) | geouser(urd-gds-volume) | > | urd-gds-geo-002 | UP | /var/mountbroker-root(OK) | > geogroup(OK) | geouser(urd-gds-volume) | > | localhost | UP | /var/mountbroker-root(OK) | > geogroup(OK) | geouser(urd-gds-volume) | > +-+-+--- > +--+--+ > > > and that is all nodes on slave cluster, so mountbroker seems ok. > > > gsyncd.log logs an error about /usr/local/sbin/gluster is missing. > > That is correct cos gluster is in /sbin/gluster and /urs/sbin/gluster > > Another error is that SSH between master and slave is broken, > > but now when I have changed permission on /var/log/glusterfs/cli.log I can > run: > > ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i > /var/lib/glusterd/geo-replication/secret.pem -p 22 geouser@urd-gds-geo-001 > gluster --xml --remote-host=localhost volume info urd-gds-volume > > as geouser and that works, which means that the ssh connection works. > > > Is the permissions on /var/log/glusterfs/cli.log changed when > geo-replication is setup? > > Is gluster supposed to be in /usr/local/sbin/gluster? > > > Do I have any options or should I remove current geo-replication and > create a new? > > How much do I need to clean up before creating a new geo-replication? > > In that case can I pause geo-replication, mount slave cluster on master > cluster and run rsync , just to speed up transfer of files? > > > Many thanks in advance! > > Marcus Pedersén > > > Part from the gsyncd.log: > > [2018-07-16 19:34:56.26287] E [syncdutils(worker > /urd-gds/gluster):749:errlog] Popen: command returned errorcmd=ssh > -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i > /var/lib/glusterd/geo-replicatio\ > n/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-WrbZ22/ > bf60c68f1a195dad59573a8dbaa309f2.sock geouser@urd-gds-geo-001 > /nonexistent/gsyncd slave urd-gds-volume geouser@urd-gds-geo-001::urd- > gds-volu\ > me --master-node urd-gds-001 --master-node-id > 912bebfd-1a7f-44dc-b0b7-f001a20d58cd > --master-brick /urd-gds/gluster --local-node urd-gds-geo-000 > --local-node-id 03075698-2bbf-43e4-a99a-65fe82f61794 --slave-timeo\ > ut 120 --slave-log-level INFO --slave-gluster-log-level INFO > --slave-gluster-command-dir /usr/local/sbin/ error=1 > [2018-07-16 19:34:56.26583] E [syncdutils(worker > /urd-gds/gluster):753:logerr] Popen: ssh> failure: execution of > "/usr/local/sbin/gluster" failed with ENOENT (No such file or directory) > [2018-07-16 19:34:56.33901] I [repce(agent /urd-gds/gluster):89:service_loop] > RepceServer: terminating on reaching EOF. > [2018-07-16 19:34:56.34307] I [monitor(monitor):262:monitor] Monitor: > worker died before establishing connectionbrick=/urd-gds/gluster > [2018-07-16 19:35:06.59412] I [monitor(monitor):158:monitor] Monitor: > starting gsyncd workerbrick=/urd-gds/gluster slave_node=urd-gds-geo-000 > [2018-07-16 19:35:06.99509] I [gsyncd(
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
-gds-volu\ me --master-node urd-gds-001 --master-node-id 912bebfd-1a7f-44dc-b0b7-f001a20d58cd --master-brick /urd-gds/gluster --local-node urd-gds-geo-000 --local-node-id 03075698-2bbf-43e4-a99a-65fe82f61794 --slave-timeo\ ut 120 --slave-log-level INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/local/sbin/ error=1 [2018-07-16 19:35:06.763398] E [syncdutils(worker /urd-gds/gluster):753:logerr] Popen: ssh> failure: execution of "/usr/local/sbin/gluster" failed with ENOENT (No such file or directory) [2018-07-16 19:35:06.771905] I [repce(agent /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF. [2018-07-16 19:35:06.772272] I [monitor(monitor):262:monitor] Monitor: worker died before establishing connection brick=/urd-gds/gluster [2018-07-16 19:35:16.786387] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 [2018-07-16 19:35:16.828056] I [gsyncd(worker /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-07-16 19:35:16.828066] I [gsyncd(agent /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-07-16 19:35:16.828912] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... [2018-07-16 19:35:16.837100] I [resource(worker /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-07-16 19:35:17.260257] E [syncdutils(worker /urd-gds/gluster):303:log_raise_exception] : connection to peer is broken Från: gluster-users-boun...@gluster.org för Marcus Pedersén Skickat: den 13 juli 2018 14:50 Till: Kotresh Hiremath Ravishankar Kopia: gluster-users@gluster.org Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work Hi Kotresh, Yes, all nodes have the same version 4.1.1 both master and slave. All glusterd are crashing on the master side. Will send logs tonight. Thanks, Marcus Marcus Pedersén Systemadministrator Interbull Centre Sent from my phone Den 13 juli 2018 11:28 skrev Kotresh Hiremath Ravishankar : Hi Marcus, Is the gluster geo-rep version is same on both master and slave? Thanks, Kotresh HR On Fri, Jul 13, 2018 at 1:26 AM, Marcus Pedersén mailto:marcus.peder...@slu.se>> wrote: Hi Kotresh, i have replaced both files (gsyncdconfig.py<https://review.gluster.org/#/c/20207/1/geo-replication/syncdaemon/gsyncdconfig.py> and repce.py<https://review.gluster.org/#/c/20207/1/geo-replication/syncdaemon/repce.py>) in all nodes both master and slave. I rebooted all servers but geo-replication status is still Stopped. I tried to start geo-replication with response Successful but status still show Stopped on all nodes. Nothing has been written to geo-replication logs since I sent the tail of the log. So I do not know what info to provide? Please, help me to find a way to solve this. Thanks! Regards Marcus Från: gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org> mailto:gluster-users-boun...@gluster.org>> för Marcus Pedersén mailto:marcus.peder...@slu.se>> Skickat: den 12 juli 2018 08:51 Till: Kotresh Hiremath Ravishankar Kopia: gluster-users@gluster.org<mailto:gluster-users@gluster.org> Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work Thanks Kotresh, I installed through the official centos channel, centos-release-gluster41. Isn't this fix included in centos install? I will have a look, test it tonight and come back to you! Thanks a lot! Regards Marcus Marcus Pedersén Systemadministrator Interbull Centre Sent from my phone Den 12 juli 2018 07:41 skrev Kotresh Hiremath Ravishankar mailto:khire...@redhat.com>>: Hi Marcus, I think the fix [1] is needed in 4.1 Could you please this out and let us know if that works for you? [1] https://review.gluster.org/#/c/20207/ Thanks, Kotresh HR On Thu, Jul 12, 2018 at 1:49 AM, Marcus Pedersén mailto:marcus.peder...@slu.se>> wrote: Hi all, I have upgraded from 3.12.9 to 4.1.1 and been following upgrade instructions for offline upgrade. I upgraded geo-replication side first 1 x (2+1) and the master side after that 2 x (2+1). Both clusters works the way they should on their own. After upgrade on master side status for all geo-replication nodes is Stopped. I tried to start the geo-replication from master node and response back was started successfully. Status again Stopped Tried to start again and get response started successfully, after that all glusterd crashed on all master nodes. After a restart of al
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
Hi Kotresh,
Yes, all nodes have the same version 4.1.1 both master and slave.
All glusterd are crashing on the master side.
Will send logs tonight.
Thanks,
Marcus
Marcus Pedersén
Systemadministrator
Interbull Centre
Sent from my phone
Den 13 juli 2018 11:28 skrev Kotresh Hiremath Ravishankar :
Hi Marcus,
Is the gluster geo-rep version is same on both master and slave?
Thanks,
Kotresh HR
On Fri, Jul 13, 2018 at 1:26 AM, Marcus Pedersén
mailto:marcus.peder...@slu.se>> wrote:
Hi Kotresh,
i have replaced both files
(gsyncdconfig.py<https://review.gluster.org/#/c/20207/1/geo-replication/syncdaemon/gsyncdconfig.py>
and
repce.py<https://review.gluster.org/#/c/20207/1/geo-replication/syncdaemon/repce.py>)
in all nodes both master and slave.
I rebooted all servers but geo-replication status is still Stopped.
I tried to start geo-replication with response Successful but status still show
Stopped on all nodes.
Nothing has been written to geo-replication logs since I sent the tail of the
log.
So I do not know what info to provide?
Please, help me to find a way to solve this.
Thanks!
Regards
Marcus
Från:
gluster-users-boun...@gluster.org<mailto:gluster-users-boun...@gluster.org>
mailto:gluster-users-boun...@gluster.org>>
för Marcus Pedersén mailto:marcus.peder...@slu.se>>
Skickat: den 12 juli 2018 08:51
Till: Kotresh Hiremath Ravishankar
Kopia: gluster-users@gluster.org<mailto:gluster-users@gluster.org>
Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
Thanks Kotresh,
I installed through the official centos channel, centos-release-gluster41.
Isn't this fix included in centos install?
I will have a look, test it tonight and come back to you!
Thanks a lot!
Regards
Marcus
Marcus Pedersén
Systemadministrator
Interbull Centre
Sent from my phone
Den 12 juli 2018 07:41 skrev Kotresh Hiremath Ravishankar
mailto:khire...@redhat.com>>:
Hi Marcus,
I think the fix [1] is needed in 4.1
Could you please this out and let us know if that works for you?
[1] https://review.gluster.org/#/c/20207/
Thanks,
Kotresh HR
On Thu, Jul 12, 2018 at 1:49 AM, Marcus Pedersén
mailto:marcus.peder...@slu.se>> wrote:
Hi all,
I have upgraded from 3.12.9 to 4.1.1 and been following upgrade instructions
for offline upgrade.
I upgraded geo-replication side first 1 x (2+1) and the master side after that
2 x (2+1).
Both clusters works the way they should on their own.
After upgrade on master side status for all geo-replication nodes is Stopped.
I tried to start the geo-replication from master node and response back was
started successfully.
Status again Stopped
Tried to start again and get response started successfully, after that all
glusterd crashed on all master nodes.
After a restart of all glusterd the master cluster was up again.
Status for geo-replication is still Stopped and every try to start it after
this gives the response successful but still status Stopped.
Please help me get the geo-replication up and running again.
Best regards
Marcus Pedersén
Part of geo-replication log from master node:
[2018-07-11 18:42:48.941760] I [changelogagent(/urd-gds/gluster):73:__init__]
ChangelogAgent: Agent listining...
[2018-07-11 18:42:48.947567] I [resource(/urd-gds/gluster):1780:connect_remote]
SSH: Initializing SSH connection between master and slave...
[2018-07-11 18:42:49.363514] E
[syncdutils(/urd-gds/gluster):304:log_raise_exception] : connection to
peer is broken
[2018-07-11 18:42:49.364279] E [resource(/urd-gds/gluster):210:errlog] Popen:
command returned errorcmd=ssh -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret\
.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-hjRhBo/7e5534547f3675a710a107722317484f.sock
geouser@urd-gds-geo-000 /nonexistent/gsyncd --session-owner
5e94eb7d-219f-4741-a179-d4ae6b50c7ee --local-id .%\
2Furd-gds%2Fgluster --local-node urd-gds-001 -N --listen --timeout 120
gluster://localhost:urd-gds-volume error=2
[2018-07-11 18:42:49.364586] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh> usage: gsyncd.py [-h]
[2018-07-11 18:42:49.364799] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh>
[2018-07-11 18:42:49.364989] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh>
{monitor-status,monitor,worker,agent,slave,status,config-check,config-get,config-set,config-reset,voluuidget,d\
elete}
[2018-07-11 18:42:49.365210] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh> ...
[2018-07-11 18:42:49.365408] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh> gsyncd.py: error: argument subcmd: invalid choice:
'5e94eb7d-219f-4741-a179-d4ae6b50c7ee' (choose from 'monitor-status', 'monit\
or', 'worker', 'agent', 'slave',
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
Hi Marcus,
Is the gluster geo-rep version is same on both master and slave?
Thanks,
Kotresh HR
On Fri, Jul 13, 2018 at 1:26 AM, Marcus Pedersén
wrote:
> Hi Kotresh,
>
> i have replaced both files (gsyncdconfig.py
> <https://review.gluster.org/#/c/20207/1/geo-replication/syncdaemon/gsyncdconfig.py>
> and repce.py
> <https://review.gluster.org/#/c/20207/1/geo-replication/syncdaemon/repce.py>)
> in all nodes both master and slave.
>
> I rebooted all servers but geo-replication status is still Stopped.
>
> I tried to start geo-replication with response Successful but status still
> show Stopped on all nodes.
>
> Nothing has been written to geo-replication logs since I sent the tail of
> the log.
>
> So I do not know what info to provide?
>
>
> Please, help me to find a way to solve this.
>
>
> Thanks!
>
>
> Regards
>
> Marcus
>
>
> --
> *Från:* gluster-users-boun...@gluster.org gluster.org> för Marcus Pedersén
> *Skickat:* den 12 juli 2018 08:51
> *Till:* Kotresh Hiremath Ravishankar
> *Kopia:* gluster-users@gluster.org
> *Ämne:* Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
>
> Thanks Kotresh,
> I installed through the official centos channel, centos-release-gluster41.
> Isn't this fix included in centos install?
> I will have a look, test it tonight and come back to you!
>
> Thanks a lot!
>
> Regards
> Marcus
>
>
> Marcus Pedersén
> Systemadministrator
> Interbull Centre
>
> Sent from my phone
>
>
> Den 12 juli 2018 07:41 skrev Kotresh Hiremath Ravishankar <
> khire...@redhat.com>:
>
> Hi Marcus,
>
> I think the fix [1] is needed in 4.1
> Could you please this out and let us know if that works for you?
>
> [1] https://review.gluster.org/#/c/20207/
>
> Thanks,
> Kotresh HR
>
> On Thu, Jul 12, 2018 at 1:49 AM, Marcus Pedersén
> wrote:
>
> Hi all,
>
> I have upgraded from 3.12.9 to 4.1.1 and been following upgrade
> instructions for offline upgrade.
>
> I upgraded geo-replication side first 1 x (2+1) and the master side after
> that 2 x (2+1).
>
> Both clusters works the way they should on their own.
>
> After upgrade on master side status for all geo-replication nodes
> is Stopped.
>
> I tried to start the geo-replication from master node and response back
> was started successfully.
>
> Status again Stopped
>
> Tried to start again and get response started successfully, after that all
> glusterd crashed on all master nodes.
>
> After a restart of all glusterd the master cluster was up again.
>
> Status for geo-replication is still Stopped and every try to start it
> after this gives the response successful but still status Stopped.
>
>
> Please help me get the geo-replication up and running again.
>
>
> Best regards
>
> Marcus Pedersén
>
>
> Part of geo-replication log from master node:
>
> [2018-07-11 18:42:48.941760] I [changelogagent(/urd-gds/gluster):73:__init__]
> ChangelogAgent: Agent listining...
> [2018-07-11 18:42:48.947567] I
> [resource(/urd-gds/gluster):1780:connect_remote]
> SSH: Initializing SSH connection between master and slave...
> [2018-07-11 18:42:49.363514] E
> [syncdutils(/urd-gds/gluster):304:log_raise_exception]
> : connection to peer is broken
> [2018-07-11 18:42:49.364279] E [resource(/urd-gds/gluster):210:errlog]
> Popen: command returned errorcmd=ssh -oPasswordAuthentication=no
> -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret\
> .pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-hjRhBo/7e5
> 534547f3675a710a107722317484f.sock geouser@urd-gds-geo-000
> /nonexistent/gsyncd --session-owner 5e94eb7d-219f-4741-a179-d4ae6b50c7ee
> --local-id .%\
> 2Furd-gds%2Fgluster --local-node urd-gds-001 -N --listen --timeout 120
> gluster://localhost:urd-gds-volume error=2
> [2018-07-11 18:42:49.364586] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh> usage: gsyncd.py [-h]
> [2018-07-11 18:42:49.364799] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh>
> [2018-07-11 18:42:49.364989] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh> {monitor-status,monitor,worker
> ,agent,slave,status,config-check,config-get,config-set,
> config-reset,voluuidget,d\
> elete}
> [2018-07-11 18:42:49.365210] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh> ...
> [2018-07-11 18:42:49.365408] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh> gsyncd.py: error: argument subcmd: invalid choice:
> '5e94eb7d-219f-4741-a179-d4ae6b50c7ee' (choose from 'monitor-status',
>
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
Hi Kotresh,
i have replaced both files
(gsyncdconfig.py<https://review.gluster.org/#/c/20207/1/geo-replication/syncdaemon/gsyncdconfig.py>
and
repce.py<https://review.gluster.org/#/c/20207/1/geo-replication/syncdaemon/repce.py>)
in all nodes both master and slave.
I rebooted all servers but geo-replication status is still Stopped.
I tried to start geo-replication with response Successful but status still show
Stopped on all nodes.
Nothing has been written to geo-replication logs since I sent the tail of the
log.
So I do not know what info to provide?
Please, help me to find a way to solve this.
Thanks!
Regards
Marcus
Från: gluster-users-boun...@gluster.org för
Marcus Pedersén
Skickat: den 12 juli 2018 08:51
Till: Kotresh Hiremath Ravishankar
Kopia: gluster-users@gluster.org
Ämne: Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
Thanks Kotresh,
I installed through the official centos channel, centos-release-gluster41.
Isn't this fix included in centos install?
I will have a look, test it tonight and come back to you!
Thanks a lot!
Regards
Marcus
Marcus Pedersén
Systemadministrator
Interbull Centre
Sent from my phone
Den 12 juli 2018 07:41 skrev Kotresh Hiremath Ravishankar :
Hi Marcus,
I think the fix [1] is needed in 4.1
Could you please this out and let us know if that works for you?
[1] https://review.gluster.org/#/c/20207/
Thanks,
Kotresh HR
On Thu, Jul 12, 2018 at 1:49 AM, Marcus Pedersén
mailto:marcus.peder...@slu.se>> wrote:
Hi all,
I have upgraded from 3.12.9 to 4.1.1 and been following upgrade instructions
for offline upgrade.
I upgraded geo-replication side first 1 x (2+1) and the master side after that
2 x (2+1).
Both clusters works the way they should on their own.
After upgrade on master side status for all geo-replication nodes is Stopped.
I tried to start the geo-replication from master node and response back was
started successfully.
Status again Stopped
Tried to start again and get response started successfully, after that all
glusterd crashed on all master nodes.
After a restart of all glusterd the master cluster was up again.
Status for geo-replication is still Stopped and every try to start it after
this gives the response successful but still status Stopped.
Please help me get the geo-replication up and running again.
Best regards
Marcus Pedersén
Part of geo-replication log from master node:
[2018-07-11 18:42:48.941760] I [changelogagent(/urd-gds/gluster):73:__init__]
ChangelogAgent: Agent listining...
[2018-07-11 18:42:48.947567] I [resource(/urd-gds/gluster):1780:connect_remote]
SSH: Initializing SSH connection between master and slave...
[2018-07-11 18:42:49.363514] E
[syncdutils(/urd-gds/gluster):304:log_raise_exception] : connection to
peer is broken
[2018-07-11 18:42:49.364279] E [resource(/urd-gds/gluster):210:errlog] Popen:
command returned errorcmd=ssh -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret\
.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-hjRhBo/7e5534547f3675a710a107722317484f.sock
geouser@urd-gds-geo-000 /nonexistent/gsyncd --session-owner
5e94eb7d-219f-4741-a179-d4ae6b50c7ee --local-id .%\
2Furd-gds%2Fgluster --local-node urd-gds-001 -N --listen --timeout 120
gluster://localhost:urd-gds-volume error=2
[2018-07-11 18:42:49.364586] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh> usage: gsyncd.py [-h]
[2018-07-11 18:42:49.364799] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh>
[2018-07-11 18:42:49.364989] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh>
{monitor-status,monitor,worker,agent,slave,status,config-check,config-get,config-set,config-reset,voluuidget,d\
elete}
[2018-07-11 18:42:49.365210] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh> ...
[2018-07-11 18:42:49.365408] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh> gsyncd.py: error: argument subcmd: invalid choice:
'5e94eb7d-219f-4741-a179-d4ae6b50c7ee' (choose from 'monitor-status', 'monit\
or', 'worker', 'agent', 'slave', 'status', 'config-check', 'config-get',
'config-set', 'config-reset', 'voluuidget', 'delete')
[2018-07-11 18:42:49.365919] I [syncdutils(/urd-gds/gluster):271:finalize]
: exiting.
[2018-07-11 18:42:49.369316] I [repce(/urd-gds/gluster):92:service_loop]
RepceServer: terminating on reaching EOF.
[2018-07-11 18:42:49.369921] I [syncdutils(/urd-gds/gluster):271:finalize]
: exiting.
[2018-07-11 18:42:49.369694] I [monitor(monitor):353:monitor] Monitor: worker
died before establishing connection brick=/urd-gds/gluster
[2018-07-11 18:42:59.492762] I [monitor(monitor):280:monitor] Monitor: starting
gsyncd worker brick=/urd-gds/gluster
slave_node=ssh://geouser@urd-gds-geo-000:gluster://localhost:urd-gds-volume
[2018-07-11 18:42:59.558491] I
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
Thanks Kotresh,
I installed through the official centos channel, centos-release-gluster41.
Isn't this fix included in centos install?
I will have a look, test it tonight and come back to you!
Thanks a lot!
Regards
Marcus
Marcus Pedersén
Systemadministrator
Interbull Centre
Sent from my phone
Den 12 juli 2018 07:41 skrev Kotresh Hiremath Ravishankar :
Hi Marcus,
I think the fix [1] is needed in 4.1
Could you please this out and let us know if that works for you?
[1] https://review.gluster.org/#/c/20207/
Thanks,
Kotresh HR
On Thu, Jul 12, 2018 at 1:49 AM, Marcus Pedersén
mailto:marcus.peder...@slu.se>> wrote:
Hi all,
I have upgraded from 3.12.9 to 4.1.1 and been following upgrade instructions
for offline upgrade.
I upgraded geo-replication side first 1 x (2+1) and the master side after that
2 x (2+1).
Both clusters works the way they should on their own.
After upgrade on master side status for all geo-replication nodes is Stopped.
I tried to start the geo-replication from master node and response back was
started successfully.
Status again Stopped
Tried to start again and get response started successfully, after that all
glusterd crashed on all master nodes.
After a restart of all glusterd the master cluster was up again.
Status for geo-replication is still Stopped and every try to start it after
this gives the response successful but still status Stopped.
Please help me get the geo-replication up and running again.
Best regards
Marcus Pedersén
Part of geo-replication log from master node:
[2018-07-11 18:42:48.941760] I [changelogagent(/urd-gds/gluster):73:__init__]
ChangelogAgent: Agent listining...
[2018-07-11 18:42:48.947567] I [resource(/urd-gds/gluster):1780:connect_remote]
SSH: Initializing SSH connection between master and slave...
[2018-07-11 18:42:49.363514] E
[syncdutils(/urd-gds/gluster):304:log_raise_exception] : connection to
peer is broken
[2018-07-11 18:42:49.364279] E [resource(/urd-gds/gluster):210:errlog] Popen:
command returned errorcmd=ssh -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret\
.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-hjRhBo/7e5534547f3675a710a107722317484f.sock
geouser@urd-gds-geo-000 /nonexistent/gsyncd --session-owner
5e94eb7d-219f-4741-a179-d4ae6b50c7ee --local-id .%\
2Furd-gds%2Fgluster --local-node urd-gds-001 -N --listen --timeout 120
gluster://localhost:urd-gds-volume error=2
[2018-07-11 18:42:49.364586] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh> usage: gsyncd.py [-h]
[2018-07-11 18:42:49.364799] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh>
[2018-07-11 18:42:49.364989] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh>
{monitor-status,monitor,worker,agent,slave,status,config-check,config-get,config-set,config-reset,voluuidget,d\
elete}
[2018-07-11 18:42:49.365210] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh> ...
[2018-07-11 18:42:49.365408] E [resource(/urd-gds/gluster):214:logerr] Popen:
ssh> gsyncd.py: error: argument subcmd: invalid choice:
'5e94eb7d-219f-4741-a179-d4ae6b50c7ee' (choose from 'monitor-status', 'monit\
or', 'worker', 'agent', 'slave', 'status', 'config-check', 'config-get',
'config-set', 'config-reset', 'voluuidget', 'delete')
[2018-07-11 18:42:49.365919] I [syncdutils(/urd-gds/gluster):271:finalize]
: exiting.
[2018-07-11 18:42:49.369316] I [repce(/urd-gds/gluster):92:service_loop]
RepceServer: terminating on reaching EOF.
[2018-07-11 18:42:49.369921] I [syncdutils(/urd-gds/gluster):271:finalize]
: exiting.
[2018-07-11 18:42:49.369694] I [monitor(monitor):353:monitor] Monitor: worker
died before establishing connection brick=/urd-gds/gluster
[2018-07-11 18:42:59.492762] I [monitor(monitor):280:monitor] Monitor: starting
gsyncd worker brick=/urd-gds/gluster
slave_node=ssh://geouser@urd-gds-geo-000:gluster://localhost:urd-gds-volume
[2018-07-11 18:42:59.558491] I [resource(/urd-gds/gluster):1780:connect_remote]
SSH: Initializing SSH connection between master and slave...
[2018-07-11 18:42:59.559056] I [changelogagent(/urd-gds/gluster):73:__init__]
ChangelogAgent: Agent listining...
[2018-07-11 18:42:59.945693] E
[syncdutils(/urd-gds/gluster):304:log_raise_exception] : connection to
peer is broken
[2018-07-11 18:42:59.946439] E [resource(/urd-gds/gluster):210:errlog] Popen:
command returned errorcmd=ssh -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret\
.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-992bk7/7e5534547f3675a710a107722317484f.sock
geouser@urd-gds-geo-000 /nonexistent/gsyncd --session-owner
5e94eb7d-219f-4741-a179-d4ae6b50c7ee --local-id .%\
2Furd-gds%2Fgluster --local-node urd-gds-001 -N --listen --timeout 120
gluster://localhost:urd-gds-volume error=2
[2018-07-11 18:42:59.946748] E
Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work
Hi Marcus,
I think the fix [1] is needed in 4.1
Could you please this out and let us know if that works for you?
[1] https://review.gluster.org/#/c/20207/
Thanks,
Kotresh HR
On Thu, Jul 12, 2018 at 1:49 AM, Marcus Pedersén
wrote:
> Hi all,
>
> I have upgraded from 3.12.9 to 4.1.1 and been following upgrade
> instructions for offline upgrade.
>
> I upgraded geo-replication side first 1 x (2+1) and the master side after
> that 2 x (2+1).
>
> Both clusters works the way they should on their own.
>
> After upgrade on master side status for all geo-replication nodes
> is Stopped.
>
> I tried to start the geo-replication from master node and response back
> was started successfully.
>
> Status again Stopped
>
> Tried to start again and get response started successfully, after that all
> glusterd crashed on all master nodes.
>
> After a restart of all glusterd the master cluster was up again.
>
> Status for geo-replication is still Stopped and every try to start it
> after this gives the response successful but still status Stopped.
>
>
> Please help me get the geo-replication up and running again.
>
>
> Best regards
>
> Marcus Pedersén
>
>
> Part of geo-replication log from master node:
>
> [2018-07-11 18:42:48.941760] I [changelogagent(/urd-gds/gluster):73:__init__]
> ChangelogAgent: Agent listining...
> [2018-07-11 18:42:48.947567] I
> [resource(/urd-gds/gluster):1780:connect_remote]
> SSH: Initializing SSH connection between master and slave...
> [2018-07-11 18:42:49.363514] E
> [syncdutils(/urd-gds/gluster):304:log_raise_exception]
> : connection to peer is broken
> [2018-07-11 18:42:49.364279] E [resource(/urd-gds/gluster):210:errlog]
> Popen: command returned errorcmd=ssh -oPasswordAuthentication=no
> -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret\
> .pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-hjRhBo/
> 7e5534547f3675a710a107722317484f.sock geouser@urd-gds-geo-000
> /nonexistent/gsyncd --session-owner 5e94eb7d-219f-4741-a179-d4ae6b50c7ee
> --local-id .%\
> 2Furd-gds%2Fgluster --local-node urd-gds-001 -N --listen --timeout 120
> gluster://localhost:urd-gds-volume error=2
> [2018-07-11 18:42:49.364586] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh> usage: gsyncd.py [-h]
> [2018-07-11 18:42:49.364799] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh>
> [2018-07-11 18:42:49.364989] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh> {monitor-status,monitor,
> worker,agent,slave,status,config-check,config-get,config-set,config-reset,
> voluuidget,d\
> elete}
> [2018-07-11 18:42:49.365210] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh> ...
> [2018-07-11 18:42:49.365408] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh> gsyncd.py: error: argument subcmd: invalid choice:
> '5e94eb7d-219f-4741-a179-d4ae6b50c7ee' (choose from 'monitor-status',
> 'monit\
> or', 'worker', 'agent', 'slave', 'status', 'config-check', 'config-get',
> 'config-set', 'config-reset', 'voluuidget', 'delete')
> [2018-07-11 18:42:49.365919] I [syncdutils(/urd-gds/gluster):271:finalize]
> : exiting.
> [2018-07-11 18:42:49.369316] I [repce(/urd-gds/gluster):92:service_loop]
> RepceServer: terminating on reaching EOF.
> [2018-07-11 18:42:49.369921] I [syncdutils(/urd-gds/gluster):271:finalize]
> : exiting.
> [2018-07-11 18:42:49.369694] I [monitor(monitor):353:monitor] Monitor:
> worker died before establishing connection brick=/urd-gds/gluster
> [2018-07-11 18:42:59.492762] I [monitor(monitor):280:monitor] Monitor:
> starting gsyncd worker brick=/urd-gds/gluster
> slave_node=ssh://geouser@urd-gds-geo-000:gluster://
> localhost:urd-gds-volume
> [2018-07-11 18:42:59.558491] I
> [resource(/urd-gds/gluster):1780:connect_remote]
> SSH: Initializing SSH connection between master and slave...
> [2018-07-11 18:42:59.559056] I [changelogagent(/urd-gds/gluster):73:__init__]
> ChangelogAgent: Agent listining...
> [2018-07-11 18:42:59.945693] E
> [syncdutils(/urd-gds/gluster):304:log_raise_exception]
> : connection to peer is broken
> [2018-07-11 18:42:59.946439] E [resource(/urd-gds/gluster):210:errlog]
> Popen: command returned errorcmd=ssh -oPasswordAuthentication=no
> -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret\
> .pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-992bk7/
> 7e5534547f3675a710a107722317484f.sock geouser@urd-gds-geo-000
> /nonexistent/gsyncd --session-owner 5e94eb7d-219f-4741-a179-d4ae6b50c7ee
> --local-id .%\
> 2Furd-gds%2Fgluster --local-node urd-gds-001 -N --listen --timeout 120
> gluster://localhost:urd-gds-volume error=2
> [2018-07-11 18:42:59.946748] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh> usage: gsyncd.py [-h]
> [2018-07-11 18:42:59.946962] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh>
> [2018-07-11 18:42:59.947150] E [resource(/urd-gds/gluster):214:logerr]
> Popen: ssh> {monitor-status,monitor,
>
