Whats current trackback please share. -Sunny
On Thu, Jun 6, 2019 at 4:53 PM deepu srinivasan <[email protected]> wrote: > > Hi Sunny > I have changed the file in /usr/libexec/glusterfs/peer_mountbroker.py as > mentioned in the patch. > Now the "gluster-mountbroker status" command is working fine. But the > geo-replication seems to be in the faulty state still. > > > Thankyou > Deepak > > On Thu, Jun 6, 2019 at 4:10 PM Sunny Kumar <[email protected]> wrote: >> >> Above error can be tracked here: >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1709248 >> >> and patch link: >> https://review.gluster.org/#/c/glusterfs/+/22716/ >> >> You can apply patch and test it however its waiting on regression to >> pass and merge. >> >> -Sunny >> >> >> On Thu, Jun 6, 2019 at 4:00 PM deepu srinivasan <[email protected]> wrote: >> > >> > Hi >> > I have followed the following steps to create the geo-replication but the >> > status seems to be in a faulty state. >> > >> > Steps : >> > >> > Installed cluster version 5.6 in totally six nodes. >> >> >> >> glusterfs 5.6 >> >> >> >> Repository revision: git://git.gluster.org/glusterfs.git >> >> >> >> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> >> >> >> >> GlusterFS comes with ABSOLUTELY NO WARRANTY. >> >> >> >> It is licensed to you under your choice of the GNU Lesser >> >> >> >> General Public License, version 3 or any later version (LGPLv3 >> >> >> >> or later), or the GNU General Public License, version 2 (GPLv2), >> >> >> >> in all cases as published by the Free Software Foundation >> > >> > >> > peer_probed the first three nodes and second three nodes. >> > >> > >> > >> > Added new volume in both the clusters >> > >> > >> > >> > execute gluster-mountbroker commands and restarted glusterd. >> >> >> >> gluster-mountbroker setup /var/mountbroker-root sas >> >> >> >> gluster-mountbroker remove --volume code-misc --user sas >> > >> > >> > configured a passwordless sssh from master to slave >> >> >> >> ssh-keygen; ssh-copy-id [email protected] >> > >> > created a common pem pub file >> >> >> >> gluster system:: execute gsec_create >> > >> > created geo-replication session. >> >> >> >> gluster volume geo-replication code-misc [email protected]::code-misc >> >> create push-pem >> > >> > executed the following command in slave >> >> >> >> /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh sas code-misc code-misc >> > >> > started the gluster geo-replication. >> >> >> >> gluster volume geo-replication code-misc [email protected]::code-misc >> >> start >> > >> > >> > Now the geo-replication works fine. >> > Tested with 2000 files All seems to sync finely. >> > >> > Now I updated all the node to version 6.2 by using rpms which were built >> > by the source code in a docker container in my personal machine. >> > >> > >> >> gluster --version >> >> >> >> glusterfs 6.2 >> >> >> >> Repository revision: git://git.gluster.org/glusterfs.git >> >> >> >> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> >> >> >> >> GlusterFS comes with ABSOLUTELY NO WARRANTY. >> >> >> >> It is licensed to you under your choice of the GNU Lesser >> >> >> >> General Public License, version 3 or any later version (LGPLv3 >> >> >> >> or later), or the GNU General Public License, version 2 (GPLv2), >> >> >> >> in all cases as published by the Free Software Foundation. >> > >> > >> > I have stopped the glusterd daemons in all the node along with the volume >> > and geo-replication. >> > Now I started the daemons, volume and geo-replication session the status >> > seems to be faulty. >> > Also noted that the result of "gluster-mountbroker status" command always >> > end in python exception like this >> >> >> >> Traceback (most recent call last): >> >> >> >> File "/usr/sbin/gluster-mountbroker", line 396, in <module> >> >> >> >> runcli() >> >> >> >> File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", >> >> line 225, in runcli >> >> >> >> cls.run(args) >> >> >> >> File "/usr/sbin/gluster-mountbroker", line 275, in run >> >> >> >> out = execute_in_peers("node-status") >> >> >> >> File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", >> >> line 127, in execute_in_peers >> >> >> >> raise GlusterCmdException((rc, out, err, " ".join(cmd))) >> >> >> >> gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Unable to end. >> >> Error : Success\n', 'gluster system:: execute mountbroker.py node-status') >> > >> > >> > Is it I or everyone gets an error for gluster-mountbroker command for >> > gluster version greater than 6.0?. Please help. >> > >> > Thank you >> > Deepak >> > >> > >> > On Thu, Jun 6, 2019 at 10:35 AM Sunny Kumar <[email protected]> wrote: >> >> >> >> Hi, >> >> >> >> Updated link for documentation : >> >> >> >> -- >> >> https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/ >> >> >> >> You can use this tool as well: >> >> http://aravindavk.in/blog/gluster-georep-tools/ >> >> >> >> -Sunny >> >> >> >> On Thu, Jun 6, 2019 at 10:29 AM Kotresh Hiremath Ravishankar >> >> <[email protected]> wrote: >> >> > >> >> > Hi, >> >> > >> >> > I think the steps to setup non-root geo-rep is not followed properly. >> >> > The following entry is missing in glusterd vol file which is required. >> >> > >> >> > The message "E [MSGID: 106061] >> >> > [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> >> > mountbroker-root' missing in glusterd vol file" repeated 33 times >> >> > between [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757] >> >> > >> >> > Could you please the steps from below? >> >> > >> >> > https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html-single/administration_guide/index#Setting_Up_the_Environment_for_a_Secure_Geo-replication_Slave >> >> > >> >> > And let us know if you still face the issue. >> >> > >> >> > >> >> > >> >> > >> >> > On Thu, Jun 6, 2019 at 10:24 AM deepu srinivasan <[email protected]> >> >> > wrote: >> >> >> >> >> >> Hi Kotresh, Sunny >> >> >> I Have mailed the logs I found in one of the slave machines. Is there >> >> >> anything to do with permission? Please help. >> >> >> >> >> >> On Wed, Jun 5, 2019 at 2:28 PM deepu srinivasan <[email protected]> >> >> >> wrote: >> >> >>> >> >> >>> Hi Kotresh, Sunny >> >> >>> Found this log in the slave machine. >> >> >>>> >> >> >>>> [2019-06-05 08:49:10.632583] I [MSGID: 106488] >> >> >>>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] >> >> >>>> 0-management: Received get vol req >> >> >>>> >> >> >>>> The message "I [MSGID: 106488] >> >> >>>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] >> >> >>>> 0-management: Received get vol req" repeated 2 times between >> >> >>>> [2019-06-05 08:49:10.632583] and [2019-06-05 08:49:10.670863] >> >> >>>> >> >> >>>> The message "I [MSGID: 106496] >> >> >>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: >> >> >>>> Received mount req" repeated 34 times between [2019-06-05 >> >> >>>> 08:48:41.005398] and [2019-06-05 08:50:37.254063] >> >> >>>> >> >> >>>> The message "E [MSGID: 106061] >> >> >>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> >> >>>> mountbroker-root' missing in glusterd vol file" repeated 34 times >> >> >>>> between [2019-06-05 08:48:41.005434] and [2019-06-05 08:50:37.254079] >> >> >>>> >> >> >>>> The message "W [MSGID: 106176] >> >> >>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: >> >> >>>> unsuccessful mount request [No such file or directory]" repeated 34 >> >> >>>> times between [2019-06-05 08:48:41.005444] and [2019-06-05 >> >> >>>> 08:50:37.254080] >> >> >>>> >> >> >>>> [2019-06-05 08:50:46.361347] I [MSGID: 106496] >> >> >>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: >> >> >>>> Received mount req >> >> >>>> >> >> >>>> [2019-06-05 08:50:46.361384] E [MSGID: 106061] >> >> >>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> >> >>>> mountbroker-root' missing in glusterd vol file >> >> >>>> >> >> >>>> [2019-06-05 08:50:46.361419] W [MSGID: 106176] >> >> >>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: >> >> >>>> unsuccessful mount request [No such file or directory] >> >> >>>> >> >> >>>> The message "I [MSGID: 106496] >> >> >>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: >> >> >>>> Received mount req" repeated 33 times between [2019-06-05 >> >> >>>> 08:50:46.361347] and [2019-06-05 08:52:34.019741] >> >> >>>> >> >> >>>> The message "E [MSGID: 106061] >> >> >>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> >> >>>> mountbroker-root' missing in glusterd vol file" repeated 33 times >> >> >>>> between [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757] >> >> >>>> >> >> >>>> The message "W [MSGID: 106176] >> >> >>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: >> >> >>>> unsuccessful mount request [No such file or directory]" repeated 33 >> >> >>>> times between [2019-06-05 08:50:46.361419] and [2019-06-05 >> >> >>>> 08:52:34.019758] >> >> >>>> >> >> >>>> [2019-06-05 08:52:44.426839] I [MSGID: 106496] >> >> >>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: >> >> >>>> Received mount req >> >> >>>> >> >> >>>> [2019-06-05 08:52:44.426886] E [MSGID: 106061] >> >> >>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> >> >>>> mountbroker-root' missing in glusterd vol file >> >> >>>> >> >> >>>> [2019-06-05 08:52:44.426896] W [MSGID: 106176] >> >> >>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: >> >> >>>> unsuccessful mount request [No such file or directory] >> >> >>> >> >> >>> >> >> >>> On Wed, Jun 5, 2019 at 1:06 AM deepu srinivasan <[email protected]> >> >> >>> wrote: >> >> >>>> >> >> >>>> Thankyou Kotresh >> >> >>>> >> >> >>>> On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar >> >> >>>> <[email protected]> wrote: >> >> >>>>> >> >> >>>>> Ccing Sunny, who was investing similar issue. >> >> >>>>> >> >> >>>>> On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan >> >> >>>>> <[email protected]> wrote: >> >> >>>>>> >> >> >>>>>> Have already added the path in bashrc . Still in faulty state >> >> >>>>>> >> >> >>>>>> On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar >> >> >>>>>> <[email protected]> wrote: >> >> >>>>>>> >> >> >>>>>>> could you please try adding /usr/sbin to $PATH for user 'sas'? If >> >> >>>>>>> it's bash, add 'export PATH=/usr/sbin:$PATH' in >> >> >>>>>>> /home/sas/.bashrc >> >> >>>>>>> >> >> >>>>>>> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan >> >> >>>>>>> <[email protected]> wrote: >> >> >>>>>>>> >> >> >>>>>>>> Hi Kortesh >> >> >>>>>>>> Please find the logs of the above error >> >> >>>>>>>> Master log snippet >> >> >>>>>>>>> >> >> >>>>>>>>> [2019-06-04 11:52:09.254731] I [resource(worker >> >> >>>>>>>>> /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: >> >> >>>>>>>>> Initializing SSH connection between master and slave... >> >> >>>>>>>>> [2019-06-04 11:52:09.308923] D [repce(worker >> >> >>>>>>>>> /home/sas/gluster/data/code-misc):196:push] RepceClient: call >> >> >>>>>>>>> 89724:139652759443264:1559649129.31 __repce_version__() ... >> >> >>>>>>>>> [2019-06-04 11:52:09.602792] E [syncdutils(worker >> >> >>>>>>>>> /home/sas/gluster/data/code-misc):311:log_raise_exception] >> >> >>>>>>>>> <top>: connection to peer is broken >> >> >>>>>>>>> [2019-06-04 11:52:09.603312] E [syncdutils(worker >> >> >>>>>>>>> /home/sas/gluster/data/code-misc):805:errlog] Popen: command >> >> >>>>>>>>> returned error cmd=ssh -oPasswordAuthentication=no >> >> >>>>>>>>> -oStrictHostKeyChecking=no -i /var/lib/ >> >> >>>>>>>>> glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto >> >> >>>>>>>>> -S >> >> >>>>>>>>> /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock >> >> >>>>>>>>> [email protected] /usr/libexec/glusterfs/gsyncd slave >> >> >>>>>>>>> code-misc sas@ 192.168.185.107::code-misc --master-node >> >> >>>>>>>>> 192.168.185.106 --master-node-id >> >> >>>>>>>>> 851b64d0-d885-4ae9-9b38-ab5b15db0fec --master-brick >> >> >>>>>>>>> /home/sas/gluster/data/code-misc --local-node 192.168.185.122 >> >> >>>>>>>>> --local-node- id bcaa7af6-c3a1-4411-8e99-4ebecb32eb6a >> >> >>>>>>>>> --slave-timeout 120 --slave-log-level DEBUG >> >> >>>>>>>>> --slave-gluster-log-level INFO --slave-gluster-command-dir >> >> >>>>>>>>> /usr/sbin error=1 >> >> >>>>>>>>> [2019-06-04 11:52:09.614996] I [repce(agent >> >> >>>>>>>>> /home/sas/gluster/data/code-misc):97:service_loop] RepceServer: >> >> >>>>>>>>> terminating on reaching EOF. >> >> >>>>>>>>> [2019-06-04 11:52:09.615545] D [monitor(monitor):271:monitor] >> >> >>>>>>>>> Monitor: worker(/home/sas/gluster/data/code-misc) connected >> >> >>>>>>>>> [2019-06-04 11:52:09.616528] I [monitor(monitor):278:monitor] >> >> >>>>>>>>> Monitor: worker died in startup phase >> >> >>>>>>>>> brick=/home/sas/gluster/data/code-misc >> >> >>>>>>>>> [2019-06-04 11:52:09.619391] I >> >> >>>>>>>>> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: >> >> >>>>>>>>> Worker Status Change status=Faulty >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> Slave log snippet >> >> >>>>>>>>> >> >> >>>>>>>>> [2019-06-04 11:50:09.782668] E [syncdutils(slave >> >> >>>>>>>>> 192.168.185.106/home/sas/gluster/data/code-misc):809:logerr] >> >> >>>>>>>>> Popen: /usr/sbin/gluster> 2 : failed with this errno (No such >> >> >>>>>>>>> file or directory) >> >> >>>>>>>>> [2019-06-04 11:50:11.188167] W [gsyncd(slave >> >> >>>>>>>>> 192.168.185.125/home/sas/gluster/data/code-misc):305:main] >> >> >>>>>>>>> <top>: Session config file not exists, using the default config >> >> >>>>>>>>> path=/var/lib/glusterd/geo-replication/code-misc_192.168.185.107_code-misc/gsyncd.conf >> >> >>>>>>>>> [2019-06-04 11:50:11.201070] I [resource(slave >> >> >>>>>>>>> 192.168.185.125/home/sas/gluster/data/code-misc):1098:connect] >> >> >>>>>>>>> GLUSTER: Mounting gluster volume locally... >> >> >>>>>>>>> [2019-06-04 11:50:11.271231] E [resource(slave >> >> >>>>>>>>> 192.168.185.125/home/sas/gluster/data/code-misc):1006:handle_mounter] >> >> >>>>>>>>> MountbrokerMounter: glusterd answered mnt= >> >> >>>>>>>>> [2019-06-04 11:50:11.271998] E [syncdutils(slave >> >> >>>>>>>>> 192.168.185.125/home/sas/gluster/data/code-misc):805:errlog] >> >> >>>>>>>>> Popen: command returned error cmd=/usr/sbin/gluster >> >> >>>>>>>>> --remote-host=localhost system:: mount sas user-map-root=sas >> >> >>>>>>>>> aux-gfid-mount acl log-level=INFO >> >> >>>>>>>>> log-file=/var/log/glusterfs/geo-replication-slaves/code-misc_192.168.185.107_code-misc/mnt-192.168.185.125-home-sas-gluster-data-code-misc.log >> >> >>>>>>>>> volfile-server=localhost volfile-id=code-misc client-pid=-1 >> >> >>>>>>>>> error=1 >> >> >>>>>>>>> [2019-06-04 11:50:11.272113] E [syncdutils(slave >> >> >>>>>>>>> 192.168.185.125/home/sas/gluster/data/code-misc):809:logerr] >> >> >>>>>>>>> Popen: /usr/sbin/gluster> 2 : failed with this errno (No such >> >> >>>>>>>>> file or directory) >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> On Tue, Jun 4, 2019 at 5:10 PM deepu srinivasan >> >> >>>>>>>> <[email protected]> wrote: >> >> >>>>>>>>> >> >> >>>>>>>>> Hi >> >> >>>>>>>>> As discussed I have upgraded gluster from 4.1 to 6.2 version. >> >> >>>>>>>>> But the Geo replication failed to start. >> >> >>>>>>>>> Stays in faulty state >> >> >>>>>>>>> >> >> >>>>>>>>> On Fri, May 31, 2019, 5:32 PM deepu srinivasan >> >> >>>>>>>>> <[email protected]> wrote: >> >> >>>>>>>>>> >> >> >>>>>>>>>> Checked the data. It remains in 2708. No progress. >> >> >>>>>>>>>> >> >> >>>>>>>>>> On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar >> >> >>>>>>>>>> <[email protected]> wrote: >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> That means it could be working and the defunct process might >> >> >>>>>>>>>>> be some old zombie one. Could you check, that data progress ? >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> On Fri, May 31, 2019 at 4:29 PM deepu srinivasan >> >> >>>>>>>>>>> <[email protected]> wrote: >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> Hi >> >> >>>>>>>>>>>> When i change the rsync option the rsync process doesnt seem >> >> >>>>>>>>>>>> to start . Only a defunt process is listed in ps aux. Only >> >> >>>>>>>>>>>> when i set rsync option to " " and restart all the process >> >> >>>>>>>>>>>> the rsync process is listed in ps aux. >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar >> >> >>>>>>>>>>>> <[email protected]> wrote: >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> Yes, rsync config option should have fixed this issue. >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> Could you share the output of the following? >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> 1. gluster volume geo-replication <MASTERVOL> >> >> >>>>>>>>>>>>> <SLAVEHOST>::<SLAVEVOL> config rsync-options >> >> >>>>>>>>>>>>> 2. ps -ef | grep rsync >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> On Fri, May 31, 2019 at 4:11 PM deepu srinivasan >> >> >>>>>>>>>>>>> <[email protected]> wrote: >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> Done. >> >> >>>>>>>>>>>>>> We got the following result . >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> 1559298781.338234 write(2, "rsync: link_stat >> >> >>>>>>>>>>>>>>> \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\" >> >> >>>>>>>>>>>>>>> failed: No such file or directory (2)", 128 >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> seems like a file is missing ? >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath >> >> >>>>>>>>>>>>>> Ravishankar <[email protected]> wrote: >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> Hi, >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> Could you take the strace with with more string size? The >> >> >>>>>>>>>>>>>>> argument strings are truncated. >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> strace -s 500 -ttt -T -p <rsync pid> >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> On Fri, May 31, 2019 at 3:17 PM deepu srinivasan >> >> >>>>>>>>>>>>>>> <[email protected]> wrote: >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> Hi Kotresh >> >> >>>>>>>>>>>>>>>> The above-mentioned work around did not work properly. >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> On Fri, May 31, 2019 at 3:16 PM deepu srinivasan >> >> >>>>>>>>>>>>>>>> <[email protected]> wrote: >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> Hi Kotresh >> >> >>>>>>>>>>>>>>>>> We have tried the above-mentioned rsync option and we >> >> >>>>>>>>>>>>>>>>> are planning to have the version upgrade to 6.0. >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath >> >> >>>>>>>>>>>>>>>>> Ravishankar <[email protected]> wrote: >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> Hi, >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> This looks like the hang because stderr buffer filled >> >> >>>>>>>>>>>>>>>>>> up with errors messages and no one reading it. >> >> >>>>>>>>>>>>>>>>>> I think this issue is fixed in latest releases. As a >> >> >>>>>>>>>>>>>>>>>> workaround, you can do following and check if it works. >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> Prerequisite: >> >> >>>>>>>>>>>>>>>>>> rsync version should be > 3.1.0 >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> Workaround: >> >> >>>>>>>>>>>>>>>>>> gluster volume geo-replication <MASTERVOL> >> >> >>>>>>>>>>>>>>>>>> <SLAVEHOST>::<SLAVEVOL> config rsync-options >> >> >>>>>>>>>>>>>>>>>> "--ignore-missing-args" >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> Thanks, >> >> >>>>>>>>>>>>>>>>>> Kotresh HR >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> On Thu, May 30, 2019 at 5:39 PM deepu srinivasan >> >> >>>>>>>>>>>>>>>>>> <[email protected]> wrote: >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> Hi >> >> >>>>>>>>>>>>>>>>>>> We were evaluating Gluster geo Replication between >> >> >>>>>>>>>>>>>>>>>>> two DCs one is in US west and one is in US east. We >> >> >>>>>>>>>>>>>>>>>>> took multiple trials for different file size. >> >> >>>>>>>>>>>>>>>>>>> The Geo Replication tends to stop replicating but >> >> >>>>>>>>>>>>>>>>>>> while checking the status it appears to be in Active >> >> >>>>>>>>>>>>>>>>>>> state. But the slave volume did not increase in size. >> >> >>>>>>>>>>>>>>>>>>> So we have restarted the geo-replication session and >> >> >>>>>>>>>>>>>>>>>>> checked the status. The status was in an active state >> >> >>>>>>>>>>>>>>>>>>> and it was in History Crawl for a long time. We have >> >> >>>>>>>>>>>>>>>>>>> enabled the DEBUG mode in logging and checked for any >> >> >>>>>>>>>>>>>>>>>>> error. >> >> >>>>>>>>>>>>>>>>>>> There was around 2000 file appeared for syncing >> >> >>>>>>>>>>>>>>>>>>> candidate. The Rsync process starts but the rsync did >> >> >>>>>>>>>>>>>>>>>>> not happen in the slave volume. Every time the rsync >> >> >>>>>>>>>>>>>>>>>>> process appears in the "ps auxxx" list but the >> >> >>>>>>>>>>>>>>>>>>> replication did not happen in the slave end. What >> >> >>>>>>>>>>>>>>>>>>> would be the cause of this problem? Is there anyway >> >> >>>>>>>>>>>>>>>>>>> to debug it? >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> We have also checked the strace of the rync program. >> >> >>>>>>>>>>>>>>>>>>> it displays something like this >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128" >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> We are using the below specs >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> Gluster version - 4.1.7 >> >> >>>>>>>>>>>>>>>>>>> Sync mode - rsync >> >> >>>>>>>>>>>>>>>>>>> Volume - 1x3 in each end (master and slave) >> >> >>>>>>>>>>>>>>>>>>> Intranet Bandwidth - 10 Gig >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> -- >> >> >>>>>>>>>>>>>>>>>> Thanks and Regards, >> >> >>>>>>>>>>>>>>>>>> Kotresh H R >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> -- >> >> >>>>>>>>>>>>>>> Thanks and Regards, >> >> >>>>>>>>>>>>>>> Kotresh H R >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> -- >> >> >>>>>>>>>>>>> Thanks and Regards, >> >> >>>>>>>>>>>>> Kotresh H R >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> -- >> >> >>>>>>>>>>> Thanks and Regards, >> >> >>>>>>>>>>> Kotresh H R >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> -- >> >> >>>>>>> Thanks and Regards, >> >> >>>>>>> Kotresh H R >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> -- >> >> >>>>> Thanks and Regards, >> >> >>>>> Kotresh H R >> >> > >> >> > >> >> > >> >> > -- >> >> > Thanks and Regards, >> >> > Kotresh H R _______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
