You should not have used this one: > > gluster-mountbroker remove --volume code-misc --user sas
-- This one is to remove volume/user from mount broker. Please try setting up mount broker once again. -Sunny On Thu, Jun 6, 2019 at 5:28 PM deepu srinivasan <[email protected]> wrote: > > Hi Sunny > Please find the logs attached >> >> The message "E [MSGID: 106061] >> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> mountbroker-root' missing in glusterd vol file" repeated 13 times between >> [2019-06-06 11:51:43.986788] and [2019-06-06 11:52:32.764546] >> >> The message "W [MSGID: 106176] >> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful >> mount request [No such file or directory]" repeated 13 times between >> [2019-06-06 11:51:43.986798] and [2019-06-06 11:52:32.764548] >> >> The message "I [MSGID: 106488] >> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: >> Received get vol req" repeated 2 times between [2019-06-06 11:53:07.064332] >> and [2019-06-06 11:53:07.303978] >> >> [2019-06-06 11:55:35.624320] I [MSGID: 106495] >> [glusterd-handler.c:3137:__glusterd_handle_getwd] 0-glusterd: Received getwd >> req >> >> [2019-06-06 11:55:35.884345] I [MSGID: 106131] >> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: quotad already >> stopped >> >> [2019-06-06 11:55:35.884373] I [MSGID: 106568] >> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: quotad service is >> stopped >> >> [2019-06-06 11:55:35.884459] I [MSGID: 106131] >> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already >> stopped >> >> [2019-06-06 11:55:35.884473] I [MSGID: 106568] >> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: bitd service is >> stopped >> >> [2019-06-06 11:55:35.884554] I [MSGID: 106131] >> [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already >> stopped >> >> [2019-06-06 11:55:35.884567] I [MSGID: 106568] >> [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: scrub service is >> stopped >> >> [2019-06-06 11:55:35.893823] I [run.c:242:runner_log] >> (-->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe8e1a) >> [0x7f7380d60e1a] >> -->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe88e5) >> [0x7f7380d608e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) >> [0x7f738cbc5df5] ) 0-management: Ran script: >> /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=code-misc -o >> features.read-only=on --gd-workdir=/var/lib/glusterd >> >> [2019-06-06 11:55:35.900465] I [run.c:242:runner_log] >> (-->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe8e1a) >> [0x7f7380d60e1a] >> -->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe88e5) >> [0x7f7380d608e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) >> [0x7f738cbc5df5] ) 0-management: Ran script: >> /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh >> --volname=code-misc -o features.read-only=on --gd-workdir=/var/lib/glusterd >> >> [2019-06-06 11:55:43.485284] I [MSGID: 106488] >> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: >> Received get vol req >> >> The message "I [MSGID: 106488] >> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: >> Received get vol req" repeated 2 times between [2019-06-06 11:55:43.485284] >> and [2019-06-06 11:55:43.512321] >> >> [2019-06-06 11:55:44.055419] I [MSGID: 106496] >> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount >> req >> >> [2019-06-06 11:55:44.055473] E [MSGID: 106061] >> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> mountbroker-root' missing in glusterd vol file >> >> [2019-06-06 11:55:44.055483] W [MSGID: 106176] >> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful >> mount request [No such file or directory] >> >> [2019-06-06 11:55:44.056695] I [MSGID: 106496] >> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount >> req >> >> [2019-06-06 11:55:44.056725] E [MSGID: 106061] >> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> mountbroker-root' missing in glusterd vol file >> >> [2019-06-06 11:55:44.056734] W [MSGID: 106176] >> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful >> mount request [No such file or directory] >> >> [2019-06-06 11:55:44.057522] I [MSGID: 106496] >> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount >> req >> >> [2019-06-06 11:55:44.057552] E [MSGID: 106061] >> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> mountbroker-root' missing in glusterd vol file >> >> [2019-06-06 11:55:44.057562] W [MSGID: 106176] >> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful >> mount request [No such file or directory] >> >> [2019-06-06 11:55:54.655681] I [MSGID: 106496] >> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount >> req >> >> [2019-06-06 11:55:54.655741] E [MSGID: 106061] >> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> mountbroker-root' missing in glusterd vol file >> >> [2019-06-06 11:55:54.655752] W [MSGID: 106176] >> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful >> mount request [No such file or directory] > > > On Thu, Jun 6, 2019 at 5:09 PM Sunny Kumar <[email protected]> wrote: >> >> Whats current trackback please share. >> >> -Sunny >> >> >> On Thu, Jun 6, 2019 at 4:53 PM deepu srinivasan <[email protected]> wrote: >> > >> > Hi Sunny >> > I have changed the file in /usr/libexec/glusterfs/peer_mountbroker.py as >> > mentioned in the patch. >> > Now the "gluster-mountbroker status" command is working fine. But the >> > geo-replication seems to be in the faulty state still. >> > >> > >> > Thankyou >> > Deepak >> > >> > On Thu, Jun 6, 2019 at 4:10 PM Sunny Kumar <[email protected]> wrote: >> >> >> >> Above error can be tracked here: >> >> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1709248 >> >> >> >> and patch link: >> >> https://review.gluster.org/#/c/glusterfs/+/22716/ >> >> >> >> You can apply patch and test it however its waiting on regression to >> >> pass and merge. >> >> >> >> -Sunny >> >> >> >> >> >> On Thu, Jun 6, 2019 at 4:00 PM deepu srinivasan <[email protected]> >> >> wrote: >> >> > >> >> > Hi >> >> > I have followed the following steps to create the geo-replication but >> >> > the status seems to be in a faulty state. >> >> > >> >> > Steps : >> >> > >> >> > Installed cluster version 5.6 in totally six nodes. >> >> >> >> >> >> glusterfs 5.6 >> >> >> >> >> >> Repository revision: git://git.gluster.org/glusterfs.git >> >> >> >> >> >> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> >> >> >> >> >> >> GlusterFS comes with ABSOLUTELY NO WARRANTY. >> >> >> >> >> >> It is licensed to you under your choice of the GNU Lesser >> >> >> >> >> >> General Public License, version 3 or any later version (LGPLv3 >> >> >> >> >> >> or later), or the GNU General Public License, version 2 (GPLv2), >> >> >> >> >> >> in all cases as published by the Free Software Foundation >> >> > >> >> > >> >> > peer_probed the first three nodes and second three nodes. >> >> > >> >> > >> >> > >> >> > Added new volume in both the clusters >> >> > >> >> > >> >> > >> >> > execute gluster-mountbroker commands and restarted glusterd. >> >> >> >> >> >> gluster-mountbroker setup /var/mountbroker-root sas >> >> >> >> >> >> gluster-mountbroker remove --volume code-misc --user sas >> >> > >> >> > >> >> > configured a passwordless sssh from master to slave >> >> >> >> >> >> ssh-keygen; ssh-copy-id [email protected] >> >> > >> >> > created a common pem pub file >> >> >> >> >> >> gluster system:: execute gsec_create >> >> > >> >> > created geo-replication session. >> >> >> >> >> >> gluster volume geo-replication code-misc >> >> >> [email protected]::code-misc create push-pem >> >> > >> >> > executed the following command in slave >> >> >> >> >> >> /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh sas code-misc code-misc >> >> > >> >> > started the gluster geo-replication. >> >> >> >> >> >> gluster volume geo-replication code-misc >> >> >> [email protected]::code-misc start >> >> > >> >> > >> >> > Now the geo-replication works fine. >> >> > Tested with 2000 files All seems to sync finely. >> >> > >> >> > Now I updated all the node to version 6.2 by using rpms which were >> >> > built by the source code in a docker container in my personal machine. >> >> > >> >> > >> >> >> gluster --version >> >> >> >> >> >> glusterfs 6.2 >> >> >> >> >> >> Repository revision: git://git.gluster.org/glusterfs.git >> >> >> >> >> >> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> >> >> >> >> >> >> GlusterFS comes with ABSOLUTELY NO WARRANTY. >> >> >> >> >> >> It is licensed to you under your choice of the GNU Lesser >> >> >> >> >> >> General Public License, version 3 or any later version (LGPLv3 >> >> >> >> >> >> or later), or the GNU General Public License, version 2 (GPLv2), >> >> >> >> >> >> in all cases as published by the Free Software Foundation. >> >> > >> >> > >> >> > I have stopped the glusterd daemons in all the node along with the >> >> > volume and geo-replication. >> >> > Now I started the daemons, volume and geo-replication session the >> >> > status seems to be faulty. >> >> > Also noted that the result of "gluster-mountbroker status" command >> >> > always end in python exception like this >> >> >> >> >> >> Traceback (most recent call last): >> >> >> >> >> >> File "/usr/sbin/gluster-mountbroker", line 396, in <module> >> >> >> >> >> >> runcli() >> >> >> >> >> >> File >> >> >> "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line >> >> >> 225, in runcli >> >> >> >> >> >> cls.run(args) >> >> >> >> >> >> File "/usr/sbin/gluster-mountbroker", line 275, in run >> >> >> >> >> >> out = execute_in_peers("node-status") >> >> >> >> >> >> File >> >> >> "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line >> >> >> 127, in execute_in_peers >> >> >> >> >> >> raise GlusterCmdException((rc, out, err, " ".join(cmd))) >> >> >> >> >> >> gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Unable to end. >> >> >> Error : Success\n', 'gluster system:: execute mountbroker.py >> >> >> node-status') >> >> > >> >> > >> >> > Is it I or everyone gets an error for gluster-mountbroker command for >> >> > gluster version greater than 6.0?. Please help. >> >> > >> >> > Thank you >> >> > Deepak >> >> > >> >> > >> >> > On Thu, Jun 6, 2019 at 10:35 AM Sunny Kumar <[email protected]> wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> Updated link for documentation : >> >> >> >> >> >> -- >> >> >> https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/ >> >> >> >> >> >> You can use this tool as well: >> >> >> http://aravindavk.in/blog/gluster-georep-tools/ >> >> >> >> >> >> -Sunny >> >> >> >> >> >> On Thu, Jun 6, 2019 at 10:29 AM Kotresh Hiremath Ravishankar >> >> >> <[email protected]> wrote: >> >> >> > >> >> >> > Hi, >> >> >> > >> >> >> > I think the steps to setup non-root geo-rep is not followed >> >> >> > properly. The following entry is missing in glusterd vol file which >> >> >> > is required. >> >> >> > >> >> >> > The message "E [MSGID: 106061] >> >> >> > [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >> >> >> > mountbroker-root' missing in glusterd vol file" repeated 33 times >> >> >> > between [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757] >> >> >> > >> >> >> > Could you please the steps from below? >> >> >> > >> >> >> > https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html-single/administration_guide/index#Setting_Up_the_Environment_for_a_Secure_Geo-replication_Slave >> >> >> > >> >> >> > And let us know if you still face the issue. >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > On Thu, Jun 6, 2019 at 10:24 AM deepu srinivasan >> >> >> > <[email protected]> wrote: >> >> >> >> >> >> >> >> Hi Kotresh, Sunny >> >> >> >> I Have mailed the logs I found in one of the slave machines. Is >> >> >> >> there anything to do with permission? Please help. >> >> >> >> >> >> >> >> On Wed, Jun 5, 2019 at 2:28 PM deepu srinivasan >> >> >> >> <[email protected]> wrote: >> >> >> >>> >> >> >> >>> Hi Kotresh, Sunny >> >> >> >>> Found this log in the slave machine. >> >> >> >>>> >> >> >> >>>> [2019-06-05 08:49:10.632583] I [MSGID: 106488] >> >> >> >>>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] >> >> >> >>>> 0-management: Received get vol req >> >> >> >>>> >> >> >> >>>> The message "I [MSGID: 106488] >> >> >> >>>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] >> >> >> >>>> 0-management: Received get vol req" repeated 2 times between >> >> >> >>>> [2019-06-05 08:49:10.632583] and [2019-06-05 08:49:10.670863] >> >> >> >>>> >> >> >> >>>> The message "I [MSGID: 106496] >> >> >> >>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: >> >> >> >>>> Received mount req" repeated 34 times between [2019-06-05 >> >> >> >>>> 08:48:41.005398] and [2019-06-05 08:50:37.254063] >> >> >> >>>> >> >> >> >>>> The message "E [MSGID: 106061] >> >> >> >>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: >> >> >> >>>> 'option mountbroker-root' missing in glusterd vol file" repeated >> >> >> >>>> 34 times between [2019-06-05 08:48:41.005434] and [2019-06-05 >> >> >> >>>> 08:50:37.254079] >> >> >> >>>> >> >> >> >>>> The message "W [MSGID: 106176] >> >> >> >>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: >> >> >> >>>> unsuccessful mount request [No such file or directory]" repeated >> >> >> >>>> 34 times between [2019-06-05 08:48:41.005444] and [2019-06-05 >> >> >> >>>> 08:50:37.254080] >> >> >> >>>> >> >> >> >>>> [2019-06-05 08:50:46.361347] I [MSGID: 106496] >> >> >> >>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: >> >> >> >>>> Received mount req >> >> >> >>>> >> >> >> >>>> [2019-06-05 08:50:46.361384] E [MSGID: 106061] >> >> >> >>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: >> >> >> >>>> 'option mountbroker-root' missing in glusterd vol file >> >> >> >>>> >> >> >> >>>> [2019-06-05 08:50:46.361419] W [MSGID: 106176] >> >> >> >>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: >> >> >> >>>> unsuccessful mount request [No such file or directory] >> >> >> >>>> >> >> >> >>>> The message "I [MSGID: 106496] >> >> >> >>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: >> >> >> >>>> Received mount req" repeated 33 times between [2019-06-05 >> >> >> >>>> 08:50:46.361347] and [2019-06-05 08:52:34.019741] >> >> >> >>>> >> >> >> >>>> The message "E [MSGID: 106061] >> >> >> >>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: >> >> >> >>>> 'option mountbroker-root' missing in glusterd vol file" repeated >> >> >> >>>> 33 times between [2019-06-05 08:50:46.361384] and [2019-06-05 >> >> >> >>>> 08:52:34.019757] >> >> >> >>>> >> >> >> >>>> The message "W [MSGID: 106176] >> >> >> >>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: >> >> >> >>>> unsuccessful mount request [No such file or directory]" repeated >> >> >> >>>> 33 times between [2019-06-05 08:50:46.361419] and [2019-06-05 >> >> >> >>>> 08:52:34.019758] >> >> >> >>>> >> >> >> >>>> [2019-06-05 08:52:44.426839] I [MSGID: 106496] >> >> >> >>>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: >> >> >> >>>> Received mount req >> >> >> >>>> >> >> >> >>>> [2019-06-05 08:52:44.426886] E [MSGID: 106061] >> >> >> >>>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: >> >> >> >>>> 'option mountbroker-root' missing in glusterd vol file >> >> >> >>>> >> >> >> >>>> [2019-06-05 08:52:44.426896] W [MSGID: 106176] >> >> >> >>>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: >> >> >> >>>> unsuccessful mount request [No such file or directory] >> >> >> >>> >> >> >> >>> >> >> >> >>> On Wed, Jun 5, 2019 at 1:06 AM deepu srinivasan >> >> >> >>> <[email protected]> wrote: >> >> >> >>>> >> >> >> >>>> Thankyou Kotresh >> >> >> >>>> >> >> >> >>>> On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar >> >> >> >>>> <[email protected]> wrote: >> >> >> >>>>> >> >> >> >>>>> Ccing Sunny, who was investing similar issue. >> >> >> >>>>> >> >> >> >>>>> On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan >> >> >> >>>>> <[email protected]> wrote: >> >> >> >>>>>> >> >> >> >>>>>> Have already added the path in bashrc . Still in faulty state >> >> >> >>>>>> >> >> >> >>>>>> On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar >> >> >> >>>>>> <[email protected]> wrote: >> >> >> >>>>>>> >> >> >> >>>>>>> could you please try adding /usr/sbin to $PATH for user 'sas'? >> >> >> >>>>>>> If it's bash, add 'export PATH=/usr/sbin:$PATH' in >> >> >> >>>>>>> /home/sas/.bashrc >> >> >> >>>>>>> >> >> >> >>>>>>> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan >> >> >> >>>>>>> <[email protected]> wrote: >> >> >> >>>>>>>> >> >> >> >>>>>>>> Hi Kortesh >> >> >> >>>>>>>> Please find the logs of the above error >> >> >> >>>>>>>> Master log snippet >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> [2019-06-04 11:52:09.254731] I [resource(worker >> >> >> >>>>>>>>> /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: >> >> >> >>>>>>>>> Initializing SSH connection between master and slave... >> >> >> >>>>>>>>> [2019-06-04 11:52:09.308923] D [repce(worker >> >> >> >>>>>>>>> /home/sas/gluster/data/code-misc):196:push] RepceClient: >> >> >> >>>>>>>>> call 89724:139652759443264:1559649129.31 __repce_version__() >> >> >> >>>>>>>>> ... >> >> >> >>>>>>>>> [2019-06-04 11:52:09.602792] E [syncdutils(worker >> >> >> >>>>>>>>> /home/sas/gluster/data/code-misc):311:log_raise_exception] >> >> >> >>>>>>>>> <top>: connection to peer is broken >> >> >> >>>>>>>>> [2019-06-04 11:52:09.603312] E [syncdutils(worker >> >> >> >>>>>>>>> /home/sas/gluster/data/code-misc):805:errlog] Popen: command >> >> >> >>>>>>>>> returned error cmd=ssh -oPasswordAuthentication=no >> >> >> >>>>>>>>> -oStrictHostKeyChecking=no -i /var/lib/ >> >> >> >>>>>>>>> glusterd/geo-replication/secret.pem -p 22 >> >> >> >>>>>>>>> -oControlMaster=auto -S >> >> >> >>>>>>>>> /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock >> >> >> >>>>>>>>> [email protected] /usr/libexec/glusterfs/gsyncd slave >> >> >> >>>>>>>>> code-misc sas@ 192.168.185.107::code-misc --master-node >> >> >> >>>>>>>>> 192.168.185.106 --master-node-id >> >> >> >>>>>>>>> 851b64d0-d885-4ae9-9b38-ab5b15db0fec --master-brick >> >> >> >>>>>>>>> /home/sas/gluster/data/code-misc --local-node >> >> >> >>>>>>>>> 192.168.185.122 --local-node- id >> >> >> >>>>>>>>> bcaa7af6-c3a1-4411-8e99-4ebecb32eb6a --slave-timeout 120 >> >> >> >>>>>>>>> --slave-log-level DEBUG --slave-gluster-log-level INFO >> >> >> >>>>>>>>> --slave-gluster-command-dir /usr/sbin error=1 >> >> >> >>>>>>>>> [2019-06-04 11:52:09.614996] I [repce(agent >> >> >> >>>>>>>>> /home/sas/gluster/data/code-misc):97:service_loop] >> >> >> >>>>>>>>> RepceServer: terminating on reaching EOF. >> >> >> >>>>>>>>> [2019-06-04 11:52:09.615545] D >> >> >> >>>>>>>>> [monitor(monitor):271:monitor] Monitor: >> >> >> >>>>>>>>> worker(/home/sas/gluster/data/code-misc) connected >> >> >> >>>>>>>>> [2019-06-04 11:52:09.616528] I >> >> >> >>>>>>>>> [monitor(monitor):278:monitor] Monitor: worker died in >> >> >> >>>>>>>>> startup phase brick=/home/sas/gluster/data/code-misc >> >> >> >>>>>>>>> [2019-06-04 11:52:09.619391] I >> >> >> >>>>>>>>> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: >> >> >> >>>>>>>>> Worker Status Change status=Faulty >> >> >> >>>>>>>> >> >> >> >>>>>>>> >> >> >> >>>>>>>> Slave log snippet >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> [2019-06-04 11:50:09.782668] E [syncdutils(slave >> >> >> >>>>>>>>> 192.168.185.106/home/sas/gluster/data/code-misc):809:logerr] >> >> >> >>>>>>>>> Popen: /usr/sbin/gluster> 2 : failed with this errno (No >> >> >> >>>>>>>>> such file or directory) >> >> >> >>>>>>>>> [2019-06-04 11:50:11.188167] W [gsyncd(slave >> >> >> >>>>>>>>> 192.168.185.125/home/sas/gluster/data/code-misc):305:main] >> >> >> >>>>>>>>> <top>: Session config file not exists, using the default >> >> >> >>>>>>>>> config >> >> >> >>>>>>>>> path=/var/lib/glusterd/geo-replication/code-misc_192.168.185.107_code-misc/gsyncd.conf >> >> >> >>>>>>>>> [2019-06-04 11:50:11.201070] I [resource(slave >> >> >> >>>>>>>>> 192.168.185.125/home/sas/gluster/data/code-misc):1098:connect] >> >> >> >>>>>>>>> GLUSTER: Mounting gluster volume locally... >> >> >> >>>>>>>>> [2019-06-04 11:50:11.271231] E [resource(slave >> >> >> >>>>>>>>> 192.168.185.125/home/sas/gluster/data/code-misc):1006:handle_mounter] >> >> >> >>>>>>>>> MountbrokerMounter: glusterd answered mnt= >> >> >> >>>>>>>>> [2019-06-04 11:50:11.271998] E [syncdutils(slave >> >> >> >>>>>>>>> 192.168.185.125/home/sas/gluster/data/code-misc):805:errlog] >> >> >> >>>>>>>>> Popen: command returned error cmd=/usr/sbin/gluster >> >> >> >>>>>>>>> --remote-host=localhost system:: mount sas user-map-root=sas >> >> >> >>>>>>>>> aux-gfid-mount acl log-level=INFO >> >> >> >>>>>>>>> log-file=/var/log/glusterfs/geo-replication-slaves/code-misc_192.168.185.107_code-misc/mnt-192.168.185.125-home-sas-gluster-data-code-misc.log >> >> >> >>>>>>>>> volfile-server=localhost volfile-id=code-misc client-pid=-1 >> >> >> >>>>>>>>> error=1 >> >> >> >>>>>>>>> [2019-06-04 11:50:11.272113] E [syncdutils(slave >> >> >> >>>>>>>>> 192.168.185.125/home/sas/gluster/data/code-misc):809:logerr] >> >> >> >>>>>>>>> Popen: /usr/sbin/gluster> 2 : failed with this errno (No >> >> >> >>>>>>>>> such file or directory) >> >> >> >>>>>>>> >> >> >> >>>>>>>> >> >> >> >>>>>>>> On Tue, Jun 4, 2019 at 5:10 PM deepu srinivasan >> >> >> >>>>>>>> <[email protected]> wrote: >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> Hi >> >> >> >>>>>>>>> As discussed I have upgraded gluster from 4.1 to 6.2 >> >> >> >>>>>>>>> version. But the Geo replication failed to start. >> >> >> >>>>>>>>> Stays in faulty state >> >> >> >>>>>>>>> >> >> >> >>>>>>>>> On Fri, May 31, 2019, 5:32 PM deepu srinivasan >> >> >> >>>>>>>>> <[email protected]> wrote: >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>> Checked the data. It remains in 2708. No progress. >> >> >> >>>>>>>>>> >> >> >> >>>>>>>>>> On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath >> >> >> >>>>>>>>>> Ravishankar <[email protected]> wrote: >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>> That means it could be working and the defunct process >> >> >> >>>>>>>>>>> might be some old zombie one. Could you check, that data >> >> >> >>>>>>>>>>> progress ? >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>> On Fri, May 31, 2019 at 4:29 PM deepu srinivasan >> >> >> >>>>>>>>>>> <[email protected]> wrote: >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> Hi >> >> >> >>>>>>>>>>>> When i change the rsync option the rsync process doesnt >> >> >> >>>>>>>>>>>> seem to start . Only a defunt process is listed in ps >> >> >> >>>>>>>>>>>> aux. Only when i set rsync option to " " and restart all >> >> >> >>>>>>>>>>>> the process the rsync process is listed in ps aux. >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> >> >> >> >>>>>>>>>>>> On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath >> >> >> >>>>>>>>>>>> Ravishankar <[email protected]> wrote: >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> Yes, rsync config option should have fixed this issue. >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> Could you share the output of the following? >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> 1. gluster volume geo-replication <MASTERVOL> >> >> >> >>>>>>>>>>>>> <SLAVEHOST>::<SLAVEVOL> config rsync-options >> >> >> >>>>>>>>>>>>> 2. ps -ef | grep rsync >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> On Fri, May 31, 2019 at 4:11 PM deepu srinivasan >> >> >> >>>>>>>>>>>>> <[email protected]> wrote: >> >> >> >>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>> Done. >> >> >> >>>>>>>>>>>>>> We got the following result . >> >> >> >>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>> 1559298781.338234 write(2, "rsync: link_stat >> >> >> >>>>>>>>>>>>>>> \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\" >> >> >> >>>>>>>>>>>>>>> failed: No such file or directory (2)", 128 >> >> >> >>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>> seems like a file is missing ? >> >> >> >>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>> On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath >> >> >> >>>>>>>>>>>>>> Ravishankar <[email protected]> wrote: >> >> >> >>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>> Hi, >> >> >> >>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>> Could you take the strace with with more string size? >> >> >> >>>>>>>>>>>>>>> The argument strings are truncated. >> >> >> >>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>> strace -s 500 -ttt -T -p <rsync pid> >> >> >> >>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>> On Fri, May 31, 2019 at 3:17 PM deepu srinivasan >> >> >> >>>>>>>>>>>>>>> <[email protected]> wrote: >> >> >> >>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>> Hi Kotresh >> >> >> >>>>>>>>>>>>>>>> The above-mentioned work around did not work properly. >> >> >> >>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>> On Fri, May 31, 2019 at 3:16 PM deepu srinivasan >> >> >> >>>>>>>>>>>>>>>> <[email protected]> wrote: >> >> >> >>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>> Hi Kotresh >> >> >> >>>>>>>>>>>>>>>>> We have tried the above-mentioned rsync option and >> >> >> >>>>>>>>>>>>>>>>> we are planning to have the version upgrade to 6.0. >> >> >> >>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>> On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath >> >> >> >>>>>>>>>>>>>>>>> Ravishankar <[email protected]> wrote: >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> Hi, >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> This looks like the hang because stderr buffer >> >> >> >>>>>>>>>>>>>>>>>> filled up with errors messages and no one reading >> >> >> >>>>>>>>>>>>>>>>>> it. >> >> >> >>>>>>>>>>>>>>>>>> I think this issue is fixed in latest releases. As >> >> >> >>>>>>>>>>>>>>>>>> a workaround, you can do following and check if it >> >> >> >>>>>>>>>>>>>>>>>> works. >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> Prerequisite: >> >> >> >>>>>>>>>>>>>>>>>> rsync version should be > 3.1.0 >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> Workaround: >> >> >> >>>>>>>>>>>>>>>>>> gluster volume geo-replication <MASTERVOL> >> >> >> >>>>>>>>>>>>>>>>>> <SLAVEHOST>::<SLAVEVOL> config rsync-options >> >> >> >>>>>>>>>>>>>>>>>> "--ignore-missing-args" >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> Thanks, >> >> >> >>>>>>>>>>>>>>>>>> Kotresh HR >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> On Thu, May 30, 2019 at 5:39 PM deepu srinivasan >> >> >> >>>>>>>>>>>>>>>>>> <[email protected]> wrote: >> >> >> >>>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>>> Hi >> >> >> >>>>>>>>>>>>>>>>>>> We were evaluating Gluster geo Replication between >> >> >> >>>>>>>>>>>>>>>>>>> two DCs one is in US west and one is in US east. >> >> >> >>>>>>>>>>>>>>>>>>> We took multiple trials for different file size. >> >> >> >>>>>>>>>>>>>>>>>>> The Geo Replication tends to stop replicating but >> >> >> >>>>>>>>>>>>>>>>>>> while checking the status it appears to be in >> >> >> >>>>>>>>>>>>>>>>>>> Active state. But the slave volume did not >> >> >> >>>>>>>>>>>>>>>>>>> increase in size. >> >> >> >>>>>>>>>>>>>>>>>>> So we have restarted the geo-replication session >> >> >> >>>>>>>>>>>>>>>>>>> and checked the status. The status was in an >> >> >> >>>>>>>>>>>>>>>>>>> active state and it was in History Crawl for a >> >> >> >>>>>>>>>>>>>>>>>>> long time. We have enabled the DEBUG mode in >> >> >> >>>>>>>>>>>>>>>>>>> logging and checked for any error. >> >> >> >>>>>>>>>>>>>>>>>>> There was around 2000 file appeared for syncing >> >> >> >>>>>>>>>>>>>>>>>>> candidate. The Rsync process starts but the rsync >> >> >> >>>>>>>>>>>>>>>>>>> did not happen in the slave volume. Every time the >> >> >> >>>>>>>>>>>>>>>>>>> rsync process appears in the "ps auxxx" list but >> >> >> >>>>>>>>>>>>>>>>>>> the replication did not happen in the slave end. >> >> >> >>>>>>>>>>>>>>>>>>> What would be the cause of this problem? Is there >> >> >> >>>>>>>>>>>>>>>>>>> anyway to debug it? >> >> >> >>>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>>> We have also checked the strace of the rync >> >> >> >>>>>>>>>>>>>>>>>>> program. >> >> >> >>>>>>>>>>>>>>>>>>> it displays something like this >> >> >> >>>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>>> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., >> >> >> >>>>>>>>>>>>>>>>>>> 128" >> >> >> >>>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>>> We are using the below specs >> >> >> >>>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>>> Gluster version - 4.1.7 >> >> >> >>>>>>>>>>>>>>>>>>> Sync mode - rsync >> >> >> >>>>>>>>>>>>>>>>>>> Volume - 1x3 in each end (master and slave) >> >> >> >>>>>>>>>>>>>>>>>>> Intranet Bandwidth - 10 Gig >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>>>>> -- >> >> >> >>>>>>>>>>>>>>>>>> Thanks and Regards, >> >> >> >>>>>>>>>>>>>>>>>> Kotresh H R >> >> >> >>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>>>> -- >> >> >> >>>>>>>>>>>>>>> Thanks and Regards, >> >> >> >>>>>>>>>>>>>>> Kotresh H R >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> >> >> >> >>>>>>>>>>>>> -- >> >> >> >>>>>>>>>>>>> Thanks and Regards, >> >> >> >>>>>>>>>>>>> Kotresh H R >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>> >> >> >> >>>>>>>>>>> -- >> >> >> >>>>>>>>>>> Thanks and Regards, >> >> >> >>>>>>>>>>> Kotresh H R >> >> >> >>>>>>> >> >> >> >>>>>>> >> >> >> >>>>>>> >> >> >> >>>>>>> -- >> >> >> >>>>>>> Thanks and Regards, >> >> >> >>>>>>> Kotresh H R >> >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> -- >> >> >> >>>>> Thanks and Regards, >> >> >> >>>>> Kotresh H R >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > Thanks and Regards, >> >> >> > Kotresh H R _______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
