Re: [Gluster-users] Geo-replication: Entry not present on master. Fixing gfid mismatch in slave

[Sunny Kumar]

Hi David,

You haven't answered my previous question regarding the type of your workload.
---
You can use the below command to enable debug log.

`gluster vol geo-rep  :: config log-level DEBUG`

and after capturing log again switch back to info mode:

`gluster vol geo-rep  :: config log-level INFO`

Please share the debug log and geo-rep config to debug further:
for config:

`gluster vol geo-rep  :: config`

/sunny


On Tue, Jun 2, 2020 at 10:18 AM Strahil Nikolov  wrote:
>
> Hi David,
>
> in which log do you see the entries ?
>
> I think I got an explanation why you see the process  only on one of the 
> master nodes -  geo-rep session is established from only 1 master node  /I 
> hope someone corrects me if I'm wrong/ to one slave node. Thus it will be 
> natural to see the high CPU  usage on only 1 master node in your situation.
>
> Do you see anything else  in the : var/log/glusterfs/geo-replication/ 
> (master nodes) or in /var/log/glusterfs/geo-replication-slaves (slaves) that 
> could hint of the exact issue. I have  a vague feeling that that python 
> script is constantly  looping over some data causing the CPU hog.
>
> Sadly, I can't find an instruction for increasing the log level of the geo 
> rep log .
>
>
> Best  Regards,
> Strahil  Nikolov
>
>
> На 2 юни 2020 г. 6:14:46 GMT+03:00, David Cunningham 
>  написа:
> >Hi Strahil and Sunny,
> >
> >Thank you for the replies. I checked the gfid on the master and slaves
> >and
> >they are the same. After moving the file away and back again it doesn't
> >seem to be having the issue with that file any more.
> >
> >We are still getting higher CPU usage on one of the master nodes than
> >the
> >others. It logs this every few seconds:
> >
> >[2020-06-02 03:10:15.637815] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1384:process] _GMaster: Entry Time
> >Taken
> >   MKD=0   MKN=0   LIN=0   SYM=0   REN=0   RMD=0CRE=0   duration=0.
> >UNL=0
> >[2020-06-02 03:10:15.638010] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1394:process] _GMaster: Data/Metadata
> >Time TakenSETA=0  SETX=0
> >meta_duration=0.data_duration=12.7878
> >   DATA=4  XATT=0
> >[2020-06-02 03:10:15.638286] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1404:process] _GMaster: Batch
> >Completed
> >changelog_end=1591067378entry_stime=(1591067167, 0)
> >changelog_start=1591067364  stime=(1591067377, 0)
> >duration=12.8068
> > num_changelogs=2mode=live_changelog
> >[2020-06-02 03:10:20.658601] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1470:crawl] _GMaster: slave's time
> > stime=(1591067377, 0)
> >[2020-06-02 03:10:34.21799] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1954:syncjob] Syncer: Sync Time Taken
> > duration=0.3826 num_files=8 job=1   return_code=0
> >[2020-06-02 03:10:46.440535] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1384:process] _GMaster: Entry Time
> >Taken
> >   MKD=0   MKN=0   LIN=0   SYM=0   REN=1   RMD=0CRE=2   duration=0.1314
> >UNL=1
> >[2020-06-02 03:10:46.440809] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1394:process] _GMaster: Data/Metadata
> >Time TakenSETA=0  SETX=0
> >meta_duration=0.data_duration=13.0171
> >   DATA=14 XATT=0
> >[2020-06-02 03:10:46.441205] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1404:process] _GMaster: Batch
> >Completed
> >changelog_end=1591067420entry_stime=(1591067419, 0)
> >changelog_start=1591067392  stime=(1591067419, 0)
> >duration=13.0322
> > num_changelogs=3mode=live_changelog
> >[2020-06-02 03:10:51.460925] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1470:crawl] _GMaster: slave's time
> > stime=(1591067419, 0)
> >
> >[2020-06-02 03:11:04.448913] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1954:syncjob] Syncer: Sync Time Taken
> >duration=0.3466 num_files=3 job=1   return_code=0
> >
> >Whereas the other master nodes only log this:
> >
> >[2020-06-02 03:11:33.886938] I [gsyncd(config-get):308:main] :
> >Using
> >session config file
> >path=/var/lib/glusterd/geo-replication/gvol0_nvfs10_gvol0/gsyncd.conf
> >[2020-06-02 03:11:33.993175] I [gsyncd(status):308:main] : Using
> >session config file
> >path=/var/lib/glusterd/geo-replication/gvol0_nvfs10_gvol0/gsyncd.conf
> >
> >Can anyone help with what might cause the high CPU usage on one master
> >node? The process is this one, and is using 70-100% of CPU:
> >
> >python2 /usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py
> >worker gvol0 nvfs10::gvol0 --feedback-fd 15 --local-path
> >/nodirectwritedata/gluster/gvol0 --local-node cafs30 --local-node-id
> >b7521445-ee93-4fed-8ced-6a609fa8c7d4 --slave-id
> >cdcdb210-839c-4306-a4dc-e696b165ed17 --rpc-fd 12,11,9,13 --subvol-num 1
> >--resource-remote nvfs30 --resource-remote-id
> >1e698ccd-aeec-4ec4-96fe-383da8fc3b78
> >
> >Thank you in advance!
> >
> >
> >
> >
> >On Sat, 30 May 2020 at 20:20, Strahil Nikolov 
> >wrote:
> >
> >> Hey David,
> >>
> >> for m

Re: [Gluster-users] Geo-replication: Entry not present on master. Fixing gfid mismatch in slave

[Sunny Kumar]

Hi David,

Looks like you are running a workload that involves lots of rename and
geo-rep is trying to handle those. you can try below patches which
will give you performance benefits.

[1]. https://review.gluster.org/#/c/glusterfs/+/23570/
[2]. https://review.gluster.org/#/c/glusterfs/+/23459/
[3]. https://review.gluster.org/#/c/glusterfs/+/22720/

/sunny

On Sat, May 30, 2020 at 9:20 AM Strahil Nikolov  wrote:
>
> Hey David,
>
> for me a gfid  mismatch means  that the file  was  replaced/recreated  -  
> just like  vim in linux does (and it is expected for config file).
>
> Have  you checked the gfid  of  the file on both source and destination,  do 
> they really match or they are different ?
>
> What happens  when you move away the file  from the slave ,  does it fixes 
> the issue ?
>
> Best Regards,
> Strahil Nikolov
>
> На 30 май 2020 г. 1:10:56 GMT+03:00, David Cunningham 
>  написа:
> >Hello,
> >
> >We're having an issue with a geo-replication process with unusually
> >high
> >CPU use and giving "Entry not present on master. Fixing gfid mismatch
> >in
> >slave" errors. Can anyone help on this?
> >
> >We have 3 GlusterFS replica nodes (we'll call the master), which also
> >push
> >data to a remote server (slave) using geo-replication. This has been
> >running fine for a couple of months, but yesterday one of the master
> >nodes
> >started having unusually high CPU use. It's this process:
> >
> >root@cafs30:/var/log/glusterfs# ps aux | grep 32048
> >root 32048 68.7  0.6 1843140 845756 ?  Rl   02:51 493:51
> >python2
> >/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py worker
> >gvol0 nvfs10::gvol0 --feedback-fd 15 --local-path
> >/nodirectwritedata/gluster/gvol0 --local-node cafs30 --local-node-id
> >b7521445-ee93-4fed-8ced-6a609fa8c7d4 --slave-id
> >cdcdb210-839c-4306-a4dc-e696b165ed17 --rpc-fd 12,11,9,13 --subvol-num 1
> >--resource-remote nvfs30 --resource-remote-id
> >1e698ccd-aeec-4ec4-96fe-383da8fc3b78
> >
> >Here's what is being logged in
> >/var/log/glusterfs/geo-replication/gvol0_nvfs10_gvol0/gsyncd.log:
> >
> >[2020-05-29 21:57:18.843524] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1470:crawl] _GMaster: slave's time
> > stime=(1590789408, 0)
> >[2020-05-29 21:57:30.626172] I [master(worker
> >/nodirectwritedata/gluster/gvol0):813:fix_possible_entry_failures]
> >_GMaster: Entry not present on master. Fixing gfid mismatch in slave.
> >Deleting the entryretry_count=1   entry=({u'uid': 108, u'gfid':
> >u'7c0b75e5-d8b7-454f-8010-112d613c599e', u'gid': 117, u'mode': 33204,
> >u'entry':
> >u'.gfid/c5422396-1578-4b50-a29d-315be2a9c5d8/00a859f7.cfg',
> >u'op': u'CREATE'}, 17, {u'slave_isdir': False, u'gfid_mismatch': True,
> >u'slave_name': None, u'slave_gfid':
> >u'ec4b0ace-2ec4-4ea5-adbc-9f519b81917c', u'name_mismatch': False,
> >u'dst':
> >False})
> >[2020-05-29 21:57:30.627893] I [master(worker
> >/nodirectwritedata/gluster/gvol0):813:fix_possible_entry_failures]
> >_GMaster: Entry not present on master. Fixing gfid mismatch in slave.
> >Deleting the entryretry_count=1   entry=({u'uid': 108, u'gfid':
> >u'a4d52e40-2e2f-4885-be5f-65fe95a8ebd7', u'gid': 117, u'mode': 33204,
> >u'entry':
> >u'.gfid/f857c42e-22f1-4ce4-8f2e-13bdadedde45/polycom_00a859f7.cfg',
> >u'op': u'CREATE'}, 17, {u'slave_isdir': False, u'gfid_mismatch': True,
> >u'slave_name': None, u'slave_gfid':
> >u'ece8da77-b5ea-45a7-9af7-7d4d8f55f74a', u'name_mismatch': False,
> >u'dst':
> >False})
> >[2020-05-29 21:57:30.629532] I [master(worker
> >/nodirectwritedata/gluster/gvol0):813:fix_possible_entry_failures]
> >_GMaster: Entry not present on master. Fixing gfid mismatch in slave.
> >Deleting the entryretry_count=1   entry=({u'uid': 108, u'gfid':
> >u'3c525ad8-aeb2-46b6-9c41-7fb4987916f8', u'gid': 117, u'mode': 33204,
> >u'entry':
> >u'.gfid/f857c42e-22f1-4ce4-8f2e-13bdadedde45/00a859f7-directory.xml',
> >u'op': u'CREATE'}, 17, {u'slave_isdir': False, u'gfid_mismatch': True,
> >u'slave_name': None, u'slave_gfid':
> >u'06717b5a-d842-495d-bd25-aab9cd454490', u'name_mismatch': False,
> >u'dst':
> >False})
> >[2020-05-29 21:57:30.659123] I [master(worker
> >/nodirectwritedata/gluster/gvol0):942:handle_entry_failures] _GMaster:
> >Sucessfully fixed entry ops with gfid mismatch retry_count=1
> >[2020-05-29 21:57:30.659343] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1194:process_change] _GMaster: Retry
> >original entries. count = 1
> >[2020-05-29 21:57:30.725810] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1197:process_change] _GMaster:
> >Sucessfully fixed all entry ops with gfid mismatch
> >[2020-05-29 21:57:31.747319] I [master(worker
> >/nodirectwritedata/gluster/gvol0):1954:syncjob] Syncer: Sync Time Taken
> >duration=0.7409 num_files=18job=1   return_code=0
> >
> >We've verified that the files like polycom_00a859f7.cfg referred to
> >in
> >the error do exist on the master nodes and slave.
> >
> >We found this bug fix:
> >https://bugzilla.redhat.

Re: [Gluster-users] Faulty staus in geo-replication session of a sub-volume

[Sunny Kumar]

Hi Naranderan,


On Sat, May 30, 2020 at 3:52 PM Strahil Nikolov  wrote:
>
> Hello Naranderan,
>
> what OS are you using ? Do you have SELINUX in enforcing mode (verify via 
> 'sestatus') ?
>
> Best Regards,
> Strahil Nikolov
>
>
>
>
>
>
> В събота, 30 май 2020 г., 13:33:05 ч. Гринуич+3, Naranderan Ramakrishnan 
>  написа:
>
>
>
>
>
> Dear Developers/Users,
>
> A geo-rep session of a sub-volume is in 'faulty' status. Please find the 
> setup and log details below.
>
> Setup Details:
> > Gluster version - 7.0
> > Volume configuration - 2x3 (DxR)
> > gysncd permission(master)  - root
> > gysncd permission(slave)  - sas (non-root)
> > glusterd, glusterfsd permissions(master) - root
> > glusterd, glusterfsd permissions(slave) - root
>
> Log details:
> In the master gyncd log, this traceback is printed repeatedly.
> > [2020-05-22 12:09:43.838727] I [master(worker 
> > /home/sas/gluster/data/code-ide):1991:syncjob] Syncer: Sync Time Taken 
> > duration=0.4240 num_files=1 job=1 return_code=0
> > [2020-05-22 12:09:43.944392] E [repce(worker 
> > /home/sas/gluster/data/code-ide):214:__call__] RepceClient: call failed 
> > call=261471:140535761106752:1590149383.8 method=entry_ops error=OSError
> > [2020-05-22 12:09:43.944746] E [syncdutils(worker 
> > /home/sas/gluster/data/code-ide):338:log_raise_exception] : FAIL:
> > Traceback (most recent call last):
> >   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in 
> > main
> > func(args)
> >   File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in 
> > subcmd_worker
> > local.service_loop(remote)
> >   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1305, 
> > in service_loop
> > g3.crawlwrap(oneshot=True)
> >   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in 
> > crawlwrap
> > self.crawl()
> >   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1592, in 
> > crawl
> > self.changelogs_batch_process(changes)
> >   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1492, in 
> > changelogs_batch_process
> > self.process(batch)
> >   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in 
> > process
> > self.process_change(change, done, retry)
> >   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in 
> > process_change
> > failures = self.slave.server.entry_ops(entries)
> >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in 
> > __call__
> > return self.ins(self.meth, *a)
> >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in 
> > __call__
> > raise res
> > OSError: [Errno 13] Permission denied: 
> > '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
> > [2020-05-22 12:09:43.968710] I [repce(agent 
> > /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating 
> > on reaching EOF.
> > [2020-05-22 12:09:44.912470] I [monitor(monitor):280:monitor] Monitor: 
> > worker died in startup phase brick=/home/sas/gluster/data/code-ide
> > [2020-05-22 12:09:44.913692] I 
> > [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status 
> > Change status=Faulty
>
> In salve end, these are printed repeatedly.
> >
> > [2020-05-22 11:23:26.65115] W [gsyncd(slave 
> > 10.47.8.153/home/sas/gluster/data/code-ide):307:main] : Session config 
> > file not exists, using the default config 
> > path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
> > [2020-05-22 11:23:26.77414] I [resource(slave 
> > 10.47.8.153/home/sas/gluster/data/code-ide):1105:connect] GLUSTER: Mounting 
> > gluster volume locally...
> > [2020-05-22 11:23:27.297466] I [resource(slave 
> > 10.47.8.153/home/sas/gluster/data/code-ide):1128:connect] GLUSTER: Mounted 
> > gluster volume duration=1.2199
> > [2020-05-22 11:23:27.298125] I [resource(slave 
> > 10.47.8.153/home/sas/gluster/data/code-ide):1155:service_loop] GLUSTER: 
> > slave listening
> > [2020-05-22 11:23:32.654939] E [repce(slave 
> > 10.47.8.153/home/sas/gluster/data/code-ide):122:worker] : call failed:
> > Traceback (most recent call last):
> >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in 
> > worker
> > res = getattr(self.obj, rmeth)(*in_data[2:])
> >   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706, in 
> > entry_ops
> > collect_failure(e, cmd_ret, uid, gid)
> >   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444, in 
> > collect_failure
> > disk_gfid)
> >   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, 
> > in get_slv_dir_path
> > [ENOENT], [ESTALE])
> >   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, 
> > in errno_wrap
> > return call(*arg)
> > OSError: [Errno 13] Permission denied: 
> > '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
> > [2020-05-22 1

[Gluster-users] Minutes of Gluster Community Meeting [28th April 2020]

[Sunny Kumar]

#
Attendance

Name (#gluster-dev alias) - company
Sunil - Red Hat
Ashit - SODA Foundation (https://sodafoundation.io)
Hari - (hgowtham) Red Hat
Rinku - (rkothiya) - Re Hat
Nikhil - Red Hat
Sheetal Pamecha (spamecha) - Red Hat
Aravinda (aravindavk) - Kadalu.IO 
Ravi (itisravi)- Red Hat
sankarshan (sankarshan) - Kadalu.IO 
Deepshikha(dkhandel) - Red Hat
Amar Tumballi - Kadalu.IO 
Sunny(sunnyk) - Red Hat
User stories
Community

   - Project metrics:

MetricsValue
Coverity  61
Clang Scan  58
Test coverage

71.2
New Issues in last 14 days
master


22
Gluster User Queries in last 14 days

49
Total Github issues  752

[Sunny]Take some time and triage the bugs. we might be able to reduce
around 100(expected)
[Sankarshan]What are the type of questions asked in the mailing list
Are the questions from slack / irc being turned into bugs if we are not
able to fix it?

   - Any release updates?

Rinku proposed to use labels to know that an issue is used for various
branches as well
[Patch by Hari]
Template for tracker :
https://review.gluster.org/#/c/glusterfs/+/24285/
(needs review)
If something is of a high priority move ahead and merge the changes. we
need not wait for the usual procedure to be followed.

[Deepshikha]Fixed the permission issues to assign labels.
Checked a smoke run on github. works fine for now.
Need volunteers for migrating build jobs repo.(Deepshikha will take care)
Need a proper config for github. (who will be the maintainers and stuff
need to be decided)
[Amar] use the maintainers list to figure out who can be made maintainers
for github.
[Deepshikha] need a final call on this to move ahead. Looking into mergify.
[Sunny] send out a mail with the doubts and we can clarify
[Deepshikha] will send out a mail.

Need review for Release notes of Release-8 :
https://review.gluster.org/#/c/glusterfs/+/24372/
[Amar] a patch on master to mark the release is pending.

   -

   Blocker issues across the project?
   - [Sunny] Frequent Geo rep failure(spurious). issue not reproducible
  locally or on softserve.
  - [Deepshikha] looked at the pattern. Have tried to disconnect the
  builders and now it looks stable. Issue might be with the AWS
builder need
  help to figure it out.
  - [Sunny]Smoke failure: no space available.
  - [Deepshikha] Misc is looking into it.
   -

   Notable thread form mailing list
   https://lists.gluster.org/pipermail/gluster-users/2020-April/037969.html
   [Ravi can give more insight into this]
   [Ravi] spurious split brain issue. Gave him a patch and looks like it
   fixed it but it resulted in a crash for which we are expecting a separate
   thread/bug.
   [Sankarshan] Is the work being done by Barak being tracked using a
   issue? (making rebalance more efficient)
   [Ravi] https://github.com/gluster/glusterfs/issues/1138

Conferences
/ Meetups

COVID-19 impact is likely for events!

   - Storage Developer Conference

Important dates:
CFP Closed : No (May 15, 2020)
Schedule Announcement :
Event Open for Registration : Yes (early bird registeration till 8/24/2020)
Last Date of Registration :
Event dates: September 21-24, 2020
Venue: Santa Clara, CA

Talks related to gluster:

   - Devconf.us 

Important dates:
CFP Closed : 27th April
Schedule Announcement : July 3, 2020
Event Open for Registration : July 3 2020
Last Date of Registration :
Event dates: September 23rd to 25th 2020
Venue:Framingham, MA.

   - https://sambaxp.org/

Important dates:
CFP Closed : Closed
Event Open for Registration : Open
Event dates: 26th - 28th of May 2020
Venue: Online

Talks related to gluster:

   - RH Sunmmit is starts today (Virtual)
  - https://www.redhat.com/en/summit

GlusterFS
- v8.0 and beyond

[Sunny] will ask Rafi to send a mail upstream regarding the ls -l
improvements in gluster
Developer
focus

   - Any design specs to discuss?
   Any effort done in the perf side of gluster? gluster vs other storage?
   any design change
   -

Component
status

   - Arbiter - Nil
   - AFR - Nil
   - DHT - Nil
   - EC - Nil
   - DOC - Nil
   - G

Re: [Gluster-users] GlusterFS geo-replication progress question

[Sunny Kumar]

Hi Alexander,

Answers inline below:

On Thu, Apr 2, 2020 at 1:08 AM Alexander Iliev  wrote:
>
> Hi all,
>
> I have a running geo-replication session between two clusters and I'm
> trying to figure out what is the current progress of the replication and
> possibly how much longer it will take.
>
> It has been running for quite a while now (> 1 month), but the thing is
> that both the hardware of the nodes and the link between the two
> clusters aren't that great (e.g., the volumes are backed by rotating
> disks) and the volume is somewhat sizeable (30-ish TB) and given these
> details I'm not really sure how long it is supposed to take normally.
>
> I have several bricks in the volume (same brick size and physical layout
> in both clusters) that are now showing up with a Changelog Crawl status
> and with a recent LAST_SYNCED date in the `gluster colume
> geo-replication status detail` command output which seems to be the
> desired state for all bricks. The rest of the bricks though are in
> Hybrid Crawl state and have been in that state forever.
>
> So I suppose my questions are - how can I tell if the replication
> session is somehow broken and if it's not, then is there are way for me
> to find out the progress and the ETA of the replication?
>
Please go through this section[1] which talks about this.
In Hybrid crawl at present we do not have any accounting information
like how much time it will take to sync data.

> In /var/log/glusterfs/geo-replication/$session_dir/gsyncd.log there are
> some errors like:
>
> [2020-03-31 11:48:47.81269] E [syncdutils(worker
> /data/gfs/store1/8/brick):822:errlog] Popen: command returned error
> cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
> /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto
> -S /tmp/gsync
> d-aux-ssh-6aDWmc/206c4b2c3eb782ea2cf49ab5142bd68b.sock x.x.x.x
> /nonexistent/gsyncd slave  x.x.x.x:: --master-node x.x.x.x
> --master-node-id 9476b8bb-d7ee-489a-b083-875805343e67 --master-brick
>  --local-node x.x.x.x
> 2 --local-node-id 426b564d-35d9-4291-980e-795903e9a386 --slave-timeout
> 120 --slave-log-level INFO --slave-gluster-log-level INFO
> --slave-gluster-command-dir /usr/sbinerror=1
> [2020-03-31 11:48:47.81617] E [syncdutils(worker
> ):826:logerr] Popen: ssh> failed with ValueError.
> [2020-03-31 11:48:47.390397] I [repce(agent
> ):97:service_loop] RepceServer: terminating on reaching EOF.
>

If you are seeing this error at a regular interval then please check
your ssh connection, it might have broken.
If possible please share full traceback form both master and slave to
debug the issue.

> In the brick logs I see stuff like:
>
> [2020-03-29 07:49:05.338947] E [fuse-bridge.c:4167:fuse_xattr_cbk]
> 0-glusterfs-fuse: extended attribute not supported by the backend storage
>
> I don't know if these are critical, from the rest of the logs it looks
> like data is traveling between the clusters.
>
> Any help will be greatly appreciated. Thank you in advance!
>
> Best regards,
> --
> alexander iliev
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
[1]. 
https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/#status

/sunny





Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo-Replication File not Found on /.glusterfs/XX/XX/XXXXXXXXXXXX

[Sunny Kumar]

Hi Senén,

By any chance you perform any operation   on slave volume; like
deleting data directly from slave volume.

Also If possible please share geo-rep slave logs.

/sunny

On Wed, Mar 25, 2020 at 9:15 AM Senén Vidal Blanco
 wrote:
>
> Hi,
> I have a problem with the Geo-Replication system.
> The first synchronization was successful a few days ago. But after a bit of
> filming I run into an error message preventing the sync from continuing.
> I summarize a little the data on the configuration:
>
> Debian 10
> Glusterfs 7.3
> Master volume: archivosvao
> Slave volume: archivossamil
>
> volume geo-replication archivosvao samil::archivossamil config
> access_mount:false
> allow_network:
> change_detector:changelog
> change_interval:5
> changelog_archive_format:%Y%m
> changelog_batch_size:727040
> changelog_log_file:/var/log/glusterfs/geo-replication/
> archivosvao_samil_archivossamil/changes-${local_id}.log
> changelog_log_level:INFO
> checkpoint:0
> cli_log_file:/var/log/glusterfs/geo-replication/cli.log
> cli_log_level:INFO
> connection_timeout:60
> georep_session_working_dir:/var/lib/glusterd/geo-replication/
> archivosvao_samil_archivossamil/
> gfid_conflict_resolution:true
> gluster_cli_options:
> gluster_command:gluster
> gluster_command_dir:/usr/sbin
> gluster_log_file:/var/log/glusterfs/geo-replication/
> archivosvao_samil_archivossamil/mnt-${local_id}.log
> gluster_log_level:INFO
> gluster_logdir:/var/log/glusterfs
> gluster_params:aux-gfid-mount acl
> gluster_rundir:/var/run/gluster
> glusterd_workdir:/var/lib/glusterd
> gsyncd_miscdir:/var/lib/misc/gluster/gsyncd
> ignore_deletes:false
> isolated_slaves:
> log_file:/var/log/glusterfs/geo-replication/archivosvao_samil_archivossamil/
> gsyncd.log
> log_level:INFO
> log_rsync_performance:false
> master_disperse_count:1
> master_distribution_count:1
> master_replica_count:1
> max_rsync_retries:10
> meta_volume_mnt:/var/run/gluster/shared_storage
> pid_file:/var/run/gluster/gsyncd-archivosvao-samil-archivossamil.pid
> remote_gsyncd:
> replica_failover_interval:1
> rsync_command:rsync
> rsync_opt_existing:true
> rsync_opt_ignore_missing_args:true
> rsync_options:
> rsync_ssh_options:
> slave_access_mount:false
> slave_gluster_command_dir:/usr/sbin
> slave_gluster_log_file:/var/log/glusterfs/geo-replication-slaves/
> archivosvao_samil_archivossamil/mnt-${master_node}-${master_brick_id}.log
> slave_gluster_log_file_mbr:/var/log/glusterfs/geo-replication-slaves/
> archivosvao_samil_archivossamil/mnt-mbr-${master_node}-${master_brick_id}.log
> slave_gluster_log_level:INFO
> slave_gluster_params:aux-gfid-mount acl
> slave_log_file:/var/log/glusterfs/geo-replication-slaves/
> archivosvao_samil_archivossamil/gsyncd.log
> slave_log_level:INFO
> slave_timeout:120
> special_sync_mode:
> ssh_command:ssh
> ssh_options:-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/
> lib/glusterd/geo-replication/secret.pem
> ssh_options_tar:-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /
> var/lib/glusterd/geo-replication/tar_ssh.pem
> ssh_port:22
> state_file:/var/lib/glusterd/geo-replication/archivosvao_samil_archivossamil/
> monitor.status
> state_socket_unencoded:
> stime_xattr_prefix:trusted.glusterfs.c7fa7778-
> f2e4-48f9-8817-5811c09964d5.8d4c7ef7-35fc-497a-9425-66f4aced159b
> sync_acls:true
> sync_jobs:3
> sync_method:rsync
> sync_xattrs:true
> tar_command:tar
> use_meta_volume:false
> use_rsync_xattrs:false
> working_dir:/var/lib/misc/gluster/gsyncd/archivosvao_samil_archivossamil/
>
>
> gluster> volume info
>
> Volume Name: archivossamil
> Type: Distribute
> Volume ID: 8d4c7ef7-35fc-497a-9425-66f4aced159b
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1
> Transport-type: tcp
> Bricks:
> Brick1: samil:/brickarchivos/archivos
> Options Reconfigured:
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> features.read-only: on
>
> Volume Name: archivosvao
> Type: Distribute
> Volume ID: c7fa7778-f2e4-48f9-8817-5811c09964d5
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1
> Transport-type: tcp
> Bricks:
> Brick1: vao:/brickarchivos/archivos
> Options Reconfigured:
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> geo-replication.indexing: on
> geo-replication.ignore-pid-check: on
> changelog.changelog: on
>
> Volume Name: home
> Type: Replicate
> Volume ID: 74522542-5d7a-4fdd-9cea-76bf1ff27e7d
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: samil:/brickhome/home
> Brick2: vao:/brickhome/home
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
>
>
> These errors appear in the master logs:
>
>
>
> .
>
> [2020-03-25 09:00:12.554226] I [master(worker /brickarchivos/archivos):
> 1991:syncjob] Syncer: Sync Time Takenjob=1   num_files=2 return_code=0
> duration=0.0483
> [2020-03-25 09:00:12

Re: [Gluster-users] Trying out changelog xlator - human readable output?

[Sunny Kumar]

Hi David,

Link is here:

Changelog parser: https://github.com/aravindavk/gluster_georep_scripts

/sunny

On Mon, Mar 16, 2020 at 10:47 AM Sunny Kumar  wrote:
>
> Hi David,
>
> You can try this Gluster Changelog Parser[1] which converts changelog
> into human readable format.
>
> [1].https://github.com/kotreshhr/gluster-geo-rep
>
> /sunny
>
> On Mon, Mar 16, 2020 at 10:41 AM David Spisla  wrote:
> >
> > Dear Gluster Community,
> > I trying out the changelog xlator to evaluate if it can be used as a kind 
> > of audit log when accessing a volume via nfs. Here is the entry in the 
> > CHANGELOG file when performing $ echo test >> file1.txt && chmod 444 
> > file1.txt :
> >
> > GlusterFS Changelog | version: v1.2 | encoding : 2
> > Ef845378a-f8d4-4dd2-a0fc-946e3d247f84234366553465534ac80f4db-3a41-42f6-b831-49cd9882a564/file1.txtDf845378a-f8d4-4dd2-a0fc-946e3d247f84Mf845378a-f8d4-4dd2-a0fc-946e3d247f8438
> >
> > How one can make it human readable e.g. with full path information?
> >
> > Regards
> > David Spisla
> >
> > ps Below my volume info!
> >
> > Here is my volume setting:
> > fs-davids-c2-n1:~ # gluster vo info repo2
> >
> > Volume Name: repo2
> > Type: Replicate
> > Volume ID: 8b7b64d3-6019-4dac-b2cf-b78e78172724
> > Status: Started
> > Snapshot Count: 0
> > Number of Bricks: 1 x (2 + 1) = 3
> > Transport-type: tcp
> > Bricks:
> > Brick1: fs-davids-c2-n1:/gluster/brick3/glusterbrick
> > Brick2: fs-davids-c2-n2:/gluster/brick3/glusterbrick
> > Brick3: fs-davids-c2-n3:/gluster/arbiter3/glusterbrick (arbiter)
> > Options Reconfigured:
> > changelog.changelog: on
> > changelog.rollover-time: 36000
> > features.scrub-freq: daily
> > features.scrub: Active
> > features.bitrot: on
> > cluster.quorum-type: auto
> > features.default-retention-period: 120
> > storage.ctime: on
> > features.utime: on
> > storage.build-pgfid: on
> > performance.write-behind: off
> > performance.write-behind-window-size: 4MB
> > performance.read-ahead: off
> > performance.cache-refresh-timeout: 10
> > performance.cache-size: 512MB
> > cluster.use-compound-fops: on
> > performance.io-thread-count: 64
> > performance.cache-ima-xattrs: on
> > performance.cache-samba-metadata: on
> > performance.md-cache-timeout: 600
> > performance.cache-invalidation: on
> > performance.stat-prefetch: on
> > cluster.lookup-optimize: on
> > server.event-threads: 32
> > client.event-threads: 32
> > performance.nl-cache-timeout: 600
> > performance.nl-cache: on
> > features.cache-invalidation-timeout: 600
> > features.cache-invalidation: on
> > network.ping-timeout: 10
> > features.retention-mode: enterprise
> > features.worm-file-level: on
> > features.worm: off
> > features.read-only: off
> > user.smb: disable
> > transport.address-family: inet
> > nfs.disable: on
> > performance.client-io-threads: off
> > changelog.capture-del-path: on
> > changelog.encoding: ascii
> >
> > Changelog settings:
> > fs-davids-c2-n1:~ # gluster vo get repo2 all | grep changelog
> > changelog.changelog on
> > changelog.changelog-dir {{ brick.path 
> > }}/.glusterfs/changelogs
> > changelog.encoding  ascii
> > changelog.rollover-time 36000
> > changelog.fsync-interval5
> > changelog.changelog-barrier-timeout 120
> > changelog.capture-del-path  on
> > 
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://bluejeans.com/441850968
> >
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users





Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Trying out changelog xlator - human readable output?

[Sunny Kumar]

Hi David,

You can try this Gluster Changelog Parser[1] which converts changelog
into human readable format.

[1].https://github.com/kotreshhr/gluster-geo-rep

/sunny

On Mon, Mar 16, 2020 at 10:41 AM David Spisla  wrote:
>
> Dear Gluster Community,
> I trying out the changelog xlator to evaluate if it can be used as a kind of 
> audit log when accessing a volume via nfs. Here is the entry in the CHANGELOG 
> file when performing $ echo test >> file1.txt && chmod 444 file1.txt :
>
> GlusterFS Changelog | version: v1.2 | encoding : 2
> Ef845378a-f8d4-4dd2-a0fc-946e3d247f84234366553465534ac80f4db-3a41-42f6-b831-49cd9882a564/file1.txtDf845378a-f8d4-4dd2-a0fc-946e3d247f84Mf845378a-f8d4-4dd2-a0fc-946e3d247f8438
>
> How one can make it human readable e.g. with full path information?
>
> Regards
> David Spisla
>
> ps Below my volume info!
>
> Here is my volume setting:
> fs-davids-c2-n1:~ # gluster vo info repo2
>
> Volume Name: repo2
> Type: Replicate
> Volume ID: 8b7b64d3-6019-4dac-b2cf-b78e78172724
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: fs-davids-c2-n1:/gluster/brick3/glusterbrick
> Brick2: fs-davids-c2-n2:/gluster/brick3/glusterbrick
> Brick3: fs-davids-c2-n3:/gluster/arbiter3/glusterbrick (arbiter)
> Options Reconfigured:
> changelog.changelog: on
> changelog.rollover-time: 36000
> features.scrub-freq: daily
> features.scrub: Active
> features.bitrot: on
> cluster.quorum-type: auto
> features.default-retention-period: 120
> storage.ctime: on
> features.utime: on
> storage.build-pgfid: on
> performance.write-behind: off
> performance.write-behind-window-size: 4MB
> performance.read-ahead: off
> performance.cache-refresh-timeout: 10
> performance.cache-size: 512MB
> cluster.use-compound-fops: on
> performance.io-thread-count: 64
> performance.cache-ima-xattrs: on
> performance.cache-samba-metadata: on
> performance.md-cache-timeout: 600
> performance.cache-invalidation: on
> performance.stat-prefetch: on
> cluster.lookup-optimize: on
> server.event-threads: 32
> client.event-threads: 32
> performance.nl-cache-timeout: 600
> performance.nl-cache: on
> features.cache-invalidation-timeout: 600
> features.cache-invalidation: on
> network.ping-timeout: 10
> features.retention-mode: enterprise
> features.worm-file-level: on
> features.worm: off
> features.read-only: off
> user.smb: disable
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
> changelog.capture-del-path: on
> changelog.encoding: ascii
>
> Changelog settings:
> fs-davids-c2-n1:~ # gluster vo get repo2 all | grep changelog
> changelog.changelog on
> changelog.changelog-dir {{ brick.path }}/.glusterfs/changelogs
> changelog.encoding  ascii
> changelog.rollover-time 36000
> changelog.fsync-interval5
> changelog.changelog-barrier-timeout 120
> changelog.capture-del-path  on
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users





Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Faulty status in Geo-replication

[Sunny Kumar]

Hi deepu,

Can you please check all bricks are up on both master and slave side.
It will also be helpful if you share geo-rep mount logs form slave.

/sunny

On Wed, Mar 11, 2020 at 6:19 AM deepu srinivasan  wrote:
>
> Hi Sunny
> Please update on this issue.
>
> On Tue, Mar 10, 2020 at 11:51 AM deepu srinivasan  wrote:
>>
>> Hi Sunny
>> Any updates on this issue?
>>
>> On Mon, Mar 9, 2020 at 3:24 PM deepu srinivasan  wrote:
>>>
>>> Hi Sunny
>>> This is what I got in slave end
>>>
>>> [2020-03-09 08:08:20.651207] W [gsyncd(slave 
>>> 10.37.xx.xxx/home/sas/gluster/data/code-ide):307:main] : Session 
>>> config file not exists, using the default config 
>>> path=/var/lib/glusterd/geo-replication/code-ide_10.47.8.152_code-ide/gsyncd.conf
>>>
>>> [2020-03-09 08:08:20.663784] I [resource(slave 
>>> 10.37.xx.xxx/home/sas/gluster/data/code-ide):1105:connect] GLUSTER: 
>>> Mounting gluster volume locally...
>>>
>>> [2020-03-09 08:08:21.885207] I [resource(slave 
>>> 10.37.xx.xxx/home/sas/gluster/data/code-ide):1128:connect] GLUSTER: Mounted 
>>> gluster volume   duration=1.2213
>>>
>>> [2020-03-09 08:08:21.885740] I [resource(slave 
>>> 10.37.xx.xxx/home/sas/gluster/data/code-ide):1155:service_loop] GLUSTER: 
>>> slave listening
>>>
>>> [2020-03-09 08:10:29.463335] E [repce(slave 
>>> 10.37.xx.xxx/home/sas/gluster/data/code-ide):122:worker] : call failed:
>>>
>>> Traceback (most recent call last):
>>>
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in 
>>> worker
>>>
>>> res = getattr(self.obj, rmeth)(*in_data[2:])
>>>
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 532, in 
>>> entry_ops
>>>
>>> er = entry_purge(op, entry, gfid, e, uid, gid)
>>>
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 395, in 
>>> entry_purge
>>>
>>> if not matching_disk_gfid(gfid, entry):
>>>
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 574, 
>>> in matching_disk_gfid
>>>
>>> disk_gfid = get_gfid_from_mnt(entry)
>>>
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 570, 
>>> in get_gfid_from_mnt
>>>
>>> GX_GFID_CANONICAL_LEN], [ENOENT], [ESTALE])
>>>
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, 
>>> in errno_wrap
>>>
>>> return call(*arg)
>>>
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/libcxattr.py", line 59, in 
>>> lgetxattr
>>>
>>> return gr_query_xattr(cls, path, siz, 'lgetxattr', attr)
>>>
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/py2py3.py", line 130, in 
>>> gr_query_xattr
>>>
>>> return cls._query_xattr(path, size, syscall, attr)
>>>
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/libcxattr.py", line 48, in 
>>> _query_xattr
>>>
>>> cls.raise_oserr()
>>>
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/libcxattr.py", line 38, in 
>>> raise_oserr
>>>
>>> raise OSError(errn, os.strerror(errn))
>>>
>>> OSError: [Errno 107] Transport endpoint is not connected
>>>
>>> [2020-03-09 08:10:29.535897] I [repce(slave 
>>> 10.37.xx.xxx/home/sas/gluster/data/code-ide):97:service_loop] RepceServer: 
>>> terminating on reaching EOF.
>>>
>>> [2020-03-09 08:10:40.747127] W [gsyncd(slave 
>>> 10.37.xx.xxx/home/sas/gluster/data/code-ide):307:main] : Session 
>>> config file not exists, using the default config 
>>> path=/var/lib/glusterd/geo-replication/code-ide_10.47.8.152_code-ide/gsyncd.conf
>>>
>>>
>>> On Mon, Mar 9, 2020 at 3:15 PM Sunny Kumar  wrote:
>>>>
>>>> Hi Deepu,
>>>> Can you share traceback from slave node.
>>>>
>>>> /sunny
>>>>
>>>>
>>>> On Mon, Mar 9, 2020 at 5:08 AM deepu srinivasan  wrote:
>>>> >
>>>> > Hi guys
>>>> > Have any update on this?
>>>> >
>>>> > On Thu, Mar 5, 2020, 12:30 PM deepu srinivasan  
>>>> > wrote:
>>>> >>
>>>> >>

Re: [Gluster-users] Faulty status in Geo-replication

[Sunny Kumar]

Hi Deepu,
Can you share traceback from slave node.

/sunny


On Mon, Mar 9, 2020 at 5:08 AM deepu srinivasan  wrote:
>
> Hi guys
> Have any update on this?
>
> On Thu, Mar 5, 2020, 12:30 PM deepu srinivasan  wrote:
>>
>> Hi Users/Devs
>>
>> We are getting a faulty status for geo-replication and below are the logs. 
>> How to solve this issue?
>>>
>>> [2020-03-05 06:52:44.858073] E [repce(worker 
>>> /home/sas/gluster/data/code-ide):214:__call__] RepceClient: call failed
>>> call=232079:140660126910272:1583391154.92   method=entry_ops
>>> error=OSError
>>> [2020-03-05 06:52:44.858309] E [syncdutils(worker 
>>> /home/sas/gluster/data/code-ide):338:log_raise_exception] : FAIL:
>>> Traceback (most recent call last):
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in 
>>> main
>>> func(args)
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in 
>>> subcmd_worker
>>> local.service_loop(remote)
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1297, 
>>> in service_loop
>>> g3.crawlwrap(oneshot=True)
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in 
>>> crawlwrap
>>> self.crawl()
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1592, in 
>>> crawl
>>> self.changelogs_batch_process(changes)
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1492, in 
>>> changelogs_batch_process
>>> self.process(batch)
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in 
>>> process
>>> self.process_change(change, done, retry)
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in 
>>> process_change
>>> failures = self.slave.server.entry_ops(entries)
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in 
>>> __call__
>>> return self.ins(self.meth, *a)
>>>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in 
>>> __call__
>>> raise res
>>> OSError: [Errno 1] Operation not permitted
>>> [2020-03-05 06:52:44.887593] I [repce(agent 
>>> /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating 
>>> on reaching EOF.
>>> [2020-03-05 06:52:44.899364] I 
>>> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status 
>>> Change status=Faulty
>>> [2020-03-05 06:52:55.288171] I 
>>> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status 
>>> Change status=Initializing...
>>
>>
>> Our Setup Details
>> We have one volume at each DC. Below is the configuration
>>
>> Volume configuration2x3
>> Replication3
>> Distribution2
>> Gluster version7.0
>> Client TypeFuse Mount
>> Glusterd process
>> GeoReplication Sessionenabled
>> Gluster daemon permission in masterroot
>> Gluster daemon permission in slaveroot
>> Gsynd user daemon in masterroot
>> Gsynd user daemon in slavenon-root
>> Geo Replication configurationroot(master) - non-root(slave)
>>





Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo-replication /var/lib space question

[Sunny Kumar]

Hi Alexander,

Yes that is geo-replication working directory and you can run the
below command to get the location.
  gluster vol geo-rep  :: config working_dir

This directory contains parsed changelogs from backend brick which are
ready to be processed. After a batch is processed it will be
automatically cleaned up for next batch processing.
It does not depends on volume size but on config value "
changelog-batch-size " the max size of changelogs to process per
batch.

/sunny

On Mon, Feb 10, 2020 at 11:07 PM Alexander Iliev
 wrote:
>
> Hello list,
>
> I have been running a geo-replication session for some time now, but at
> some point I noticed that the /var/lib/misc/gluster is eating up the
> storage on my root partition.
>
> I moved the folder away to another partition, but I don't seem to
> remember reading any specific space requirement for /var/lib and
> geo-replication. Did I miss it in the documentation?
>
> Also, does the space used in /var/lib/misc/gluster depend on the
> geo-replicated volume size? What exactly is stored there? (I'm guessing
> that's where gsyncd keeps track of the replicatation progress.)
>
> (I'm running gluster 6.6 on CentOS 7.7.)
>
> Thanks!
> --
> alexander iliev
> 
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] GeoReplication takes too log in hybrid crawl and no sync happens in changelog mode for 2x3 volume.

[Sunny Kumar]

Hi Deepu,

I mean to say rename and deletes on master node will not be sync to
slave node during hybrid crawl.

/sunny

On Wed, Feb 12, 2020 at 5:24 PM deepu srinivasan  wrote:
>
> Thank you for the clarification.  If when will the rename and delete will 
> happen to the slave end.
>
> On Wed, Feb 12, 2020, 10:49 PM Sunny Kumar  wrote:
>>
>> Hi Deepu,
>>
>> Basically in Hybrid crawl, the sync daemon will crawl through full
>> file system in your case 127GB to generating pseudo changelog  which
>> it uses to sync the data, that's why it can take some time.
>>
>> On Wed, Feb 12, 2020 at 8:15 AM deepu srinivasan  wrote:
>> >
>> > Hi Users/Developers
>> > We have a geo-replication session and it takes too long in hybrid crawl 
>> > mode. With 127GB of data in our volume, it takes nearly two days to change 
>> > to changelog mode. And during changelog, the sync does not happen 
>> > properly. How to debug the issue and does it take this long to hybrid mode.
>> What do you mean by sync not happening properly.
>> Also please note when the sync is in hybrid mode it will not sync
>> renames and detetes to  the slave volume.
>>
>> > We have a 2x3 volume and we have enabled geo replication in it .
>> /sunny
>>



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] GeoReplication takes too log in hybrid crawl and no sync happens in changelog mode for 2x3 volume.

[Sunny Kumar]

Hi Deepu,

Basically in Hybrid crawl, the sync daemon will crawl through full
file system in your case 127GB to generating pseudo changelog  which
it uses to sync the data, that's why it can take some time.

On Wed, Feb 12, 2020 at 8:15 AM deepu srinivasan  wrote:
>
> Hi Users/Developers
> We have a geo-replication session and it takes too long in hybrid crawl mode. 
> With 127GB of data in our volume, it takes nearly two days to change to 
> changelog mode. And during changelog, the sync does not happen properly. How 
> to debug the issue and does it take this long to hybrid mode.
What do you mean by sync not happening properly.
Also please note when the sync is in hybrid mode it will not sync
renames and detetes to  the slave volume.

> We have a 2x3 volume and we have enabled geo replication in it .
/sunny



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] GlusterFS problems & alternatives

[Sunny Kumar]

Hi Stefan,

Adding to what Hari said; I want to share a link [1] which talks about
the future releases.
Apart from that, I would suggest you to join community meeting and
share your thoughts.

Hi Strahil,
We have separate version system for "Red Hat's client" and community,
so it's not Gluster-V3.
We encourage everyone to use the latest release which we believe is
more stable and comes with the latest fixes.

[1] https://lists.gluster.org/pipermail/gluster-devel/2020-January/056779.html


/sunny

On Wed, Feb 12, 2020 at 7:23 AM Hari Gowtham  wrote:
>
> Hi,
>
> Stefan, sorry to hear that things are breaking a lot in your cluster, please 
> file a bug(s) with the necessary information so we can take a look.
> If already filed, share it here so we are reminded of it. Fixing broken 
> cluster state should be easy with Gluster.
> There are a few older threads you should be able to find regarding the same.
>
> Do consider the facts that the devs are limited in bandwidth. we do look at 
> the issues and are fixing them actively.
> We may take some time expecting the community to help each other as well. If 
> they couldn't resolve it we get in try to sort it out.
> FYI: You can see dozens of bugs being worked on even in the past 2 days: 
> https://review.gluster.org/#/q/status:open+project:glusterfs
> And there are other activities happening around as well to make gluster 
> project healthier. Like Glusto. We are working on this testing framework
> to cover as many cases as possible. If you can send out a test case, it will 
> be beneficial for you as well as the community.
>
> We don't see many people sending out mails that their cluster is healthy and 
> they are happy (not sure if they think they are spamming.
> which they won't be. It helps us understand how well things are going).
> Thanks Erik and Strahi, for sharing your experience. It means a lot to us :)
> People usually prefer to send a mail when something breaks and that's one 
> main reason all the threads you read are creating negativity.
>
> Do let us know what is the issue and we will try our best to help you out.
>
> Regards,
> Hari.
>
>
>
> On Wed, Feb 12, 2020 at 11:58 AM Strahil Nikolov  
> wrote:
>>
>> On February 12, 2020 12:28:14 AM GMT+02:00, Erik Jacobson 
>>  wrote:
>> >> looking through the last couple of week on this mailing list and
>> >reflecting our own experiences, I have to ask: what is the status of
>> >GlusterFS? So many people here reporting bugs and no solutions are in
>> >sight. GlusterFS clusters break left and right, reboots of a node have
>> >become a warrant for instability and broken clusters, no way to fix
>> >broken clusters. And all of that with recommended settings, and in our
>> >case, enterprise hardware underneath.
>> >
>> >
>> >I have been one of the people asking questions. I sometimes get an
>> >answer, which I appreciate. Other times not. But I'm not paying for
>> >support in this forum so I appreciate what I can get. My questions
>> >are sometimes very hard to summarize and I can't say I've been offering
>> >help as much as I ask. I think I will try to do better.
>> >
>> >
>> >Just to counter with something cool
>> >As we speak now, I'm working on a 2,000 node cluster that will soon be
>> >a
>> >5120 node cluster. We're validating it with the newest version of our
>> >cluster manager.
>> >
>> >It has 12 leader nodes (soon to have 24) that are gluster servers and
>> >gnfs servers.
>> >
>> >I am validating Gluster7.2 (updating from 4.6). Things are looking very
>> >good. 5120 nodes using RO NFS root with RW NFS overmounts (for things
>> >like /var, /etc, ...)...
>> >- boot 1 (where each node creates a RW XFS image on top of NFS for its
>> >  writable area then syncs /var, /etc, etc) -- full boot is 15-16
>> >  minutes for 2007 nodes.
>> >- boot 2 (where the writable area pre-exists and is reused, just
>> >  re-rsynced) -- 8-9 minutes to boot 2007 nodes.
>> >
>> >This is similar to gluster 4, but I think it's saying something to not
>> >have had any failures in this setup on the bleeding edge release level.
>> >
>> >We also use a different volume shared between the leaders and the head
>> >node for shared-storage consoles and system logs. It's working great.
>> >
>> >I haven't had time to test other solutions. Our old solution from SGI
>> >days (ICE, ICE X, etc) was a different model where each leader served
>> >a set of nodes and NFS-booted 288 or so. No shared storage.
>> >
>> >Like you, I've wondered if something else matches this solution. We
>> >like
>> >the shared storage and the ability for a leader to drop and not take
>> >288 noes with it.
>> >
>> >(All nodes running RHEL8.0, Glusterfs 72, CTDB 4.9.1)
>> >
>> >
>> >
>> >So we can say gluster is providing the network boot solution for now
>> >two
>> >supercomputers.
>> >
>> >
>> >
>> >Erik
>> >
>> >
>> >Community Meeting Calendar:
>> >
>> >APAC Schedule -
>> >Every 2nd and 4th Tuesday at 11:30 AM IST
>> >Bridge: https://bluejeans.com/441850968

Re: [Gluster-users] [Gluster-devel] Community Meeting: Make it more reachable

[Sunny Kumar]

On Thu, Feb 6, 2020 at 11:47 AM sankarshan  wrote:
>
> On Thu, 6 Feb 2020 at 16:24, Yati Padia  wrote:
> >
> > Hi,
> > In response to the discussion that we had about the timings of the 
> > community meeting, I would like to propose that we can have it at 3PM/4PM 
> > IST on 11th February to accommodate EMEA/NA zone and if it suits all, we 
> > can fix the timing for next meetings as well.
> > If anyone has any objections regarding this, it can be discussed in this 
> > thread so that we can come up with a fixed timing for the meeting.
> >
>
> I'm not remarkably inconvenienced by the proposed time. 1600
> (UTC+0530) is still 0530 EST (at present day) - so that's perhaps
> early for those in that TZ. I'll defer to the participants from there
> to have their feedback heard.
>
Agree with you Sankarshan; it will be too early for NA TZ.  We can
work for a balanced/overlapping time probably between 1800 to 2000
(UTC+530).
I think in the past it was being hosted at the same time.

We can discuss about it in upcoming meeting.

/sunny
> However, if we do want to switch over, we will have to:
>
> + modify the text on the website
> + modify the email footers
>
> We already have had users pointing out that mixed/confusing messages
> about the meeting time slices make it difficult to understand.
> --
> sankars...@kadalu.io | TZ: UTC+0530
> kadalu.io : Making it easy to provision storage in k8s!
> ___
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-devel mailing list
> gluster-de...@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Community Meeting: Make it more reachable

[Sunny Kumar]

On Wed, Jan 29, 2020 at 9:50 AM Sankarshan Mukhopadhyay
 wrote:
>
> On Wed, 29 Jan 2020 at 15:03, Sunny Kumar  wrote:
> >
> > Hello folks,
> >
> > I would like to propose moving community meeting to a time which is
> > more suitable for EMEA/NA zone, that is merging both of the zone
> > specific meetings into a single one.
> >
>
> I am aware that there are 2 sets of meetings now - APAC and EMEA/NA.
> This came about to ensure that users and community at these TZs have
> the opportunity to participate in time slices that are more
> comfortable. I have never managed to attend a NA/EMEA instance - is
> that not seeing enough participation? There is only a very thin time

I usually join but no one turns ups in meeting, I guess there is some
sort of problem most likely no one is aware/hosting/communication gap.

> slice that overlaps APAC, EMEA and NA. If we want to do this, we would
> need to have a doodle/whenisgood set of options to see how this pans
> out.

Yes, we have to come up with a time which can cover most of TZs.
>
> > It will be really helpful for people who wish to join form these
> > meetings form other time zones and will help users to collaborate with
> > developers in APAC region.
> >
> > Please share your thoughts.
> >
> > /sunny
> >
> ___
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-devel mailing list
> gluster-de...@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Community Meeting: Make it more reachable

[Sunny Kumar]

Hello folks,

I would like to propose moving community meeting to a time which is
more suitable for EMEA/NA zone, that is merging both of the zone
specific meetings into a single one.

It will be really helpful for people who wish to join form these
meetings form other time zones and will help users to collaborate with
developers in APAC region.

Please share your thoughts.

/sunny



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo-Replication Issue while upgrading

[Sunny Kumar]

Thanks Deepu.

I will investigate this can you just summarize the steps which can be
helpful in reproducing this issue.

/sunny

On Fri, Nov 29, 2019 at 7:29 AM deepu srinivasan  wrote:
>
> Hi Sunny
> The issue seems to be a bug.
> The issue got fixed when I restarted the glusterd daemon in the slave 
> machines. The logs in the slave end reported that the mount-broker folder was 
> not in the vol file. So when I restarted the machine it got fixed.
> This might be some race condition.
>
> On Thu, Nov 28, 2019 at 9:00 PM deepu srinivasan  wrote:
>>
>> Hi Sunny
>> I Also got this error in slave end
>>>
>>> [2019-11-28 15:30:12.520461] I [resource(slave 
>>> 192.168.185.89/home/sas/gluster/data/code-misc):1105:connect] GLUSTER: 
>>> Mounting gluster volume locally...
>>>
>>> [2019-11-28 15:30:12.649425] E [resource(slave 
>>> 192.168.185.89/home/sas/gluster/data/code-misc):1013:handle_mounter] 
>>> MountbrokerMounter: glusterd answered   mnt=
>>>
>>> [2019-11-28 15:30:12.650573] E [syncdutils(slave 
>>> 192.168.185.89/home/sas/gluster/data/code-misc):805:errlog] Popen: command 
>>> returned error  cmd=/usr/sbin/gluster --remote-host=localhost system:: 
>>> mount sas user-map-root=sas aux-gfid-mount acl log-level=INFO 
>>> log-file=/var/log/glusterfs/geo-replication-slaves/code-misc_192.168.185.118_code-misc/mnt-192.168.185.89-home-sas-gluster-data-code-misc.log
>>>  volfile-server=localhost volfile-id=code-misc client-pid=-1  error=1
>>>
>>> [2019-11-28 15:30:12.650742] E [syncdutils(slave 
>>> 192.168.185.89/home/sas/gluster/data/code-misc):809:logerr] Popen: 
>>> /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
>>
>>
>> On Thu, Nov 28, 2019 at 6:45 PM deepu srinivasan  wrote:
>>>
>>> root@192.168.185.101/var/log/glusterfs#ssh -oPasswordAuthentication=no 
>>> -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem 
>>> -p 22 sas@192.168.185.118 "sudo gluster volume status"
>>>
>>> **
>>>
>>> WARNING: This system is a restricted access system.  All activity on this 
>>> system is subject to monitoring.  If information collected reveals possible 
>>> criminal activity or activity that exceeds privileges, evidence of such 
>>> activity may be providedto the relevant authorities for further action.
>>>
>>> By continuing past this point, you expressly consent to   this monitoring
>>>
>>> **
>>>
>>> invoking sudo in restricted SSH session is not allowed
>>>
>>>
>>> On Thu, Nov 28, 2019 at 6:04 PM Sunny Kumar  wrote:
>>>>
>>>> Hi Deepu,
>>>>
>>>> Can you try this:
>>>>
>>>> ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
>>>> /var/lib/glusterd/geo-replication/secret.pem -p 22
>>>> sas@192.168.185.118 "sudo gluster volume status"
>>>>
>>>> /sunny
>>>>
>>>>
>>>> On Thu, Nov 28, 2019 at 12:14 PM deepu srinivasan  
>>>> wrote:
>>>> >>
>>>> >> MASTER NODEMASTER VOLMASTER BRICK
>>>> >> SLAVE USERSLAVE SLAVE NODE 
>>>> >> STATUS CRAWL STATUSLAST_SYNCED
>>>> >>
>>>> >> -
>>>> >>
>>>> >> 192.168.185.89 code-misc /home/sas/gluster/data/code-misc
>>>> >> sas   sas@192.168.185.118::code-miscN/A
>>>> >> Faulty N/A N/A
>>>> >>
>>>> >> 192.168.185.101code-misc /home/sas/gluster/data/code-misc
>>>> >> sas   sas@192.168.185.118::code-misc192.168.185.118
>>>> >> PassiveN/A N/A
>>>> >>
>>>> >> 192.168.185.93 code-misc /home/sas/gluster/data/code-misc
>>>> >> sas   sas@192.168.185.118::code-miscN/A 

Re: [Gluster-users] Geo-Replication Issue while upgrading

[Sunny Kumar]

Hi Deepu,

Can you try this:

ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22
sas@192.168.185.118 "sudo gluster volume status"

/sunny


On Thu, Nov 28, 2019 at 12:14 PM deepu srinivasan  wrote:
>>
>> MASTER NODEMASTER VOLMASTER BRICKSLAVE 
>> USERSLAVE SLAVE NODE STATUS 
>> CRAWL STATUSLAST_SYNCED
>>
>> -
>>
>> 192.168.185.89 code-misc /home/sas/gluster/data/code-miscsas 
>>   sas@192.168.185.118::code-miscN/AFaulty N/A
>>  N/A
>>
>> 192.168.185.101code-misc /home/sas/gluster/data/code-miscsas 
>>   sas@192.168.185.118::code-misc192.168.185.118PassiveN/A
>>  N/A
>>
>> 192.168.185.93 code-misc /home/sas/gluster/data/code-miscsas 
>>   sas@192.168.185.118::code-miscN/AFaulty N/A
>>  N/A
>
>
> On Thu, Nov 28, 2019 at 5:43 PM deepu srinivasan  wrote:
>>
>> I Think its configured properly. Should i check something else..
>>
>> root@192.168.185.89/var/log/glusterfs#ssh sas@192.168.185.118 "sudo gluster 
>> volume info"
>>
>> **
>>
>> WARNING: This system is a restricted access system.  All activity on this 
>> system is subject to monitoring.  If information collected reveals possible 
>> criminal activity or activity that exceeds privileges, evidence of such 
>> activity may be providedto the relevant authorities for further action.
>>
>> By continuing past this point, you expressly consent to   this monitoring.-
>>
>> **
>>
>>
>>
>> Volume Name: code-misc
>>
>> Type: Replicate
>>
>> Volume ID: e9b6fbed-fcd0-42a9-ab11-02ec39c2ee07
>>
>> Status: Started
>>
>> Snapshot Count: 0
>>
>> Number of Bricks: 1 x 3 = 3
>>
>> Transport-type: tcp
>>
>> Bricks:
>>
>> Brick1: 192.168.185.118:/home/sas/gluster/data/code-misc
>>
>> Brick2: 192.168.185.45:/home/sas/gluster/data/code-misc
>>
>> Brick3: 192.168.185.84:/home/sas/gluster/data/code-misc
>>
>> Options Reconfigured:
>>
>> features.read-only: enable
>>
>> transport.address-family: inet
>>
>> nfs.disable: on
>>
>> performance.client-io-threads: off
>>
>>
>> On Thu, Nov 28, 2019 at 5:40 PM Sunny Kumar  wrote:
>>>
>>> Hi Deepu,
>>>
>>> Looks like this is error generated due to ssh restrictions:
>>> Can you please check and confirm ssh is properly configured?
>>>
>>>
>>> 2019-11-28 11:59:12.934436] E [syncdutils(worker
>>> /home/sas/gluster/data/code-misc):809:logerr] Popen: ssh>
>>> **
>>>
>>> [2019-11-28 11:59:12.934703] E [syncdutils(worker
>>> /home/sas/gluster/data/code-misc):809:logerr] Popen: ssh> WARNING:
>>> This system is a restricted access system.  All activity on this
>>> system is subject to monitoring.  If information collected reveals
>>> possible criminal activity or activity that exceeds privileges,
>>> evidence of such activity may be providedto the relevant authorities
>>> for further action.
>>>
>>> [2019-11-28 11:59:12.934967] E [syncdutils(worker
>>> /home/sas/gluster/data/code-misc):809:logerr] Popen: ssh> By
>>> continuing past this point, you expressly consent to   this
>>> monitoring.- ZOHO Corporation
>>>
>>> [2019-11-28 11:59:12.935194] E [syncdutils(worker
>>> /home/sas/gluster/data/code-misc):809:logerr] Popen: ssh>
>>> **
>>>
>>> 2019-11-28 11:59:12.944369] I [repce(agent
>>> /home/sas/gluster/data/code-misc):97:service_loop] RepceServer:
>>> terminating on reaching EOF.
>>>
>>> /sunny
>>>
>>> On Thu, Nov 28, 2019 at 12:03 PM deepu

Re: [Gluster-users] Geo-Replication Issue while upgrading

[Sunny Kumar]

Hi Deepu,

Looks like this is error generated due to ssh restrictions:
Can you please check and confirm ssh is properly configured?


2019-11-28 11:59:12.934436] E [syncdutils(worker
/home/sas/gluster/data/code-misc):809:logerr] Popen: ssh>
**

[2019-11-28 11:59:12.934703] E [syncdutils(worker
/home/sas/gluster/data/code-misc):809:logerr] Popen: ssh> WARNING:
This system is a restricted access system.  All activity on this
system is subject to monitoring.  If information collected reveals
possible criminal activity or activity that exceeds privileges,
evidence of such activity may be providedto the relevant authorities
for further action.

[2019-11-28 11:59:12.934967] E [syncdutils(worker
/home/sas/gluster/data/code-misc):809:logerr] Popen: ssh> By
continuing past this point, you expressly consent to   this
monitoring.- ZOHO Corporation

[2019-11-28 11:59:12.935194] E [syncdutils(worker
/home/sas/gluster/data/code-misc):809:logerr] Popen: ssh>
**

2019-11-28 11:59:12.944369] I [repce(agent
/home/sas/gluster/data/code-misc):97:service_loop] RepceServer:
terminating on reaching EOF.

/sunny

On Thu, Nov 28, 2019 at 12:03 PM deepu srinivasan  wrote:
>
>
>
> -- Forwarded message -
> From: deepu srinivasan 
> Date: Thu, Nov 28, 2019 at 5:32 PM
> Subject: Geo-Replication Issue while upgrading
> To: gluster-users 
>
>
> Hi Users/Developers
> I hope you remember the last issue we faced regarding the geo-replication 
> goes to the faulty state while stopping and starting the geo-replication.
>>
>> [2019-11-16 17:29:43.536881] I [gsyncdstatus(worker 
>> /home/sas/gluster/data/code-misc6):281:set_active] GeorepStatus: Worker 
>> Status Change   status=Active
>> [2019-11-16 17:29:43.629620] I [gsyncdstatus(worker 
>> /home/sas/gluster/data/code-misc6):253:set_worker_crawl_status] 
>> GeorepStatus: Crawl Status Change   status=History Crawl
>> [2019-11-16 17:29:43.630328] I [master(worker 
>> /home/sas/gluster/data/code-misc6):1517:crawl] _GMaster: starting history 
>> crawl   turns=1 stime=(1573924576, 0)   entry_stime=(1573924576, 0) 
>> etime=1573925383
>> [2019-11-16 17:29:44.636725] I [master(worker 
>> /home/sas/gluster/data/code-misc6):1546:crawl] _GMaster: slave's time 
>> stime=(1573924576, 0)
>> [2019-11-16 17:29:44.778966] I [master(worker 
>> /home/sas/gluster/data/code-misc6):898:fix_possible_entry_failures] 
>> _GMaster: Fixing ENOENT error in slave. Parent does not exist on master. 
>> Safe to ignore, take out entry   retry_count=1   entry=({'uid': 0, 
>> 'gfid': 'c02519e0-0ead-4fe8-902b-dcae72ef83a3', 'gid': 0, 'mode': 33188, 
>> 'entry': '.gfid/d60aa0d5-4fdf-4721-97dc-9e3e50995dab/368307802', 'op': 
>> 'CREATE'}, 2, {'slave_isdir': False, 'gfid_mismatch': False, 'slave_name': 
>> None, 'slave_gfid': None, 'name_mismatch': False, 'dst': False})
>> [2019-11-16 17:29:44.779306] I [master(worker 
>> /home/sas/gluster/data/code-misc6):942:handle_entry_failures] _GMaster: 
>> Sucessfully fixed entry ops with gfid mismatchretry_count=1
>> [2019-11-16 17:29:44.779516] I [master(worker 
>> /home/sas/gluster/data/code-misc6):1194:process_change] _GMaster: Retry 
>> original entries. count = 1
>> [2019-11-16 17:29:44.879321] E [repce(worker 
>> /home/sas/gluster/data/code-misc6):214:__call__] RepceClient: call failed  
>> call=151945:140353273153344:1573925384.78   method=entry_ops
>> error=OSError
>> [2019-11-16 17:29:44.879750] E [syncdutils(worker 
>> /home/sas/gluster/data/code-misc6):338:log_raise_exception] : FAIL:
>> Traceback (most recent call last):
>>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 322, in 
>> main
>> func(args)
>>   File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 82, in 
>> subcmd_worker
>> local.service_loop(remote)
>>   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1277, in 
>> service_loop
>> g3.crawlwrap(oneshot=True)
>>   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 599, in 
>> crawlwrap
>> self.crawl()
>>   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1555, in 
>> crawl
>> self.changelogs_batch_process(changes)
>>   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1455, in 
>> changelogs_batch_process
>> self.process(batch)
>>   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1290, in 
>> process
>> self.process_change(change, done, retry)
>>   File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1195, in 
>> process_change
>> failures = self.slave.server.entry_ops(entries)
>>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in 
>> __call__
>> return self.ins(self.meth, *a)
>>   File "/usr/libexec/glusterfs/p

[Gluster-users] Fwd: Gluster meetup: India

[Sunny Kumar]

-- Forwarded message -
From: Sunny Kumar 
Date: Thu, Sep 26, 2019 at 10:09 AM
Subject: Re: Gluster meetup: India
To:
Cc: A mailing list of Red Hat associates involved in development,
testing and production of RHGS , storage-eng
, RHS team in Bangalore



Thanks everyone for making this meetup successful! It was a great
event and total participation was 40+. People who joined meetup in
person had a chance to grab gluster goodies.

We started meetup with Atin's talk on current status of gluster. In
second session Yaniv talked about possible candidates of features
landing in Gluster X. 3rd session where Aravinda and Amar talked about
KaDalu (Rethinking gluster management) followed by a demo by Aravinda
on Gluster Dashboard experiment.

We concluded meetup with discussion around user story and migration
from gerrit to github.  I found all session interesting and
informative.

Please find recording of session here[1]. It is divided into 4
separate chapters (sessions).


[1].https://bluejeans.com/s/rotg2

/sunny


On Tue, Sep 24, 2019 at 4:51 PM Sunny Kumar  wrote:
>
> Hello folks,
>
> Please find final agenda for gluster meetup here[1] and bluejeans link[2].
>
> [1]. https://www.meetup.com/glusterfs-India/events/264366771/.
>
> [2].  https://bluejeans.com/2114306332.
>
> /sunny
>
> On Tue, Sep 24, 2019 at 2:49 PM Niels de Vos  wrote:
> >
> > On Tue, Sep 24, 2019 at 01:32:04PM +0530, Sunny Kumar wrote:
> > > Hello folks,
> > >
> > > For people who are not able to join meetup in person have good news as
> > > I will be hosting this meeting on bluejeans and all session will be
> > > recorded.
> > >
> > > Details:
> > >
> > > https://bluejeans.com/2114306332
> >
> > That is great, thanks! Could you share the agenda with the time
> > schedule?
> >
> > Niels
> >
> >
> > >
> > >
> > > To join from a Red Hat Deskphone or Softphone, dial: 84336.
> > > Join Meeting
> > > (Join from computer or phone)
> > > 
> > >
> > > Connecting directly from a room system?
> > >
> > > 1.) Dial: 199.48.152.152 or bjn.vc
> > > 2.) Enter Meeting ID: 2114306332
> > >
> > > Just want to dial in on your phone?
> > >
> > > 1.) Dial one of the following numbers:
> > > 408-915-6466 (US)
> > > See all numbers
> > > 2.) Enter Meeting ID: 2114306332
> > > 3.) Press #
> > >
> > > 
> > > Description:
> > > GlusterFS Bangalore Meetup
> > >
> > > Agenda:
> > >
> > > *) Gluster- X Speaker: Yaniv Kaul
> > >
> > > *) Kadalu - k8s storage with Gluster- Speakers: Aravinda & Amar
> > >
> > > *) Discussion Q/A
> > > ____
> > > Want to test your video connection?
> > > https://bluejeans.com/111
> > >
> > >
> > >
> > > /sunny
> > > On Mon, Sep 23, 2019 at 2:01 PM Sunny Kumar  wrote:
> > > >
> > > > Hello folks,
> > > >
> > > > A gentle reminder!
> > > > Please do RSVP, if planning to attained.
> > > >
> > > > /sunny
> > > >
> > > > On Wed, Sep 18, 2019 at 11:09 AM Sunny Kumar  
> > > > wrote:
> > > > >
> > > > > -- Forwarded message -
> > > > > From: Sunny Kumar 
> > > > > Date: Wed, Aug 28, 2019 at 6:21 PM
> > > > > Subject: Gluster meetup: India
> > > > > To: gluster-users , Gluster Devel
> > > > > 
> > > > > Cc: Yaniv Kaul , Atin Mukherjee
> > > > > , Tumballi, Amar ,
> > > > > Udayakumar Chandrashekhar , Neha Kulkarni
> > > > > 
> > > > >
> > > > >
> > > > > Hello folks,
> > > > >
> > > > > We are hosting Gluster meetup at our office (Redhat-BLR-IN) on 25th
> > > > > September 2019.
> > > > >
> > > > > Please find the agenda and location detail here [1] and plan 
> > > > > accordingly.
> > > > >
> > > > > The highlight of this event will be Gluster -X we will keep on
> > > > > updating agenda with topics, so keep an eye on it.
> > > > >
> > > > > Note:
> > > > >  * RSVP as YES if attending, this will help us to organize the
> > > > > facilities better.
> > > > >
> > > > > If you have any question, please reach out to me or comment on the
> > > > > event page [1].
> > > > >
> > > > > Feel free to share this meetup via other channels.
> > > > >
> > > > > [1]. https://www.meetup.com/glusterfs-India/events/264366771/
> > > > >
> > > > >
> > > > > /sunny
> > >
> > > ---
> > > Note: This list is intended for discussions relating to Red Hat Storage 
> > > products, customers and/or support. Discussions on GlusterFS and Ceph 
> > > architecture, design and engineering should go to relevant upstream 
> > > mailing lists.



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster meetup: India

[Sunny Kumar]

Hi folks,

A gentle reminder!
Please do RSVP, if planning to attained.

/sunny

On Wed, Aug 28, 2019 at 6:21 PM Sunny Kumar  wrote:
>
> Hello folks,
>
> We are hosting Gluster meetup at our office (Redhat-BLR-IN) on 25th
> September 2019.
>
> Please find the agenda and location detail here [1] and plan accordingly.
>
> The highlight of this event will be Gluster -X we will keep on
> updating agenda with topics, so keep an eye on it.
>
> Note:
>  * RSVP as YES if attending, this will help us to organize the
> facilities better.
>
> If you have any question, please reach out to me or comment on the
> event page [1].
>
> Feel free to share this meetup via other channels.
>
> [1]. https://www.meetup.com/glusterfs-India/events/264366771/
>
>
> /sunny



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Issues with Geo-replication (GlusterFS 6.3 on Ubuntu 18.04)

[Sunny Kumar]

Thank you for the explanation Kaleb.

Alexander,

This fix will be available with next release for all supported versions.

/sunny

On Mon, Sep 2, 2019 at 6:47 PM Kaleb Keithley  wrote:
>
> Fixes on master (before or after the release-7 branch was taken) almost 
> certainly warrant a backport IMO to at least release-6, and probably 
> release-5 as well.
>
> We used to have a "tracker" BZ for each minor release (e.g. 6.6) to keep 
> track of backports by cloning the original BZ and changing the Version, and 
> adding that BZ to the tracker. I'm not sure what happened to that practice. 
> The last ones I can find are for 6.3 and 5.7;  
> https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-6.3 and 
> https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-5.7
>
> It isn't enough to just backport recent fixes on master to release-7. We are 
> supposedly continuing to maintain release-6 and release-5 after release-7 
> GAs. If that has changed, I haven't seen an announcement to that effect. I 
> don't know why our developers don't automatically backport to all the 
> actively maintained releases.
>
> Even if there isn't a tracker BZ, you can always create a backport BZ by 
> cloning the original BZ and change the release to 6. That'd be a good place 
> to start.
>
> On Sun, Sep 1, 2019 at 8:45 AM Alexander Iliev  
> wrote:
>>
>> Hi Strahil,
>>
>> Yes, this might be right, but I would still expect fixes like this to be
>> released for all supported major versions (which should include 6.) At
>> least that's how I understand https://www.gluster.org/release-schedule/.
>>
>> Anyway, let's wait for Sunny to clarify.
>>
>> Best regards,
>> alexander iliev
>>
>> On 9/1/19 2:07 PM, Strahil Nikolov wrote:
>> > Hi Alex,
>> >
>> > I'm not very deep into bugzilla stuff, but for me NEXTRELEASE means v7.
>> >
>> > Sunny,
>> > Am I understanding it correctly ?
>> >
>> > Best Regards,
>> > Strahil Nikolov
>> >
>> > В неделя, 1 септември 2019 г., 14:27:32 ч. Гринуич+3, Alexander Iliev
>> >  написа:
>> >
>> >
>> > Hi Sunny,
>> >
>> > Thank you for the quick response.
>> >
>> > It's not clear to me however if the fix has been already released or not.
>> >
>> > The bug status is CLOSED NEXTRELEASE and according to [1] the
>> > NEXTRELEASE resolution means that the fix will be included in the next
>> > supported release. The bug is logged against the mainline version
>> > though, so I'm not sure what this means exactly.
>> >
>> >  From the 6.4[2] and 6.5[3] release notes it seems it hasn't been
>> > released yet.
>> >
>> > Ideally I would not like to patch my systems locally, so if you have an
>> > ETA on when this will be out officially I would really appreciate it.
>> >
>> > Links:
>> > [1] https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_status
>> > [2] https://docs.gluster.org/en/latest/release-notes/6.4/
>> > [3] https://docs.gluster.org/en/latest/release-notes/6.5/
>> >
>> > Thank you!
>> >
>> > Best regards,
>> >
>> > alexander iliev
>> >
>> > On 8/30/19 9:22 AM, Sunny Kumar wrote:
>> >  > Hi Alexander,
>> >  >
>> >  > Thanks for pointing that out!
>> >  >
>> >  > But this issue is fixed now you can see below link for bz-link and 
>> > patch.
>> >  >
>> >  > BZ - https://bugzilla.redhat.com/show_bug.cgi?id=1709248
>> >  >
>> >  > Patch - https://review.gluster.org/#/c/glusterfs/+/22716/
>> >  >
>> >  > Hope this helps.
>> >  >
>> >  > /sunny
>> >  >
>> >  > On Fri, Aug 30, 2019 at 2:30 AM Alexander Iliev
>> >  > mailto:glus...@mamul.org>> wrote:
>> >  >>
>> >  >> Hello dear GlusterFS users list,
>> >  >>
>> >  >> I have been trying to set up geo-replication between two clusters for
>> >  >> some time now. The desired state is (Cluster #1) being replicated to
>> >  >> (Cluster #2).
>> >  >>
>> >  >> Here are some details about the setup:
>> >  >>
>> >  >> Cluster #1: three nodes connected via a local network (172.31.35.0/24),
>> >  >> one replicated (3 replica) volume.
>> >  >>
>> >  >> Cluster #2: three nodes connected 

Re: [Gluster-users] Issues with Geo-replication (GlusterFS 6.3 on Ubuntu 18.04)

[Sunny Kumar]

Hi Alexander,

Thanks for pointing that out!

But this issue is fixed now you can see below link for bz-link and patch.

BZ - https://bugzilla.redhat.com/show_bug.cgi?id=1709248

Patch - https://review.gluster.org/#/c/glusterfs/+/22716/

Hope this helps.

/sunny

On Fri, Aug 30, 2019 at 2:30 AM Alexander Iliev
 wrote:
>
> Hello dear GlusterFS users list,
>
> I have been trying to set up geo-replication between two clusters for
> some time now. The desired state is (Cluster #1) being replicated to
> (Cluster #2).
>
> Here are some details about the setup:
>
> Cluster #1: three nodes connected via a local network (172.31.35.0/24),
> one replicated (3 replica) volume.
>
> Cluster #2: three nodes connected via a local network (172.31.36.0/24),
> one replicated (3 replica) volume.
>
> The two clusters are connected to the Internet via separate network
> adapters.
>
> Only SSH (port 22) is open on cluster #2 nodes' adapters connected to
> the Internet.
>
> All nodes are running Ubuntu 18.04 and GlusterFS 6.3 installed from [1].
>
> The first time I followed the guide[2] everything went fine up until I
> reached the "Create the session" step. That was like a month ago, then I
> had to temporarily stop working in this and now I am coming back to it.
>
> Currently, if I try to see the mountbroker status I get the following:
>
> > # gluster-mountbroker status
> > Traceback (most recent call last):
> >   File "/usr/sbin/gluster-mountbroker", line 396, in 
> > runcli()
> >   File "/usr/lib/python3/dist-packages/gluster/cliutils/cliutils.py", line 
> > 225, in runcli
> > cls.run(args)
> >   File "/usr/sbin/gluster-mountbroker", line 275, in run
> > out = execute_in_peers("node-status")
> >   File "/usr/lib/python3/dist-packages/gluster/cliutils/cliutils.py",
> line 127, in execute_in_peers
> > raise GlusterCmdException((rc, out, err, " ".join(cmd)))
> > gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Unable to
> end. Error : Success\n', 'gluster system:: execute mountbroker.py
> node-status')
>
> And in /var/log/gluster/glusterd.log I have:
>
> > [2019-08-10 15:24:21.418834] E [MSGID: 106336]
> [glusterd-geo-rep.c:5413:glusterd_op_sys_exec] 0-management: Unable to
> end. Error : Success
> > [2019-08-10 15:24:21.418908] E [MSGID: 106122]
> [glusterd-syncop.c:1445:gd_commit_op_phase] 0-management: Commit of
> operation 'Volume Execute system commands' failed on localhost : Unable
> to end. Error : Success
>
> So, I have two questions right now:
>
> 1) Is there anything wrong with my setup (networking, open ports, etc.)?
> Is it expected to work with this setup or should I redo it in a
> different way?
> 2) How can I troubleshoot the current status of my setup? Can I find out
> what's missing/wrong and continue from there or should I just start from
> scratch?
>
> Links:
> [1] http://ppa.launchpad.net/gluster/glusterfs-6/ubuntu
> [2]
> https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/
>
> Thank you!
>
> Best regards,
> --
> alexander iliev
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster meetup: India

[Sunny Kumar]

Hello folks,

We are hosting Gluster meetup at our office (Redhat-BLR-IN) on 25th
September 2019.

Please find the agenda and location detail here [1] and plan accordingly.

The highlight of this event will be Gluster -X we will keep on
updating agenda with topics, so keep an eye on it.

Note:
 * RSVP as YES if attending, this will help us to organize the
facilities better.

If you have any question, please reach out to me or comment on the
event page [1].

Feel free to share this meetup via other channels.

[1]. https://www.meetup.com/glusterfs-India/events/264366771/


/sunny
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo Replication Failure: libgfchangelog.so: cannot open shared object file

[Sunny Kumar]

Thanks Andy for detailed answer.

Cédric,

If possible can you share your updated document/patch which can be
merged to gluster doc[1] and made available to everyone[2].

[1]. https://github.com/gluster/glusterdocs/
[2] https://docs.gluster.org/en/latest/
.
/sunny


On Wed, Aug 28, 2019 at 12:13 PM ROUVRAIS Cedric ResgBscRscDef
 wrote:
>
> Hi Andy,
>
>
>
> Thanks for your reply, I actually fixed the issue – shared library wasn’t 
> installed, so I reinstalled it.
>
>
>
> I realize that the documentation for geo replication would need more 
> information
>
>
>
> I will try a clean install on a new image and document it more precisely.
>
>
>
> Thanks,
>
>
>
> Cédric
>
>
>
>
>
> De : Andy Coates 
> Envoyé : mercredi 28 août 2019 05:18
> À : ROUVRAIS Cedric ResgBscRscDef 
> Cc : gluster-users@gluster.org
> Objet : Re: [Gluster-users] Geo Replication Failure: libgfchangelog.so: 
> cannot open shared object file
>
>
>
> We saw this with 4.1.x RPM on OEL (can't recall which specific version and 
> haven't checked if its fixed in later, at least up to 4.1.6), but the issue 
> seemed to be it just wasn't symlinked for some reason, so we symlinked 
> libgfchangelog.so to /lib64/libgfchangelog.so.0
>
>
>
> Not sure if the python code is meant to ask for libgfchangelog.so.0 in the 
> first place, or if the RPM is meant to symlink it post-install.  In 7.x the 
> script seems to use a different method for finding it too.
>
>
>
> Andy.
>
>
>
> On Tue, 27 Aug 2019 at 04:21, ROUVRAIS Cedric ResgBscRscDef 
>  wrote:
>
> Hello,
>
>
>
> Having some slight difficulties with GeoReplication across 2 servers where 
> the user is root on both sides: libgfchangelog.so: cannot open shared object 
> file: No such file or directory
>
>
>
> I get this error on the master node. I haven’t been able to get around it 
> (I’ve deleted and recreated the configuration ex-nihilo a couple of time to 
> no avail).
>
>
>
> [2019-08-26 17:43:24.213577] I [gsyncdstatus(monitor):248:set_worker_status] 
> GeorepStatus: Worker Status Change status=Initializing...
>
> [2019-08-26 17:43:24.213959] I [monitor(monitor):157:monitor] Monitor: 
> starting gsyncd worker   brick=/vol/gluster/fr/brick1/gv_master 
> slave_node=srv_slave
>
> [2019-08-26 17:43:24.259780] I [gsyncd(agent 
> /vol/gluster/fr/brick1/gv_master):309:main] : Using session config file  
> 
> path=/var/lib/glusterd/geo-replication/gv_master_srv_slave_gv_slave/gsyncd.conf
>
> [2019-08-26 17:43:24.261590] I [changelogagent(agent 
> /vol/gluster/fr/brick1/gv_master):72:__init__] ChangelogAgent: Agent 
> listining...
>
> [2019-08-26 17:43:24.266072] I [gsyncd(worker 
> /vol/gluster/fr/brick1/gv_master):309:main] : Using session config file  
>
> path=/var/lib/glusterd/geo-replication/gv_master_srv_slave_gv_slave/gsyncd.conf
>
> [2019-08-26 17:43:24.280029] I [resource(worker 
> /vol/gluster/fr/brick1/gv_master):1379:connect_remote] SSH: Initializing SSH 
> connection between master and slave...
>
> [2019-08-26 17:43:25.749739] I [resource(worker 
> /vol/gluster/fr/brick1/gv_master):1426:connect_remote] SSH: SSH connection 
> between master and slave established.   duration=1.4696
>
> [2019-08-26 17:43:25.749941] I [resource(worker 
> /vol/gluster/fr/brick1/gv_master):1098:connect] GLUSTER: Mounting gluster 
> volume locally...
>
> [2019-08-26 17:43:26.824810] I [resource(worker 
> /vol/gluster/fr/brick1/gv_master):1121:connect] GLUSTER: Mounted gluster 
> volumeduration=1.0747
>
> [2019-08-26 17:43:26.825088] I [subcmds(worker 
> /vol/gluster/fr/brick1/gv_master):80:subcmd_worker] : Worker spawn 
> successful. Acknowledging back to monitor
>
> [2019-08-26 17:43:26.888922] E [repce(agent 
> /vol/gluster/fr/brick1/gv_master):122:worker] : call failed:
>
> Traceback (most recent call last):
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in 
> worker
>
> res = getattr(self.obj, rmeth)(*in_data[2:])
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 37, 
> in init
>
> return Changes.cl_init()
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 21, 
> in __getattr__
>
> from libgfchangelog import Changes as LChanges
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 18, 
> in 
>
> class Changes(object):
>
>  File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 20, 
> in Changes
>
> use_errno=True)
>
>   File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__
>
> self._handle = _dlopen(self._name, mode)
>
> OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> directory
>
>
>
>
>
> Thank you for any insight,
>
>
>
> Cédric
>
>
>
> =
>
> Ce message et toutes les pieces jointes (ci-apres le "message")
> sont confidentiels et susceptibles de contenir des informations
> couvertes par le secret professionnel. Ce message est etabli
> a l'intentio

Re: [Gluster-users] Brick missing trusted.glusterfs.dht xattr

[Sunny Kumar]

HI Matthew,

Can you share geo-rep logs and one more log file
(changes-.log) it will help to pinpoint actual reason
behind failure.

/sunny

On Mon, Jul 29, 2019 at 9:13 AM Nithya Balachandran  wrote:
>
>
>
> On Sat, 27 Jul 2019 at 02:31, Matthew Benstead  wrote:
>>
>> Ok thank-you for explaining everything - that makes sense.
>>
>> Currently the brick file systems are pretty evenly distributed so I probably 
>> won't run the fix-layout right now.
>>
>> Would this state have any impact on geo-replication? I'm trying to 
>> geo-replicate this volume, but am getting a weird error: "Changelog register 
>> failed error=[Errno 21] Is a directory"
>
>
> It should not. Sunny, can you comment on this?
>
> Regards,
> Nithya
>>
>>
>> I assume this is related to something else, but I wasn't sure.
>>
>> Thanks,
>>  -Matthew
>>
>> --
>> Matthew Benstead
>> System Administrator
>> Pacific Climate Impacts Consortium
>> University of Victoria, UH1
>> PO Box 1800, STN CSC
>> Victoria, BC, V8W 2Y2
>> Phone: +1-250-721-8432
>> Email: matth...@uvic.ca
>>
>> On 7/26/19 12:02 AM, Nithya Balachandran wrote:
>>
>>
>>
>> On Fri, 26 Jul 2019 at 01:56, Matthew Benstead  wrote:
>>>
>>> Hi Nithya,
>>>
>>> Hmm... I don't remember if I did, but based on what I'm seeing it sounds 
>>> like I probably didn't run rebalance or fix-layout.
>>>
>>> It looks like folders that haven't had any new files created have a dht of 
>>> 0, while other folders have non-zero values.
>>>
>>> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex 
>>> /mnt/raid6-storage/storage/ | grep dht
>>> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex 
>>> /mnt/raid6-storage/storage/home | grep dht
>>> trusted.glusterfs.dht=0x
>>> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex 
>>> /mnt/raid6-storage/storage/home/matthewb | grep dht
>>> trusted.glusterfs.dht=0x00014924921a6db6dbc7
>>>
>>> If I just run the fix-layout command will it re-create all of the dht 
>>> values or just the missing ones?
>>
>>
>> A fix-layout will recalculate the layouts entirely so files all the values 
>> will change. No files will be moved.
>> A rebalance will recalculate the layouts like the fix-layout but will also 
>> move files to their new locations based on the new layout ranges. This could 
>> take a lot of time depending on the number of files/directories on the 
>> volume. If you do this, I would recommend that you turn off lookup-optimize 
>> until the rebalance is over.
>>
>>>
>>> Since the brick is already fairly size balanced could I get away with 
>>> running fix-layout but not rebalance? Or would the new dht layout mean 
>>> slower accesses since the files may be expected on different bricks?
>>
>>
>> The first access for a file will be slower. The next one will be faster as 
>> the location will be cached in the client's in-memory structures.
>> You may not need to run either a fix-layout or a rebalance if new file 
>> creations will be in directories created after the add-brick. Gluster will 
>> automatically include all 7 bricks for those directories.
>>
>> Regards,
>> Nithya
>>
>>>
>>> Thanks,
>>>  -Matthew
>>>
>>> --
>>> Matthew Benstead
>>> System Administrator
>>> Pacific Climate Impacts Consortium
>>> University of Victoria, UH1
>>> PO Box 1800, STN CSC
>>> Victoria, BC, V8W 2Y2
>>> Phone: +1-250-721-8432
>>> Email: matth...@uvic.ca
>>>
>>> On 7/24/19 9:30 PM, Nithya Balachandran wrote:
>>>
>>>
>>>
>>> On Wed, 24 Jul 2019 at 22:12, Matthew Benstead  wrote:

 So looking more closely at the trusted.glusterfs.dht attributes from the 
 bricks it looks like they cover the entire range... and there is no range 
 left for gluster07.

 The first 6 bricks range from 0x to 0x - so... is there a 
 way to re-calculate what the dht values should be? Each of the bricks 
 should have a gap

 Gluster05  -> 2aa9
 Gluster06 2aaa -> 5553
 Gluster01 5554 -> 7ffd
 Gluster02 7ffe -> aaa7
 Gluster03 aaa8 -> d551
 Gluster04 d552 -> 
 Gluster07 None

 If we split the range into 7 servers that would be a gap of about 
 0x24924924 for each server.

 Now in terms of the gluster07 brick, about 2 years ago the RAID array the 
 brick was stored on became corrupted. I ran the remove-brick force 
 command, then provisioned a new server, ran the add-brick command and then 
 restored the missing files from backup by copying them back to the main 
 gluster mount (not the brick).

>>>
>>> Did you run a rebalance after performing the add-brick? Without a 
>>> rebalance/fix-layout , the layout for existing directories on the volume 
>>> will not  be updated to use the new brick as well.
>>>
>>> That the layout does not include the new brick in the root dir is in itself 
>>> is not a problem. Do you create a lot of files directly in the root of the 
>>> volum

Re: [Gluster-users] Geo Replication stops replicating

[Sunny Kumar]

l 
>> mount request [No such file or directory]
>>
>> [2019-06-06 11:55:44.057522] I [MSGID: 106496] 
>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount 
>> req
>>
>> [2019-06-06 11:55:44.057552] E [MSGID: 106061] 
>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
>> mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-06 11:55:44.057562] W [MSGID: 106176] 
>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
>> mount request [No such file or directory]
>>
>> [2019-06-06 11:55:54.655681] I [MSGID: 106496] 
>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount 
>> req
>>
>> [2019-06-06 11:55:54.655741] E [MSGID: 106061] 
>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
>> mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-06 11:55:54.655752] W [MSGID: 106176] 
>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
>> mount request [No such file or directory]
>
>
> On Thu, Jun 6, 2019 at 5:09 PM Sunny Kumar  wrote:
>>
>> Whats current trackback please share.
>>
>> -Sunny
>>
>>
>> On Thu, Jun 6, 2019 at 4:53 PM deepu srinivasan  wrote:
>> >
>> > Hi Sunny
>> > I have changed the file in /usr/libexec/glusterfs/peer_mountbroker.py as 
>> > mentioned in the patch.
>> > Now the "gluster-mountbroker status" command is working fine. But the 
>> > geo-replication seems to be in the faulty state still.
>> >
>> >
>> > Thankyou
>> > Deepak
>> >
>> > On Thu, Jun 6, 2019 at 4:10 PM Sunny Kumar  wrote:
>> >>
>> >> Above error can be tracked here:
>> >>
>> >> https://bugzilla.redhat.com/show_bug.cgi?id=1709248
>> >>
>> >> and patch link:
>> >> https://review.gluster.org/#/c/glusterfs/+/22716/
>> >>
>> >> You can apply patch and test it however its waiting on regression to
>> >> pass and merge.
>> >>
>> >> -Sunny
>> >>
>> >>
>> >> On Thu, Jun 6, 2019 at 4:00 PM deepu srinivasan  
>> >> wrote:
>> >> >
>> >> > Hi
>> >> > I have followed the following steps to create the geo-replication but 
>> >> > the status seems to be in a faulty state.
>> >> >
>> >> > Steps :
>> >> >
>> >> > Installed cluster version 5.6 in totally six nodes.
>> >> >>
>> >> >> glusterfs 5.6
>> >> >>
>> >> >> Repository revision: git://git.gluster.org/glusterfs.git
>> >> >>
>> >> >> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
>> >> >>
>> >> >> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> >> >>
>> >> >> It is licensed to you under your choice of the GNU Lesser
>> >> >>
>> >> >> General Public License, version 3 or any later version (LGPLv3
>> >> >>
>> >> >> or later), or the GNU General Public License, version 2 (GPLv2),
>> >> >>
>> >> >> in all cases as published by the Free Software Foundation
>> >> >
>> >> >
>> >> > peer_probed the first three nodes and second three nodes.
>> >> >
>> >> >
>> >> >
>> >> > Added new volume in both the clusters
>> >> >
>> >> >
>> >> >
>> >> > execute gluster-mountbroker commands and restarted glusterd.
>> >> >>
>> >> >> gluster-mountbroker setup /var/mountbroker-root sas
>> >> >>
>> >> >> gluster-mountbroker remove --volume code-misc --user sas
>> >> >
>> >> >
>> >> > configured a passwordless sssh from master to slave
>> >> >>
>> >> >> ssh-keygen; ssh-copy-id sas@192.168.185.107
>> >> >
>> >> > created a common pem pub file
>> >> >>
>> >> >> gluster system:: execute gsec_create
>> >> >
>> >> > created geo-replication session.
>> >> >>
>> >> >> gluster volume geo-replication code-misc 
>> >> >> sas@192.168.185.107::code-misc create push-pem
>> >

Re: [Gluster-users] Geo Replication stops replicating

[Sunny Kumar]

Whats current trackback please share.

-Sunny


On Thu, Jun 6, 2019 at 4:53 PM deepu srinivasan  wrote:
>
> Hi Sunny
> I have changed the file in /usr/libexec/glusterfs/peer_mountbroker.py as 
> mentioned in the patch.
> Now the "gluster-mountbroker status" command is working fine. But the 
> geo-replication seems to be in the faulty state still.
>
>
> Thankyou
> Deepak
>
> On Thu, Jun 6, 2019 at 4:10 PM Sunny Kumar  wrote:
>>
>> Above error can be tracked here:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1709248
>>
>> and patch link:
>> https://review.gluster.org/#/c/glusterfs/+/22716/
>>
>> You can apply patch and test it however its waiting on regression to
>> pass and merge.
>>
>> -Sunny
>>
>>
>> On Thu, Jun 6, 2019 at 4:00 PM deepu srinivasan  wrote:
>> >
>> > Hi
>> > I have followed the following steps to create the geo-replication but the 
>> > status seems to be in a faulty state.
>> >
>> > Steps :
>> >
>> > Installed cluster version 5.6 in totally six nodes.
>> >>
>> >> glusterfs 5.6
>> >>
>> >> Repository revision: git://git.gluster.org/glusterfs.git
>> >>
>> >> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
>> >>
>> >> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> >>
>> >> It is licensed to you under your choice of the GNU Lesser
>> >>
>> >> General Public License, version 3 or any later version (LGPLv3
>> >>
>> >> or later), or the GNU General Public License, version 2 (GPLv2),
>> >>
>> >> in all cases as published by the Free Software Foundation
>> >
>> >
>> > peer_probed the first three nodes and second three nodes.
>> >
>> >
>> >
>> > Added new volume in both the clusters
>> >
>> >
>> >
>> > execute gluster-mountbroker commands and restarted glusterd.
>> >>
>> >> gluster-mountbroker setup /var/mountbroker-root sas
>> >>
>> >> gluster-mountbroker remove --volume code-misc --user sas
>> >
>> >
>> > configured a passwordless sssh from master to slave
>> >>
>> >> ssh-keygen; ssh-copy-id sas@192.168.185.107
>> >
>> > created a common pem pub file
>> >>
>> >> gluster system:: execute gsec_create
>> >
>> > created geo-replication session.
>> >>
>> >> gluster volume geo-replication code-misc sas@192.168.185.107::code-misc 
>> >> create push-pem
>> >
>> >  executed the following command in slave
>> >>
>> >> /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh sas code-misc code-misc
>> >
>> > started the gluster geo-replication.
>> >>
>> >> gluster volume geo-replication code-misc sas@192.168.185.107::code-misc 
>> >> start
>> >
>> >
>> > Now the geo-replication works fine.
>> > Tested with 2000 files All seems to sync finely.
>> >
>> > Now I updated all the node to version 6.2 by using rpms which were built 
>> > by the source code in a docker container in my personal machine.
>> >
>> >
>> >> gluster --version
>> >>
>> >> glusterfs 6.2
>> >>
>> >> Repository revision: git://git.gluster.org/glusterfs.git
>> >>
>> >> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
>> >>
>> >> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> >>
>> >> It is licensed to you under your choice of the GNU Lesser
>> >>
>> >> General Public License, version 3 or any later version (LGPLv3
>> >>
>> >> or later), or the GNU General Public License, version 2 (GPLv2),
>> >>
>> >> in all cases as published by the Free Software Foundation.
>> >
>> >
>> > I have stopped the glusterd daemons in all the node along with the volume 
>> > and geo-replication.
>> > Now I started the daemons, volume and geo-replication session the status 
>> > seems to be faulty.
>> > Also noted that the result of "gluster-mountbroker status" command always 
>> > end in python exception like this
>> >>
>> >> Traceback (most recent call last):
>> >>
>> >>   File "/usr/sbin/gluster-mountbroker", line 396, in 
>>

Re: [Gluster-users] Geo Replication stops replicating

[Sunny Kumar]

Above error can be tracked here:

https://bugzilla.redhat.com/show_bug.cgi?id=1709248

and patch link:
https://review.gluster.org/#/c/glusterfs/+/22716/

You can apply patch and test it however its waiting on regression to
pass and merge.

-Sunny


On Thu, Jun 6, 2019 at 4:00 PM deepu srinivasan  wrote:
>
> Hi
> I have followed the following steps to create the geo-replication but the 
> status seems to be in a faulty state.
>
> Steps :
>
> Installed cluster version 5.6 in totally six nodes.
>>
>> glusterfs 5.6
>>
>> Repository revision: git://git.gluster.org/glusterfs.git
>>
>> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
>>
>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>>
>> It is licensed to you under your choice of the GNU Lesser
>>
>> General Public License, version 3 or any later version (LGPLv3
>>
>> or later), or the GNU General Public License, version 2 (GPLv2),
>>
>> in all cases as published by the Free Software Foundation
>
>
> peer_probed the first three nodes and second three nodes.
>
>
>
> Added new volume in both the clusters
>
>
>
> execute gluster-mountbroker commands and restarted glusterd.
>>
>> gluster-mountbroker setup /var/mountbroker-root sas
>>
>> gluster-mountbroker remove --volume code-misc --user sas
>
>
> configured a passwordless sssh from master to slave
>>
>> ssh-keygen; ssh-copy-id sas@192.168.185.107
>
> created a common pem pub file
>>
>> gluster system:: execute gsec_create
>
> created geo-replication session.
>>
>> gluster volume geo-replication code-misc sas@192.168.185.107::code-misc 
>> create push-pem
>
>  executed the following command in slave
>>
>> /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh sas code-misc code-misc
>
> started the gluster geo-replication.
>>
>> gluster volume geo-replication code-misc sas@192.168.185.107::code-misc start
>
>
> Now the geo-replication works fine.
> Tested with 2000 files All seems to sync finely.
>
> Now I updated all the node to version 6.2 by using rpms which were built by 
> the source code in a docker container in my personal machine.
>
>
>> gluster --version
>>
>> glusterfs 6.2
>>
>> Repository revision: git://git.gluster.org/glusterfs.git
>>
>> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
>>
>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>>
>> It is licensed to you under your choice of the GNU Lesser
>>
>> General Public License, version 3 or any later version (LGPLv3
>>
>> or later), or the GNU General Public License, version 2 (GPLv2),
>>
>> in all cases as published by the Free Software Foundation.
>
>
> I have stopped the glusterd daemons in all the node along with the volume and 
> geo-replication.
> Now I started the daemons, volume and geo-replication session the status 
> seems to be faulty.
> Also noted that the result of "gluster-mountbroker status" command always end 
> in python exception like this
>>
>> Traceback (most recent call last):
>>
>>   File "/usr/sbin/gluster-mountbroker", line 396, in 
>>
>> runcli()
>>
>>   File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 
>> 225, in runcli
>>
>> cls.run(args)
>>
>>   File "/usr/sbin/gluster-mountbroker", line 275, in run
>>
>> out = execute_in_peers("node-status")
>>
>>   File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 
>> 127, in execute_in_peers
>>
>> raise GlusterCmdException((rc, out, err, " ".join(cmd)))
>>
>> gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Unable to end. Error 
>> : Success\n', 'gluster system:: execute mountbroker.py node-status')
>
>
> Is it I or everyone gets an error for gluster-mountbroker command for gluster 
> version greater than 6.0?. Please help.
>
> Thank you
> Deepak
>
>
> On Thu, Jun 6, 2019 at 10:35 AM Sunny Kumar  wrote:
>>
>> Hi,
>>
>> Updated link for documentation :
>>
>> --  
>> https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/
>>
>> You can use this tool as well:
>> http://aravindavk.in/blog/gluster-georep-tools/
>>
>> -Sunny
>>
>> On Thu, Jun 6, 2019 at 10:29 AM Kotresh Hiremath Ravishankar
>>  wrote:
>> >
>> > Hi,
>> >
>> > I think the steps to setup no

Re: [Gluster-users] Geo Replication stops replicating

[Sunny Kumar]

Hi,

Updated link for documentation :

--  https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/

You can use this tool as well:
http://aravindavk.in/blog/gluster-georep-tools/

-Sunny

On Thu, Jun 6, 2019 at 10:29 AM Kotresh Hiremath Ravishankar
 wrote:
>
> Hi,
>
> I think the steps to setup non-root geo-rep is not followed properly. The 
> following entry is missing in glusterd vol file which is required.
>
> The message "E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 
> 0-management: 'option mountbroker-root' missing in glusterd vol file" 
> repeated 33 times between [2019-06-05 08:50:46.361384] and [2019-06-05 
> 08:52:34.019757]
>
> Could you please the steps from below?
>
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html-single/administration_guide/index#Setting_Up_the_Environment_for_a_Secure_Geo-replication_Slave
>
> And let us know if you still face the issue.
>
>
>
>
> On Thu, Jun 6, 2019 at 10:24 AM deepu srinivasan  wrote:
>>
>> Hi Kotresh, Sunny
>> I Have mailed the logs I found in one of the slave machines. Is there 
>> anything to do with permission? Please help.
>>
>> On Wed, Jun 5, 2019 at 2:28 PM deepu srinivasan  wrote:
>>>
>>> Hi Kotresh, Sunny
>>> Found this log in the slave machine.

 [2019-06-05 08:49:10.632583] I [MSGID: 106488] 
 [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: 
 Received get vol req

 The message "I [MSGID: 106488] 
 [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: 
 Received get vol req" repeated 2 times between [2019-06-05 
 08:49:10.632583] and [2019-06-05 08:49:10.670863]

 The message "I [MSGID: 106496] 
 [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received 
 mount req" repeated 34 times between [2019-06-05 08:48:41.005398] and 
 [2019-06-05 08:50:37.254063]

 The message "E [MSGID: 106061] 
 [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
 mountbroker-root' missing in glusterd vol file" repeated 34 times between 
 [2019-06-05 08:48:41.005434] and [2019-06-05 08:50:37.254079]

 The message "W [MSGID: 106176] 
 [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
 mount request [No such file or directory]" repeated 34 times between 
 [2019-06-05 08:48:41.005444] and [2019-06-05 08:50:37.254080]

 [2019-06-05 08:50:46.361347] I [MSGID: 106496] 
 [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received 
 mount req

 [2019-06-05 08:50:46.361384] E [MSGID: 106061] 
 [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
 mountbroker-root' missing in glusterd vol file

 [2019-06-05 08:50:46.361419] W [MSGID: 106176] 
 [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
 mount request [No such file or directory]

 The message "I [MSGID: 106496] 
 [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received 
 mount req" repeated 33 times between [2019-06-05 08:50:46.361347] and 
 [2019-06-05 08:52:34.019741]

 The message "E [MSGID: 106061] 
 [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
 mountbroker-root' missing in glusterd vol file" repeated 33 times between 
 [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757]

 The message "W [MSGID: 106176] 
 [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
 mount request [No such file or directory]" repeated 33 times between 
 [2019-06-05 08:50:46.361419] and [2019-06-05 08:52:34.019758]

 [2019-06-05 08:52:44.426839] I [MSGID: 106496] 
 [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received 
 mount req

 [2019-06-05 08:52:44.426886] E [MSGID: 106061] 
 [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option 
 mountbroker-root' missing in glusterd vol file

 [2019-06-05 08:52:44.426896] W [MSGID: 106176] 
 [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful 
 mount request [No such file or directory]
>>>
>>>
>>> On Wed, Jun 5, 2019 at 1:06 AM deepu srinivasan  wrote:

 Thankyou Kotresh

 On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar 
  wrote:
>
> Ccing Sunny, who was investing similar issue.
>
> On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan  
> wrote:
>>
>> Have already added the path in bashrc . Still in faulty state
>>
>> On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar 
>>  wrote:
>>>
>>> could you please try adding /usr/sbin to $PATH for user 'sas'? If it's 
>>> bash, add 'export PATH=/usr/sbin:$PATH' in
>>> /home/sas/.bashrc
>>>
>>> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan  
>>> wrote:

 Hi Kort

Re: [Gluster-users] Geo-replication status always on 'Created'

[Sunny Kumar]

Hi Maurya,

Looks like hook script is failed to set permissions for azureuser on
"/var/log/glusterfs".
You can assign permission manually for directory then it will work.

-Sunny

On Fri, Mar 22, 2019 at 2:07 PM Maurya M  wrote:
>
> hi Sunny,
>  Passwordless ssh to :
>
> ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i 
> /var/lib/glusterd/geo-replication/secret.pem -p 22 azureuser@172.16.201.35
>
> is login, but when the whole command is run getting permission issues again::
>
> ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i 
> /var/lib/glusterd/geo-replication/secret.pem -p 22 azureuser@172.16.201.35 
> gluster --xml --remote-host=localhost volume info 
> vol_a5aee81a873c043c99a938adcb5b5781 -v
> ERROR: failed to create logfile "/var/log/glusterfs/cli.log" (Permission 
> denied)
> ERROR: failed to open logfile /var/log/glusterfs/cli.log
>
> any idea here ?
>
> thanks,
> Maurya
>
>
> On Thu, Mar 21, 2019 at 2:43 PM Maurya M  wrote:
>>
>> hi Sunny,
>>  i did use the [1] link for the setup, when i encountered this error during 
>> ssh-copy-id : (so setup the passwordless ssh, by manually copied the 
>> private/ public keys to all the nodes , both master & slave)
>>
>> [root@k8s-agentpool1-24779565-1 ~]# ssh-copy-id geou...@xxx.xx.xxx.x
>> /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: 
>> "/root/.ssh/id_rsa.pub"
>> The authenticity of host ' xxx.xx.xxx.x   ( xxx.xx.xxx.x  )' can't be 
>> established.
>> ECDSA key fingerprint is SHA256:B2rNaocIcPjRga13oTnopbJ5KjI/7l5fMANXc+KhA9s.
>> ECDSA key fingerprint is MD5:1b:70:f9:7a:bf:35:33:47:0c:f2:c1:cd:21:e2:d3:75.
>> Are you sure you want to continue connecting (yes/no)? yes
>> /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to 
>> filter out any that are already installed
>> /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are 
>> prompted now it is to install the new keys
>> Permission denied (publickey).
>>
>> To start afresh what all needs to teardown / delete, do we have any script 
>> for it ? where all the pem keys do i need to delete?
>>
>> thanks,
>> Maurya
>>
>> On Thu, Mar 21, 2019 at 2:12 PM Sunny Kumar  wrote:
>>>
>>> Hey you can start a fresh I think you are not following proper setup steps.
>>>
>>> Please follow these steps [1] to create geo-rep session, you can
>>> delete the old one and do a fresh start. Or alternative you can use
>>> this tool[2] to setup geo-rep.
>>>
>>>
>>> [1]. 
>>> https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/
>>> [2]. http://aravindavk.in/blog/gluster-georep-tools/
>>>
>>>
>>> /Sunny
>>>
>>> On Thu, Mar 21, 2019 at 11:28 AM Maurya M  wrote:
>>> >
>>> > Hi Sunil,
>>> >  I did run the on the slave node :
>>> >  /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh azureuser 
>>> > vol_041afbc53746053368a1840607636e97 vol_a5aee81a873c043c99a938adcb5b5781
>>> > getting this message "/home/azureuser/common_secret.pem.pub not present. 
>>> > Please run geo-replication command on master with push-pem option to 
>>> > generate the file"
>>> >
>>> > So went back and created the session again, no change, so manually copied 
>>> > the common_secret.pem.pub to /home/azureuser/ but still the 
>>> > set_geo_rep_pem_keys.sh is looking the pem file in different name :  
>>> > COMMON_SECRET_PEM_PUB=${master_vol}_${slave_vol}_common_secret.pem.pub , 
>>> > change the name of pem , ran the command again :
>>> >
>>> >  /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh azureuser 
>>> > vol_041afbc53746053368a1840607636e97 vol_a5aee81a873c043c99a938adcb5b5781
>>> > Successfully copied file.
>>> > Command executed successfully.
>>> >
>>> >
>>> > - went back and created the session , start the geo-replication , still 
>>> > seeing the  same error in logs. Any ideas ?
>>> >
>>> > thanks,
>>> > Maurya
>>> >
>>> >
>>> >
>>> > On Wed, Mar 20, 2019 at 11:07 PM Sunny Kumar  wrote:
>>> >>
>>> >> Hi Maurya,
>>> >>
>>> >> I guess you missed last trick to distribute keys in slave node. I see
>>> >> this is non-root geo-rep setup so please try this:
>>> >>
>>> >&g

Re: [Gluster-users] Geo-replication status always on 'Created'

[Sunny Kumar]

Hey you can start a fresh I think you are not following proper setup steps.

Please follow these steps [1] to create geo-rep session, you can
delete the old one and do a fresh start. Or alternative you can use
this tool[2] to setup geo-rep.


[1]. https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/
[2]. http://aravindavk.in/blog/gluster-georep-tools/


/Sunny

On Thu, Mar 21, 2019 at 11:28 AM Maurya M  wrote:
>
> Hi Sunil,
>  I did run the on the slave node :
>  /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh azureuser 
> vol_041afbc53746053368a1840607636e97 vol_a5aee81a873c043c99a938adcb5b5781
> getting this message "/home/azureuser/common_secret.pem.pub not present. 
> Please run geo-replication command on master with push-pem option to generate 
> the file"
>
> So went back and created the session again, no change, so manually copied the 
> common_secret.pem.pub to /home/azureuser/ but still the 
> set_geo_rep_pem_keys.sh is looking the pem file in different name :  
> COMMON_SECRET_PEM_PUB=${master_vol}_${slave_vol}_common_secret.pem.pub , 
> change the name of pem , ran the command again :
>
>  /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh azureuser 
> vol_041afbc53746053368a1840607636e97 vol_a5aee81a873c043c99a938adcb5b5781
> Successfully copied file.
> Command executed successfully.
>
>
> - went back and created the session , start the geo-replication , still 
> seeing the  same error in logs. Any ideas ?
>
> thanks,
> Maurya
>
>
>
> On Wed, Mar 20, 2019 at 11:07 PM Sunny Kumar  wrote:
>>
>> Hi Maurya,
>>
>> I guess you missed last trick to distribute keys in slave node. I see
>> this is non-root geo-rep setup so please try this:
>>
>>
>> Run the following command as root in any one of Slave node.
>>
>> /usr/local/libexec/glusterfs/set_geo_rep_pem_keys.sh  
>>  
>>
>> - Sunny
>>
>> On Wed, Mar 20, 2019 at 10:47 PM Maurya M  wrote:
>> >
>> > Hi all,
>> >  Have setup a 3 master nodes - 3 slave nodes (gluster 4.1) for 
>> > geo-replication, but once have the geo-replication configure the status is 
>> > always on "Created',
>> > even after have force start the session.
>> >
>> > On close inspect of the logs on the master node seeing this error:
>> >
>> > "E [syncdutils(monitor):801:errlog] Popen: command returned error   
>> > cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i 
>> > /var/lib/glusterd/geo-replication/secret.pem -p 22 
>> > azureu...@x...xxx. gluster --xml --remote-host=localhost volume 
>> > info vol_a5ae34341a873c043c99a938adcb5b5781  error=255"
>> >
>> > Any ideas what is issue?
>> >
>> > thanks,
>> > Maurya
>> >
>> > ___
>> > Gluster-users mailing list
>> > Gluster-users@gluster.org
>> > https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo-replication status always on 'Created'

[Sunny Kumar]

Hi Maurya,

I guess you missed last trick to distribute keys in slave node. I see
this is non-root geo-rep setup so please try this:


Run the following command as root in any one of Slave node.

/usr/local/libexec/glusterfs/set_geo_rep_pem_keys.sh  
 

- Sunny

On Wed, Mar 20, 2019 at 10:47 PM Maurya M  wrote:
>
> Hi all,
>  Have setup a 3 master nodes - 3 slave nodes (gluster 4.1) for 
> geo-replication, but once have the geo-replication configure the status is 
> always on "Created',
> even after have force start the session.
>
> On close inspect of the logs on the master node seeing this error:
>
> "E [syncdutils(monitor):801:errlog] Popen: command returned error   cmd=ssh 
> -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i 
> /var/lib/glusterd/geo-replication/secret.pem -p 22 azureu...@x...xxx. 
> gluster --xml --remote-host=localhost volume info 
> vol_a5ae34341a873c043c99a938adcb5b5781  error=255"
>
> Any ideas what is issue?
>
> thanks,
> Maurya
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] File renaming not geo-replicated

[Sunny Kumar]

Hi Arnaud,

To analyse this behaviour I need log from slave and mount log also
from slave and not just snips please share complete log.
You can find logs from master - var/log/glusterfs/geo-replication/*
and for slave var/log/glusterfs/geo-replication-slave/* on slave node.

- Sunny
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] File renaming not geo-replicated

[Sunny Kumar]

Hi Arnaud,

Can you please share geo-replication log for master and mount log form slave.

- Sunny

On Mon, Dec 17, 2018 at 4:09 PM Arnaud Launay  wrote:
>
> Hello Sunny,
>
> Thanks for the feedback.
>
> Le Mon, Dec 17, 2018 at 03:59:07PM +0530, Sunny Kumar a écrit:
> > This is how it supposed to work on any file system including GlusterFS.
> > root@prod01:/srv/www# touch foo.txt && mv foo.txt bar.txt
> > Here foo.txt is created and then it is renamed to bar.txt
> > root@prod01:/srv/www# ls -l foo.txt bar.txt
> > ls: cannot access 'foo.txt': No such file or directory
> > -- SO this is correct foo should not exist as it is renamed to bar.
> > -rw-rw-r-- 1 root root 0 Dec 10 12:34 bar.txt
>
> This is /not/ my problem, that is what I see (and want) on prod01
> and prod02 (basic replication), but on test (georep), foo.txt
> still exists and bar.txt is never present: the rename never takes place.
>
> Arnaud.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] File renaming not geo-replicated

[Sunny Kumar]

Hi Arnaud,

This is how it supposed to work on any file system including GlusterFS.

root@prod01:/srv/www# touch foo.txt && mv foo.txt bar.txt

Here foo.txt is created and then it is renamed to bar.txt

root@prod01:/srv/www# ls -l foo.txt bar.txt
ls: cannot access 'foo.txt': No such file or directory

-- SO this is correct foo should not exist as it is renamed to bar.
-rw-rw-r-- 1 root root 0 Dec 10 12:34 bar.txt

- Sunny
On Mon, Dec 17, 2018 at 3:36 PM Arnaud Launay  wrote:
>
> Hello,
>
> Should I open a bug somewhere for hoping to get an answer ?
>
> Thanks,
> Arnaud.
>
> Le Mon, Dec 10, 2018 at 01:15:30PM +0100, Arnaud Launay a écrit:
> > We have a simple glusterfs setup, with one volume replicated on two
> > servers, and on a third one using georeplication.
> >
> > Environment:
> > - Debian 9 stretch uptodate
> > - gluster repository:
> > deb [arch=amd64] 
> > https://download.gluster.org/pub/gluster/glusterfs/4.1/LATEST/Debian/stretch/amd64/apt
> >  stretch main
> >
> > Last version:
> >
> > root@prod01:~# gluster --version
> > glusterfs 4.1.6
> > Repository revision: git://git.gluster.org/glusterfs.git
> >
> > (same on all three servers)
> >
> > Basic brick, lvm-mapper with an XFS and ACL filesystem:
> > /dev/mapper/data-glusterwww on /var/lib/glusterbricks/gbrick1 type xfs 
> > (rw,noatime,nodiratime,attr2,inode64,noquota)
> >
> > Then mounted locally via NFS (slightly quicker access time):
> > 127.0.0.1:/www0 on /srv/www type nfs 
> > (rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=38465,mountproto=tcp,local_lock=none,addr=127.0.0.1)
> >
> >
> >
> > We have a problem with the georep one, when renaming a file:
> >
> > root@prod01:/srv/www# touch foo.txt && mv foo.txt bar.txt
> > root@prod01:/srv/www# ls -l foo.txt bar.txt
> > ls: cannot access 'foo.txt': No such file or directory
> > -rw-rw-r-- 1 root root 0 Dec 10 12:34 bar.txt
> >
> > It's perfectly replicated on the second one:
> >
> > root@prod02:/srv/www# ls -l foo.txt bar.txt
> > ls: cannot access 'foo.txt': No such file or directory
> > -rw-rw-r-- 1 root root 0 Dec 10 12:34 bar.txt
> >
> >
> >
> > But the georeplicated one does not match:
> >
> > root@test:/srv/mirror-prod-www# ls -l foo.txt bar.txt
> > ls: cannot access 'bar.txt': No such file or directory
> > -rw-rw-r-- 2 root root 0 Dec 10 12:34 foo.txt
> >
> >
> >
> > The touched file (still) exists, but the mv has not been passed along.
> >
> > We tried multiple things, including deleting the georeplicated brick and
> > creating it again from scratch, but the problem is still present.
> >
> >
> > We saw that this has been the object of a few bugs in the past, but they 
> > seem
> > to have been corrected. Any idea ?
> >
> >   Arnaud.
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
>
> Arnaud.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster meetup: India

[Sunny Kumar]

Hello folks,

Gluster meetup is back and we will be hosting it in our office (Redhat-BLR-IN).
Please find the agenda and location detail here [1] and plan accordingly.

The highlight of this event will be Gluster in container world,
involving discussion around GCS (Gluster Container Storage) [2]. You
can try out latest GCS release here[3] and provide feedback.

Note:
 * RSVP as YES if attending, this will help us to organize the
facilities better.

If you have any question, please reach out to me or comment on the
event page [4].

1. http://meetu.ps/e/FZ9GH/zB8xy/f
2. https://github.com/gluster/gcs.
3. https://lists.gluster.org/pipermail/gluster-users/2018-December/035457.html
4. https://www.meetup.com/glusterfs-India/events/255792144/

- Sunny
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] geo replication issue

[Sunny Kumar]

Hi Krishna,

Please check for this file existance
'/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py' at slave.

- Sunny
On Wed, Oct 24, 2018 at 4:36 PM Krishna Verma  wrote:
>
>
>
>
>
>
>
> Hi Everyone,
>
>
>
> I have created a 4*4 distributed gluster but when I am starting the start the 
> session its get failed with below errors.
>
>
>
> [2018-10-24 10:02:03.857861] I [gsyncdstatus(monitor):245:set_worker_status] 
> GeorepStatus: Worker Status Change status=Initializing...
>
> [2018-10-24 10:02:03.858133] I [monitor(monitor):155:monitor] Monitor: 
> starting gsyncd worker   brick=/gfs1/brick1/gv1  slave_node=sj-gluster02
>
> [2018-10-24 10:02:03.954746] I [gsyncd(agent /gfs1/brick1/gv1):297:main] 
> : Using session config file   
> path=/var/lib/glusterd/geo-replication/gv1_sj-gluster01_gv1/gsyncd.conf
>
> [2018-10-24 10:02:03.956724] I [changelogagent(agent 
> /gfs1/brick1/gv1):72:__init__] ChangelogAgent: Agent listining...
>
> [2018-10-24 10:02:03.958110] I [gsyncd(worker /gfs1/brick1/gv1):297:main] 
> : Using session config file  
> path=/var/lib/glusterd/geo-replication/gv1_sj-gluster01_gv1/gsyncd.conf
>
> [2018-10-24 10:02:03.975778] I [resource(worker 
> /gfs1/brick1/gv1):1377:connect_remote] SSH: Initializing SSH connection 
> between master and slave...
>
> [2018-10-24 10:02:07.413379] E [syncdutils(worker 
> /gfs1/brick1/gv1):305:log_raise_exception] : connection to peer is broken
>
> [2018-10-24 10:02:07.414144] E [syncdutils(worker 
> /gfs1/brick1/gv1):801:errlog] Popen: command returned error   cmd=ssh 
> -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i 
> /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S 
> /tmp/gsyncd-aux-ssh-OE_W1C/cf9a66dce686717c4a5ef9a7c3a7f8be.sock sj-gluster01 
> /nonexistent/gsyncd slave gv1 sj-gluster01::gv1 --master-node noida-gluster01 
> --master-node-id 08925454-9fea-4b24-8f82-9d7ad917b870 --master-brick 
> /gfs1/brick1/gv1 --local-node sj-gluster02 --local-node-id 
> f592c041-dcae-493c-b5a0-31e376a5be34 --slave-timeout 120 --slave-log-level 
> INFO --slave-gluster-log-level INFO --slave-gluster-command-dir 
> /usr/local/sbin/  error=2
>
> [2018-10-24 10:02:07.414386] E [syncdutils(worker 
> /gfs1/brick1/gv1):805:logerr] Popen: ssh> /usr/bin/python2: can't open file 
> '/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py': [Errno 2] No such file 
> or directory
>
> [2018-10-24 10:02:07.422688] I [repce(agent 
> /gfs1/brick1/gv1):80:service_loop] RepceServer: terminating on reaching EOF.
>
> [2018-10-24 10:02:07.422842] I [monitor(monitor):266:monitor] Monitor: worker 
> died before establishing connection   brick=/gfs1/brick1/gv1
>
> [2018-10-24 10:02:07.435054] I [gsyncdstatus(monitor):245:set_worker_status] 
> GeorepStatus: Worker Status Change status=Faulty
>
>
>
>
>
> MASTER NODE  MASTER VOLMASTER BRICKSLAVE USERSLAVE
> SLAVE NODESTATUSCRAWL STATUSLAST_SYNCED
>
> 
>
> noida-gluster01  gv1   /gfs1/brick1/gv1root  
> sj-gluster01::gv1N/A   FaultyN/A N/A
>
> noida-gluster02  gv1   /gfs1/brick1/gv1root  
> sj-gluster01::gv1N/A   FaultyN/A N/A
>
> gluster-poc-noidagv1   /gfs1/brick1/gv1root  
> sj-gluster01::gv1N/A   FaultyN/A N/A
>
> noi-poc-gluster  gv1   /gfs1/brick1/gv1root  
> sj-gluster01::gv1N/A   FaultyN/A N/A
>
>
>
>
>
> Could someone please help?
>
>
>
> /Krishna
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo-replication keeps failing.

[Sunny Kumar]

Hi Alvin,

Please share complete log and also master mount log.
 - Sunny
On Thu, Sep 13, 2018 at 5:07 PM Alvin Starr  wrote:
>
> We are running glusterfs-3.8.9-1.el7.x86_64 with geo-replication.
>
> I have been having ongoing problems with the replication failing after
> some time.
>
> Once it has failed restarting it results in the attached logfile snippet.
>
>
> --
> Alvin Starr   ||   land:  (905)513-7688
> Netvel Inc.   ||   Cell:  (416)806-0133
> al...@netvel.net  ||
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Sunny Kumar]

On Tue, Aug 28, 2018 at 3:55 PM Krishna Verma  wrote:
>
> Hi Sunny,
>
> [root@gluster-poc-noida ~]# ldconfig /usr/local/lib
> [root@gluster-poc-noida ~]# ldconfig -p /usr/local/lib | grep libgf
> libgfxdr.so.0 (libc6,x86-64) => /lib64/libgfxdr.so.0
> libgfrpc.so.0 (libc6,x86-64) => /lib64/libgfrpc.so.0
> libgfortran.so.3 (libc6,x86-64) => /lib64/libgfortran.so.3
> libgfortran.so.1 (libc6,x86-64) => /lib64/libgfortran.so.1
> libgfdb.so.0 (libc6,x86-64) => /lib64/libgfdb.so.0
> libgfchangelog.so.0 (libc6,x86-64) => /lib64/libgfchangelog.so.0
This is linked properly so check for geo-rep It should be working
> libgfapi.so.0 (libc6,x86-64) => /lib64/libgfapi.so.0
> [root@gluster-poc-noida ~]#
>
> -Original Message-
> From: Sunny Kumar 
> Sent: Tuesday, August 28, 2018 3:53 PM
> To: Krishna Verma 
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work
>
> EXTERNAL MAIL
>
>
> can you do ldconfig /usr/local/lib and share the output of ldconfig -p 
> /usr/local/lib | grep libgf On Tue, Aug 28, 2018 at 3:45 PM Krishna Verma 
>  wrote:
> >
> > Hi Sunny,
> >
> > I did the mentioned changes given in patch and restart the session for 
> > geo-replication. But again same errors in the logs.
> >
> > I have attaching the config files and logs here.
> >
> >
> > [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep
> > gluster-poc-sj::glusterep stop Stopping geo-replication session
> > between glusterep & gluster-poc-sj::glusterep has been successful
> > [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep
> > gluster-poc-sj::glusterep delete Deleting geo-replication session
> > between glusterep & gluster-poc-sj::glusterep has been successful
> > [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep
> > gluster-poc-sj::glusterep create push-pem force Creating
> > geo-replication session between glusterep & gluster-poc-sj::glusterep
> > has been successful [root@gluster-poc-noida ~]# gluster volume
> > geo-replication glusterep gluster-poc-sj::glusterep start
> > geo-replication start failed for glusterep gluster-poc-sj::glusterep
> > geo-replication command failed [root@gluster-poc-noida ~]# gluster
> > volume geo-replication glusterep gluster-poc-sj::glusterep start
> > geo-replication start failed for glusterep gluster-poc-sj::glusterep
> > geo-replication command failed [root@gluster-poc-noida ~]# vim
> > /usr/libexec/glusterfs/python/syncdaemon/repce.py
> > [root@gluster-poc-noida ~]# systemctl restart glusterd
> > [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep
> > gluster-poc-sj::glusterep start Starting geo-replication session
> > between glusterep & gluster-poc-sj::glusterep has been successful
> > [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep
> > gluster-poc-sj::glusterep status
> >
> > MASTER NODE  MASTER VOLMASTER BRICK SLAVE USERSLAVE 
> >SLAVE NODESTATUSCRAWL STATUSLAST_SYNCED
> > -
> > gluster-poc-noidaglusterep /data/gluster/gv0root  
> > gluster-poc-sj::glusterepN/A   FaultyN/A N/A
> > noi-poc-gluster  glusterep /data/gluster/gv0root  
> > gluster-poc-sj::glusterepN/A   FaultyN/A N/A
> > [root@gluster-poc-noida ~]#
> >
> >
> > /Krishna.
> >
> > -Original Message-
> > From: Sunny Kumar 
> > Sent: Tuesday, August 28, 2018 3:17 PM
> > To: Krishna Verma 
> > Cc: gluster-users@gluster.org
> > Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> > work
> >
> > EXTERNAL MAIL
> >
> >
> > With same log message ?
> >
> > Can you please verify that
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__review.gluster.org_-23_c_glusterfs_-2B_20207_&d=DwIBaQ&c=aUq983L2pue2FqKFoP6PGHMJQyoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=F0ExtFUfa_YCktOGvy82x3IAxvi2GrbPR72jZ8beuYk&s=fGtkmezHJj5YoLN3dUeVUCcYFnREHyOSk36mRjbTTEQ&e=
> >  patch is present if not can you please apply that.
> > and try with symlinking ln -s /usr/lib64/libgfchangelog.so.0 
> > /usr/lib64/libgfchangelog.so.
> >
> > Please share the log also.
> >
> > Regards,
>

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Sunny Kumar]

can you do ldconfig /usr/local/lib and share the output of ldconfig -p
/usr/local/lib | grep libgf
On Tue, Aug 28, 2018 at 3:45 PM Krishna Verma  wrote:
>
> Hi Sunny,
>
> I did the mentioned changes given in patch and restart the session for 
> geo-replication. But again same errors in the logs.
>
> I have attaching the config files and logs here.
>
>
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep 
> gluster-poc-sj::glusterep stop
> Stopping geo-replication session between glusterep & 
> gluster-poc-sj::glusterep has been successful
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep 
> gluster-poc-sj::glusterep delete
> Deleting geo-replication session between glusterep & 
> gluster-poc-sj::glusterep has been successful
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep 
> gluster-poc-sj::glusterep create push-pem force
> Creating geo-replication session between glusterep & 
> gluster-poc-sj::glusterep has been successful
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep 
> gluster-poc-sj::glusterep start
> geo-replication start failed for glusterep gluster-poc-sj::glusterep
> geo-replication command failed
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep 
> gluster-poc-sj::glusterep start
> geo-replication start failed for glusterep gluster-poc-sj::glusterep
> geo-replication command failed
> [root@gluster-poc-noida ~]# vim 
> /usr/libexec/glusterfs/python/syncdaemon/repce.py
> [root@gluster-poc-noida ~]# systemctl restart glusterd
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep 
> gluster-poc-sj::glusterep start
> Starting geo-replication session between glusterep & 
> gluster-poc-sj::glusterep has been successful
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep 
> gluster-poc-sj::glusterep status
>
> MASTER NODE  MASTER VOLMASTER BRICK SLAVE USERSLAVE   
>  SLAVE NODESTATUSCRAWL STATUSLAST_SYNCED
> -
> gluster-poc-noidaglusterep /data/gluster/gv0root  
> gluster-poc-sj::glusterepN/A   FaultyN/A N/A
> noi-poc-gluster  glusterep /data/gluster/gv0root  
> gluster-poc-sj::glusterepN/A   FaultyN/A N/A
> [root@gluster-poc-noida ~]#
>
>
> /Krishna.
>
> -Original Message-
> From: Sunny Kumar 
> Sent: Tuesday, August 28, 2018 3:17 PM
> To: Krishna Verma 
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work
>
> EXTERNAL MAIL
>
>
> With same log message ?
>
> Can you please verify that
> https://urldefense.proofpoint.com/v2/url?u=https-3A__review.gluster.org_-23_c_glusterfs_-2B_20207_&d=DwIBaQ&c=aUq983L2pue2FqKFoP6PGHMJQyoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=F0ExtFUfa_YCktOGvy82x3IAxvi2GrbPR72jZ8beuYk&s=fGtkmezHJj5YoLN3dUeVUCcYFnREHyOSk36mRjbTTEQ&e=
>  patch is present if not can you please apply that.
> and try with symlinking ln -s /usr/lib64/libgfchangelog.so.0 
> /usr/lib64/libgfchangelog.so.
>
> Please share the log also.
>
> Regards,
> Sunny
> On Tue, Aug 28, 2018 at 3:02 PM Krishna Verma  wrote:
> >
> > Hi Sunny,
> >
> > Thanks for your response, I tried both, but still I am getting the same 
> > error.
> >
> >
> > [root@noi-poc-gluster ~]# ldconfig /usr/lib [root@noi-poc-gluster ~]#
> >
> > [root@noi-poc-gluster ~]# ln -s /usr/lib64/libgfchangelog.so.1
> > /usr/lib64/libgfchangelog.so [root@noi-poc-gluster ~]# ls -l
> > /usr/lib64/libgfchangelog.so lrwxrwxrwx. 1 root root 30 Aug 28 14:59
> > /usr/lib64/libgfchangelog.so -> /usr/lib64/libgfchangelog.so.1
> >
> > /Krishna
> >
> > -Original Message-
> > From: Sunny Kumar 
> > Sent: Tuesday, August 28, 2018 2:55 PM
> > To: Krishna Verma 
> > Cc: gluster-users@gluster.org
> > Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> > work
> >
> > EXTERNAL MAIL
> >
> >
> > Hi Krish,
> >
> > You can run -
> > #ldconfig /usr/lib
> >
> > If that still does not solves your problem you can do manual symlink
> > like - ln -s /usr/lib64/libgfchangelog.so.1
> > /usr/lib64/libgfchangelog.so
> >
> > Thanks,
> > Sunny Kumar
> > On Tue, Aug 28, 2018 at 1:47 PM Krishna Verma  wrote:
> > >
> > > Hi
&

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Sunny Kumar]

With same log message ?

Can you please verify that
https://review.gluster.org/#/c/glusterfs/+/20207/ patch is present if
not can you please apply that.
and try with symlinking ln -s /usr/lib64/libgfchangelog.so.0
/usr/lib64/libgfchangelog.so.

Please share the log also.

Regards,
Sunny
On Tue, Aug 28, 2018 at 3:02 PM Krishna Verma  wrote:
>
> Hi Sunny,
>
> Thanks for your response, I tried both, but still I am getting the same error.
>
>
> [root@noi-poc-gluster ~]# ldconfig /usr/lib
> [root@noi-poc-gluster ~]#
>
> [root@noi-poc-gluster ~]# ln -s /usr/lib64/libgfchangelog.so.1 
> /usr/lib64/libgfchangelog.so
> [root@noi-poc-gluster ~]# ls -l /usr/lib64/libgfchangelog.so
> lrwxrwxrwx. 1 root root 30 Aug 28 14:59 /usr/lib64/libgfchangelog.so -> 
> /usr/lib64/libgfchangelog.so.1
>
> /Krishna
>
> -Original Message-
> From: Sunny Kumar 
> Sent: Tuesday, August 28, 2018 2:55 PM
> To: Krishna Verma 
> Cc: gluster-users@gluster.org
> Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work
>
> EXTERNAL MAIL
>
>
> Hi Krish,
>
> You can run -
> #ldconfig /usr/lib
>
> If that still does not solves your problem you can do manual symlink like - 
> ln -s /usr/lib64/libgfchangelog.so.1 /usr/lib64/libgfchangelog.so
>
> Thanks,
> Sunny Kumar
> On Tue, Aug 28, 2018 at 1:47 PM Krishna Verma  wrote:
> >
> > Hi
> >
> >
> >
> > I am getting below error in gsyncd.log
> >
> >
> >
> > OSError: libgfchangelog.so: cannot open shared object file: No such
> > file or directory
> >
> > [2018-08-28 07:19:41.446785] E [repce(worker 
> > /data/gluster/gv0):197:__call__] RepceClient: call failed   
> > call=26469:139794524604224:1535440781.44method=init 
> > error=OSError
> >
> > [2018-08-28 07:19:41.447041] E [syncdutils(worker 
> > /data/gluster/gv0):330:log_raise_exception] : FAIL:
> >
> > Traceback (most recent call last):
> >
> >   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311,
> > in main
> >
> > func(args)
> >
> >   File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72,
> > in subcmd_worker
> >
> > local.service_loop(remote)
> >
> >   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line
> > 1236, in service_loop
> >
> > changelog_agent.init()
> >
> >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 216,
> > in __call__
> >
> > return self.ins(self.meth, *a)
> >
> >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 198,
> > in __call__
> >
> > raise res
> >
> > OSError: libgfchangelog.so: cannot open shared object file: No such
> > file or directory
> >
> > [2018-08-28 07:19:41.457555] I [repce(agent 
> > /data/gluster/gv0):80:service_loop] RepceServer: terminating on reaching 
> > EOF.
> >
> > [2018-08-28 07:19:42.440184] I [monitor(monitor):272:monitor] Monitor:
> > worker died in startup phase  brick=/data/gluster/gv0
> >
> >
> >
> > Below is my file location:
> >
> >
> >
> > /usr/lib64/libgfchangelog.so.0
> >
> > /usr/lib64/libgfchangelog.so.0.0.1
> >
> >
> >
> > What I can do to fix it ?
> >
> >
> >
> > /Krish
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gluster.org
> > _mailman_listinfo_gluster-2Dusers&d=DwIBaQ&c=aUq983L2pue2FqKFoP6PGHMJQ
> > yoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=_u6
> > vGRjlVsype7Z8hXDgCONilqVe4sIWkXNqqz2n3IQ&s=i0EUwtUHurhJHyw9UPpepCdLB70
> > 1mkxoNZWYvU7XXug&e=
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Sunny Kumar]

Hi Krish,

You can run -
#ldconfig /usr/lib

If that still does not solves your problem you can do manual symlink like -
ln -s /usr/lib64/libgfchangelog.so.1 /usr/lib64/libgfchangelog.so

Thanks,
Sunny Kumar
On Tue, Aug 28, 2018 at 1:47 PM Krishna Verma  wrote:
>
> Hi
>
>
>
> I am getting below error in gsyncd.log
>
>
>
> OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> directory
>
> [2018-08-28 07:19:41.446785] E [repce(worker /data/gluster/gv0):197:__call__] 
> RepceClient: call failed   call=26469:139794524604224:1535440781.44   
>  method=init error=OSError
>
> [2018-08-28 07:19:41.447041] E [syncdutils(worker 
> /data/gluster/gv0):330:log_raise_exception] : FAIL:
>
> Traceback (most recent call last):
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, in main
>
> func(args)
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72, in 
> subcmd_worker
>
> local.service_loop(remote)
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1236, in 
> service_loop
>
> changelog_agent.init()
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 216, in 
> __call__
>
> return self.ins(self.meth, *a)
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 198, in 
> __call__
>
> raise res
>
> OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> directory
>
> [2018-08-28 07:19:41.457555] I [repce(agent 
> /data/gluster/gv0):80:service_loop] RepceServer: terminating on reaching EOF.
>
> [2018-08-28 07:19:42.440184] I [monitor(monitor):272:monitor] Monitor: worker 
> died in startup phase  brick=/data/gluster/gv0
>
>
>
> Below is my file location:
>
>
>
> /usr/lib64/libgfchangelog.so.0
>
> /usr/lib64/libgfchangelog.so.0.0.1
>
>
>
> What I can do to fix it ?
>
>
>
> /Krish
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo-replication stops after 4-5 hours

[Sunny Kumar]

Hi Marcus,

Can you please share mount log from slave (You can find it at
"/var/log/glusterfs/geo-replication-slaves/hostname/mnt.log").

- Sunny
On Tue, Aug 14, 2018 at 12:48 AM Marcus Pedersén  wrote:
>
> Hi again,
>
> New changes in behaviour, both master master nodes that are active toggles to 
> failure and the logs repeat the same over and over again.
>
>
> Part of log, node1:
>
> [2018-08-13 18:24:44.701711] I [gsyncdstatus(worker 
> /urd-gds/gluster):276:set_active] GeorepStatus: Worker Status Change
> status=Active
> [2018-08-13 18:24:44.704360] I [gsyncdstatus(worker 
> /urd-gds/gluster):248:set_worker_crawl_status] GeorepStatus: Crawl Status 
> Changestatus=History Crawl
> [2018-08-13 18:24:44.705162] I [master(worker /urd-gds/gluster):1448:crawl] 
> _GMaster: starting history crawlturns=1 stime=(1523907056, 0)   
> entry_stime=Noneetime=1534184684
> [2018-08-13 18:24:45.717072] I [master(worker /urd-gds/gluster):1477:crawl] 
> _GMaster: slave's time  stime=(1523907056, 0)
> [2018-08-13 18:24:45.904958] E [repce(worker /urd-gds/gluster):197:__call__] 
> RepceClient: call failed   call=5919:140339726538560:1534184685.88 
> method=entry_opserror=GsyncdError
> [2018-08-13 18:24:45.905111] E [syncdutils(worker 
> /urd-gds/gluster):298:log_raise_exception] : execution of "gluster" 
> failed with ENOENT (No such file or directory)
> [2018-08-13 18:24:45.919265] I [repce(agent 
> /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF.
> [2018-08-13 18:24:46.553194] I [monitor(monitor):272:monitor] Monitor: worker 
> died in startup phase brick=/urd-gds/gluster
> [2018-08-13 18:24:46.561784] I [gsyncdstatus(monitor):243:set_worker_status] 
> GeorepStatus: Worker Status Change status=Faulty
> [2018-08-13 18:24:56.581748] I [monitor(monitor):158:monitor] Monitor: 
> starting gsyncd worker   brick=/urd-gds/gluster  slave_node=urd-gds-geo-000
> [2018-08-13 18:24:56.655164] I [gsyncd(worker /urd-gds/gluster):297:main] 
> : Using session config file  
> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> [2018-08-13 18:24:56.655193] I [gsyncd(agent /urd-gds/gluster):297:main] 
> : Using session config file   
> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> [2018-08-13 18:24:56.655889] I [changelogagent(agent 
> /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
> [2018-08-13 18:24:56.664628] I [resource(worker 
> /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection 
> between master and slave...
> [2018-08-13 18:24:58.347415] I [resource(worker 
> /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master and 
> slave established.duration=1.6824
> [2018-08-13 18:24:58.348151] I [resource(worker 
> /urd-gds/gluster):1067:connect] GLUSTER: Mounting gluster volume locally...
> [2018-08-13 18:24:59.463598] I [resource(worker 
> /urd-gds/gluster):1090:connect] GLUSTER: Mounted gluster volume 
> duration=1.1150
> [2018-08-13 18:24:59.464184] I [subcmds(worker 
> /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. 
> Acknowledging back to monitor
> [2018-08-13 18:25:01.549007] I [master(worker 
> /urd-gds/gluster):1534:register] _GMaster: Working dir
> path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster
> [2018-08-13 18:25:01.549606] I [resource(worker 
> /urd-gds/gluster):1253:service_loop] GLUSTER: Register time 
> time=1534184701
> [2018-08-13 18:25:01.593946] I [gsyncdstatus(worker 
> /urd-gds/gluster):276:set_active] GeorepStatus: Worker Status Change
> status=Active
>
>
> Part of log, node2:
>
> [2018-08-13 18:25:14.554233] I [gsyncdstatus(monitor):243:set_worker_status] 
> GeorepStatus: Worker Status Change status=Faulty
> [2018-08-13 18:25:24.568727] I [monitor(monitor):158:monitor] Monitor: 
> starting gsyncd worker   brick=/urd-gds/gluster  slave_node=urd-gds-geo-000
> [2018-08-13 18:25:24.609642] I [gsyncd(agent /urd-gds/gluster):297:main] 
> : Using session config file   
> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> [2018-08-13 18:25:24.609678] I [gsyncd(worker /urd-gds/gluster):297:main] 
> : Using session config file  
> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> [2018-08-13 18:25:24.610362] I [changelogagent(agent 
> /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
> [2018-08-13 18:25:24.621551] I [resource(worker 
> /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection 
> between master and slave...
> [2018-08-13 18:25:26.164855] I [resource(worker 
> /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master and 
> slave established.duration=1.5431
> [2018-08-13 18:25:26.165124] I [resource(worker 
> /urd-gds/gluster):1067:connec

Re: [Gluster-users] georeplication woes

[Sunny Kumar]

Hi,
If you run this below command on master

gluster vol geo-rep   config
slave-gluster-command-dir 

on slave run "which gluster" to know gluster-binary-location on slave

It will make the same entry in gsyncd.conf file please recheck and
confirm both entries are same and also can you confirm that both
master and slave have same gluster version.

- Sunny


On Mon, Jul 23, 2018 at 5:50 PM Maarten van Baarsel
 wrote:
>
> On 23/07/18 13:48, Sunny Kumar wrote:
>
> Hi Sunny,
>
> thanks again for replying!
>
>
> >> Can I test something else? Is the command normally run in a jail?
>
> > Please share gsyncd.log form master.
>
> [2018-07-23 12:18:19.773240] I [monitor(monitor):158:monitor] Monitor: 
> starting gsyncd worker   brick=/var/lib/gluster  slave_node=gluster-4.glstr
> [2018-07-23 12:18:19.832611] I [gsyncd(agent /var/lib/gluster):297:main] 
> : Using session config file   
> path=/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf
> [2018-07-23 12:18:19.832674] I [gsyncd(worker /var/lib/gluster):297:main] 
> : Using session config file  
> path=/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf
> [2018-07-23 12:18:19.834259] I [changelogagent(agent 
> /var/lib/gluster):72:__init__] ChangelogAgent: Agent listining...
> [2018-07-23 12:18:19.848596] I [resource(worker 
> /var/lib/gluster):1345:connect_remote] SSH: Initializing SSH connection 
> between master and slave...
> [2018-07-23 12:18:20.387191] E [syncdutils(worker 
> /var/lib/gluster):301:log_raise_exception] : connection to peer is broken
> [2018-07-23 12:18:20.387592] E [syncdutils(worker 
> /var/lib/gluster):747:errlog] Popen: command returned error   cmd=ssh 
> -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i 
> /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S 
> /tmp/gsyncd-aux-ssh-nN8_GE/2648484453eaadd9d3042ceba9bafa6a.sock 
> georep@gluster-4.glstr /nonexistent/gsyncd slave gl0 
> georep@gluster-4.glstr::glbackup --master-node gluster-3.glstr 
> --master-node-id 9650e965-bf4f-4544-a42b-f4d540d23a1f --master-brick 
> /var/lib/gluster --local-node gluster-4.glstr --local-node-id 
> 736f6431-2f9c-4115-9790-68f9a88d99a7 --slave-timeout 120 --slave-log-level 
> INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin/
>   error=1
> [2018-07-23 12:18:20.37] I [repce(agent 
> /var/lib/gluster):80:service_loop] RepceServer: terminating on reaching EOF.
> [2018-07-23 12:18:21.389723] I [monitor(monitor):266:monitor] Monitor: worker 
> died in startup phase brick=/var/lib/gluster
>
> repeated again and again.
>
> Maarten.
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work

[Sunny Kumar]

Hi Marcus,

Okay first apologies for wrong pattern here, please run

# find /usr/ -name libgfchangelog.so

- Sunny
On Mon, Jul 23, 2018 at 6:25 PM Marcus Pedersén  wrote:
>
> Hi,
>  #find /usr/ -name libglusterfs.so
> Gives nothing.
>
> #find /usr/ -name libglusterfs.so*
> Gives:
> /usr/lib64/libglusterfs.so.0
> /usr/lib64/libglusterfs.so.0.0.1
>
> Thanks!
> Marcus
>
> 
> Marcus Pedersén
> Systemadministrator
> Interbull Centre
> 
> Sent from my phone
> ####
>
> Den 23 juli 2018 14:17 skrev Sunny Kumar :
>
> Hi,
>
> Can you confirm the location for libgfchangelog.so
> by sharing output of following command -
> # find /usr/ -name libglusterfs.so
>
> - Sunny
>
> On Mon, Jul 23, 2018 at 5:12 PM Marcus Pedersén  
> wrote:
> >
> > Hi Sunny,
> > Here comes a part of gsyncd.log (The same info is repeated over and over 
> > again):
> >
> >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in 
> > __call__
> > raise res
> > OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> > directory
> > [2018-07-23 11:33:09.254915] I [repce(agent 
> > /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF.
> > [2018-07-23 11:33:10.225150] I [monitor(monitor):272:monitor] Monitor: 
> > worker died in startup phase brick=/urd-gds/gluster
> > [2018-07-23 11:33:20.250036] I [monitor(monitor):158:monitor] Monitor: 
> > starting gsyncd worker   brick=/urd-gds/gluster  slave_node=urd-gds-geo-000
> > [2018-07-23 11:33:20.326205] I [gsyncd(agent /urd-gds/gluster):297:main] 
> > : Using session config file   
> > path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> > [2018-07-23 11:33:20.326282] I [gsyncd(worker /urd-gds/gluster):297:main] 
> > : Using session config file  
> > path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> > [2018-07-23 11:33:20.327152] I [changelogagent(agent 
> > /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
> > [2018-07-23 11:33:20.335777] I [resource(worker 
> > /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection 
> > between master and slave...
> > [2018-07-23 11:33:22.11188] I [resource(worker 
> > /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master 
> > and slave established. duration=1.6752
> > [2018-07-23 11:33:22.11744] I [resource(worker 
> > /urd-gds/gluster):1067:connect] GLUSTER: Mounting gluster volume locally...
> > [2018-07-23 11:33:23.101602] I [resource(worker 
> > /urd-gds/gluster):1090:connect] GLUSTER: Mounted gluster volume 
> > duration=1.0894
> > [2018-07-23 11:33:23.102168] I [subcmds(worker 
> > /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. 
> > Acknowledging back to monitor
> > [2018-07-23 11:33:23.119129] E [repce(agent /urd-gds/gluster):114:worker] 
> > : call failed:
> > Traceback (most recent call last):
> >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 110, in 
> > worker
> > res = getattr(self.obj, rmeth)(*in_data[2:])
> >   File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 
> > 37, in init
> > return Changes.cl_init()
> >   File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 
> > 21, in __getattr__
> > from libgfchangelog import Changes as LChanges
> >   File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 
> > 17, in 
> > class Changes(object):
> >   File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 
> > 19, in Changes
> > use_errno=True)
> >   File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__
> > self._handle = _dlopen(self._name, mode)
> > OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> > directory
> > [2018-07-23 11:33:23.119609] E [repce(worker 
> > /urd-gds/gluster):206:__call__] RepceClient: call failed   
> > call=29589:140155686246208:1532345603.11method=init 
> > error=OSError
> > [2018-07-23 11:33:23.119708] E [syncdutils(worker 
> > /urd-gds/gluster):330:log_raise_exception] : FAIL:
> > Traceback (most recent call last):
> >   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, in 
> > main
> > func(args)
> >   File "/usr/libexec/glusterf

Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work

[Sunny Kumar]

Hi,

Can you confirm the location for libgfchangelog.so
by sharing output of following command -
# find /usr/ -name libglusterfs.so

- Sunny

On Mon, Jul 23, 2018 at 5:12 PM Marcus Pedersén  wrote:
>
> Hi Sunny,
> Here comes a part of gsyncd.log (The same info is repeated over and over 
> again):
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in 
> __call__
> raise res
> OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> directory
> [2018-07-23 11:33:09.254915] I [repce(agent 
> /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF.
> [2018-07-23 11:33:10.225150] I [monitor(monitor):272:monitor] Monitor: worker 
> died in startup phase brick=/urd-gds/gluster
> [2018-07-23 11:33:20.250036] I [monitor(monitor):158:monitor] Monitor: 
> starting gsyncd worker   brick=/urd-gds/gluster  slave_node=urd-gds-geo-000
> [2018-07-23 11:33:20.326205] I [gsyncd(agent /urd-gds/gluster):297:main] 
> : Using session config file   
> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> [2018-07-23 11:33:20.326282] I [gsyncd(worker /urd-gds/gluster):297:main] 
> : Using session config file  
> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> [2018-07-23 11:33:20.327152] I [changelogagent(agent 
> /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
> [2018-07-23 11:33:20.335777] I [resource(worker 
> /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection 
> between master and slave...
> [2018-07-23 11:33:22.11188] I [resource(worker 
> /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master and 
> slave established. duration=1.6752
> [2018-07-23 11:33:22.11744] I [resource(worker 
> /urd-gds/gluster):1067:connect] GLUSTER: Mounting gluster volume locally...
> [2018-07-23 11:33:23.101602] I [resource(worker 
> /urd-gds/gluster):1090:connect] GLUSTER: Mounted gluster volume 
> duration=1.0894
> [2018-07-23 11:33:23.102168] I [subcmds(worker 
> /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. 
> Acknowledging back to monitor
> [2018-07-23 11:33:23.119129] E [repce(agent /urd-gds/gluster):114:worker] 
> : call failed:
> Traceback (most recent call last):
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 110, in 
> worker
> res = getattr(self.obj, rmeth)(*in_data[2:])
>   File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 37, 
> in init
> return Changes.cl_init()
>   File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 21, 
> in __getattr__
> from libgfchangelog import Changes as LChanges
>   File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 17, 
> in 
> class Changes(object):
>   File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 19, 
> in Changes
> use_errno=True)
>   File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__
> self._handle = _dlopen(self._name, mode)
> OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> directory
> [2018-07-23 11:33:23.119609] E [repce(worker /urd-gds/gluster):206:__call__] 
> RepceClient: call failed   call=29589:140155686246208:1532345603.11
> method=init error=OSError
> [2018-07-23 11:33:23.119708] E [syncdutils(worker 
> /urd-gds/gluster):330:log_raise_exception] : FAIL:
> Traceback (most recent call last):
>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, in main
> func(args)
>   File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72, in 
> subcmd_worker
> local.service_loop(remote)
>   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1236, in 
> service_loop
> changelog_agent.init()
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in 
> __call__
> return self.ins(self.meth, *a)
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in 
> __call__
> raise res
> OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> directory
> [2018-07-23 11:33:23.130100] I [repce(agent 
> /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF.
> [2018-07-23 11:33:24.104176] I [monitor(monitor):272:monitor] Monitor: worker 
> died in startup phase brick=/urd-gds/gluster
>
> Thanks, Sunny!!
>
> Regards
> Marcus Pedersén
>
> 
> Från: Sunny Kumar 
> Skickat: den 23

Re: [Gluster-users] georeplication woes

[Sunny Kumar]

Hi Maarten,



On Mon, Jul 23, 2018 at 3:37 PM Maarten van Baarsel
 wrote:
>
> Hi Sunny,
>
> >> Can't run that command on the slave, it doesn't know the gl0 volume:
> >>
> >> root@gluster-4:/home/mrten# gluster volume geo-rep gl0 
> >> ssh://georep@gluster-4.glstr::glbackup config
>
> > please do not use ssh://
> > gluster volume geo-rep gl0 georep@gluster-4.glstr::glbackup config
> > just use it like config command.
>
> Sorry, does not make a difference on geo-rep slave side:
>
> root@gluster-4:/home/mrten# gluster volume geo-rep gl0 
> georep@gluster-4.glstr::glbackup config
> Volume gl0 does not exist
> geo-replication command failed

Apologies, if my statement to on slave you mistook to to set for slave.

This above command should run on master only like -

gluster vol geo-rep   config
slave-gluster-command-dir 

on slave run "which gluster" to know gluster-binary-location.

>
>
> Does work on the master side but the output is the same:
>
> root@gluster-3:/home/mrten# gluster volume geo-rep gl0 
> georep@gluster-4.glstr::glbackup config
> access_mount:false
> allow_network:
> change_detector:changelog
> change_interval:5
> changelog_archive_format:%Y%m
> changelog_batch_size:727040
> changelog_log_file:/var/log/glusterfs/geo-replication/gl0_gluster-4.glstr_glbackup/changes-${local_id}.log
> changelog_log_level:INFO
> checkpoint:0
> chnagelog_archive_format:%Y%m
> cli_log_file:/var/log/glusterfs/geo-replication/cli.log
> cli_log_level:INFO
> connection_timeout:60
> georep_session_working_dir:/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/
> gluster_cli_options:
> gluster_command:gluster
> gluster_command_dir:/usr/sbin/
> gluster_log_file:/var/log/glusterfs/geo-replication/gl0_gluster-4.glstr_glbackup/mnt-${local_id}.log
> gluster_log_level:INFO
> gluster_logdir:/var/log/glusterfs
> gluster_params:aux-gfid-mount acl
> gluster_rundir:/var/run/gluster
> glusterd_workdir:/var/lib/glusterd
> gsyncd_miscdir:/var/lib/misc/gluster/gsyncd
> ignore_deletes:false
> isolated_slaves:
> log_file:/var/log/glusterfs/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.log
> log_level:INFO
> log_rsync_performance:false
> master_disperse_count:1
> master_replica_count:1
> max_rsync_retries:10
> meta_volume_mnt:/var/run/gluster/shared_storage
> pid_file:/var/run/gluster/gsyncd-gl0-gluster-4.glstr-glbackup.pid
> remote_gsyncd:
> replica_failover_interval:1
> rsync_command:rsync
> rsync_opt_existing:
> rsync_opt_ignore_missing_args:
> rsync_options:
> rsync_ssh_options:
> slave_access_mount:false
> slave_gluster_command_dir:/usr/sbin/
> slave_gluster_log_file:/var/log/glusterfs/geo-replication-slaves/gl0_gluster-4.glstr_glbackup/mnt-${master_node}-${master_brick_id}.log
> slave_gluster_log_file_mbr:/var/log/glusterfs/geo-replication-slaves/gl0_gluster-4.glstr_glbackup/mnt-mbr-${master_node}-${master_brick_id}.log
> slave_gluster_log_level:INFO
> slave_gluster_params:aux-gfid-mount acl
> slave_log_file:/var/log/glusterfs/geo-replication-slaves/gl0_gluster-4.glstr_glbackup/gsyncd.log
> slave_log_level:INFO
> slave_timeout:120
> special_sync_mode:
> ssh_command:ssh
> ssh_options:-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i 
> /var/lib/glusterd/geo-replication/secret.pem
> ssh_options_tar:-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i 
> /var/lib/glusterd/geo-replication/tar_ssh.pem
> ssh_port:22
> state_file:/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/monitor.status
> state_socket_unencoded:
> stime_xattr_prefix:trusted.glusterfs.4054e7ad-7eb9-41fe-94cf-b52a690bb655.f7ce9a54-0ce4-4056-9958-4fa3f1630154
> sync_acls:true
> sync_jobs:3
> sync_xattrs:true
> tar_command:tar
> use_meta_volume:true
> use_rsync_xattrs:false
> use_tarssh:false
> working_dir:/var/lib/misc/gluster/gsyncd/gl0_gluster-4.glstr_glbackup/
>
>
> This should be non-root georeplication, should this work?
>
> georep@gluster-4:~$ /usr/sbin/gluster volume status all
> Connection failed. Please check if gluster daemon is operational.
no need for that
>
> (run as the geo-replication user on the slave side)
>
>
> Can I test something else? Is the command normally run in a jail?
Please share gsyncd.log form master.
>
>
>
> Maarten.
- Sunny
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work

[Sunny Kumar]

Hi Marcus,

On Mon, Jul 23, 2018 at 4:04 PM Marcus Pedersén  wrote:
>
> Hi Sunny,
> ldconfig -p /usr/local/lib | grep libgf
> Output:
> libgfxdr.so.0 (libc6,x86-64) => /lib64/libgfxdr.so.0  
>   libgfrpc.so.0 (libc6,x86-64) => 
> /lib64/libgfrpc.so.0
> libgfdb.so.0 (libc6,x86-64) => /lib64/libgfdb.so.0
>   libgfchangelog.so.0 (libc6,x86-64) => 
> /lib64/libgfchangelog.so.0
> libgfapi.so.0 (libc6,x86-64) => /lib64/libgfapi.so.0
>
> So that seems to be alright,  right?
>
Yes, this seems wright can you share the gsyncd.log again
> Best regards
> Marcus
>
> 
> Marcus Pedersén
> Systemadministrator
> Interbull Centre
> ####
> Sent from my phone
> 
>
> Den 23 juli 2018 11:17 skrev Sunny Kumar :
>
> Hi Marcus,
>
> On Wed, Jul 18, 2018 at 4:08 PM Marcus Pedersén  
> wrote:
> >
> > Hi Kotresh,
> >
> > I ran:
> >
> > #ldconfig /usr/lib
> can you do -
> ldconfig /usr/local/lib
>
>
> Output:
>
> >
> > on all nodes in both clusters but I still get the same error.
> >
> > What to do?
> >
> >
> > Output for:
> >
> > # ldconfig -p /usr/lib | grep libgf
> >
> > libgfxdr.so.0 (libc6,x86-64) => /lib64/libgfxdr.so.0
> > libgfrpc.so.0 (libc6,x86-64) => /lib64/libgfrpc.so.0
> > libgfdb.so.0 (libc6,x86-64) => /lib64/libgfdb.so.0
> > libgfchangelog.so.0 (libc6,x86-64) => /lib64/libgfchangelog.so.0
> > libgfapi.so.0 (libc6,x86-64) => /lib64/libgfapi.so.0
> >
> >
> > I read somewere that you could change some settings for geo-replication to 
> > speed up sync.
> >
> > I can not remember where I saw that and what config parameters.
> >
> > When geo-replication works I have 30TB on master cluster that has to be 
> > synced to slave nodes,
> >
> > and that will take a while before the slave nodes have catched up.
> >
> >
> > Thanks and regards
> >
> > Marcus Pedersén
> >
> >
> > Part of gsyncd.log:
> >
> >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in 
> > __call__
> > raise res
> > OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> > directory
> > [2018-07-18 10:23:52.305119] I [repce(agent 
> > /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF.
> > [2018-07-18 10:23:53.273298] I [monitor(monitor):272:monitor] Monitor: 
> > worker died in startup phase brick=/urd-gds/gluster
> > [2018-07-18 10:24:03.294312] I [monitor(monitor):158:monitor] Monitor: 
> > starting gsyncd worker   brick=/urd-gds/gluster  slave_node=urd-gds-geo-000
> > [2018-07-18 10:24:03.334563] I [gsyncd(agent /urd-gds/gluster):297:main] 
> > : Using session config file   
> > path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> > [2018-07-18 10:24:03.334702] I [gsyncd(worker /urd-gds/gluster):297:main] 
> > : Using session config file  
> > path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> > [2018-07-18 10:24:03.335380] I [changelogagent(agent 
> > /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
> > [2018-07-18 10:24:03.343605] I [resource(worker 
> > /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection 
> > between master and slave...
> > [2018-07-18 10:24:04.881148] I [resource(worker 
> > /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master 
> > and slave established.duration=1.5373
> > [2018-07-18 10:24:04.881707] I [resource(worker 
> > /urd-gds/gluster):1067:connect] GLUSTER: Mounting gluster volume locally...
> > [2018-07-18 10:24:05.967451] I [resource(worker 
> > /urd-gds/gluster):1090:connect] GLUSTER: Mounted gluster volume 
> > duration=1.0853
> > [2018-07-18 10:24:05.968028] I [subcmds(worker 
> > /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. 
> > Acknowledging back to monitor
> > [2018-07-18 10:24:05.984179] E [repce(agent /urd-gds/gluster):114:worker] 
> > : call failed:
> > Traceback (most recent call last):
> >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 110, in 
> > worker
> > res = getattr(self.obj, rmeth)(*in_data[2:])
> >   File "/usr/libexec/glusterfs/python/syncdaemon/change

Re: [Gluster-users] Upgrade to 4.1.1 geo-replication does not work

[Sunny Kumar]

Hi Marcus,

On Wed, Jul 18, 2018 at 4:08 PM Marcus Pedersén  wrote:
>
> Hi Kotresh,
>
> I ran:
>
> #ldconfig /usr/lib
can you do -
ldconfig /usr/local/lib
>
> on all nodes in both clusters but I still get the same error.
>
> What to do?
>
>
> Output for:
>
> # ldconfig -p /usr/lib | grep libgf
>
> libgfxdr.so.0 (libc6,x86-64) => /lib64/libgfxdr.so.0
> libgfrpc.so.0 (libc6,x86-64) => /lib64/libgfrpc.so.0
> libgfdb.so.0 (libc6,x86-64) => /lib64/libgfdb.so.0
> libgfchangelog.so.0 (libc6,x86-64) => /lib64/libgfchangelog.so.0
> libgfapi.so.0 (libc6,x86-64) => /lib64/libgfapi.so.0
>
>
> I read somewere that you could change some settings for geo-replication to 
> speed up sync.
>
> I can not remember where I saw that and what config parameters.
>
> When geo-replication works I have 30TB on master cluster that has to be 
> synced to slave nodes,
>
> and that will take a while before the slave nodes have catched up.
>
>
> Thanks and regards
>
> Marcus Pedersén
>
>
> Part of gsyncd.log:
>
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in 
> __call__
> raise res
> OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> directory
> [2018-07-18 10:23:52.305119] I [repce(agent 
> /urd-gds/gluster):89:service_loop] RepceServer: terminating on reaching EOF.
> [2018-07-18 10:23:53.273298] I [monitor(monitor):272:monitor] Monitor: worker 
> died in startup phase brick=/urd-gds/gluster
> [2018-07-18 10:24:03.294312] I [monitor(monitor):158:monitor] Monitor: 
> starting gsyncd worker   brick=/urd-gds/gluster  slave_node=urd-gds-geo-000
> [2018-07-18 10:24:03.334563] I [gsyncd(agent /urd-gds/gluster):297:main] 
> : Using session config file   
> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> [2018-07-18 10:24:03.334702] I [gsyncd(worker /urd-gds/gluster):297:main] 
> : Using session config file  
> path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf
> [2018-07-18 10:24:03.335380] I [changelogagent(agent 
> /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...
> [2018-07-18 10:24:03.343605] I [resource(worker 
> /urd-gds/gluster):1348:connect_remote] SSH: Initializing SSH connection 
> between master and slave...
> [2018-07-18 10:24:04.881148] I [resource(worker 
> /urd-gds/gluster):1395:connect_remote] SSH: SSH connection between master and 
> slave established.duration=1.5373
> [2018-07-18 10:24:04.881707] I [resource(worker 
> /urd-gds/gluster):1067:connect] GLUSTER: Mounting gluster volume locally...
> [2018-07-18 10:24:05.967451] I [resource(worker 
> /urd-gds/gluster):1090:connect] GLUSTER: Mounted gluster volume 
> duration=1.0853
> [2018-07-18 10:24:05.968028] I [subcmds(worker 
> /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. 
> Acknowledging back to monitor
> [2018-07-18 10:24:05.984179] E [repce(agent /urd-gds/gluster):114:worker] 
> : call failed:
> Traceback (most recent call last):
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 110, in 
> worker
> res = getattr(self.obj, rmeth)(*in_data[2:])
>   File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 37, 
> in init
> return Changes.cl_init()
>   File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line 21, 
> in __getattr__
> from libgfchangelog import Changes as LChanges
>   File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 17, 
> in 
> class Changes(object):
>   File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line 19, 
> in Changes
> use_errno=True)
>   File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__
> self._handle = _dlopen(self._name, mode)
> OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> directory
> [2018-07-18 10:24:05.984647] E [repce(worker /urd-gds/gluster):206:__call__] 
> RepceClient: call failed   call=1146:139672481965888:1531909445.98 
> method=init error=OSError
> [2018-07-18 10:24:05.984747] E [syncdutils(worker 
> /urd-gds/gluster):330:log_raise_exception] : FAIL:
> Traceback (most recent call last):
>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 311, in main
> func(args)
>   File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 72, in 
> subcmd_worker
> local.service_loop(remote)
>   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1236, in 
> service_loop
> changelog_agent.init()
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 225, in 
> __call__
> return self.ins(self.meth, *a)
>   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 207, in 
> __call__
> raise res
> OSError: libgfchangelog.so: cannot open shared object file: No such file or 
> directory
I think then you will not see this.
> [2018-07-18 10:24:05.994826] I [repce(agent 
> /urd-gds/gluster):89:servic

Re: [Gluster-users] georeplication woes

[Sunny Kumar]

Hi Maarten,

On Sat, Jul 21, 2018 at 12:34 AM Maarten van Baarsel
 wrote:
>
>
> Hi Sunny,
>
> thanks for your reply!
>
> > I think setup for gluster binary location is wrongly setup here.
> >
> > Can you please try setting it up by using following command
> >
> > on master
> > #gluster vol geo-rep   config
> > gluster-command-dir 
> >
> > and on slave
> > #gluster vol geo-rep   config
> > slave-gluster-command-dir 
>
> Is there a difference between _ and -? I only seem to have the _ ones:
>
> (gluster-3 is one of the master 3-replica, gluster-4 is the slave)
>
> root@gluster-3:/home/mrten# gluster volume geo-rep gl0 
> ssh://georep@gluster-4.glstr::glbackup config | fgrep command
> gluster_command:gluster
> gluster_command_dir:/usr/sbin/
> rsync_command:rsync
> slave_gluster_command_dir:/usr/sbin/
> ssh_command:ssh
> tar_command:tar
>
> Can't run that command on the slave, it doesn't know the gl0 volume:
>
> root@gluster-4:/home/mrten# gluster volume geo-rep gl0 
> ssh://georep@gluster-4.glstr::glbackup config
please do not use ssh://
gluster volume geo-rep gl0 georep@gluster-4.glstr::glbackup config
just use it like config command.
> Volume gl0 does not exist
> geo-replication command failed
>
>
>
> But the location is correct:
>
> master:
>
> root@gluster-3:/home/mrten# which gluster
> /usr/sbin/gluster
>
> slave:
>
> root@gluster-4:/home/mrten# which gluster
> /usr/sbin/gluster
>
>
> and they work:
>
> root@gluster-3:/home/mrten# /usr/sbin/gluster
> gluster>
>
> root@gluster-4:/home/mrten# /usr/sbin/gluster
> gluster>
>
>
> >
> >> [2018-07-20 12:32:13.404048] W [gsyncd(slave 
> >> gluster-2.glstr/var/lib/gluster):293:main] : Session config file not 
> >> exists, using the default config 
> >> path=/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf
>
> this one is weird though; there is nothing in
> /var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf
> but
>
>
> [vars]
> stime-xattr-prefix = 
> trusted.glusterfs.4054e7ad-7eb9-41fe-94cf-b52a690bb655.f7ce9a54-0ce4-4056-9958-4fa3f1630154
> use-meta-volume = true
>
> Maarten.
- Sunny
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] georeplication woes

[Sunny Kumar]

Hi Maarten,

On Fri, Jul 20, 2018 at 9:24 PM Maarten van Baarsel  wrote:
>
> Hi,
>
> I've upgraded my gluster 3.13 cluster to 4.0 on an Ubuntu
> server and it broke my geo-replication because of this missing
> file:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1601532
>
> However, between geo-replication dying and me finding that bugreport
> I've tried multiple other things including setting up geo-rep again
> so I've still got a Faulty georeplication now.
>
> Removed it, created it again with the suggested fix in place but at
> the moment I see this error on the georep slave, in
> /var/log/glusterfs/glusterd.log:
>
> E [MSGID: 106581] [glusterd-mountbroker.c:579:glusterd_do_mount] 
> 0-management: Missing mspec: Check the corresponding option in glusterd vol 
> file for mountbroker user: georep [No such file or directory]
>
>
> These messages also seem relevant from 
> /var/log/glusterfs/geo-replication-slaves/.../gsyncd.log:
>
> [2018-07-20 12:32:11.452244] I [resource(slave 
> gluster-1.glstr/var/lib/gluster):1064:connect] GLUSTER: Mounting gluster 
> volume locally...
> [2018-07-20 12:32:11.503766] E [resource(slave 
> gluster-1.glstr/var/lib/gluster):973:handle_mounter] MountbrokerMounter: 
> glusterd answered   mnt=
> [2018-07-20 12:32:11.504313] E [syncdutils(slave 
> gluster-1.glstr/var/lib/gluster):747:errlog] Popen: command returned error
>  cmd=/usr/sbin/gluster --remote-host=localhost system:: mount georep 
> user-map-root=georep aux-gfid-mount acl log-level=INFO 
> log-file=/var/log/glusterfs/geo-replication-slaves/gl0_gluster-4.glstr_glbackup/mnt-gluster-1.glstr-var-lib-gluster.log
>  volfile-server=localhost volfile-id=glbackup client-pid=-1   error=1
> [2018-07-20 12:32:11.504446] E [syncdutils(slave 
> gluster-1.glstr/var/lib/gluster):751:logerr] Popen: /usr/sbin/gluster> 2 : 
> failed with this errno (No such file or directory)

I think setup for gluster binary location is wrongly setup here.

Can you please try setting it up by using following command

on master
#gluster vol geo-rep   config
gluster-command-dir 

and on slave
#gluster vol geo-rep   config
slave-gluster-command-dir 

> [2018-07-20 12:32:13.404048] W [gsyncd(slave 
> gluster-2.glstr/var/lib/gluster):293:main] : Session config file not 
> exists, using the default config 
> path=/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf
>
>
> Is there anyone out there that can tell me what to do with these
> error messages? I've resorted to checking out the source code
> but that road did not lead to enlightment either :(
>
> thanks in advance,
> Maarten.
>
>
> [1] The gluster master is a 3-replica; the slave is a single
> volume at another physical location, geo-replication user is
> georep. main volume is called 'gl0', backup 'glbackup'.
>
>
> root@gluster-4:/var/log/glusterfs# gluster-mountbroker status
> +---+-+-++---+
> |NODE   | NODE STATUS |MOUNT ROOT   |   GROUP|
>USERS   |
> +---+-+-++---+
> | localhost |  UP | /var/lib/georep-mountbroker(OK) | georep(OK) | 
> georep(glbackup)  |
> +---+-+-++---+
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

- Sunny
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Geo-Replication memory leak on slave node

[Sunny Kumar]

Hi Mark,

Currently I am looking at this issue (Kotresh is busy with some other
work) so can you please share the latest log with me.

Thanks,
Sunny


On Fri, Jul 13, 2018 at 12:41 PM Mark Betham  wrote:
>
> Hi Kotresh,
>
> I was wondering if you had found any time t take a look at the issue I am 
> currently experiencing with geo-replication and memory usage.
>
> If you require any further information then please do not hesitate to ask.
>
> Many thanks,
>
> Mark Betham
>
>
> On Wed, 20 Jun 2018 at 11:27, Mark Betham 
>  wrote:
>>
>> Hi Kotresh,
>>
>> Many thanks for your prompt response.  No need to apologise, any help you 
>> can provide is greatly appreciated.
>>
>> I look forward to receiving your update next week.
>>
>> Many thanks,
>>
>> Mark Betham
>>
>> On Wed, 20 Jun 2018 at 10:55, Kotresh Hiremath Ravishankar 
>>  wrote:
>>>
>>> Hi Mark,
>>>
>>> Sorry, I was busy and could not take a serious look at the logs. I can 
>>> update you on Monday.
>>>
>>> Thanks,
>>> Kotresh HR
>>>
>>> On Wed, Jun 20, 2018 at 12:32 PM, Mark Betham 
>>>  wrote:

 Hi Kotresh,

 I was wondering if you had made any progress with regards to the issue I 
 am currently experiencing with geo-replication.

 For info the fault remains and effectively requires a restart of the 
 geo-replication service on a daily basis to reclaim the used memory on the 
 slave node.

 If you require any further information then please do not hesitate to ask.

 Many thanks,

 Mark Betham


 On Mon, 11 Jun 2018 at 08:24, Mark Betham 
  wrote:
>
> Hi Kotresh,
>
> Many thanks.  I will shortly setup a share on my GDrive and send the link 
> directly to yourself.
>
> For Info;
> The Geo-Rep slave failed again over the weekend but it did not recover 
> this time.  It looks to have become unresponsive at around 14:40 UTC on 
> 9th June.  I have attached an image showing the mem usage and you can see 
> from this when the system failed.  The system was totally unresponsive 
> and required a cold power off and then power on in order to recover the 
> server.
>
> Many thanks for your help.
>
> Mark Betham.
>
> On 11 June 2018 at 05:53, Kotresh Hiremath Ravishankar 
>  wrote:
>>
>> Hi Mark,
>>
>> Google drive works for me.
>>
>> Thanks,
>> Kotresh HR
>>
>> On Fri, Jun 8, 2018 at 3:00 PM, Mark Betham 
>>  wrote:
>>>
>>> Hi Kotresh,
>>>
>>> The memory issue re-occurred again.  This is indicating it will occur 
>>> around once a day.
>>>
>>> Again no traceback listed in the log, the only update in the log was as 
>>> follows;
>>> [2018-06-08 08:26:43.404261] I [resource(slave):1020:service_loop] 
>>> GLUSTER: connection inactive, stopping timeout=120
>>> [2018-06-08 08:29:19.357615] I [syncdutils(slave):271:finalize] : 
>>> exiting.
>>> [2018-06-08 08:31:02.432002] I [resource(slave):1502:connect] GLUSTER: 
>>> Mounting gluster volume locally...
>>> [2018-06-08 08:31:03.716967] I [resource(slave):1515:connect] GLUSTER: 
>>> Mounted gluster volume duration=1.2729
>>> [2018-06-08 08:31:03.717411] I [resource(slave):1012:service_loop] 
>>> GLUSTER: slave listening
>>>
>>> I have attached an image showing the latest memory usage pattern.
>>>
>>> Can you please advise how I can pass the log data across to you?  As 
>>> soon as I know this I will get the data uploaded for your review.
>>>
>>> Thanks,
>>>
>>> Mark Betham
>>>
>>>
>>>
>>>
>>> On 7 June 2018 at 08:19, Mark Betham 
>>>  wrote:

 Hi Kotresh,

 Many thanks for your prompt response.

 Below are my responses to your questions;

 1. Is this trace back consistently hit? I just wanted to confirm 
 whether it's transient which occurs once in a while and gets back to 
 normal?
 It appears not.  As soon as the geo-rep recovered yesterday from the 
 high memory usage it immediately began rising again until it consumed 
 all of the available ram.  But this time nothing was committed to the 
 log file.
 I would like to add here that this current instance of geo-rep was 
 only brought online at the start of this week due to the issues with 
 glibc on CentOS 7.5.  This is the first time I have had geo-rep 
 running with Gluster ver 3.12.9, both storage clusters at each 
 physical site were only rebuilt approx. 4 weeks ago, due to the 
 previous version in use going EOL.  Prior to this I had been running 
 3.13.2 (3.13.X now EOL) at each of the sites and it is worth noting 
 that the same behaviour was also seen on this version of Gluster, 
 unfortunately I do not have any of the log data from then but I do not 
>>