[Gluster-users] ubuntu 18.04 PPA and fuse3
I took my pointers from https://docs.gluster.org/en/latest/Install-Guide/Install/ installing 10.1 from the PPA but I stumble: root@gluster-1:/# apt install glusterfs-server Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation:The following packages have unmet dependencies: glusterfs-server : Depends: glusterfs-client (>= 10.0-ubuntu1~bionic3) but it is not going to be installed E: Unable to correct problems, you have held broken packages. which is because of glusterfs-client: [...] The following packages have unmet dependencies: glusterfs-client : Depends: fuse3 but it is not installable E: Unable to correct problems, you have held broken packages. There /is/ no fuse3 in 18.04. https://packages.ubuntu.com/search?keywords=fuse3 Where do I report this, besides on this list? thanks, Maarten. Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] trigger georeplication resync
Hi, I seem to have gotten my geo-replication back to working again, but if I compare the outputs of 'find -type f' from the georep master and the slave there are numerous differences. This is 400G of data (and a 4 megabit WAN link) so I can't just pull over the data again, so: How do I trigger a full resync/rsync? If not possible, is there a way to intervene manually, say, by running rsync --xattr? thanks, Maarten. ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Subject: Help needed in improving monitoring in Gluster
On 23/7/2018 18:25 , Sankarshan Mukhopadhyay wrote: Regarding monitoring; I would love to see in my monitoring that geo-replication is working as intended; at the moment I'm faking georep monitoring by having a process touch a file (every server involved in gluster touches another file) on every volume and checking mtime on the slave. I'd like to request that if possible, you elaborate on how you'd like to see the "as intended" situation. What kind of data points and/or visualization would aid you in arriving at that conclusion? for this particular example: perhaps last sync time, and number of files on both sides. i realize this is a difficult problem... number of files touched by sync per run? currently there is a 'started/stopped/faulty' indicator for the geo-rep, could be exposed as well. a monitoring interface that is guaranteed to be non-blocking would be a great enhancement. M. ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] georeplication woes
On 24-07-2018 06:12:25, Kotresh Hiremath Ravishankar wrote:
Hi Kotresh,
Looks like gsyncd on slave is failing for some reason.
Please run the below cmd on the master.
#ssh -i /var/lib/glusterd/geo-replication/secret.pem georep@gluster-4.glstr
It should run gsyncd on the slave. If there is error, it should be fixed.
Please share the output of above cmd.
here we go:
root@gluster-3:/home/mrten# ssh -i
/var/lib/glusterd/geo-replication/secret.pem georep@gluster-4.glstr
usage: gsyncd.py [-h]
{monitor-status,monitor,worker,agent,slave,status,config-check,config-get,config-set,config-reset,voluuidget,delete}
...
gsyncd.py: error: too few arguments
Connection to gluster-4.glstr closed.
Maarten.
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] georeplication woes
On 23/07/18 15:28, Sunny Kumar wrote: > Hi, > If you run this below command on master > > gluster vol geo-rep config > slave-gluster-command-dir > > on slave run "which gluster" to know gluster-binary-location on slave Done that, repeatedly, no change :( > It will make the same entry in gsyncd.conf file please recheck and (what gsyncd.conf? the one in /etc or someplace else?) > confirm both entries are same and also can you confirm that both > master and slave have same gluster version. slave: root@gluster-4:~$ /usr/sbin/gluster --version glusterfs 4.0.2 master: root@gluster-3:/home/mrten# /usr/sbin/gluster --version glusterfs 4.0.2 Looking at the slaves' cli.log: [2018-07-23 15:53:26.187547] I [cli.c:767:main] 0-cli: Started running /usr/sbin/gluster with version 4.0.2 [2018-07-23 15:53:26.187611] I [cli.c:646:cli_rpc_init] 0-cli: Connecting to remote glusterd at localhost [2018-07-23 15:53:26.229756] I [MSGID: 101190] [event-epoll.c:609:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2018-07-23 15:53:26.229871] W [rpc-clnt.c:1739:rpc_clnt_submit] 0-glusterfs: error returned while attempting to connect to host:(null), port:0 [2018-07-23 15:53:26.229963] I [socket.c:2625:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2018-07-23 15:53:26.230640] I [cli-rpc-ops.c:8785:gf_cli_mount_cbk] 0-cli: Received resp to mount [2018-07-23 15:53:26.230825] I [input.c:31:cli_batch] 0-: Exiting with: -1 there's a weird warning there with host:(null), port:0 M. ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Subject: Help needed in improving monitoring in Gluster
On 23/07/18 16:03, Pranith Kumar Karampuri wrote:
We want gluster's monitoring/observability to be as easy as possible
going forward. As part of reaching this goal we are starting this
initiative to add improvements to existing apis/commands and create new
apis/commands to gluster so that the admin can integrate it with
whichever monitoring tool he/she likes. The gluster-prometheus project
hosted at https://github.com/gluster/gluster-prometheus is the direction
in which we feel metrics can be collected and distributed from a Gluster
cluster enabling analysis and visualization.
As a first step we want to hear from you what you feel needs to be
addressed.
Regarding monitoring; I would love to see in my monitoring that
geo-replication is working as intended; at the moment I'm faking georep
monitoring by having a process touch a file (every server involved in
gluster touches another file) on every volume and checking mtime on the
slave.
However, I discovered that this is not foolproof: if the georep run
stops for whatever reason the mtime of the monitored file is being kept
updated, probably because it's updated to often, but the georep is not
complete.
I've also seen that a crashed glusterd escapes this monitoring.
What would also be fun is some kind of monitoring where you can find out
why gluster is running at X MB/sec where Y MB/sec is expected (bit large
target, that)
I've once tried monitoring 'gluster volume status all' output but that
only works if everything is OK; with some network problems you can wait
for hours for output which then causes more problems.
Also, I've checked the example output at
https://github.com/gluster/gluster-prometheus:
would JSON or something like that be more friendy to parse instead of
the "[parameter] { [details] } [number]" format?
thanks,
Maarten.
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] georeplication woes
On 23/07/18 13:48, Sunny Kumar wrote: Hi Sunny, thanks again for replying! >> Can I test something else? Is the command normally run in a jail? > Please share gsyncd.log form master. [2018-07-23 12:18:19.773240] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/var/lib/gluster slave_node=gluster-4.glstr [2018-07-23 12:18:19.832611] I [gsyncd(agent /var/lib/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf [2018-07-23 12:18:19.832674] I [gsyncd(worker /var/lib/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf [2018-07-23 12:18:19.834259] I [changelogagent(agent /var/lib/gluster):72:__init__] ChangelogAgent: Agent listining... [2018-07-23 12:18:19.848596] I [resource(worker /var/lib/gluster):1345:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-07-23 12:18:20.387191] E [syncdutils(worker /var/lib/gluster):301:log_raise_exception] : connection to peer is broken [2018-07-23 12:18:20.387592] E [syncdutils(worker /var/lib/gluster):747:errlog] Popen: command returned error cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-nN8_GE/2648484453eaadd9d3042ceba9bafa6a.sock georep@gluster-4.glstr /nonexistent/gsyncd slave gl0 georep@gluster-4.glstr::glbackup --master-node gluster-3.glstr --master-node-id 9650e965-bf4f-4544-a42b-f4d540d23a1f --master-brick /var/lib/gluster --local-node gluster-4.glstr --local-node-id 736f6431-2f9c-4115-9790-68f9a88d99a7 --slave-timeout 120 --slave-log-level INFO --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin/ error=1 [2018-07-23 12:18:20.37] I [repce(agent /var/lib/gluster):80:service_loop] RepceServer: terminating on reaching EOF. [2018-07-23 12:18:21.389723] I [monitor(monitor):266:monitor] Monitor: worker died in startup phase brick=/var/lib/gluster repeated again and again. Maarten. ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] georeplication woes
Hi Sunny,
>> Can't run that command on the slave, it doesn't know the gl0 volume:
>>
>> root@gluster-4:/home/mrten# gluster volume geo-rep gl0
>> ssh://georep@gluster-4.glstr::glbackup config
> please do not use ssh://
> gluster volume geo-rep gl0 georep@gluster-4.glstr::glbackup config
> just use it like config command.
Sorry, does not make a difference on geo-rep slave side:
root@gluster-4:/home/mrten# gluster volume geo-rep gl0
georep@gluster-4.glstr::glbackup config
Volume gl0 does not exist
geo-replication command failed
Does work on the master side but the output is the same:
root@gluster-3:/home/mrten# gluster volume geo-rep gl0
georep@gluster-4.glstr::glbackup config
access_mount:false
allow_network:
change_detector:changelog
change_interval:5
changelog_archive_format:%Y%m
changelog_batch_size:727040
changelog_log_file:/var/log/glusterfs/geo-replication/gl0_gluster-4.glstr_glbackup/changes-${local_id}.log
changelog_log_level:INFO
checkpoint:0
chnagelog_archive_format:%Y%m
cli_log_file:/var/log/glusterfs/geo-replication/cli.log
cli_log_level:INFO
connection_timeout:60
georep_session_working_dir:/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/
gluster_cli_options:
gluster_command:gluster
gluster_command_dir:/usr/sbin/
gluster_log_file:/var/log/glusterfs/geo-replication/gl0_gluster-4.glstr_glbackup/mnt-${local_id}.log
gluster_log_level:INFO
gluster_logdir:/var/log/glusterfs
gluster_params:aux-gfid-mount acl
gluster_rundir:/var/run/gluster
glusterd_workdir:/var/lib/glusterd
gsyncd_miscdir:/var/lib/misc/gluster/gsyncd
ignore_deletes:false
isolated_slaves:
log_file:/var/log/glusterfs/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.log
log_level:INFO
log_rsync_performance:false
master_disperse_count:1
master_replica_count:1
max_rsync_retries:10
meta_volume_mnt:/var/run/gluster/shared_storage
pid_file:/var/run/gluster/gsyncd-gl0-gluster-4.glstr-glbackup.pid
remote_gsyncd:
replica_failover_interval:1
rsync_command:rsync
rsync_opt_existing:
rsync_opt_ignore_missing_args:
rsync_options:
rsync_ssh_options:
slave_access_mount:false
slave_gluster_command_dir:/usr/sbin/
slave_gluster_log_file:/var/log/glusterfs/geo-replication-slaves/gl0_gluster-4.glstr_glbackup/mnt-${master_node}-${master_brick_id}.log
slave_gluster_log_file_mbr:/var/log/glusterfs/geo-replication-slaves/gl0_gluster-4.glstr_glbackup/mnt-mbr-${master_node}-${master_brick_id}.log
slave_gluster_log_level:INFO
slave_gluster_params:aux-gfid-mount acl
slave_log_file:/var/log/glusterfs/geo-replication-slaves/gl0_gluster-4.glstr_glbackup/gsyncd.log
slave_log_level:INFO
slave_timeout:120
special_sync_mode:
ssh_command:ssh
ssh_options:-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem
ssh_options_tar:-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/tar_ssh.pem
ssh_port:22
state_file:/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/monitor.status
state_socket_unencoded:
stime_xattr_prefix:trusted.glusterfs.4054e7ad-7eb9-41fe-94cf-b52a690bb655.f7ce9a54-0ce4-4056-9958-4fa3f1630154
sync_acls:true
sync_jobs:3
sync_xattrs:true
tar_command:tar
use_meta_volume:true
use_rsync_xattrs:false
use_tarssh:false
working_dir:/var/lib/misc/gluster/gsyncd/gl0_gluster-4.glstr_glbackup/
This should be non-root georeplication, should this work?
georep@gluster-4:~$ /usr/sbin/gluster volume status all
Connection failed. Please check if gluster daemon is operational.
(run as the geo-replication user on the slave side)
Can I test something else? Is the command normally run in a jail?
Maarten.
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] georeplication woes
Hi Sunny, thanks for your reply! > I think setup for gluster binary location is wrongly setup here. > > Can you please try setting it up by using following command > > on master > #gluster vol geo-rep config > gluster-command-dir > > and on slave > #gluster vol geo-rep config > slave-gluster-command-dir Is there a difference between _ and -? I only seem to have the _ ones: (gluster-3 is one of the master 3-replica, gluster-4 is the slave) root@gluster-3:/home/mrten# gluster volume geo-rep gl0 ssh://georep@gluster-4.glstr::glbackup config | fgrep command gluster_command:gluster gluster_command_dir:/usr/sbin/ rsync_command:rsync slave_gluster_command_dir:/usr/sbin/ ssh_command:ssh tar_command:tar Can't run that command on the slave, it doesn't know the gl0 volume: root@gluster-4:/home/mrten# gluster volume geo-rep gl0 ssh://georep@gluster-4.glstr::glbackup config Volume gl0 does not exist geo-replication command failed But the location is correct: master: root@gluster-3:/home/mrten# which gluster /usr/sbin/gluster slave: root@gluster-4:/home/mrten# which gluster /usr/sbin/gluster and they work: root@gluster-3:/home/mrten# /usr/sbin/gluster gluster> root@gluster-4:/home/mrten# /usr/sbin/gluster gluster> > >> [2018-07-20 12:32:13.404048] W [gsyncd(slave >> gluster-2.glstr/var/lib/gluster):293:main] : Session config file not >> exists, using the default config >> path=/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf this one is weird though; there is nothing in /var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf but [vars] stime-xattr-prefix = trusted.glusterfs.4054e7ad-7eb9-41fe-94cf-b52a690bb655.f7ce9a54-0ce4-4056-9958-4fa3f1630154 use-meta-volume = true Maarten. ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] georeplication woes
Hi, I've upgraded my gluster 3.13 cluster to 4.0 on an Ubuntu server and it broke my geo-replication because of this missing file: https://bugzilla.redhat.com/show_bug.cgi?id=1601532 However, between geo-replication dying and me finding that bugreport I've tried multiple other things including setting up geo-rep again so I've still got a Faulty georeplication now. Removed it, created it again with the suggested fix in place but at the moment I see this error on the georep slave, in /var/log/glusterfs/glusterd.log: E [MSGID: 106581] [glusterd-mountbroker.c:579:glusterd_do_mount] 0-management: Missing mspec: Check the corresponding option in glusterd vol file for mountbroker user: georep [No such file or directory] These messages also seem relevant from /var/log/glusterfs/geo-replication-slaves/.../gsyncd.log: [2018-07-20 12:32:11.452244] I [resource(slave gluster-1.glstr/var/lib/gluster):1064:connect] GLUSTER: Mounting gluster volume locally... [2018-07-20 12:32:11.503766] E [resource(slave gluster-1.glstr/var/lib/gluster):973:handle_mounter] MountbrokerMounter: glusterd answered mnt= [2018-07-20 12:32:11.504313] E [syncdutils(slave gluster-1.glstr/var/lib/gluster):747:errlog] Popen: command returned error cmd=/usr/sbin/gluster --remote-host=localhost system:: mount georep user-map-root=georep aux-gfid-mount acl log-level=INFO log-file=/var/log/glusterfs/geo-replication-slaves/gl0_gluster-4.glstr_glbackup/mnt-gluster-1.glstr-var-lib-gluster.log volfile-server=localhost volfile-id=glbackup client-pid=-1 error=1 [2018-07-20 12:32:11.504446] E [syncdutils(slave gluster-1.glstr/var/lib/gluster):751:logerr] Popen: /usr/sbin/gluster> 2 : failed with this errno (No such file or directory) [2018-07-20 12:32:13.404048] W [gsyncd(slave gluster-2.glstr/var/lib/gluster):293:main] : Session config file not exists, using the default config path=/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf Is there anyone out there that can tell me what to do with these error messages? I've resorted to checking out the source code but that road did not lead to enlightment either :( thanks in advance, Maarten. [1] The gluster master is a 3-replica; the slave is a single volume at another physical location, geo-replication user is georep. main volume is called 'gl0', backup 'glbackup'. root@gluster-4:/var/log/glusterfs# gluster-mountbroker status +---+-+-++---+ |NODE | NODE STATUS |MOUNT ROOT | GROUP| USERS | +---+-+-++---+ | localhost | UP | /var/lib/georep-mountbroker(OK) | georep(OK) | georep(glbackup) | +---+-+-++---+ ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
