Re: [Gluster-users] how to get the true used capacity of the volume

You will get weird results like these if you put two bricks on a single filesystem. In use case one (presumably replica 2) the data gets written to both bricks, which means there are two copies on the disk and so twice the disk space consumed. In the second case there is some overhead involved

Re: [Gluster-users] [Gluster-devel] Announcing Glusterfs release 3.12.2 (Long Term Maintenance)

seemed fine - no errors nothing unexpected. On 23 October 2017 at 17:42, Niels de Vos <nde...@redhat.com> wrote: > On Mon, Oct 23, 2017 at 02:12:53PM -0400, Alastair Neil wrote: > > Any idea when these packages will be in the CentOS mirrors? there is no &

Re: [Gluster-users] SMB copies failing with GlusterFS 3.10

What is the volume configuration, is it replicated, distributed, distribute-replicate, disperse? have you tried setting: performance.strict-write-ordering to on? On 14 November 2017 at 06:24, Brett Randall wrote: > Hi all > > We've got a brand new 6-node GlusterFS 3.10

Re: [Gluster-users] [Gluster-devel] Request for Comments: Upgrades from 3.x to 4.0+

Ahh OK I see, thanks On 6 November 2017 at 00:54, Kaushal M <kshlms...@gmail.com> wrote: > On Fri, Nov 3, 2017 at 8:50 PM, Alastair Neil <ajneil.t...@gmail.com> > wrote: > > Just so I am clear the upgrade process will be as follows: > > > > upgrade all client

Re: [Gluster-users] [Gluster-devel] Request for Comments: Upgrades from 3.x to 4.0+

Just so I am clear the upgrade process will be as follows: upgrade all clients to 4.0 rolling upgrade all servers to 4.0 (with GD1) kill all GD1 daemons on all servers and run upgrade script (new clients unable to connect at this point) start GD2 ( necessary or does the upgrade script do

Re: [Gluster-users] brick is down but gluster volume status says it's fine

.export-brick7-digitalcorpora -p /var/lib/glusterd/vols/digitalcorpor [root@gluster-2 ~]# On 24 October 2017 at 13:56, Atin Mukherjee <amukh...@redhat.com> wrote: > > > On Tue, Oct 24, 2017 at 11:13 PM, Alastair Neil <ajneil.t...@gmail.com> > wrote: > >> gluster

[Gluster-users] brick is down but gluster volume status says it's fine

gluster version 3.10.6, replica 3 volume, daemon is present but does not appear to be functioning peculiar behaviour. If I kill the glusterfs brick daemon and restart glusterd then the brick becomes available - but one of my other volumes bricks on the same server goes down in the same way it's

Re: [Gluster-users] [Gluster-devel] Announcing Glusterfs release 3.12.2 (Long Term Maintenance)

Any idea when these packages will be in the CentOS mirrors? there is no sign of them on download.gluster.org. On 13 October 2017 at 08:45, Jiffin Tony Thottan wrote: > The Gluster community is pleased to announce the release of Gluster 3.12.2 > (packages available at

Re: [Gluster-users] warning spam in the logs after tiering experiment

forgot to mention Gluster version 3.10.6 On 18 October 2017 at 13:26, Alastair Neil <ajneil.t...@gmail.com> wrote: > a short while ago I experimented with tiering on one of my volumes. I > decided it was not working out so I removed the tier. I now have spam in > the glust

[Gluster-users] warning spam in the logs after tiering experiment

a short while ago I experimented with tiering on one of my volumes. I decided it was not working out so I removed the tier. I now have spam in the glusterd.log evert 7 seconds: [2017-10-18 17:17:29.578327] W [socket.c:3207:socket_connect] 0-tierd: Ignore failed connection attempt on

Re: [Gluster-users] gluster volume + lvm : recommendation or neccessity ?

LVM is also good if you want to add ssd cache. It is more flexible and easier to manage and expand than bcache. On 11 October 2017 at 04:00, Mohammed Rafi K C wrote: > > Volumes are aggregation of bricks, so I would consider bricks as a > unique entity here rather than

Re: [Gluster-users] small files performance

I just tried setting: performance.parallel-readdir on features.cache-invalidation on features.cache-invalidation-timeout 600 performance.stat-prefetch performance.cache-invalidation performance.md-cache-timeout 600 network.inode-lru-limit 5 performance.cache-invalidation on and clients could

Re: [Gluster-users] Access from multiple hosts where users have different uid/gid

nfsv4 id mapping is based on username not uid so you could use ganesha nfs to share the files. On 5 October 2017 at 04:42, Frizz wrote: > I have a setup with multiple hosts, each of them are administered > separately. So there are no unified uid/gid for the users. >

Re: [Gluster-users] GlusterFS as virtual machine storage

ing to do IO. > > https://gluster.readthedocs.io/en/latest/Administrator% > 20Guide/Split%20brain%20and%20ways%20to%20deal%20with% > 20it/#1-replica-3-volume > > > > On Sep 7, 2017 17:52, "Alastair Neil" <ajneil.t...@gmail.com> wrote: > >> *shrug*

Re: [Gluster-users] GlusterFS as virtual machine storage

t 03:59:14PM -0400, Alastair Neil wrote: > > you need to set > > > > cluster.server-quorum-ratio 51% > > > > On 6 September 2017 at 10:12, Pavel Szalbot <pavel.szal...@gmail.com> > wrote: > > > > > Hi all, > > > > > >

Re: [Gluster-users] GlusterFS as virtual machine storage

you need to set cluster.server-quorum-ratio 51% On 6 September 2017 at 10:12, Pavel Szalbot wrote: > Hi all, > > I have promised to do some testing and I finally find some time and > infrastructure. > > So I have 3 servers with Gluster 3.10.5 on CentOS 7. I

Re: [Gluster-users] RECOMMENDED CONFIGURATIONS - DISPERSED VOLUME

Dmitri the recommendation from redhat is likely because it is recommended to have the data stripes be a power of two otherwise there is a performance penalty. On 31 July 2017 at 14:28, Dmitri Chebotarov <4dim...@gmail.com> wrote: > Hi > > I'm looking for an advise to configure a dispersed

Re: [Gluster-users] Multi petabyte gluster

I can ask our other engineer but I don't have those figues. -Alastair On 30 June 2017 at 13:52, Serkan Çoban <cobanser...@gmail.com> wrote: > Did you test healing by increasing disperse.shd-max-threads? > What is your heal times per brick now? > > On Fri, Jun 30, 2017 at 8:01

Re: [Gluster-users] Multi petabyte gluster

We are using 3.10 and have a 7 PB cluster. We decided against 16+3 as the rebuild time are bottlenecked by matrix operations which scale as the square of the number of data stripes. There are some savings because of larger data chunks but we ended up using 8+3 and heal times are about half

[Gluster-users] issue with trash feature and arbiter volumes

Gluster 3.10.2 I have a replica 3 (2+1) volume and I have just seen both data bricks go down (arbiter stayed up). I had to disable trash feature to get the bricks to start. I had a quick look on bugzilla but did not see anything that looked similar. I just wanted to check that I was not

Re: [Gluster-users] disperse volume brick counts limits in RHES

pating in an EC >> set number >> > of parallel heals increase. Is the heal speed you saw improved per >> file or >> > the over all time it took to heal the data? >> > >> >> >> >> >> >> >> &g

Re: [Gluster-users] disperse volume brick counts limits in RHES

;> >> speed. > >>> >> I tested heal speed with 8+2 and 16+4 on 3.9.0 and see that heals on > >>> >> 8+2 is faster by 2x. > >>> > > >>> > > >>> > As you increase number of nodes that are participating in an EC set >

[Gluster-users] disperse volume brick counts limits in RHES

Hi we are deploying a large (24node/45brick) cluster and noted that the RHES guidelines limit the number of data bricks in a disperse set to 8. Is there any reason for this. I am aware that you want this to be a power of 2, but as we have a large number of nodes we were planning on going with

[Gluster-users] advice needed on configuring large gluster cluster

Hi we have a new gluster cluster we are planning on deploying. We will have 24 nodes each with JBOD, 39 8TB drives and 6, 900GB SSDs, and FDR IB We will not be using all of this as one volume , but I thought initially of using a distributed disperse volume. Never having attempted anything on

Re: [Gluster-users] 90 Brick/Server suggestions?

We have 12 on order. Actually the DSS7000 has two nodes in the chassis, and each accesses 45 bricks. We will be using an erasure code scheme probably 24:3 or 24:4, we have not sat down and really thought about the exact scheme we will use. On 15 February 2017 at 14:04, Serkan Çoban

[Gluster-users] 3.8.5 replica 3 volumes: I/O error on file on fuse mounts

Would apprecaite any insight into this issue: replica 3 volume, it is showing a number of files on two of the bricks as needing healed, when you examine the files on the fuse mounts they generate I/O errors. No files listed in split brain, but if I look at one of the files it looks to me like they

Re: [Gluster-users] Looking for use cases / opinions

Serkan I'd be interested to know how your disks are attached (SAS?)? Do you use any hardware RAID, or zfs and do you have and SSDs in there? On 9 November 2016 at 06:17, Serkan Çoban wrote: > Hi, I am using 26x8TB disks per server. There are 60 servers in gluster >

Re: [Gluster-users] Performance

What version of Gluster? Are you using glusterfs or nfs mount? Any other traffic on the network, is the cluster quiescent apart from your dd test? It does seem slow. I have a three server cluster, using straight xfs over 10G with Gluster 3.8 and glusterfs mounts and I see: [root@sb-c

Re: [Gluster-users] How gluster parallelize reads

I think this might give you something like the behaviour you are looking for, it will not balance blocks across different servers but will distribute reads from clients across all the servers. cluster.read-hash-mode 2 0 means use the first server to respond I think - at least that's my guess of

Re: [Gluster-users] New cluster - first experience

It does not seem to me that this is a gluster issue. I just quickly reviewed the thread and you said that you saw 60 MB/s with plain nfs to the bricks and with gluster and no sharding you got 59 MB/s With plain NFS (no gluster involved) i'm getting almost the same > speed: about 60MB/s > >

Re: [Gluster-users] New cluster - first experience

I am not sure if your nics support it but you could try balance-alb (bonding mode 6), this does not require special switch support and I have had good results with it. As Lindsey said the switch configuration could be limiting the bandwidth between nodes in balance-rr. On 14 July 2016 at 05:21,

Re: [Gluster-users] GlusterFS Storage Interruption at Node Loss

Nic I believe this is normal expected behaviour. The network timeout is there because it is expensive to tear down the sockets etc. so you only want to do it if a node has really failed and not for some transitory network blip. On 8 July 2016 at 20:29, Nic Seltzer

Re: [Gluster-users] New cluster - first experience

Also remember with a single transfer you will not see 2000 gb/s only 1000 gb/s On 8 July 2016 at 15:14, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > 2016-07-08 20:43 GMT+02:00 : > > Gluster, and in particular the fuse mounter, do not operate on small >

[Gluster-users] Fedora upgrade to f24 installed 3.8.0 client and broke mounting

I upgraded my fedora 23 system to f24 a couple of days ago, now I am unable to mount my gluster cluster. The update installed: glusterfs-3.8.0-1.fc24.x86_64 glusterfs-libs-3.8.0-1.fc24.x86_64 glusterfs-fuse-3.8.0-1.fc24.x86_64 glusterfs-client-xlators-3.8.0-1.fc24.x86_64 the gluster is running

Re: [Gluster-users] snapshot removal failed on one node how to recover (3.7.11)

No one has any suggestions? Would this scenario I have been toying with work: remove the brick from the node with the out of sync snapshots, destroy all associated logical volumes, and then add the brick back as an arbiter node? On 1 June 2016 at 13:40, Alastair Neil <ajneil.t...@gmail.

[Gluster-users] snapshot removal failed on one node how to recover (3.7.11)

I have a replica 3 volume that has snapshot scheduled using snap_scheduler.py I recently tried to remove a snapshot and the command failed on one node: snapshot delete: failed: Commit failed on gluster0.vsnet.gmu.edu. Please > check log file for details. > Snapshot command failed How do I

Re: [Gluster-users] Questions about healing

yes it's configurable with: network.ping-timeout and default is 42 seconds I believe. On 22 May 2016 at 03:39, Kevin Lemonnier wrote: > > Let's assume 10.000 shard on a server being healed. > > Gluster heal 1 shard at once, so the other 9.999 pieces would be read > >

Re: [Gluster-users] Questions about healing

ve found that yes some VMs will experience IO timeouts, freeze, and then need to be restarted. However, at least you don't need to wait hours before you can do that. On 20 May 2016 at 15:20, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > Il 20 mag 2016 20:14,

Re: [Gluster-users] Questions about healing

I think you are confused about what sharding does. In a sharded replica 3 volume all the shards exist on all the replicas so there is no distribution. Might you be getting confused with erasure coding? The upshot of sharding is that if you have a failure, instead of healing multiple gigabyte

Re: [Gluster-users] VM disks corruption on 3.7.11

I am slightly confused you say you have image file corruption but then you say the qemu-img check says there is no corruption. If what you mean is that you see I/O errors during a heal this is likely to be due to io starvation, something that is a well know issue. There is work happening to

Re: [Gluster-users] "gluster volume heal full" locking all files after adding a brick

what are the quorum setting on the volumes? On 27 April 2016 at 08:08, Tomer Paretsky wrote: > Hi all > > i am currently running two replica 3 volumes acting as storage for VM > images. > due to some issues with glusterfs over ext4 filesystem (kernel panics), i > tried

[Gluster-users] self service snapshot access broken with 3.7.11

I just upgraded my cluster to 3.7.11 from 3.7.10 and access to the .snaps directories now fail with bash: cd: .snaps: Transport endpoint is not connected in the volume log file on the client I see: 016-04-22 21:08:28.005854] I [rpc-clnt.c:1847:rpc_clnt_reconfig] > 2-homes-snapd-client:

[Gluster-users] non interactive use of glsuter snapshot delete

I am crafting a script (well actually I am modifying gcron.py) to retain a configurable number of Hourly, Daily,Weekly and Monthly snapshots. One issue is that gluster snapshot delete "snapname" does not seem to have a command line switch (like "-y" in yum) to attempt the operation

Re: [Gluster-users] How to recover after one node breakdown

hopefully you have a back up of /var/lib/glusterd/glusterd.info and /var/lib/glusterd/peers, if so I think you can copy them back to and restart glusterd and the volume info should get populated from the other node. If not you can probably reconstruct these from these files on the other node.

Re: [Gluster-users] Issue with storage access when brick goes down

I thought it was to do with the expense of tearing down and setting up the connections, so the timeout is there to avoid an expensive operation if it is not really necessary. On 7 December 2015 at 22:15, Bipin Kunal wrote: > I assume that this is because the host which went

Re: [Gluster-users] gluster 3.7.5 low speed on write and heavy load

is that a typo 120mb/s? Is this volume replica 2 or distributed or both? On 16 October 2015 at 08:05, Kandalf ® wrote: > Hi, > > I have 2 server with centos 7 and I want to make a replicate storage > cluster for the esxi. > In both server I have 4 disk in raid 5 mdadm. The

Re: [Gluster-users] gluster 3.7.5 low speed on write and heavy load

aw file like > esxi does in vmdk, than the speed drops to 20-40mb/s. So my issue is low > speed when I write into raw image file (using losetup and DD on linux , or > esxi vmdk - vm guest) > > > > On Friday, October 16, 2015 8:31 PM, Alastair Neil <ajneil.t...@gmail.com>

Re: [Gluster-users] gluster 3.7.5 low speed on write and heavy load

; > > If I write with DD in linux locally to the same cluster node... or via > network I have 120MB/s. the full speed of te 1 Gbps. > But if I write to an RAW Image File... like vmdk in vmware... I have > average of 20-40MB/s only. > > > > On Friday, October 16, 2015 11:31

Re: [Gluster-users] Suggested method for replacing an entire node

I think you should back up /var/lib/glusterd and then restore it after the reinstall and installation of glusterfs packages. Assuming the node will have the same hostname and ip addresses and you are installing the same version gluster bits, I think it should be fine. I am assuming you are not

Re: [Gluster-users] Suggested method for replacing an entire node

Ahh that is good to know. On 8 October 2015 at 09:50, Atin Mukherjee <atin.mukherje...@gmail.com> wrote: > -Atin > Sent from one plus one > On Oct 8, 2015 7:17 PM, "Alastair Neil" <ajneil.t...@gmail.com> wrote: > > > > I think you should back up

Re: [Gluster-users] Scaling a network - Error while adding new brick to a existing cluster.

Yes it seems likely you have a duplicate conflicting gluster configuration. Fixing it should be fairly easy: clear out /var/lib/glusterd, and then reinstall the glusterfs packages. You will also have to clear all the extended attributes and files from any brick directories, it may be easier to

Re: [Gluster-users] GlusterFS 3.7

glusterd should handle syncing any changes you make with the "gluster" command to the peers, obviously if you make local changes to the volume file on one server you are likely to break things unless you copy rsync the changes to the other server. On 22 September 2015 at 04:02, Andreas Hollaus

Re: [Gluster-users] multiple volumes on same brick?

you mean create multiple top level directories on a mounted files system and use each directory as a brick in a different volume? if the underlying block device is a thinly provisioned lvm2 volume and you want to use snapshots you cannot do this, otherwise I don't think there is a technical

Re: [Gluster-users] autosnap feature?

abels automatically, when necessary. So some sort of ability to create and manipulate labels on snapshots and then expose them as links in the .snaps directory would probably be a start. -Alastair On 15 September 2015 at 01:35, Rajesh Joseph <rjos...@redhat.com> wrote: > > >

Re: [Gluster-users] GlusterFS disk full.

If you have an active snapshot, I expect the space will not be freed until you remove the snapshot. On 11 September 2015 at 01:44, Fujii Yasuhiro wrote: > Hi. > > I have a question. > > GlusterFS hdd space can't be recovered automatically after glusterfs > is disk full and

[Gluster-users] autosnap feature?

Wondering if there were any plans for a fexible and easy to use snapshotting feature along the lines of zfs autosnap scipts. I imagine at the least it would need the ability to rename snapshots. ___ Gluster-users mailing list Gluster-users@gluster.org

[Gluster-users] questions about snapshots and cifs

I have been working on providing user serviceable snap shots and I am confused by the documentation about allowing access from windows explorer: For snapshots to be accessible from windows, below 2 options can be used. > A) The glusterfs plugin for samba should give the option >

Re: [Gluster-users] Cannot upgrade from 3.6.3 to 3.7.3

Did you mean the option rpc-auth-allow-insecure on setting? I just did a rolling upgrade from 3.6 to 3.7 without issue, however, I had enabled insecure connections because I had some clients running 3.7. -Alastair On 27 August 2015 at 10:04, Andreas Mather andr...@allaboutapps.at wrote: Hi

Re: [Gluster-users] backupvolfile-server fqdn vs short

_netdev,use-readdirp=no,backupvolfile-server=gluster1.vsnet.gmu.edu:g luster-2.vsnet.gmu.edu 0 0 On 23 July 2015 at 12:30, Atin Mukherjee atin.mukherje...@gmail.com wrote: -Atin Sent from one plus one On Jul 23, 2015 9:07 PM, Alastair Neil ajneil.t...@gmail.com wrote: I just had

[Gluster-users] backupvolfile-server fqdn vs short

I just had a curious failure. I have a gluster 3.6.3 replica 3 volume which was mounted via an 3.6.3 client from one of the nodes with the other two specified in the backupvolfile-server mount option. In the fstab entry all the nodes are referenced by their fully qualified domain names. When I

Re: [Gluster-users] 3.6.3 split brain on web browser cache dir w. replica 3 volume

, Alastair Neil wrote: I have a replica 3 volume I am using to serve my home directory. I have notices a couple of split-brains recently on files used by browsers(for the most recent see below, I had an earlier one on .config/google-chrome/Default/Session Storage/) . When I was running replica 2

[Gluster-users] 3.6.3 split brain on web browser cache dir w. replica 3 volume

I have a replica 3 volume I am using to serve my home directory. I have notices a couple of split-brains recently on files used by browsers(for the most recent see below, I had an earlier one on .config/google-chrome/Default/Session Storage/) . When I was running replica 2 I don't recall seeing

Re: [Gluster-users] fault tolerance

For the client's not to notice the fault you would need to have three servers in replica 3, otherwise the filesystem will become read-only with one node down. On 17 May 2015 at 03:25, Markus Ueberall markus.ueber...@gmail.com wrote: Dear Carlos, Have a look at http://bit.ly/gadmstp (section

Re: [Gluster-users] fault tolerance

True, but since split-brain resolution is tedious, I assumed he would have the quorum set to 51% On 21 May 2015 at 00:57, Joe Julian j...@julianfamily.org wrote: On 05/20/2015 09:51 PM, Alastair Neil wrote: For the client's not to notice the fault you would need to have three servers

[Gluster-users] Unable to add brick from new fresh installed server into upgraded cluster

I have a 2 server CentOS 6 replicated cluster which started out as version 3.3 and has been progressively updated and is now on version 3.6.1. Yesterday I added a new freshly installed CentOS 6.6 host and wanted to convert to replica 3 on one of my volumes, however I was unable to add the brick

[Gluster-users] Installing glusterfs 3.6 client on EL7

I'm looking for repos to install on EL7 since RH only supplies 3.4. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] confused about replicated volumes and sparse files

I am trying to understand how verify that a replicated volume is up to date. Here is my scenario. I have a gluster cluster with two nodes serving vm images to ovirt. I have a volume called vm-store with a brick from each of the nodes: Volume Name: vm-store Type: Replicate Volume ID: