Re: [Gluster-users] High-availability with KVM?

On 20 January 2017 at 19:26, Lindsay Mathieson wrote: > This I think, highlights one of glusters few weaknesses - the > inflexibility of brick layout. It would be really nice if you could just > arbitrarily add bricks to distributed-replicate volumes and have files

Re: [Gluster-users] High-availability with KVM?

On 21/01/2017 6:52 AM, Gambit15 wrote: With that setup, you lose quorum if you lose any one node. Brick 1 replicates to brick 2, and brick 3 replicates to brick 4. If any one of those goes down, quorum falls to <51%, which locks the brick under the default settings. This I think, highlights

Re: [Gluster-users] High-availability with KVM?

On 21/01/2017 8:12 AM, Ziemowit Pierzycki wrote: Can I not add a brick that is already used in volume just a different location? I don't think so, I believe you have to earse it, add the brick and let a full heal complete. I'd wait for confirmation from a dev first though. -- Lindsay

Re: [Gluster-users] High-availability with KVM?

Can't afford downtime... I see starting with version 3.8 I can add them while they're online, but when I try I get: gluster> v add-brick virt0 replica 3 arbiter 1 dalpinfglt01:/vol/virt/arbt0 volume add-brick: failed: Operation failed Can I not add a brick that is already used in volume just a

Re: [Gluster-users] High-availability with KVM?

Ah yeah, I did wonder about that but I thought quorum was based on all bricks, not just the replica set. Guess my first instinct was right ! > I see. So if I switch to a different cluster quorum model, I may get > split-brain which would need manual intervention should a node go > missing? Yes

Re: [Gluster-users] High-availability with KVM?

I see. So if I switch to a different cluster quorum model, I may get split-brain which would need manual intervention should a node go missing? Adding an arbiter involves creating a volume from scratch? You can't just add it to an existing volume. On Fri, Jan 20, 2017 at 2:52 PM, Gambit15

Re: [Gluster-users] High-availability with KVM?

> > Type: Distributed-Replicate > Number of Bricks: 2 x 2 = 4 > With that setup, you lose quorum if you lose any one node. Brick 1 replicates to brick 2, and brick 3 replicates to brick 4. If any one of those goes down, quorum falls to <51%, which locks the brick under the default settings. If

Re: [Gluster-users] High-availability with KVM?

> Also, should there be anything I have to do after bringing the node > back up and letting the heal process run? No, everything should be automatic. Did you try looking at the gfapi logs of the VMs ? They should be showed on the standard output of the VM process, if that's something you have

Re: [Gluster-users] Convert to Shard - Setting Guidance

> > I would be interested to hear how you did this while running. On my > test setup, I have gone through the copy (rename) and it does work but > like you said it took quite awhile. I went into my proxmox web interface, selected the disk and clicked "move" :) It just uses some qemu command

Re: [Gluster-users] Convert to Shard - Setting Guidance

I had a few different data points on the 512MB size as well as setting the heal algorithm to full. Some of this information is old though so I appreciate the feedback that you have given on what you are using. I see Lindsay confirmed what I have witnessed while testing these settings

Re: [Gluster-users] High-availability with KVM?

Yes, only the VMs that reside on the node start having issues. Also, should there be anything I have to do after bringing the node back up and letting the heal process run? Thanks. On Fri, Jan 20, 2017 at 2:26 PM, Gambit15 wrote: > As long as there are enough nodes to

Re: [Gluster-users] High-availability with KVM?

I'm using libgfapi. Gluster version is 3.8.1-1. Here is the volume info: Volume Name: virt0 Type: Distributed-Replicate Volume ID: fb9f428e-b1b5-4136-8b59-19d680237302 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: elkpinfglt01:/vol/virt/virt0 Brick2:

Re: [Gluster-users] High-availability with KVM?

> So, is the setup wrong or does gluster not provide high availability? How exactly is it setup ? libgfapi ? fuse ? NFS mount ? It should work, we're using proxmox at work (which uses KVM) with gluster and it does work well. What version of gluster are you using ? -- Kevin Lemonnier PGP

[Gluster-users] High-availability with KVM?

Hi, I have a 4-node gluster with distributed-replicated volumes that serve out VM images. The purpose of using gluster was to provide highly available storage for our virtualization platform. When performing maintenance on one of the gluster nodes, I noticed the VM storage becomes unavailable

Re: [Gluster-users] Convert to Shard - Setting Guidance

> > Just the files that need healing and for sharded volumes thats the > individual shards. > Yep, in our case it's just syncing a bunch of 64 Mb files, it works a lot better than the default algo, at least it did last time I tried the default one. Now I've been staying on 3.7.12 so maybe it's

Re: [Gluster-users] Convert to Shard - Setting Guidance

On 21/01/2017 1:26 AM, Gambit15 wrote: And according to the docs, "full" forces a resync of the entire volume, not just the out-of-sync files. Seems a bit overkill for anything other than a brick replacement... Just the files that need healing and for sharded volumes thats the individual

Re: [Gluster-users] Convert to Shard - Setting Guidance

> If your images easily fit within the bricks, why do you need sharding in >> the first place? It adds an extra layer of complexity & removes the cool >> feature of having entire files on each brick, making DR & things a lot >> easier. > > > Because healing with large VM images completes orders of

Re: [Gluster-users] Convert to Shard - Setting Guidance

> > > data-self-heal-algorithm full > > There was a bug in the default algo, at least for VM hosting, > not that long ago. Not sure if it was fixed but I know we were > told here to use full instead, I'm guessing that's why he's using it too. > Huh, not heard of that. Do you have any useful links

Re: [Gluster-users] Convert to Shard - Setting Guidance

On 21/01/2017 12:57 AM, Kevin Lemonnier wrote: It will only affect new files, so you'll need to copy the current images to new names. That's why I was speaking about livre migrating the disk to the same volume, that's how I did it last year. Good to know, thanks Kevin. -- Lindsay Mathieson

Re: [Gluster-users] Convert to Shard - Setting Guidance

> - or change the shard setting inplace? (I don't think that would work) It will only affect new files, so you'll need to copy the current images to new names. That's why I was speaking about livre migrating the disk to the same volume, that's how I did it last year. -- Kevin Lemonnier PGP

Re: [Gluster-users] Convert to Shard - Setting Guidance

On 20/01/2017 1:11 PM, Gustave Dahl wrote: One: It may take months to convert my current VM images to shard’s, do you see any issues with this? My priority is to make sure future images are distributed as shards. Two: Settings, my intent is to set it as follows based on guidance on the

Re: [Gluster-users] Convert to Shard - Setting Guidance

On 21/01/2017 12:07 AM, Gambit15 wrote: If your images easily fit within the bricks, why do you need sharding in the first place? It adds an extra layer of complexity & removes the cool feature of having entire files on each brick, making DR & things a lot easier. Because healing with large

Re: [Gluster-users] Convert to Shard - Setting Guidance

> data-self-heal-algorithm full There was a bug in the default algo, at least for VM hosting, not that long ago. Not sure if it was fixed but I know we were told here to use full instead, I'm guessing that's why he's using it too. > If your images easily fit within the bricks, why do you need

Re: [Gluster-users] Convert to Shard - Setting Guidance

It's a given, but test it well before going into production. People have occasionally had problems with corruption when converting to shards. In my initial tests, enabling sharding took our I/O down to 15Kbps from 300Mpbs without. data-self-heal-algorithm full > That could be painful. Any

Re: [Gluster-users] RDMA transport problems in GLUSTER on host with MIC

One thing to note here is that rdma uses srq, which I see as disabled in both devices. Regards Rafi KC On 01/20/2017 05:05 PM, Anoop C S wrote: > On Fri, 2017-01-20 at 11:53 +0100, Fedele Stabile wrote: >> Thank you for your help, >> I will answer to your questions: >> >> Il giorno ven,

Re: [Gluster-users] RDMA transport problems in GLUSTER on host with MIC

On Fri, 2017-01-20 at 11:53 +0100, Fedele Stabile wrote: > > > Thank you for your help,  > I will answer to your questions: > > Il giorno ven, 20/01/2017 alle 12.58 +0530, Anoop C S ha scritto: > > On Wed, 2017-01-18 at 12:56 +0100, Fedele Stabile wrote: > > > Hi, > > > it happens that RDMA

Re: [Gluster-users] RDMA transport problems in GLUSTER on host with MIC

# rpm -qa | grep glusterfs-rdma glusterfs-rdma-3.8.5-1.el6.x86_64 glusterfs-rdma is installed and # glusterd -V glusterfs 3.8.5 built on Oct 13 2016 14:04:36 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. GlusterFS comes

Re: [Gluster-users] RDMA transport problems in GLUSTER on host with MIC

On Fri, 2017-01-20 at 11:53 +0100, Fedele Stabile wrote: > Thank you for your help,  > I will answer to your questions: > > Il giorno ven, 20/01/2017 alle 12.58 +0530, Anoop C S ha scritto: > > On Wed, 2017-01-18 at 12:56 +0100, Fedele Stabile wrote: > > > Hi, > > > it happens that RDMA gluster

Re: [Gluster-users] RDMA transport problems in GLUSTER on host with MIC

Thank you for your help, I will answer to your questions: Il giorno ven, 20/01/2017 alle 12.58 +0530, Anoop C S ha scritto: > On Wed, 2017-01-18 at 12:56 +0100, Fedele Stabile wrote: > > Hi, > > it happens that RDMA gluster transport does not works anymore > > after I have configured ibscif

Re: [Gluster-users] Convert to Shard - Setting Guidance

> > One: It may take months to convert my current VM images to shard's, do you > see any issues with this? My priority is to make sure future images are > distributed as shards. You should be able to do that while your VMs are running. I guess it depends on your hypvervisor, but with KVM just

Re: [Gluster-users] [Gluster-devel] Lot of EIO errors in disperse volume

On 20/01/17 08:55, Ankireddypalle Reddy wrote: Xavi, Thanks. Please let me know the functions that we need to track for any inconsistencies in the return codes from multiple bricks for issue 1. I will start doing that. 1. Why the write fails in first place The best way