Re: [Gluster-users] Change underlying brick on node

[David Gossage]

On Mon, Aug 8, 2016 at 5:24 PM, Joe Julian  wrote:

>
>
> On 08/08/2016 02:56 PM, David Gossage wrote:
>
> On Mon, Aug 8, 2016 at 4:37 PM, David Gossage  > wrote:
>
>> On Mon, Aug 8, 2016 at 4:23 PM, Joe Julian  wrote:
>>
>>>
>>>
>>> On 08/08/2016 01:39 PM, David Gossage wrote:
>>>
>>> So now that I have my cluster on 3.7.14 and sharded and working I am of
>>> course looking for what to break next.
>>>
>>> Currently each of 3 nodes is on a 6 disk (WD Red 1TB) raidz6 (zil on
>>> mirrored ssd), which I am thinking is more protection than I may need with
>>> a 3 way replica.  I was going to one by one change them to basically raid10
>>> letting it heal in between.
>>>
>>> Is best way to do that a systemctl stop glusterd, should I just kill the
>>> brick process to simulate a brick dying, or is their an actual brick
>>> maintenance command?
>>>
>>>
>>> Just kill (-15) the brick process. That'll close the TCP connections and
>>> the clients will just go right on functioning off the remaining replica.
>>> When you format and recreate your filesystem, it'll be missing the
>>> volume-id extended attributes so to start it you'll need to force it:
>>>
>>
> Also could I just do this from different node?
>
> getfattr  -n trusted.glusterfs.volume-id /srv/.bricks/www
>
> Then on node with new raid10 backed disks
>
> setfattr -n trusted.glusterfs.volume-id -v 'value_from_other_brick'
> /srv/.bricks/www
>
>
> Sure, but that's a lot more keystrokes and a lot more potential for human
> error.
>

So far working on test machine.  my one vm  still plodding away no packet
loss


>
>
>
>
>>>gluster volume start $volname start force
>>>
>>
>> If I left volume started when brick process is killed and clients are
>> still (in theory) connected to volume wouldn't that just give me an error
>> that volume is already started?
>>
>>
>> Likely I would shut down the volume and do downtime for this anyway
>> though letting heals go on with VM's off.
>>
>>
>>>
>>> If /etc/glusterfs is unchanged and /var/lib/glusterd is unchanged will
>>> doing a heal full after reboot or restarting glusterd take care of
>>> everything if I recreate the expected brick path first?
>>>
>>>
>>> Once started, perform a full heal to re-replicate.
>>>
>>>
>>> Are the improvements in 3.8 for sharding significant enough I should
>>> first look at updating to 3.8.2 when released in few days?
>>>
>>>
>>> Yes.
>>>
>>>
>>>
>>> *David Gossage*
>>> *Carousel Checks Inc. | System Administrator*
>>> *Office* 708.613.2284
>>>
>>>
>>> ___
>>> Gluster-users mailing 
>>> listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>> ___ Gluster-users mailing
>>> list Gluster-users@gluster.org http://www.gluster.org/mailman
>>> /listinfo/gluster-users
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[David Gossage]

On Tue, Aug 9, 2016 at 2:18 AM, Lindsay Mathieson <
lindsay.mathie...@gmail.com> wrote:

> On 9 August 2016 at 12:23, David Gossage 
> wrote:
> > Since my dev is now on 3.8 and has granular enabled I'm feeling too lazy
> to
> > roll back so will just wait till 3.8.2 is released in few days that fixes
> > the bugs mentioned to me and then test this few times on my dev.
> >
> > Would be nice if I could get to a point where I could have one brick dead
> > and doing a full heal and not have every VM pause while shards heal, but
> I
> > may be asking too much when dealing with a rather heavy recovery.
>
>
> I tested Joe's method with my dev cluster, worked perfectly. The *one*
> running VM on it didn't seem to badly affected, but it was already
> very slow :)
>

3.7 or 3.8?  And if 3.8 did you have that new granular-entry-heal  feature
enabled?


> Thanks Joe.
>
> --
> Lindsay
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[Lindsay Mathieson]

On 9 August 2016 at 12:23, David Gossage  wrote:
> Since my dev is now on 3.8 and has granular enabled I'm feeling too lazy to
> roll back so will just wait till 3.8.2 is released in few days that fixes
> the bugs mentioned to me and then test this few times on my dev.
>
> Would be nice if I could get to a point where I could have one brick dead
> and doing a full heal and not have every VM pause while shards heal, but I
> may be asking too much when dealing with a rather heavy recovery.


I tested Joe's method with my dev cluster, worked perfectly. The *one*
running VM on it didn't seem to badly affected, but it was already
very slow :)

Thanks Joe.

-- 
Lindsay
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[David Gossage]

On Mon, Aug 8, 2016 at 9:15 PM, Lindsay Mathieson <
lindsay.mathie...@gmail.com> wrote:

> On 9 August 2016 at 07:23, Joe Julian  wrote:
> > Just kill (-15) the brick process. That'll close the TCP connections and
> the
> > clients will just go right on functioning off the remaining replica. When
> > you format and recreate your filesystem, it'll be missing the volume-id
> > extended attributes so to start it you'll need to force it:
> >
> >gluster volume start $volname start force
>
>
> Just to clarify I'm interpreting this correctly, to replace a brick
> and preserve its mount point you can:
>
> 1. kill the brick process (glusterfsd)
>
> 2. Do your disk maintenance. Eventually you have a clean (erased) brick
> mount
>
> 3. Force the bricks process start. This will recreate all the meta
> data and start a full heal that will replicate all the data from the
> other bricks.
>
> Looks like the easiest way to replace a brick to me :)
>
>
> Since my dev is now on 3.8 and has granular enabled I'm feeling too lazy
to roll back so will just wait till 3.8.2 is released in few days that
fixes the bugs mentioned to me and then test this few times on my dev.

Would be nice if I could get to a point where I could have one brick dead
and doing a full heal and not have every VM pause while shards heal, but I
may be asking too much when dealing with a rather heavy recovery.



> thanks,
>
> --
> Lindsay
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[Lindsay Mathieson]

On 9 August 2016 at 07:23, Joe Julian  wrote:
> Just kill (-15) the brick process. That'll close the TCP connections and the
> clients will just go right on functioning off the remaining replica. When
> you format and recreate your filesystem, it'll be missing the volume-id
> extended attributes so to start it you'll need to force it:
>
>gluster volume start $volname start force


Just to clarify I'm interpreting this correctly, to replace a brick
and preserve its mount point you can:

1. kill the brick process (glusterfsd)

2. Do your disk maintenance. Eventually you have a clean (erased) brick mount

3. Force the bricks process start. This will recreate all the meta
data and start a full heal that will replicate all the data from the
other bricks.

Looks like the easiest way to replace a brick to me :)



thanks,

-- 
Lindsay
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[David Gossage]

On Mon, Aug 8, 2016 at 5:24 PM, Joe Julian  wrote:

>
>
> On 08/08/2016 02:56 PM, David Gossage wrote:
>
> On Mon, Aug 8, 2016 at 4:37 PM, David Gossage  > wrote:
>
>> On Mon, Aug 8, 2016 at 4:23 PM, Joe Julian  wrote:
>>
>>>
>>>
>>> On 08/08/2016 01:39 PM, David Gossage wrote:
>>>
>>> So now that I have my cluster on 3.7.14 and sharded and working I am of
>>> course looking for what to break next.
>>>
>>> Currently each of 3 nodes is on a 6 disk (WD Red 1TB) raidz6 (zil on
>>> mirrored ssd), which I am thinking is more protection than I may need with
>>> a 3 way replica.  I was going to one by one change them to basically raid10
>>> letting it heal in between.
>>>
>>> Is best way to do that a systemctl stop glusterd, should I just kill the
>>> brick process to simulate a brick dying, or is their an actual brick
>>> maintenance command?
>>>
>>>
>>> Just kill (-15) the brick process. That'll close the TCP connections and
>>> the clients will just go right on functioning off the remaining replica.
>>> When you format and recreate your filesystem, it'll be missing the
>>> volume-id extended attributes so to start it you'll need to force it:
>>>
>>
> Also could I just do this from different node?
>
> getfattr  -n trusted.glusterfs.volume-id /srv/.bricks/www
>
> Then on node with new raid10 backed disks
>
> setfattr -n trusted.glusterfs.volume-id -v 'value_from_other_brick'
> /srv/.bricks/www
>
>
> Sure, but that's a lot more keystrokes and a lot more potential for human
> error.
>

This is true.


>
>
>
>
>
>>>gluster volume start $volname start force
>>>
>>
>> If I left volume started when brick process is killed and clients are
>> still (in theory) connected to volume wouldn't that just give me an error
>> that volume is already started?
>>
>>
>> Likely I would shut down the volume and do downtime for this anyway
>> though letting heals go on with VM's off.
>>
>>
>>>
>>> If /etc/glusterfs is unchanged and /var/lib/glusterd is unchanged will
>>> doing a heal full after reboot or restarting glusterd take care of
>>> everything if I recreate the expected brick path first?
>>>
>>>
>>> Once started, perform a full heal to re-replicate.
>>>
>>>
>>> Are the improvements in 3.8 for sharding significant enough I should
>>> first look at updating to 3.8.2 when released in few days?
>>>
>>>
>>> Yes.
>>>
>>>
>>>
>>> *David Gossage*
>>> *Carousel Checks Inc. | System Administrator*
>>> *Office* 708.613.2284
>>>
>>>
>>> ___
>>> Gluster-users mailing 
>>> listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>> ___ Gluster-users mailing
>>> list Gluster-users@gluster.org http://www.gluster.org/mailman
>>> /listinfo/gluster-users
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[Joe Julian]

On 08/08/2016 02:56 PM, David Gossage wrote:
On Mon, Aug 8, 2016 at 4:37 PM, David Gossage 
> wrote:


On Mon, Aug 8, 2016 at 4:23 PM, Joe Julian > wrote:



On 08/08/2016 01:39 PM, David Gossage wrote:

So now that I have my cluster on 3.7.14 and sharded and
working I am of course looking for what to break next.

Currently each of 3 nodes is on a 6 disk (WD Red 1TB) raidz6
(zil on mirrored ssd), which I am thinking is more protection
than I may need with a 3 way replica.  I was going to one by
one change them to basically raid10 letting it heal in between.

Is best way to do that a systemctl stop glusterd, should I
just kill the brick process to simulate a brick dying, or is
their an actual brick maintenance command?


Just kill (-15) the brick process. That'll close the TCP
connections and the clients will just go right on functioning
off the remaining replica. When you format and recreate your
filesystem, it'll be missing the volume-id extended attributes
so to start it you'll need to force it:


Also could I just do this from different node?

getfattr  -n trusted.glusterfs.volume-id /srv/.bricks/www

Then on node with new raid10 backed disks

setfattr -n trusted.glusterfs.volume-id -v 'value_from_other_brick' 
/srv/.bricks/www


Sure, but that's a lot more keystrokes and a lot more potential for 
human error.







   gluster volume start $volname start force


If I left volume started when brick process is killed and clients
are still (in theory) connected to volume wouldn't that just give
me an error that volume is already started?


Likely I would shut down the volume and do downtime for this
anyway though letting heals go on with VM's off.




If /etc/glusterfs is unchanged and /var/lib/glusterd is
unchanged will doing a heal full after reboot or restarting
glusterd take care of everything if I recreate the expected
brick path first?


Once started, perform a full heal to re-replicate.



Are the improvements in 3.8 for sharding significant enough I
should first look at updating to 3.8.2 when released in few days?


Yes.




*/David Gossage/*/*
*/
//*Carousel Checks Inc.| System Administrator*
*Office*708.613.2284


___
Gluster-users mailing list
Gluster-users@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users


___ Gluster-users
mailing list Gluster-users@gluster.org

http://www.gluster.org/mailman/listinfo/gluster-users
 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[Joe Julian]

On 08/08/2016 02:37 PM, David Gossage wrote:
On Mon, Aug 8, 2016 at 4:23 PM, Joe Julian > wrote:




On 08/08/2016 01:39 PM, David Gossage wrote:

So now that I have my cluster on 3.7.14 and sharded and working I
am of course looking for what to break next.

Currently each of 3 nodes is on a 6 disk (WD Red 1TB) raidz6 (zil
on mirrored ssd), which I am thinking is more protection than I
may need with a 3 way replica.  I was going to one by one change
them to basically raid10 letting it heal in between.

Is best way to do that a systemctl stop glusterd, should I just
kill the brick process to simulate a brick dying, or is their an
actual brick maintenance command?


Just kill (-15) the brick process. That'll close the TCP
connections and the clients will just go right on functioning off
the remaining replica. When you format and recreate your
filesystem, it'll be missing the volume-id extended attributes so
to start it you'll need to force it:

   gluster volume start $volname start force


If I left volume started when brick process is killed and clients are 
still (in theory) connected to volume wouldn't that just give me an 
error that volume is already started?




No, it will just force-start the missing brick.



Likely I would shut down the volume and do downtime for this anyway 
though letting heals go on with VM's off.





If /etc/glusterfs is unchanged and /var/lib/glusterd is unchanged
will doing a heal full after reboot or restarting glusterd take
care of everything if I recreate the expected brick path first?


Once started, perform a full heal to re-replicate.



Are the improvements in 3.8 for sharding significant enough I
should first look at updating to 3.8.2 when released in few days?


Yes.




*/David Gossage/*/*
*/
//*Carousel Checks Inc.| System Administrator*
*Office*708.613.2284


___
Gluster-users mailing list
Gluster-users@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users


___ Gluster-users
mailing list Gluster-users@gluster.org

http://www.gluster.org/mailman/listinfo/gluster-users
 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[David Gossage]

On Mon, Aug 8, 2016 at 4:37 PM, David Gossage 
wrote:

> On Mon, Aug 8, 2016 at 4:23 PM, Joe Julian  wrote:
>
>>
>>
>> On 08/08/2016 01:39 PM, David Gossage wrote:
>>
>> So now that I have my cluster on 3.7.14 and sharded and working I am of
>> course looking for what to break next.
>>
>> Currently each of 3 nodes is on a 6 disk (WD Red 1TB) raidz6 (zil on
>> mirrored ssd), which I am thinking is more protection than I may need with
>> a 3 way replica.  I was going to one by one change them to basically raid10
>> letting it heal in between.
>>
>> Is best way to do that a systemctl stop glusterd, should I just kill the
>> brick process to simulate a brick dying, or is their an actual brick
>> maintenance command?
>>
>>
>> Just kill (-15) the brick process. That'll close the TCP connections and
>> the clients will just go right on functioning off the remaining replica.
>> When you format and recreate your filesystem, it'll be missing the
>> volume-id extended attributes so to start it you'll need to force it:
>>
>
Also could I just do this from different node?

getfattr  -n trusted.glusterfs.volume-id /srv/.bricks/www

Then on node with new raid10 backed disks

setfattr -n trusted.glusterfs.volume-id -v 'value_from_other_brick'
/srv/.bricks/www




>>gluster volume start $volname start force
>>
>
> If I left volume started when brick process is killed and clients are
> still (in theory) connected to volume wouldn't that just give me an error
> that volume is already started?
>
>
> Likely I would shut down the volume and do downtime for this anyway though
> letting heals go on with VM's off.
>
>
>>
>> If /etc/glusterfs is unchanged and /var/lib/glusterd is unchanged will
>> doing a heal full after reboot or restarting glusterd take care of
>> everything if I recreate the expected brick path first?
>>
>>
>> Once started, perform a full heal to re-replicate.
>>
>>
>> Are the improvements in 3.8 for sharding significant enough I should
>> first look at updating to 3.8.2 when released in few days?
>>
>>
>> Yes.
>>
>>
>>
>> *David Gossage*
>> *Carousel Checks Inc. | System Administrator*
>> *Office* 708.613.2284
>>
>>
>> ___
>> Gluster-users mailing 
>> listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[David Gossage]

On Mon, Aug 8, 2016 at 4:23 PM, Joe Julian  wrote:

>
>
> On 08/08/2016 01:39 PM, David Gossage wrote:
>
> So now that I have my cluster on 3.7.14 and sharded and working I am of
> course looking for what to break next.
>
> Currently each of 3 nodes is on a 6 disk (WD Red 1TB) raidz6 (zil on
> mirrored ssd), which I am thinking is more protection than I may need with
> a 3 way replica.  I was going to one by one change them to basically raid10
> letting it heal in between.
>
> Is best way to do that a systemctl stop glusterd, should I just kill the
> brick process to simulate a brick dying, or is their an actual brick
> maintenance command?
>
>
> Just kill (-15) the brick process. That'll close the TCP connections and
> the clients will just go right on functioning off the remaining replica.
> When you format and recreate your filesystem, it'll be missing the
> volume-id extended attributes so to start it you'll need to force it:
>
>gluster volume start $volname start force
>

If I left volume started when brick process is killed and clients are still
(in theory) connected to volume wouldn't that just give me an error that
volume is already started?


Likely I would shut down the volume and do downtime for this anyway though
letting heals go on with VM's off.


>
> If /etc/glusterfs is unchanged and /var/lib/glusterd is unchanged will
> doing a heal full after reboot or restarting glusterd take care of
> everything if I recreate the expected brick path first?
>
>
> Once started, perform a full heal to re-replicate.
>
>
> Are the improvements in 3.8 for sharding significant enough I should first
> look at updating to 3.8.2 when released in few days?
>
>
> Yes.
>
>
>
> *David Gossage*
> *Carousel Checks Inc. | System Administrator*
> *Office* 708.613.2284
>
>
> ___
> Gluster-users mailing 
> listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[David Gossage]

On Mon, Aug 8, 2016 at 4:06 PM, Lindsay Mathieson <
lindsay.mathie...@gmail.com> wrote:

> On 9/08/2016 6:39 AM, David Gossage wrote:
>
>> Currently each of 3 nodes is on a 6 disk (WD Red 1TB) raidz6 (zil on
>> mirrored ssd), which I am thinking is more protection than I may need with
>> a 3 way replica.  I was going to one by one change them to basically raid10
>> letting it heal in between.
>>
>
> Wouldn't RAID10 be more protection than Raidz6? not that there is anything
> wrong with that, all my bricks are on top of a RAIDZ10 pool, as much for
> the improved IOPS as the redundancy, though it does ease the maintance of
> bricks quite a bit. Have had two drive failures where I just hotswapped the
> drive, 0 downtime.
>

RAID10 you can lose as many drives as mirror pairs set as long as they
aren't in same mirror set.  Raidz6/raid6 you can lose any 2 drives and
still stay up regardless of position so it's less crossing fingers if
multiple drives fail back to back.  However performance is better for
raid10.  So I am basically looking at slightly increasing chance of one
brick/node dropping if I had 2 drives die that happened to be in same
mirror set, in order to squeeze a little more performance out of setup.

>
> As a matter of curiosity what SSD's are you using for the ZIL and what
> size are they?
>

Samsung Pro 850's.  small lvm's partitioned to mirror for zil, other 2
larger partitions as l2arc.  Im seeing same you are though with poort hit
ratio and may just drop their use.

>
> Do you have compression enabled? lz4?
>

No, I wasn't that concerned with space usage.  WD Red's are fairly cheap
and I have 12-14 drive bays free in the 4U servers used if I want to expand
storage

>
>
>
>> Is best way to do that a systemctl stop glusterd, should I just kill the
>> brick process to simulate a brick dying, or is their an actual brick
>> maintenance command?
>>
>
> There is a gluster replace brick command:
>
>volume replace-brick{commit force}
>
> One annoyance is the new brick mount can't be the same as the old one. If
> you can I'd setup a test volume and try it out first.


That's what I used when replacing the server with a bad nic short while
ago, but wasn't certain if it would just heal whole brick since gluster
config and directories would still consider it part of the volume just with
no data in folder.

My single server dev could likely test it.  I'd guess I'd kill brick
process delete that whole brick layout directory to remove all files and
directories.  Recreate brick path.  restart gluster or server and see what
happens.  If heal kicks off or if I need to just give it a new directory
path and do a replace-brick on it.


>
>
> --
> Lindsay Mathieson
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[Joe Julian]

On 08/08/2016 01:39 PM, David Gossage wrote:
So now that I have my cluster on 3.7.14 and sharded and working I am 
of course looking for what to break next.


Currently each of 3 nodes is on a 6 disk (WD Red 1TB) raidz6 (zil on 
mirrored ssd), which I am thinking is more protection than I may need 
with a 3 way replica.  I was going to one by one change them to 
basically raid10 letting it heal in between.


Is best way to do that a systemctl stop glusterd, should I just kill 
the brick process to simulate a brick dying, or is their an actual 
brick maintenance command?


Just kill (-15) the brick process. That'll close the TCP connections and 
the clients will just go right on functioning off the remaining replica. 
When you format and recreate your filesystem, it'll be missing the 
volume-id extended attributes so to start it you'll need to force it:


   gluster volume start $volname start force



If /etc/glusterfs is unchanged and /var/lib/glusterd is unchanged will 
doing a heal full after reboot or restarting glusterd take care of 
everything if I recreate the expected brick path first?


Once started, perform a full heal to re-replicate.



Are the improvements in 3.8 for sharding significant enough I should 
first look at updating to 3.8.2 when released in few days?


Yes.




*/David Gossage/*/*
*/
//*Carousel Checks Inc.| System Administrator*
*Office*708.613.2284


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Change underlying brick on node

[Lindsay Mathieson]

On 9/08/2016 6:39 AM, David Gossage wrote:
Currently each of 3 nodes is on a 6 disk (WD Red 1TB) raidz6 (zil on 
mirrored ssd), which I am thinking is more protection than I may need 
with a 3 way replica.  I was going to one by one change them to 
basically raid10 letting it heal in between.


Wouldn't RAID10 be more protection than Raidz6? not that there is 
anything wrong with that, all my bricks are on top of a RAIDZ10 pool, as 
much for the improved IOPS as the redundancy, though it does ease the 
maintance of bricks quite a bit. Have had two drive failures where I 
just hotswapped the drive, 0 downtime.


As a matter of curiosity what SSD's are you using for the ZIL and what 
size are they?


Do you have compression enabled? lz4?




Is best way to do that a systemctl stop glusterd, should I just kill 
the brick process to simulate a brick dying, or is their an actual 
brick maintenance command?


There is a gluster replace brick command:

   volume replace-brick{commit force}

One annoyance is the new brick mount can't be the same as the old one. 
If you can I'd setup a test volume and try it out first.



--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Change underlying brick on node

[David Gossage]

So now that I have my cluster on 3.7.14 and sharded and working I am of
course looking for what to break next.

Currently each of 3 nodes is on a 6 disk (WD Red 1TB) raidz6 (zil on
mirrored ssd), which I am thinking is more protection than I may need with
a 3 way replica.  I was going to one by one change them to basically raid10
letting it heal in between.

Is best way to do that a systemctl stop glusterd, should I just kill the
brick process to simulate a brick dying, or is their an actual brick
maintenance command?

If /etc/glusterfs is unchanged and /var/lib/glusterd is unchanged will
doing a heal full after reboot or restarting glusterd take care of
everything if I recreate the expected brick path first?

Are the improvements in 3.8 for sharding significant enough I should first
look at updating to 3.8.2 when released in few days?


*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users