Re: [Gluster-devel] Puppet-Gluster+ThinP

2014-04-24 Thread James
On Sun, Apr 20, 2014 at 8:44 PM, Ric Wheeler rwhee...@redhat.com wrote:
 On 04/20/2014 05:11 PM, James wrote:

 On Sun, Apr 20, 2014 at 7:59 PM, Ric Wheeler rwhee...@redhat.com wrote:

 The amount of space you set aside is very much workload dependent (rate
 of
 change, rate of deletion, rate of notifying the storage about the freed
 space).

  From the Puppet-Gluster perspective, this will be configurable. I
 would like to set a vaguely sensible default though, which I don't
 have at the moment.


 This will require a bit of thinking as you have noticed, but let's start
 with some definitions.

 The basic use case is one file system backed by an exclusive dm-thinp target
 (no other file system writing to that dm-thinp pool or contending for
 allocation).

 The goal is to get an alert in time to intervene before things get ugly, so
 we are hoping to get a sense of rate of change in the file system and how
 long any snapshot will be retained for.

 For example, if we have a 10TB file system (presented as such to the user)
 and we write say 500GB of new data/day, daily snapshots will need that space
 for as long as we retain them.  If you write much less (5GB/day), it will
 clearly take a lot less.

 The above makes this all an effort to predict the future, but that is where
 the watermark alert kicks in to help us recover from a bad prediction.

 Maybe we use a default of setting aside 20% of raw capacity for snapshots
 and set that watermark at 90% full?  For a lot of use people, I suspect a
 fairly low rate of change and that means pretty skinny snapshots.

 We will clearly need to have a lot of effort here in helping explain this to
 users so they can make the trade off for their particular use case.



 Keep in mind with snapshots (and thinly provisioned storage, whether
 using a
 software target or thinly provisioned array) we need to issue the
 discard
 commands down the IO stack in order to let the storage target reclaim
 space.

 That typically means running the fstrim command on the local file system
 (XFS, ext4, btrfs, etc) every so often. Less typically, you can mount
 your
 local file system with -o discard to do it inband (but that comes at a
 performance penalty usually).

 Do you think it would make sense to have Puppet-Gluster add a cron job
 to do this operation?
 Exactly what command should run, and how often? (Again for having
 sensible defaults.)


 I think that we should probably run fstrim once a day or so (hopefully late
 at night or off peak)?  Adding in Lukas who lead a lot of the discard work.

I decided I'd kick off this party by writing a patch, and opening a
bug against my own product (is it cool to do that?)
Bug is: https://bugzilla.redhat.com/show_bug.cgi?id=1090757
Patch is: 
https://github.com/purpleidea/puppet-gluster/commit/1444914fe5988cc285cd572e3ed1042365d58efd
Please comment on the bug if you have any advice or recommendations
about fstrim.

Thanks!




 There is also a event mechanism to help us get notified when we hit a
 target
 configurable watermark (help, we are running short on real disk, add
 more
 or clean up!).

 Can you point me to some docs about this feature?


 My quick google search only shows my own very shallow talk slides, so let me
 dig around for something better :)



 Definitely worth following up with the LVM/device mapper people on how to
 do
 this best,

 Ric

 Thanks for the comments. From everyone I've talked to, it seems some
 of the answers are still in progress. The good news is, that I'm ahead
 of the curve for being ready for when this becomes more mainstream. I
 think Paul is in the same position too.

 James


 This is all new stuff - even not with gluster on top of it - so this will
 mean hitting a few bumps I fear.  Definitely worth putting thought into this
 now and working on the documentation,

 Ric


___
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] Puppet-Gluster+ThinP

2014-04-20 Thread James
On Sun, Apr 20, 2014 at 7:59 PM, Ric Wheeler rwhee...@redhat.com wrote:
 The amount of space you set aside is very much workload dependent (rate of
 change, rate of deletion, rate of notifying the storage about the freed
 space).

From the Puppet-Gluster perspective, this will be configurable. I
would like to set a vaguely sensible default though, which I don't
have at the moment.


 Keep in mind with snapshots (and thinly provisioned storage, whether using a
 software target or thinly provisioned array) we need to issue the discard
 commands down the IO stack in order to let the storage target reclaim space.

 That typically means running the fstrim command on the local file system
 (XFS, ext4, btrfs, etc) every so often. Less typically, you can mount your
 local file system with -o discard to do it inband (but that comes at a
 performance penalty usually).

Do you think it would make sense to have Puppet-Gluster add a cron job
to do this operation?
Exactly what command should run, and how often? (Again for having
sensible defaults.)


 There is also a event mechanism to help us get notified when we hit a target
 configurable watermark (help, we are running short on real disk, add more
 or clean up!).
Can you point me to some docs about this feature?


 Definitely worth following up with the LVM/device mapper people on how to do
 this best,

 Ric

Thanks for the comments. From everyone I've talked to, it seems some
of the answers are still in progress. The good news is, that I'm ahead
of the curve for being ready for when this becomes more mainstream. I
think Paul is in the same position too.

James

___
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] Puppet-Gluster+ThinP

2014-04-20 Thread Ric Wheeler

On 04/09/2014 03:18 PM, Paul Cuzner wrote:


I'm really interested in the thinp best practices too. gluster-deploy has had 
thinp support for a while now - and I asked the question about best practices 
a while back - but nothing came back..


Hopefully - you're timing is better than mine!

cc'ing Rajesh since the thinp is all about snapshot enablement.


The amount of space you set aside is very much workload dependent (rate of 
change, rate of deletion, rate of notifying the storage about the freed space).


Keep in mind with snapshots (and thinly provisioned storage, whether using a 
software target or thinly provisioned array) we need to issue the discard 
commands down the IO stack in order to let the storage target reclaim space.


That typically means running the fstrim command on the local file system (XFS, 
ext4, btrfs, etc) every so often. Less typically, you can mount your local file 
system with -o discard to do it inband (but that comes at a performance 
penalty usually).


There is also a event mechanism to help us get notified when we hit a target 
configurable watermark (help, we are running short on real disk, add more or 
clean up!).


Definitely worth following up with the LVM/device mapper people on how to do 
this best,


Ric





*From: *James purplei...@gmail.com
*To: *Gluster Devel gluster-devel@nongnu.org
*Sent: *Thursday, 10 April, 2014 3:13:40 AM
*Subject: *[Gluster-devel] Puppet-Gluster+ThinP

Okay,

Here's a first draft of puppet-gluster w/ thin-p. This patch includes
documentation updates too! (w00t!)

https://github.com/purpleidea/puppet-gluster/tree/feat/thinp

FYI: I'll probably rebase this branch.
FYI: Somewhat untested. Read the commit message.

Comments welcome :)

I'm most interested to hear about if everyone is pleased with the way I
run the thin-p lv command. I think this makes the most sense, but let me
know if anyone has improvements. Also I'd love to hear about what the
default values for the parameters should be, but that's a one line
patch, so no rush for me.

Cheers,
James




___
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] Puppet-Gluster+ThinP

2014-04-20 Thread Ric Wheeler

On 04/20/2014 05:11 PM, James wrote:

On Sun, Apr 20, 2014 at 7:59 PM, Ric Wheeler rwhee...@redhat.com wrote:

The amount of space you set aside is very much workload dependent (rate of
change, rate of deletion, rate of notifying the storage about the freed
space).

 From the Puppet-Gluster perspective, this will be configurable. I
would like to set a vaguely sensible default though, which I don't
have at the moment.


This will require a bit of thinking as you have noticed, but let's start with 
some definitions.


The basic use case is one file system backed by an exclusive dm-thinp target (no 
other file system writing to that dm-thinp pool or contending for allocation).


The goal is to get an alert in time to intervene before things get ugly, so we 
are hoping to get a sense of rate of change in the file system and how long any 
snapshot will be retained for.


For example, if we have a 10TB file system (presented as such to the user) and 
we write say 500GB of new data/day, daily snapshots will need that space for as 
long as we retain them.  If you write much less (5GB/day), it will clearly take 
a lot less.


The above makes this all an effort to predict the future, but that is where the 
watermark alert kicks in to help us recover from a bad prediction.


Maybe we use a default of setting aside 20% of raw capacity for snapshots and 
set that watermark at 90% full?  For a lot of use people, I suspect a fairly low 
rate of change and that means pretty skinny snapshots.


We will clearly need to have a lot of effort here in helping explain this to 
users so they can make the trade off for their particular use case.





Keep in mind with snapshots (and thinly provisioned storage, whether using a
software target or thinly provisioned array) we need to issue the discard
commands down the IO stack in order to let the storage target reclaim space.

That typically means running the fstrim command on the local file system
(XFS, ext4, btrfs, etc) every so often. Less typically, you can mount your
local file system with -o discard to do it inband (but that comes at a
performance penalty usually).

Do you think it would make sense to have Puppet-Gluster add a cron job
to do this operation?
Exactly what command should run, and how often? (Again for having
sensible defaults.)


I think that we should probably run fstrim once a day or so (hopefully late at 
night or off peak)?  Adding in Lukas who lead a lot of the discard work.





There is also a event mechanism to help us get notified when we hit a target
configurable watermark (help, we are running short on real disk, add more
or clean up!).

Can you point me to some docs about this feature?


My quick google search only shows my own very shallow talk slides, so let me dig 
around for something better :)





Definitely worth following up with the LVM/device mapper people on how to do
this best,

Ric

Thanks for the comments. From everyone I've talked to, it seems some
of the answers are still in progress. The good news is, that I'm ahead
of the curve for being ready for when this becomes more mainstream. I
think Paul is in the same position too.

James


This is all new stuff - even not with gluster on top of it - so this will mean 
hitting a few bumps I fear.  Definitely worth putting thought into this now and 
working on the documentation,


Ric


___
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] Puppet-Gluster+ThinP

2014-04-10 Thread James
On Thu, 2014-04-10 at 00:28 -0400, Paul Cuzner wrote:
 I can do that :) 
lol... okay, what are they??

 
 Perhaps this could be a topic of conversation at the hackathon on
 Sunday in SF? 



signature.asc
Description: This is a digitally signed message part
___
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] Puppet-Gluster+ThinP

2014-04-09 Thread James
On Wed, 2014-04-09 at 11:56 -0400, Keith Schincke wrote:
 Here is a quick set of reviews.
 
 What package and distro version includes the lvmthin man page?
I think it's still in git, but it's available online.

 On line 653 of Documentation.md, you refer to man -7 thin. This should 
 be man -7 lvmthin
Good catch, thanks!

 This same is done in the README.md. Never mind. I see the sym link.
Yeah. This makes the github people happy.

  From line 178 to 208, you are doing a lot of work to build the thin lvm 
 command line. Should you wrap this in a if $lvm_thinp conditional. This 
 will keep all the thin provisioning code  and a possible vgs system call 
 from occurring if $lvm_thinp is not true.
Actually it's all declarative, so it's safe without it. The conditional
is here at 211:

$lvm_lvcreate = $lvm_thinp ? {
true = ${lvm_thinp_lvcreate},
default = /sbin/lvcreate --extents 100%PVS -n ${lvm_lvname}
${lvm_vgname},
}


 
 Could you also update your lines 218 - 220 to this:
 
 $dev2 = $lvm ? {
  true = /dev/${lvm_vgname}/${lvm_lvname}} ,
  default = ${dev1},
 }
 
 The long if with a short/trivial else is almost an oh, by the way 
 kind of statement. It can be easy to overlook. The separate conditional 
 can help an other reviewer follow along more easily.
Yeah, I like this actually. Thanks for the idea.

Branch updated (rebased):
https://github.com/purpleidea/puppet-gluster/tree/feat/thinp

Commit at:
https://github.com/purpleidea/puppet-gluster/commit/d204fe5c4b80f0bc6d7850da6ccc90cb695c5873

Also I added a small warning if someone enables thinp but disables LVM.

James

 
 Keith
 
 
 On 04/09/2014 11:13 AM, James wrote:
  Okay,
 
  Here's a first draft of puppet-gluster w/ thin-p. This patch includes
  documentation updates too! (w00t!)
 
  https://github.com/purpleidea/puppet-gluster/tree/feat/thinp
 
  FYI: I'll probably rebase this branch.
  FYI: Somewhat untested. Read the commit message.
 
  Comments welcome :)
 
  I'm most interested to hear about if everyone is pleased with the way I
  run the thin-p lv command. I think this makes the most sense, but let me
  know if anyone has improvements. Also I'd love to hear about what the
  default values for the parameters should be, but that's a one line
  patch, so no rush for me.
 
  Cheers,
  James
 
 
 
 



signature.asc
Description: This is a digitally signed message part
___
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] Puppet-Gluster+ThinP

2014-04-09 Thread Keith Schincke

James,

Here is an other small one:
The number of extents used on the lvcreate should be a changable value. 
The RHS 2.1 Admin guide recommends leaving 15% to 20% of the space 
available for future snapshotting. There may also be other times where 
the admin does not wish to use all of the remaining space withing the VG


Keith


On 04/09/2014 12:39 PM, James wrote:

On Wed, 2014-04-09 at 11:56 -0400, Keith Schincke wrote:

Here is a quick set of reviews.

What package and distro version includes the lvmthin man page?

I think it's still in git, but it's available online.


On line 653 of Documentation.md, you refer to man -7 thin. This should
be man -7 lvmthin

Good catch, thanks!


This same is done in the README.md. Never mind. I see the sym link.

Yeah. This makes the github people happy.


  From line 178 to 208, you are doing a lot of work to build the thin lvm
command line. Should you wrap this in a if $lvm_thinp conditional. This
will keep all the thin provisioning code  and a possible vgs system call
from occurring if $lvm_thinp is not true.

Actually it's all declarative, so it's safe without it. The conditional
is here at 211:

$lvm_lvcreate = $lvm_thinp ? {
true = ${lvm_thinp_lvcreate},
default = /sbin/lvcreate --extents 100%PVS -n ${lvm_lvname}
${lvm_vgname},
}



Could you also update your lines 218 - 220 to this:

$dev2 = $lvm ? {
  true = /dev/${lvm_vgname}/${lvm_lvname}} ,
  default = ${dev1},
}

The long if with a short/trivial else is almost an oh, by the way
kind of statement. It can be easy to overlook. The separate conditional
can help an other reviewer follow along more easily.

Yeah, I like this actually. Thanks for the idea.

Branch updated (rebased):
https://github.com/purpleidea/puppet-gluster/tree/feat/thinp

Commit at:
https://github.com/purpleidea/puppet-gluster/commit/d204fe5c4b80f0bc6d7850da6ccc90cb695c5873

Also I added a small warning if someone enables thinp but disables LVM.

James


Keith


On 04/09/2014 11:13 AM, James wrote:

Okay,

Here's a first draft of puppet-gluster w/ thin-p. This patch includes
documentation updates too! (w00t!)

https://github.com/purpleidea/puppet-gluster/tree/feat/thinp

FYI: I'll probably rebase this branch.
FYI: Somewhat untested. Read the commit message.

Comments welcome :)

I'm most interested to hear about if everyone is pleased with the way I
run the thin-p lv command. I think this makes the most sense, but let me
know if anyone has improvements. Also I'd love to hear about what the
default values for the parameters should be, but that's a one line
patch, so no rush for me.

Cheers,
James





--
Keith Schincke
Senior Software Engineer (RHCA)
Systems Engineering


___
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] Puppet-Gluster+ThinP

2014-04-09 Thread Paul Cuzner
I'm really interested in the thinp best practices too. gluster-deploy has had 
thinp support for a while now - and I asked the question about best practices a 
while back - but nothing came back.. 

Hopefully - you're timing is better than mine! 

cc'ing Rajesh since the thinp is all about snapshot enablement. 

- Original Message -

 From: James purplei...@gmail.com
 To: Gluster Devel gluster-devel@nongnu.org
 Sent: Thursday, 10 April, 2014 3:13:40 AM
 Subject: [Gluster-devel] Puppet-Gluster+ThinP

 Okay,

 Here's a first draft of puppet-gluster w/ thin-p. This patch includes
 documentation updates too! (w00t!)

 https://github.com/purpleidea/puppet-gluster/tree/feat/thinp

 FYI: I'll probably rebase this branch.
 FYI: Somewhat untested. Read the commit message.

 Comments welcome :)

 I'm most interested to hear about if everyone is pleased with the way I
 run the thin-p lv command. I think this makes the most sense, but let me
 know if anyone has improvements. Also I'd love to hear about what the
 default values for the parameters should be, but that's a one line
 patch, so no rush for me.

 Cheers,
 James

 ___
 Gluster-devel mailing list
 Gluster-devel@nongnu.org
 https://lists.nongnu.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] Puppet-Gluster+ThinP

2014-04-09 Thread James
On Wed, 2014-04-09 at 18:18 -0400, Paul Cuzner wrote:
 I'm really interested in the thinp best practices too. gluster-deploy
 has had thinp support for a while now
Can you paste the list of commands that gluster deploy runs to setup
physical storage including thinp LVM?

  - and I asked the question about best practices a while back - but
 nothing came back.. 
 
 Hopefully - you're timing is better than mine! 
 
 cc'ing Rajesh since the thinp is all about snapshot enablement. 



signature.asc
Description: This is a digitally signed message part
___
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel


Re: [Gluster-devel] Puppet-Gluster+ThinP

2014-04-09 Thread Paul Cuzner
I can do that :) 

Perhaps this could be a topic of conversation at the hackathon on Sunday in SF? 

- Original Message -

 From: James purplei...@gmail.com
 To: Paul Cuzner pcuz...@redhat.com
 Cc: Gluster Devel gluster-devel@nongnu.org, Rajesh Joseph
 rjos...@redhat.com
 Sent: Thursday, 10 April, 2014 10:19:24 AM
 Subject: Re: [Gluster-devel] Puppet-Gluster+ThinP

 On Wed, 2014-04-09 at 18:18 -0400, Paul Cuzner wrote:
  I'm really interested in the thinp best practices too. gluster-deploy
  has had thinp support for a while now
 Can you paste the list of commands that gluster deploy runs to setup
 physical storage including thinp LVM?

  - and I asked the question about best practices a while back - but
  nothing came back..
 
  Hopefully - you're timing is better than mine!
 
  cc'ing Rajesh since the thinp is all about snapshot enablement.
___
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel