Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Chris Friesen]

On 09/18/2015 12:11 PM, John Griffith wrote:



On Fri, Sep 18, 2015 at 11:31 AM, Chris Friesen > wrote:

On 09/18/2015 11:01 AM, John Griffith wrote:



On Fri, Sep 18, 2015 at 9:06 AM, Chris Friesen

>> wrote:

 On 09/18/2015 06:57 AM, Eric Harney wrote:

 On 09/17/2015 06:06 PM, John Griffith wrote:


 Having the "global conf" settings intermixed with the
backend sections
 caused a number of issues when we first started working on
this.  That's
 part of why we require the "self.configuration" usage all
over in the
 drivers.  Each driver instantiation is it's own independent
entity.


 Yes, each driver instantiation is independent, but that would
still be
 the case if these settings inherited values set in [DEFAULT]
when they
 aren't set in the backend section.


 Agreed.  If I explicitly set something in the [DEFAULT] section,
that should
 carry through and apply to all the backends unless overridden in 
the
 backend-specific section.



Bottom line "yes", ideally in the case of drivers we would check
global/default
setting, and then override it if something was provided in the driver
specific
setting, or if the driver itself set a different default.  That seems
like the
right way to be doing it anyway.  I've looked at that a bit this
morning, the
issue is that currently we don't even pass any of those higher level 
conf
settings in to the drivers init methods anywhere.  Need to figure out 
how to
change that, then it should be a relatively simple fix.


Actually, I think it should be slightly different.  If I set a variable in
the global/default section of the config file, then that should override any
defaults in the code for a driver.

Hmm... well, on the bright side that might be easier to implement at least :). I
guess I don't agree on the meaning of "DEFAULT", to me a "DEFAULT" section
means, "these are the defaults if you don't specify something else", no?  Your
proposal seems really counter-intuitive to me.


That's what I proposed.

If you specify anything in the driver-specific portion of the config file, that 
takes priority over everything.  If you specify something in the DEFAULT portion 
of the config file, then that takes priority over the in-code defaults.  If you 
specify a default value in the driver-specific code, that takes priority over 
any defaults specified in the global/generic code.


Chris


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[John Griffith]

On Fri, Sep 18, 2015 at 12:32 PM, Chris Friesen  wrote:

> On 09/18/2015 12:11 PM, John Griffith wrote:
>
>>
>>
>> On Fri, Sep 18, 2015 at 11:31 AM, Chris Friesen <
>> chris.frie...@windriver.com
>> > wrote:
>>
>> On 09/18/2015 11:01 AM, John Griffith wrote:
>>
>>
>>
>> On Fri, Sep 18, 2015 at 9:06 AM, Chris Friesen
>> 
>> >
>> >> wrote:
>>
>>  On 09/18/2015 06:57 AM, Eric Harney wrote:
>>
>>  On 09/17/2015 06:06 PM, John Griffith wrote:
>>
>>
>>  Having the "global conf" settings intermixed with the
>> backend sections
>>  caused a number of issues when we first started
>> working on
>> this.  That's
>>  part of why we require the "self.configuration"
>> usage all
>> over in the
>>  drivers.  Each driver instantiation is it's own
>> independent
>> entity.
>>
>>
>>  Yes, each driver instantiation is independent, but that
>> would
>> still be
>>  the case if these settings inherited values set in
>> [DEFAULT]
>> when they
>>  aren't set in the backend section.
>>
>>
>>  Agreed.  If I explicitly set something in the [DEFAULT]
>> section,
>> that should
>>  carry through and apply to all the backends unless
>> overridden in the
>>  backend-specific section.
>>
>>
>>
>> Bottom line "yes", ideally in the case of drivers we would check
>> global/default
>> setting, and then override it if something was provided in the
>> driver
>> specific
>> setting, or if the driver itself set a different default.  That
>> seems
>> like the
>> right way to be doing it anyway.  I've looked at that a bit this
>> morning, the
>> issue is that currently we don't even pass any of those higher
>> level conf
>> settings in to the drivers init methods anywhere.  Need to figure
>> out how to
>> change that, then it should be a relatively simple fix.
>>
>>
>> Actually, I think it should be slightly different.  If I set a
>> variable in
>> the global/default section of the config file, then that should
>> override any
>> defaults in the code for a driver.
>>
>> Hmm... well, on the bright side that might be easier to implement at
>> least :). I
>> guess I don't agree on the meaning of "DEFAULT", to me a "DEFAULT" section
>> means, "these are the defaults if you don't specify something else", no?
>> Your
>> proposal seems really counter-intuitive to me.
>>
>
> That's what I proposed.
>
> If you specify anything in the driver-specific portion of the config file,
> that takes priority over everything.  If you specify something in the
> DEFAULT portion of the config file, then that takes priority over the
> in-code defaults.  If you specify a default value in the driver-specific
> code, that takes priority over any defaults specified in the global/generic
> code.
>
>
> Chris
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
​Ahh... ok, yeah; sorry, I interpreted that differently when I read it.
Thanks for clarifying.

So the good news is we most definitely agree there, and that I've got
something that gets us at least part way there.  Now, to just figure out
how to determine if the value in the driver section was set explicitly or
is just parsing out the default value in the opt declaration.

Thanks!!
John​
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[John Griffith]

On Fri, Sep 18, 2015 at 12:52 PM, Eric Harney  wrote:

> On 09/18/2015 01:01 PM, John Griffith wrote:
> > On Fri, Sep 18, 2015 at 9:06 AM, Chris Friesen <
> chris.frie...@windriver.com>
> > wrote:
> >
> >> On 09/18/2015 06:57 AM, Eric Harney wrote:
> >>
> >>> On 09/17/2015 06:06 PM, John Griffith wrote:
> >>>
> >>
> >> Having the "global conf" settings intermixed with the backend sections
>  caused a number of issues when we first started working on this.
> That's
>  part of why we require the "self.configuration" usage all over in the
>  drivers.  Each driver instantiation is it's own independent entity.
> 
> 
> >>> Yes, each driver instantiation is independent, but that would still be
> >>> the case if these settings inherited values set in [DEFAULT] when they
> >>> aren't set in the backend section.
> >>>
> >>
> >> Agreed.  If I explicitly set something in the [DEFAULT] section, that
> >> should carry through and apply to all the backends unless overridden in
> the
> >> backend-specific section.
> >>
> >> Chris
> >>
> >>
> > Meh I don't know about the "have to modify the code", the config file
> works
> > you just need to add that line to your driver section and configure the
> > backend correctly.
> >
>
> My point is that there doesn't seem to be a justification for "you just
> need to add that line to your driver section", which seems to counter
> what most people's expectation would be.
>
​There certainly is, I don't want to force the same options against all
backends.  Perfect example is the issues with some distros in the past that
DID use global settings and stomp over any driver; which in turn broke
those that weren't compatible with that conf setting even though in the
driver section they overrode it.​


>
> People can and do fail to do that, because they assume that [DEFAULT]
> settings are treated as defaults.
>

​Bad assumption, we should probably document this until we fix it (making a
very large assumption that we'll ever agree on how to fix it).​

>
> To help people who make that assumption, yes, you have to modify the
> code, because the code supplies a default value that you cannot supply
> in the same way via config files.
>

​Or you could just fill out the config file properly:
[lvm-1]
iscsi_helper = lioadm

I didn't have to modify any code.
​


>
> > Regardless, I see your point (but I still certainly don't agree that it's
> > "blatantly wrong").
> >
>
> You can substitute "very confusing" for "blatantly wrong" but I think
> those are about the same thing when talking about usability issues with
> how to configure a service.
>

​Fair enough.  Call it whatever you like.​


>
> Look at options like:
>  - strict_ssh_host_key_policy
>  - sio_verify_server_certificate
>  - driver_ssl_cert_verify


> All of these default to False, and if turned on, enable protections
> against MITM attacks.  All of them also fail to turn on for the relevant
> drivers if set in [DEFAULT].  These should, if set in DEFAULT when using
> multi-backend, issue a warning so the admin knows that they are not
> getting the intended security guarantees.  Instead, nothing happens and
> Cinder and the storage works.  Confusion is dangerous.
>

​Yeah, so is crappy documentation lack of understanding.​


>
> > Bottom line "yes", ideally in the case of drivers we would check
> > global/default setting, and then override it if something was provided in
> > the driver specific setting, or if the driver itself set a different
> > default.  That seems like the right way to be doing it anyway.  I've
> looked
> > at that a bit this morning, the issue is that currently we don't even
> pass
> > any of those higher level conf settings in to the drivers init methods
> > anywhere.  Need to figure out how to change that, then it should be a
> > relatively simple fix.
> >
>
> What I was getting at earlier though, is that I'm not really sure there
> is a simple fix.  It may be simple to change the behavior to more
> predictable behavior, but doing that in a way that doesn't introduce
> upgrade problems for deployments relying on the current defaults seems
> difficult to me.
>
​Agreed, but honestly I'd like to at least try.  Especially when people use
terms like "blatantly wrong" and "dangerous", kinda prompts one to think ​

​that perhaps it should be looked at.  If nothing else, we shouldn't have
driver settings in the DEFAULT section, we should just create a driver
section, but we still need to figure out how to deal with things outside of
the "general" section vs the backend stanza.

Also, I'd argue that the behavior your arguing for is MORE dangerous and
troublesome.  The LIO being in the global CONF was a perfect example where
it broke third party devices on a specific distro because it assumed that
EVERYTHING on the system was using the lio methods and in that case you
really couldn't do anything but modify code.

I've pinged you a number of times on IRC, maybe we can chat there a 

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Eric Harney]

On 09/18/2015 03:33 PM, John Griffith wrote:
> On Fri, Sep 18, 2015 at 12:52 PM, Eric Harney  wrote:
> 
>> On 09/18/2015 01:01 PM, John Griffith wrote:
>>> On Fri, Sep 18, 2015 at 9:06 AM, Chris Friesen <
>> chris.frie...@windriver.com>
>>> wrote:
>>>
 On 09/18/2015 06:57 AM, Eric Harney wrote:

> On 09/17/2015 06:06 PM, John Griffith wrote:
>

 Having the "global conf" settings intermixed with the backend sections
>> caused a number of issues when we first started working on this.
>> That's
>> part of why we require the "self.configuration" usage all over in the
>> drivers.  Each driver instantiation is it's own independent entity.
>>
>>
> Yes, each driver instantiation is independent, but that would still be
> the case if these settings inherited values set in [DEFAULT] when they
> aren't set in the backend section.
>

 Agreed.  If I explicitly set something in the [DEFAULT] section, that
 should carry through and apply to all the backends unless overridden in
>> the
 backend-specific section.

 Chris


>>> Meh I don't know about the "have to modify the code", the config file
>> works
>>> you just need to add that line to your driver section and configure the
>>> backend correctly.
>>>
>>
>> My point is that there doesn't seem to be a justification for "you just
>> need to add that line to your driver section", which seems to counter
>> what most people's expectation would be.
>>
> ​There certainly is, I don't want to force the same options against all
> backends.  Perfect example is the issues with some distros in the past that
> DID use global settings and stomp over any driver; which in turn broke
> those that weren't compatible with that conf setting even though in the
> driver section they overrode it.​
> 
> 
>>
>> People can and do fail to do that, because they assume that [DEFAULT]
>> settings are treated as defaults.
>>
> 
> ​Bad assumption, we should probably document this until we fix it (making a
> very large assumption that we'll ever agree on how to fix it).​
> 
>>
>> To help people who make that assumption, yes, you have to modify the
>> code, because the code supplies a default value that you cannot supply
>> in the same way via config files.
>>
> 
> ​Or you could just fill out the config file properly:
> [lvm-1]
> iscsi_helper = lioadm
> 
> I didn't have to modify any code.
> ​
> 
> 

In the use case I was describing, I'm shipping a package, as a
distribution, with a default configuration file. The deployer (not me)
is the only one that knows about config sections that they want for
multi-backend. I don't think it's fair to require them to fill out
things like iscsi_helper, because there is only one correct value for
iscsi_helper on the platform I support, and defaulting to a different
one is not useful.

The fact that we don't inherit [DEFAULT] settings means that it is not
possible for me to ship a package with the correct defaults without
changing the hard-coded default value, in the code, to customize it for
my platform. I want to set iscsi_helper = lioadm in a configuration file
and have that be the default for any enabled_backend.


>>
>>> Regardless, I see your point (but I still certainly don't agree that it's
>>> "blatantly wrong").
>>>
>>
>> You can substitute "very confusing" for "blatantly wrong" but I think
>> those are about the same thing when talking about usability issues with
>> how to configure a service.
>>
> 
> ​Fair enough.  Call it whatever you like.​
> 
> 
>>
>> Look at options like:
>>  - strict_ssh_host_key_policy
>>  - sio_verify_server_certificate
>>  - driver_ssl_cert_verify
> 
> 
>> All of these default to False, and if turned on, enable protections
>> against MITM attacks.  All of them also fail to turn on for the relevant
>> drivers if set in [DEFAULT].  These should, if set in DEFAULT when using
>> multi-backend, issue a warning so the admin knows that they are not
>> getting the intended security guarantees.  Instead, nothing happens and
>> Cinder and the storage works.  Confusion is dangerous.
>>
> 
> ​Yeah, so is crappy documentation lack of understanding.​
> 
> 

I can't make my customers read documentation and test them for
understanding.  I can make software that's more robust and less prone to
misuse.  Warning people with "hey, you're using multi-backend and have
set this security-related option in a section where it will never have
an effect in your deployment" is one way to do this that we could do today.

>>
>>> Bottom line "yes", ideally in the case of drivers we would check
>>> global/default setting, and then override it if something was provided in
>>> the driver specific setting, or if the driver itself set a different
>>> default.  That seems like the right way to be doing it anyway.  I've
>> looked
>>> at that a bit this morning, the issue is that currently we don't even
>> pass
>>> any of those higher level conf 

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Eric Harney]

On 09/18/2015 01:01 PM, John Griffith wrote:
> On Fri, Sep 18, 2015 at 9:06 AM, Chris Friesen 
> wrote:
> 
>> On 09/18/2015 06:57 AM, Eric Harney wrote:
>>
>>> On 09/17/2015 06:06 PM, John Griffith wrote:
>>>
>>
>> Having the "global conf" settings intermixed with the backend sections
 caused a number of issues when we first started working on this.  That's
 part of why we require the "self.configuration" usage all over in the
 drivers.  Each driver instantiation is it's own independent entity.


>>> Yes, each driver instantiation is independent, but that would still be
>>> the case if these settings inherited values set in [DEFAULT] when they
>>> aren't set in the backend section.
>>>
>>
>> Agreed.  If I explicitly set something in the [DEFAULT] section, that
>> should carry through and apply to all the backends unless overridden in the
>> backend-specific section.
>>
>> Chris
>>
>>
> Meh I don't know about the "have to modify the code", the config file works
> you just need to add that line to your driver section and configure the
> backend correctly.
> 

My point is that there doesn't seem to be a justification for "you just
need to add that line to your driver section", which seems to counter
what most people's expectation would be.

People can and do fail to do that, because they assume that [DEFAULT]
settings are treated as defaults.

To help people who make that assumption, yes, you have to modify the
code, because the code supplies a default value that you cannot supply
in the same way via config files.

> Regardless, I see your point (but I still certainly don't agree that it's
> "blatantly wrong").
> 

You can substitute "very confusing" for "blatantly wrong" but I think
those are about the same thing when talking about usability issues with
how to configure a service.

Look at options like:
 - strict_ssh_host_key_policy
 - sio_verify_server_certificate
 - driver_ssl_cert_verify

All of these default to False, and if turned on, enable protections
against MITM attacks.  All of them also fail to turn on for the relevant
drivers if set in [DEFAULT].  These should, if set in DEFAULT when using
multi-backend, issue a warning so the admin knows that they are not
getting the intended security guarantees.  Instead, nothing happens and
Cinder and the storage works.  Confusion is dangerous.

> Bottom line "yes", ideally in the case of drivers we would check
> global/default setting, and then override it if something was provided in
> the driver specific setting, or if the driver itself set a different
> default.  That seems like the right way to be doing it anyway.  I've looked
> at that a bit this morning, the issue is that currently we don't even pass
> any of those higher level conf settings in to the drivers init methods
> anywhere.  Need to figure out how to change that, then it should be a
> relatively simple fix.
> 

What I was getting at earlier though, is that I'm not really sure there
is a simple fix.  It may be simple to change the behavior to more
predictable behavior, but doing that in a way that doesn't introduce
upgrade problems for deployments relying on the current defaults seems
difficult to me.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[John Griffith]

On Fri, Sep 18, 2015 at 11:31 AM, Chris Friesen  wrote:

> On 09/18/2015 11:01 AM, John Griffith wrote:
>
>>
>>
>> On Fri, Sep 18, 2015 at 9:06 AM, Chris Friesen <
>> chris.frie...@windriver.com
>> > wrote:
>>
>> On 09/18/2015 06:57 AM, Eric Harney wrote:
>>
>> On 09/17/2015 06:06 PM, John Griffith wrote:
>>
>>
>> Having the "global conf" settings intermixed with the backend
>> sections
>> caused a number of issues when we first started working on
>> this.  That's
>> part of why we require the "self.configuration" usage all
>> over in the
>> drivers.  Each driver instantiation is it's own independent
>> entity.
>>
>>
>> Yes, each driver instantiation is independent, but that would
>> still be
>> the case if these settings inherited values set in [DEFAULT] when
>> they
>> aren't set in the backend section.
>>
>>
>> Agreed.  If I explicitly set something in the [DEFAULT] section, that
>> should
>> carry through and apply to all the backends unless overridden in the
>> backend-specific section.
>>
>
>
> Bottom line "yes", ideally in the case of drivers we would check
>> global/default
>> setting, and then override it if something was provided in the driver
>> specific
>> setting, or if the driver itself set a different default.  That seems
>> like the
>> right way to be doing it anyway.  I've looked at that a bit this morning,
>> the
>> issue is that currently we don't even pass any of those higher level conf
>> settings in to the drivers init methods anywhere.  Need to figure out how
>> to
>> change that, then it should be a relatively simple fix.
>>
>
> Actually, I think it should be slightly different.  If I set a variable in
> the global/default section of the config file, then that should override
> any defaults in the code for a driver.
>

Hmm... well, on the bright side that might be easier to implement at least
:). I guess I don't agree on the meaning of "DEFAULT", to me a "DEFAULT"
section means, "these are the defaults if you don't specify something
else", no?  Your proposal seems really counter-intuitive to me.

​Anyway, I've come up with a way to read the DEFAULT section of CONF on
init in the driver so that's good.  The trick now though is when there's a
difference in value between the driver section and the default section;
knowing what was set explicitly and what wasn't.  In other words I don't
have any way of knowing for sure if the setting came from reading in the
defaults in the declaration options or if it was explicitly set in the
config file.​

​Still working on it, but curious if anybody might know how to get around
this little sticking point.

Thanks,
John​




> So the order of descending precedence would go:
>
> 1) setting specified in driver-specific section of config file
> 2) setting specified in global/default section of config file
> 3) setting specified in driver-specific code
> 4) setting specified in global/default code (not sure if this exists)
>
>
>
> Chris
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[John Griffith]

On Fri, Sep 18, 2015 at 2:46 PM, Eric Harney  wrote:

> On 09/18/2015 03:33 PM, John Griffith wrote:
> > On Fri, Sep 18, 2015 at 12:52 PM, Eric Harney 
> wrote:
> >
> >> On 09/18/2015 01:01 PM, John Griffith wrote:
> >>> On Fri, Sep 18, 2015 at 9:06 AM, Chris Friesen <
> >> chris.frie...@windriver.com>
> >>> wrote:
> >>>
>  On 09/18/2015 06:57 AM, Eric Harney wrote:
> 
> > On 09/17/2015 06:06 PM, John Griffith wrote:
> >
> 
>  Having the "global conf" settings intermixed with the backend sections
> >> caused a number of issues when we first started working on this.
> >> That's
> >> part of why we require the "self.configuration" usage all over in
> the
> >> drivers.  Each driver instantiation is it's own independent entity.
> >>
> >>
> > Yes, each driver instantiation is independent, but that would still
> be
> > the case if these settings inherited values set in [DEFAULT] when
> they
> > aren't set in the backend section.
> >
> 
>  Agreed.  If I explicitly set something in the [DEFAULT] section, that
>  should carry through and apply to all the backends unless overridden
> in
> >> the
>  backend-specific section.
> 
>  Chris
> 
> 
> >>> Meh I don't know about the "have to modify the code", the config file
> >> works
> >>> you just need to add that line to your driver section and configure the
> >>> backend correctly.
> >>>
> >>
> >> My point is that there doesn't seem to be a justification for "you just
> >> need to add that line to your driver section", which seems to counter
> >> what most people's expectation would be.
> >>
> > ​There certainly is, I don't want to force the same options against all
> > backends.  Perfect example is the issues with some distros in the past
> that
> > DID use global settings and stomp over any driver; which in turn broke
> > those that weren't compatible with that conf setting even though in the
> > driver section they overrode it.​
> >
> >
> >>
> >> People can and do fail to do that, because they assume that [DEFAULT]
> >> settings are treated as defaults.
> >>
> >
> > ​Bad assumption, we should probably document this until we fix it
> (making a
> > very large assumption that we'll ever agree on how to fix it).​
> >
> >>
> >> To help people who make that assumption, yes, you have to modify the
> >> code, because the code supplies a default value that you cannot supply
> >> in the same way via config files.
> >>
> >
> > ​Or you could just fill out the config file properly:
> > [lvm-1]
> > iscsi_helper = lioadm
> >
> > I didn't have to modify any code.
> > ​
> >
> >
>
> In the use case I was describing, I'm shipping a package, as a
> distribution, with a default configuration file. The deployer (not me)
> is the only one that knows about config sections that they want for
> multi-backend. I don't think it's fair to require them to fill out
> things like iscsi_helper, because there is only one correct value for
> iscsi_helper on the platform I support, and defaulting to a different
> one is not useful.
>
​Ahh, ok; so back to one of the problems with OpenStack IMO, too many
options/choices.  Regardless though yes I can see where you're coming from
now.  In your case there is only one supported/correct option here so that
creates a problem.
​


>
> The fact that we don't inherit [DEFAULT] settings means that it is not
> possible for me to ship a package with the correct defaults without
> changing the hard-coded default value, in the code, to customize it for
> my platform. I want to set iscsi_helper = lioadm in a configuration file
> and have that be the default for any enabled_backend.
>
​Yes, now I see the case you're referring to, thanks!  This is why I tried
to grab you on IRC to make sure I actually followed what your particular
case was.​



>
> >>
> >>> Regardless, I see your point (but I still certainly don't agree that
> it's
> >>> "blatantly wrong").
> >>>
> >>
> >> You can substitute "very confusing" for "blatantly wrong" but I think
> >> those are about the same thing when talking about usability issues with
> >> how to configure a service.
> >>
> >
> > ​Fair enough.  Call it whatever you like.​
> >
> >
> >>
> >> Look at options like:
> >>  - strict_ssh_host_key_policy
> >>  - sio_verify_server_certificate
> >>  - driver_ssl_cert_verify
> >
> >
> >> All of these default to False, and if turned on, enable protections
> >> against MITM attacks.  All of them also fail to turn on for the relevant
> >> drivers if set in [DEFAULT].  These should, if set in DEFAULT when using
> >> multi-backend, issue a warning so the admin knows that they are not
> >> getting the intended security guarantees.  Instead, nothing happens and
> >> Cinder and the storage works.  Confusion is dangerous.
> >>
> >
> > ​Yeah, so is crappy documentation lack of understanding.​
> >
> >
>
> I can't make my customers read documentation and test them for
> 

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Chris Friesen]

On 09/18/2015 06:57 AM, Eric Harney wrote:

On 09/17/2015 06:06 PM, John Griffith wrote:



Having the "global conf" settings intermixed with the backend sections
caused a number of issues when we first started working on this.  That's
part of why we require the "self.configuration" usage all over in the
drivers.  Each driver instantiation is it's own independent entity.



Yes, each driver instantiation is independent, but that would still be
the case if these settings inherited values set in [DEFAULT] when they
aren't set in the backend section.


Agreed.  If I explicitly set something in the [DEFAULT] section, that should 
carry through and apply to all the backends unless overridden in the 
backend-specific section.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Eric Harney]

On 09/17/2015 06:06 PM, John Griffith wrote:
> On Thu, Sep 17, 2015 at 11:31 AM, Eric Harney  wrote:
> 
>> On 09/17/2015 05:00 AM, Duncan Thomas wrote:
>>> On 16 September 2015 at 23:43, Eric Harney  wrote:
>>>
 Currently, at least some options set in [DEFAULT] don't apply to
 per-driver sections, and require you to set them in the driver section
 as well.

>>>
>>> This is extremely confusing behaviour. Do you have any examples? I'm not
>>> sure if we can fix it without breaking people's existing configs but I
>>> think it is worth trying. I'll add it to the list of things to talk about
>>> briefly in Tokyo.
>>>
>>
>> The most recent place this bit me was with iscsi_helper.
>>
>> If cinder.conf has:
>>
>> [DEFAULT]
>> iscsi_helper = lioadm
>> enabled_backends = lvm1
>>
>> [lvm1]
>> volume_driver = ...LVMISCSIDriver
>> # no iscsi_helper setting
>>
>>
>> You end up with c-vol showing "iscsi_helper = lioadm", and
>> "lvm1.iscsi_helper = tgtadm", which is the default in the code, and not
>> the default in the configuration file.
>>
>> I agree that this is confusing, I think it's also blatantly wrong.  I'm
>> not sure how to fix it, but I think it's some combination of your
>> suggestions above and possibly having to introduce new option names.
>>
> ​
> I'm not sure why that's "blatantly wrong', this is a side effect of having
> multiple backends enabled, it's by design really.  Any option that is
> defined in driver.py needs to be set in the actual enabled-backend stanza
> IIRC.  This includes iscsi_helper, volume_clear etc.
> 

I think it's wrong because it's not predictable for someone configuring
Cinder.  I understand that this is a side effect of multi-backend, but
I'm not sure what the reasoning is if it's intentional design.  I think
most people would expect a setting set in a [DEFAULT] section to be
treated as a default rather than being ignored.

This is particularly odd in the case of "iscsi_helper", where I want to
ship packages configured to use LIO since tgt doesn't exist on the
platform, and is never the right value for my packages.

This isn't possible without patching the code directly, which seems like
a shortfall in our configuration system.

> Having the "global conf" settings intermixed with the backend sections
> caused a number of issues when we first started working on this.  That's
> part of why we require the "self.configuration" usage all over in the
> drivers.  Each driver instantiation is it's own independent entity.
> 

Yes, each driver instantiation is independent, but that would still be
the case if these settings inherited values set in [DEFAULT] when they
aren't set in the backend section.

> I haven't looked at this for a long time, but if something has changed or
> I'm missing something my apologies.  We can certainly consider changing it,
> but because of the way we do multi-backend I'm not exactly sure how you
> would do this, or honestly why you would want to.
> 
> John
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[John Griffith]

On Fri, Sep 18, 2015 at 9:06 AM, Chris Friesen 
wrote:

> On 09/18/2015 06:57 AM, Eric Harney wrote:
>
>> On 09/17/2015 06:06 PM, John Griffith wrote:
>>
>
> Having the "global conf" settings intermixed with the backend sections
>>> caused a number of issues when we first started working on this.  That's
>>> part of why we require the "self.configuration" usage all over in the
>>> drivers.  Each driver instantiation is it's own independent entity.
>>>
>>>
>> Yes, each driver instantiation is independent, but that would still be
>> the case if these settings inherited values set in [DEFAULT] when they
>> aren't set in the backend section.
>>
>
> Agreed.  If I explicitly set something in the [DEFAULT] section, that
> should carry through and apply to all the backends unless overridden in the
> backend-specific section.
>
> Chris
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
Meh I don't know about the "have to modify the code", the config file works
you just need to add that line to your driver section and configure the
backend correctly.

Regardless, I see your point (but I still certainly don't agree that it's
"blatantly wrong").

Bottom line "yes", ideally in the case of drivers we would check
global/default setting, and then override it if something was provided in
the driver specific setting, or if the driver itself set a different
default.  That seems like the right way to be doing it anyway.  I've looked
at that a bit this morning, the issue is that currently we don't even pass
any of those higher level conf settings in to the drivers init methods
anywhere.  Need to figure out how to change that, then it should be a
relatively simple fix.

Thanks,
John
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Chris Friesen]

On 09/18/2015 11:01 AM, John Griffith wrote:



On Fri, Sep 18, 2015 at 9:06 AM, Chris Friesen > wrote:

On 09/18/2015 06:57 AM, Eric Harney wrote:

On 09/17/2015 06:06 PM, John Griffith wrote:


Having the "global conf" settings intermixed with the backend 
sections
caused a number of issues when we first started working on this.  
That's
part of why we require the "self.configuration" usage all over in 
the
drivers.  Each driver instantiation is it's own independent entity.


Yes, each driver instantiation is independent, but that would still be
the case if these settings inherited values set in [DEFAULT] when they
aren't set in the backend section.


Agreed.  If I explicitly set something in the [DEFAULT] section, that should
carry through and apply to all the backends unless overridden in the
backend-specific section.




Bottom line "yes", ideally in the case of drivers we would check global/default
setting, and then override it if something was provided in the driver specific
setting, or if the driver itself set a different default.  That seems like the
right way to be doing it anyway.  I've looked at that a bit this morning, the
issue is that currently we don't even pass any of those higher level conf
settings in to the drivers init methods anywhere.  Need to figure out how to
change that, then it should be a relatively simple fix.


Actually, I think it should be slightly different.  If I set a variable in the 
global/default section of the config file, then that should override any 
defaults in the code for a driver.


So the order of descending precedence would go:

1) setting specified in driver-specific section of config file
2) setting specified in global/default section of config file
3) setting specified in driver-specific code
4) setting specified in global/default code (not sure if this exists)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Duncan Thomas]

On 16 September 2015 at 23:43, Eric Harney  wrote:

> Currently, at least some options set in [DEFAULT] don't apply to
> per-driver sections, and require you to set them in the driver section
> as well.
>

This is extremely confusing behaviour. Do you have any examples? I'm not
sure if we can fix it without breaking people's existing configs but I
think it is worth trying. I'll add it to the list of things to talk about
briefly in Tokyo.

-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Eric Harney]

On 09/17/2015 05:00 AM, Duncan Thomas wrote:
> On 16 September 2015 at 23:43, Eric Harney  wrote:
> 
>> Currently, at least some options set in [DEFAULT] don't apply to
>> per-driver sections, and require you to set them in the driver section
>> as well.
>>
> 
> This is extremely confusing behaviour. Do you have any examples? I'm not
> sure if we can fix it without breaking people's existing configs but I
> think it is worth trying. I'll add it to the list of things to talk about
> briefly in Tokyo.
> 

The most recent place this bit me was with iscsi_helper.

If cinder.conf has:

[DEFAULT]
iscsi_helper = lioadm
enabled_backends = lvm1

[lvm1]
volume_driver = ...LVMISCSIDriver
# no iscsi_helper setting


You end up with c-vol showing "iscsi_helper = lioadm", and
"lvm1.iscsi_helper = tgtadm", which is the default in the code, and not
the default in the configuration file.

I agree that this is confusing, I think it's also blatantly wrong.  I'm
not sure how to fix it, but I think it's some combination of your
suggestions above and possibly having to introduce new option names.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[John Griffith]

On Thu, Sep 17, 2015 at 11:31 AM, Eric Harney  wrote:

> On 09/17/2015 05:00 AM, Duncan Thomas wrote:
> > On 16 September 2015 at 23:43, Eric Harney  wrote:
> >
> >> Currently, at least some options set in [DEFAULT] don't apply to
> >> per-driver sections, and require you to set them in the driver section
> >> as well.
> >>
> >
> > This is extremely confusing behaviour. Do you have any examples? I'm not
> > sure if we can fix it without breaking people's existing configs but I
> > think it is worth trying. I'll add it to the list of things to talk about
> > briefly in Tokyo.
> >
>
> The most recent place this bit me was with iscsi_helper.
>
> If cinder.conf has:
>
> [DEFAULT]
> iscsi_helper = lioadm
> enabled_backends = lvm1
>
> [lvm1]
> volume_driver = ...LVMISCSIDriver
> # no iscsi_helper setting
>
>
> You end up with c-vol showing "iscsi_helper = lioadm", and
> "lvm1.iscsi_helper = tgtadm", which is the default in the code, and not
> the default in the configuration file.
>
> I agree that this is confusing, I think it's also blatantly wrong.  I'm
> not sure how to fix it, but I think it's some combination of your
> suggestions above and possibly having to introduce new option names.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
​
I'm not sure why that's "blatantly wrong', this is a side effect of having
multiple backends enabled, it's by design really.  Any option that is
defined in driver.py needs to be set in the actual enabled-backend stanza
IIRC.  This includes iscsi_helper, volume_clear etc.

Having the "global conf" settings intermixed with the backend sections
caused a number of issues when we first started working on this.  That's
part of why we require the "self.configuration" usage all over in the
drivers.  Each driver instantiation is it's own independent entity.

I haven't looked at this for a long time, but if something has changed or
I'm missing something my apologies.  We can certainly consider changing it,
but because of the way we do multi-backend I'm not exactly sure how you
would do this, or honestly why you would want to.

John
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Duncan Thomas]

On 16 Sep 2015 20:42, "yang, xing"  wrote:

> On 9/16/15, 1:20 PM, "Eric Harney"  wrote:

> >This sounds like a good idea, I'm just not sure how to structure it yet
> >without creating a very confusing set of config options.
>
> I’m thinking we could have a prefix with vendor name for this and it also
> requires documentation by driver maintainers if they are using a different
> config option.  I proposed a topic to discuss about this at the summit.

We already have per-backend config values in cinder.conf. I'm not sure how
the config code will need to be  structured to achieve it, but ideally I'd
like a single config option that can be:

(i) set in the default section if desired
(in) overridden in the per driver section, and (iii) have a default set in
each driver.

I don't think oslo.config lets us do (I'll) yet though.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Eric Harney]

On 09/16/2015 04:25 PM, Duncan Thomas wrote:
> On 16 Sep 2015 20:42, "yang, xing"  wrote:
> 
>> On 9/16/15, 1:20 PM, "Eric Harney"  wrote:
> 
>>> This sounds like a good idea, I'm just not sure how to structure it yet
>>> without creating a very confusing set of config options.
>>
>> I’m thinking we could have a prefix with vendor name for this and it also
>> requires documentation by driver maintainers if they are using a different
>> config option.  I proposed a topic to discuss about this at the summit.
> 
> We already have per-backend config values in cinder.conf. I'm not sure how
> the config code will need to be  structured to achieve it, but ideally I'd
> like a single config option that can be:
> 
> (i) set in the default section if desired
> (in) overridden in the per driver section, and (iii) have a default set in
> each driver.
> 
> I don't think oslo.config lets us do (I'll) yet though.
> 

I think there may be other issues to sort through to do that.
Currently, at least some options set in [DEFAULT] don't apply to
per-driver sections, and require you to set them in the driver section
as well.

If we keep that behavior (which I think is broken, personally), then
trying to do option (iii) may be pretty confusing, because the deployer
won't know which of the global vs. driver defaults are actually going to
be applied.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Eric Harney]

On 09/15/2015 04:56 PM, yang, xing wrote:
> Hi Eric,
> 
> Regarding the default max_over_subscription_ratio, I initially set the
> default to 1 while working on oversubscription, and changed it to 2 after
> getting review comments.  After it was merged, I got feedback that 2 is
> too small and 20 is more appropriated, so I changed it to 20.  So it looks
> like we can¹t find a default value that makes everyone happy.
> 

I'm curious about how this is used in real-world deployments.  Are we
making the assumption that the admin has some external monitoring
configured to send alarms if the storage is nearing capacity?

> If we can decide what is the best default value for LVM, we can change the
> default max_over_subscription_ratio, but we should also allow other
> drivers to specify a different config option if a different default value
> is more appropriate for them.

This sounds like a good idea, I'm just not sure how to structure it yet
without creating a very confusing set of config options.


> On 9/15/15, 1:38 PM, "Eric Harney"  wrote:
> 
>> On 09/15/2015 01:00 PM, Chris Friesen wrote:
>>> I'm currently trying to work around an issue where activating LVM
>>> snapshots created through cinder takes potentially a long time.
>>> (Linearly related to the amount of data that differs between the
>>> original volume and the snapshot.)  On one system I tested it took about
>>> one minute per 25GB of data, so the worst-case boot delay can become
>>> significant.
>>>
>>> According to Zdenek Kabelac on the LVM mailing list, LVM snapshots were
>>> not intended to be kept around indefinitely, they were supposed to be
>>> used only until the backup was taken and then deleted.  He recommends
>>> using thin provisioning for long-lived snapshots due to differences in
>>> how the metadata is maintained.  (He also says he's heard reports of
>>> volume activation taking half an hour, which is clearly crazy when
>>> instances are waiting to access their volumes.)
>>>
>>> Given the above, is there any reason why we couldn't make thin
>>> provisioning the default?
>>>
>>
>>
>> My intention is to move toward thin-provisioned LVM as the default -- it
>> is definitely better suited to our use of LVM.  Previously this was less
>> easy, since some older Ubuntu platforms didn't support it, but in
>> Liberty we added the ability to specify lvm_type = "auto" [1] to use
>> thin if it is supported on the platform.
>>
>> The other issue preventing using thin by default is that we default the
>> max oversubscription ratio to 20.  IMO that isn't a safe thing to do for
>> the reference implementation, since it means that people who deploy
>> Cinder LVM on smaller storage configurations can easily fill up their
>> volume group and have things grind to halt.  I think we want something
>> closer to the semantics of thick LVM for the default case.
>>
>> We haven't thought through a reasonable migration strategy for how to
>> handle that.  I'm not sure we can change the default oversubscription
>> ratio without breaking deployments using other drivers.  (Maybe I'm
>> wrong about this?)
>>
>> If we sort out that issue, I don't see any reason we can't switch over
>> in Mitaka.
>>
>> [1] https://review.openstack.org/#/c/104653/
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[yang, xing]

Hi Eric,

Please see my replies inline below.

Thanks,
Xing


On 9/16/15, 1:20 PM, "Eric Harney"  wrote:

>On 09/15/2015 04:56 PM, yang, xing wrote:
>> Hi Eric,
>> 
>> Regarding the default max_over_subscription_ratio, I initially set the
>> default to 1 while working on oversubscription, and changed it to 2
>>after
>> getting review comments.  After it was merged, I got feedback that 2 is
>> too small and 20 is more appropriated, so I changed it to 20.  So it
>>looks
>> like we can¹t find a default value that makes everyone happy.
>> 
>
>I'm curious about how this is used in real-world deployments.  Are we
>making the assumption that the admin has some external monitoring
>configured to send alarms if the storage is nearing capacity?

We can have Ceilometer integration for capacity notifications.  See the
following patches on capacity headroom:

https://review.openstack.org/#/c/170380/
https://review.openstack.org/#/c/206923/

>
>> If we can decide what is the best default value for LVM, we can change
>>the
>> default max_over_subscription_ratio, but we should also allow other
>> drivers to specify a different config option if a different default
>>value
>> is more appropriate for them.
>
>This sounds like a good idea, I'm just not sure how to structure it yet
>without creating a very confusing set of config options.

I’m thinking we could have a prefix with vendor name for this and it also
requires documentation by driver maintainers if they are using a different
config option.  I proposed a topic to discuss about this at the summit.

>
>
>> On 9/15/15, 1:38 PM, "Eric Harney"  wrote:
>> 
>>> On 09/15/2015 01:00 PM, Chris Friesen wrote:
 I'm currently trying to work around an issue where activating LVM
 snapshots created through cinder takes potentially a long time.
 (Linearly related to the amount of data that differs between the
 original volume and the snapshot.)  On one system I tested it took
about
 one minute per 25GB of data, so the worst-case boot delay can become
 significant.

 According to Zdenek Kabelac on the LVM mailing list, LVM snapshots
were
 not intended to be kept around indefinitely, they were supposed to be
 used only until the backup was taken and then deleted.  He recommends
 using thin provisioning for long-lived snapshots due to differences in
 how the metadata is maintained.  (He also says he's heard reports of
 volume activation taking half an hour, which is clearly crazy when
 instances are waiting to access their volumes.)

 Given the above, is there any reason why we couldn't make thin
 provisioning the default?

>>>
>>>
>>> My intention is to move toward thin-provisioned LVM as the default --
>>>it
>>> is definitely better suited to our use of LVM.  Previously this was
>>>less
>>> easy, since some older Ubuntu platforms didn't support it, but in
>>> Liberty we added the ability to specify lvm_type = "auto" [1] to use
>>> thin if it is supported on the platform.
>>>
>>> The other issue preventing using thin by default is that we default the
>>> max oversubscription ratio to 20.  IMO that isn't a safe thing to do
>>>for
>>> the reference implementation, since it means that people who deploy
>>> Cinder LVM on smaller storage configurations can easily fill up their
>>> volume group and have things grind to halt.  I think we want something
>>> closer to the semantics of thick LVM for the default case.
>>>
>>> We haven't thought through a reasonable migration strategy for how to
>>> handle that.  I'm not sure we can change the default oversubscription
>>> ratio without breaking deployments using other drivers.  (Maybe I'm
>>> wrong about this?)
>>>
>>> If we sort out that issue, I don't see any reason we can't switch over
>>> in Mitaka.
>>>
>>> [1] https://review.openstack.org/#/c/104653/
>>>
>>> 
>>>
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> 
>>_
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Philipp Marek]

> > I'm currently trying to work around an issue where activating LVM
> > snapshots created through cinder takes potentially a long time. 
[[ thick LVM snapshot performance problem ]]
> > Given the above, is there any reason why we couldn't make thin
> > provisioning the default?
> 
> My intention is to move toward thin-provisioned LVM as the default -- it
> is definitely better suited to our use of LVM.
...
> The other issue preventing using thin by default is that we default the
> max oversubscription ratio to 20.  IMO that isn't a safe thing to do for
> the reference implementation, since it means that people who deploy
> Cinder LVM on smaller storage configurations can easily fill up their
> volume group and have things grind to halt.  I think we want something
> closer to the semantics of thick LVM for the default case.
The DRBDmanage backend has to deal with the same problem.

We decided to provide 3 different storage strategies:

 * Use Thick LVs - with the known performance implications when using
   snapshots.
 * Use one Thin Pool for the volumes - this uses the available space
   "optimally", but gives the oversubscription problem mentioned above.
 * Use multiple Thin Pools, one for each volume.
   This provides efficient snapshots *and* space reservation for each
   volume.
   
The last strategy is no panacea, though - something needs to check the free 
space in the pool, because the snapshots can still fill it up...
Without impacting the other volumes, though.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Chris Friesen]

I'm currently trying to work around an issue where activating LVM snapshots 
created through cinder takes potentially a long time.  (Linearly related to the 
amount of data that differs between the original volume and the snapshot.)  On 
one system I tested it took about one minute per 25GB of data, so the worst-case 
boot delay can become significant.


According to Zdenek Kabelac on the LVM mailing list, LVM snapshots were not 
intended to be kept around indefinitely, they were supposed to be used only 
until the backup was taken and then deleted.  He recommends using thin 
provisioning for long-lived snapshots due to differences in how the metadata is 
maintained.  (He also says he's heard reports of volume activation taking half 
an hour, which is clearly crazy when instances are waiting to access their volumes.)


Given the above, is there any reason why we couldn't make thin provisioning the 
default?


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[Eric Harney]

On 09/15/2015 01:00 PM, Chris Friesen wrote:
> I'm currently trying to work around an issue where activating LVM
> snapshots created through cinder takes potentially a long time. 
> (Linearly related to the amount of data that differs between the
> original volume and the snapshot.)  On one system I tested it took about
> one minute per 25GB of data, so the worst-case boot delay can become
> significant.
> 
> According to Zdenek Kabelac on the LVM mailing list, LVM snapshots were
> not intended to be kept around indefinitely, they were supposed to be
> used only until the backup was taken and then deleted.  He recommends
> using thin provisioning for long-lived snapshots due to differences in
> how the metadata is maintained.  (He also says he's heard reports of
> volume activation taking half an hour, which is clearly crazy when
> instances are waiting to access their volumes.)
> 
> Given the above, is there any reason why we couldn't make thin
> provisioning the default?
> 


My intention is to move toward thin-provisioned LVM as the default -- it
is definitely better suited to our use of LVM.  Previously this was less
easy, since some older Ubuntu platforms didn't support it, but in
Liberty we added the ability to specify lvm_type = "auto" [1] to use
thin if it is supported on the platform.

The other issue preventing using thin by default is that we default the
max oversubscription ratio to 20.  IMO that isn't a safe thing to do for
the reference implementation, since it means that people who deploy
Cinder LVM on smaller storage configurations can easily fill up their
volume group and have things grind to halt.  I think we want something
closer to the semantics of thick LVM for the default case.

We haven't thought through a reasonable migration strategy for how to
handle that.  I'm not sure we can change the default oversubscription
ratio without breaking deployments using other drivers.  (Maybe I'm
wrong about this?)

If we sort out that issue, I don't see any reason we can't switch over
in Mitaka.

[1] https://review.openstack.org/#/c/104653/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[John Griffith]

On Tue, Sep 15, 2015 at 11:38 AM, Eric Harney  wrote:

> On 09/15/2015 01:00 PM, Chris Friesen wrote:
> > I'm currently trying to work around an issue where activating LVM
> > snapshots created through cinder takes potentially a long time.
> > (Linearly related to the amount of data that differs between the
> > original volume and the snapshot.)  On one system I tested it took about
> > one minute per 25GB of data, so the worst-case boot delay can become
> > significant.
>
​Sadly the addition of the whole activate/deactivate has been problematic
ever since it was introduced.  I'd like to better understand why this is
needed and why the long delay.
​


> >
> > According to Zdenek Kabelac on the LVM mailing list, LVM snapshots were
> > not intended to be kept around indefinitely, they were supposed to be
> > used only until the backup was taken and then deleted.  He recommends
>
​Correct, and FWIW this has also been the recommendation from Cinder's
perspective for a long time as well.  Snapshots are NOT backups and
shouldn't be treated as such.
​


> > using thin provisioning for long-lived snapshots due to differences in
> > how the metadata is maintained.  (He also says he's heard reports of
> > volume activation taking half an hour, which is clearly crazy when
> > instances are waiting to access their volumes.)

>
> > Given the above, is there any reason why we couldn't make thin
> > provisioning the default?
>
​I tried, it was rejected.  I think it's crazy not to fix things up and do
this at this point.
​


> >
>
>
> My intention is to move toward thin-provisioned LVM as the default -- it
> is definitely better suited to our use of LVM.  Previously this was less
> easy, since some older Ubuntu platforms didn't support it, but in
> Liberty we added the ability to specify lvm_type = "auto" [1] to use
> thin if it is supported on the platform.
>
> The other issue preventing using thin by default is that we default the
> max oversubscription ratio to 20.  IMO that isn't a safe thing to do for
> the reference implementation, since it means that people who deploy
> Cinder LVM on smaller storage configurations can easily fill up their
> volume group and have things grind to halt.  I think we want something
> closer to the semantics of thick LVM for the default case.
>
> We haven't thought through a reasonable migration strategy for how to
> handle that.  I'm not sure we can change the default oversubscription
> ratio without breaking deployments using other drivers.  (Maybe I'm
> wrong about this?)
>
> If we sort out that issue, I don't see any reason we can't switch over
> in Mitaka.
>
> [1] https://review.openstack.org/#/c/104653/
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] LVM snapshot performance issue -- why isn't thin provisioning the default?

[yang, xing]

Hi Eric,

Regarding the default max_over_subscription_ratio, I initially set the
default to 1 while working on oversubscription, and changed it to 2 after
getting review comments.  After it was merged, I got feedback that 2 is
too small and 20 is more appropriated, so I changed it to 20.  So it looks
like we can¹t find a default value that makes everyone happy.

If we can decide what is the best default value for LVM, we can change the
default max_over_subscription_ratio, but we should also allow other
drivers to specify a different config option if a different default value
is more appropriate for them.

Thanks,
Xing


On 9/15/15, 1:38 PM, "Eric Harney"  wrote:

>On 09/15/2015 01:00 PM, Chris Friesen wrote:
>> I'm currently trying to work around an issue where activating LVM
>> snapshots created through cinder takes potentially a long time.
>> (Linearly related to the amount of data that differs between the
>> original volume and the snapshot.)  On one system I tested it took about
>> one minute per 25GB of data, so the worst-case boot delay can become
>> significant.
>> 
>> According to Zdenek Kabelac on the LVM mailing list, LVM snapshots were
>> not intended to be kept around indefinitely, they were supposed to be
>> used only until the backup was taken and then deleted.  He recommends
>> using thin provisioning for long-lived snapshots due to differences in
>> how the metadata is maintained.  (He also says he's heard reports of
>> volume activation taking half an hour, which is clearly crazy when
>> instances are waiting to access their volumes.)
>> 
>> Given the above, is there any reason why we couldn't make thin
>> provisioning the default?
>> 
>
>
>My intention is to move toward thin-provisioned LVM as the default -- it
>is definitely better suited to our use of LVM.  Previously this was less
>easy, since some older Ubuntu platforms didn't support it, but in
>Liberty we added the ability to specify lvm_type = "auto" [1] to use
>thin if it is supported on the platform.
>
>The other issue preventing using thin by default is that we default the
>max oversubscription ratio to 20.  IMO that isn't a safe thing to do for
>the reference implementation, since it means that people who deploy
>Cinder LVM on smaller storage configurations can easily fill up their
>volume group and have things grind to halt.  I think we want something
>closer to the semantics of thick LVM for the default case.
>
>We haven't thought through a reasonable migration strategy for how to
>handle that.  I'm not sure we can change the default oversubscription
>ratio without breaking deployments using other drivers.  (Maybe I'm
>wrong about this?)
>
>If we sort out that issue, I don't see any reason we can't switch over
>in Mitaka.
>
>[1] https://review.openstack.org/#/c/104653/
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev