Re: [openstack-dev] [nova] solving API case sensitivity issues

[Sean Dague]

On 02/24/2016 07:48 AM, Sean Dague wrote:
> We have a specific bug around aggregrate metadata setting in Nova which
> exposes a larger issue with our mysql schema.
> https://bugs.launchpad.net/nova/+bug/1538011
> 
> On mysql the following will explode with a 500:
> 
>> nova aggregate-create agg1
>> nova aggregate-set-metadata agg1 abc=1
>> nova aggregate-set-metadata agg1 ABC=2
> 
> mysql (by default) treats abc == ABC. However the python code does not.

This is reviving an old thread, because we ended up deadlocked on the
case folding issue. Late in the day on Friday at the summit we revisited
this issue in the Nova room, and came up with a new approach:

https://review.openstack.org/#/c/311529/

The TL;DR is to change the input validation in a microversion, provide
and audit / fix tool to adminstrators, and close bugs using uppercase
metadata keys and seeing issues as Won't Fix.

Anyone with strong opinions should spend some time looking there.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[melanie witt]

On Feb 25, 2016, at 8:35, Michał Dulko  wrote:

> We've faced similar issues in Cinder and as solution we've moved
> filtering to Python code. Like in for example [1] or [2]. But no, we
> haven't had UNIQUE constraint on the DB column in these cases, only on IDs.

This is an interesting option.

I see how option 1 (API case fold) is appealing, since our underlying storage 
(in the mysql case), is case insensitive. And when I think about it, I could 
see how for example "abc" is essentially the same thing as "ABC" and would be 
resilient to end users making differences in capitalization of metadata keys 
(imagine a typo of "DriveType" vs "Drivetype" where a user expects to set a 
value for the key and they are treated as different keys).

The only concern I have about the case folding is when the data is visible to 
the user. That is, if a user sets a value for "DriveType" and they do 'nova 
aggregate-details' and see "drivetype" displayed, different than they specified 
or expected. In the case of aggregate metadata it doesn't seem too impactful 
since nova is supposed to be the only consumer of the metadata. But are we 
considering this as the consistent behavior across all APIs?

-melanie








signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[Michał Dulko]

On 02/24/2016 01:48 PM, Sean Dague wrote:
> We have a specific bug around aggregrate metadata setting in Nova which
> exposes a larger issue with our mysql schema.
> https://bugs.launchpad.net/nova/+bug/1538011
>
> On mysql the following will explode with a 500:
>
>> nova aggregate-create agg1
>> nova aggregate-set-metadata agg1 abc=1
>> nova aggregate-set-metadata agg1 ABC=2
> mysql (by default) treats abc == ABC. However the python code does not.
>
> We have a couple of options:
>
> 1) make the API explicitly case fold
>
> 2) update the mysql DB to use latin_bin collation for these columns
>
> 3) make this a 400 error because duplicates were found
>
>
> Options 1 & 2 make all OpenStack environments consistent regardless of
> backend.
>
> Option 2 is potentially expensive TABLE alter.
>
> Option 3 gets rid of the 500 error, however at the risk that the
> behavior for this API is different depending on DB backend. Which is
> less than ideal.
>
>
> My preference is slightly towards #1. It's taken a long time for someone
> to report this issue, so I think it's an edge case, and people weren't
> think about this being case sensitive. It has the risk of impacting
> someone on an odd db platform that has been using that feature.
>
> There are going to be a few other APIs to clean up in a similar way. I
> don't think this comes in under a microversion because of how deep in
> the db api layer this is, and it's just not viable to keep both paths.
>
>   -Sean

We've faced similar issues in Cinder and as solution we've moved
filtering to Python code. Like in for example [1] or [2]. But no, we
haven't had UNIQUE constraint on the DB column in these cases, only on IDs.

[1] https://review.openstack.org/225024
[2] https://review.openstack.org/#/c/274589/12/cinder/db/sqlalchemy/api.py

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[Sean Dague]

On 02/24/2016 06:56 PM, Chris Friesen wrote:
> On 02/24/2016 02:34 PM, Sean Dague wrote:
> 
>> There are no urls stored here. This is content coming through the body.
>> While I do appreciate many folks weighing in that haven't read the bug
>> yet, I would be even better if people did read the bug first.
>>
>> We have the ability to decide at the API level what the behavior is of
>> payloads in POST / GET. Given the majority of our users are on mysql,
>> they've never had access to case sensitive overlapping metadata on
>> aggregates before. Doing so seems odd and potentially confusing.
> 
> On the other hand, coming from a UNIX environment where case sensitivity
> is the default, it seems odd that we'd intentionally become
> case-insensitive.
> 
> For option 2 do we need to actually update the DB column description? 
> Or is there a way to alter the query to be case-insensitive without
> altering the column itself (using collate maybe)?

Collate is a change in the way the data is stored, it's a real alter.
The data structures look different if they are designed to be searched
with case folding or as binary.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[Chris Friesen]

On 02/24/2016 02:34 PM, Sean Dague wrote:


There are no urls stored here. This is content coming through the body.
While I do appreciate many folks weighing in that haven't read the bug
yet, I would be even better if people did read the bug first.

We have the ability to decide at the API level what the behavior is of
payloads in POST / GET. Given the majority of our users are on mysql,
they've never had access to case sensitive overlapping metadata on
aggregates before. Doing so seems odd and potentially confusing.


On the other hand, coming from a UNIX environment where case sensitivity is the 
default, it seems odd that we'd intentionally become case-insensitive.


For option 2 do we need to actually update the DB column description?  Or is 
there a way to alter the query to be case-insensitive without altering the 
column itself (using collate maybe)?


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[Sean Dague]

On 02/24/2016 03:13 PM, Mooney, Sean K wrote:
> 
> 
>> -Original Message-
>> From: James Bottomley [mailto:james.bottom...@hansenpartnership.com]
>> Sent: Wednesday, February 24, 2016 5:46 PM
>> To: Sean Dague <s...@dague.net>; openstack-dev@lists.openstack.org
>> Subject: Re: [openstack-dev] [nova] solving API case sensitivity issues
>>
>> On Wed, 2016-02-24 at 11:40 -0500, Sean Dague wrote:
>>> On 02/24/2016 11:28 AM, James Bottomley wrote:
>>>> On Wed, 2016-02-24 at 07:48 -0500, Sean Dague wrote:
>>>>> We have a specific bug around aggregrate metadata setting in Nova
>>>>> which exposes a larger issue with our mysql schema.
>>>>> https://bugs.launchpad.net/nova/+bug/1538011
>>>>>
>>>>> On mysql the following will explode with a 500:
>>>>>
>>>>>> nova aggregate-create agg1
>>>>>> nova aggregate-set-metadata agg1 abc=1 nova
>>>>>> aggregate-set-metadata agg1 ABC=2
>>>>>
>>>>> mysql (by default) treats abc == ABC. However the python code does
>>>>> not.
> Personally I would argue that the python code is correct
> and they should not be considered the same. ABC and abc are two different keys
> in the aggregate metadata and I do not think it is correct to treat them the 
> same.
> Assuming the above commands I would expect the metadata to contain  two key 
> pairs [abc=1,ABC=2]
> 
>>>>>
>>>>> We have a couple of options:
>>>>>
>>>>> 1) make the API explicitly case fold
>>>>>
>>>>> 2) update the mysql DB to use latin_bin collation for these
>>>>> columns
> This should not be latin_bin as Unicode is allowed in URLs this should really 
> be utf8_bin

There are no urls stored here. This is content coming through the body.
While I do appreciate many folks weighing in that haven't read the bug
yet, I would be even better if people did read the bug first.

We have the ability to decide at the API level what the behavior is of
payloads in POST / GET. Given the majority of our users are on mysql,
they've never had access to case sensitive overlapping metadata on
aggregates before. Doing so seems odd and potentially confusing.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[Mooney, Sean K]

> -Original Message-
> From: James Bottomley [mailto:james.bottom...@hansenpartnership.com]
> Sent: Wednesday, February 24, 2016 5:46 PM
> To: Sean Dague <s...@dague.net>; openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [nova] solving API case sensitivity issues
> 
> On Wed, 2016-02-24 at 11:40 -0500, Sean Dague wrote:
> > On 02/24/2016 11:28 AM, James Bottomley wrote:
> > > On Wed, 2016-02-24 at 07:48 -0500, Sean Dague wrote:
> > > > We have a specific bug around aggregrate metadata setting in Nova
> > > > which exposes a larger issue with our mysql schema.
> > > > https://bugs.launchpad.net/nova/+bug/1538011
> > > >
> > > > On mysql the following will explode with a 500:
> > > >
> > > > > nova aggregate-create agg1
> > > > > nova aggregate-set-metadata agg1 abc=1 nova
> > > > > aggregate-set-metadata agg1 ABC=2
> > > >
> > > > mysql (by default) treats abc == ABC. However the python code does
> > > > not.
Personally I would argue that the python code is correct
and they should not be considered the same. ABC and abc are two different keys
in the aggregate metadata and I do not think it is correct to treat them the 
same.
Assuming the above commands I would expect the metadata to contain  two key 
pairs [abc=1,ABC=2]

> > > >
> > > > We have a couple of options:
> > > >
> > > > 1) make the API explicitly case fold
> > > >
> > > > 2) update the mysql DB to use latin_bin collation for these
> > > > columns
This should not be latin_bin as Unicode is allowed in URLs this should really 
be utf8_bin
> > > >
> > > > 3) make this a 400 error because duplicates were found
> > > >
> > > >
> > > > Options 1 & 2 make all OpenStack environments consistent
> > > > regardless of backend.
> > > >
> > > > Option 2 is potentially expensive TABLE alter.
> > > >
> > > > Option 3 gets rid of the 500 error, however at the risk that the
> > > > behavior for this API is different depending on DB backend. Which
> > > > is less than ideal.
> > > >
> > > >
> > > > My preference is slightly towards #1. It's taken a long time for
> > > > someone to report this issue, so I think it's an edge case, and
> > > > people weren't think about this being case sensitive. It has the
> > > > risk of impacting someone on an odd db platform that has been
> > > > using that feature.
> > > >
> > > > There are going to be a few other APIs to clean up in a similar
> > > > way.
> > > > I don't think this comes in under a microversion because of how
> > > > deep in the db api layer this is, and it's just not viable to keep
> > > > both paths.
> > >
> > > This is actually one of the curses wished on us by REST.  Since the
> > > intent is to use web requests for the API, the API name must follow
> > > the case sensitivity rules for URL matching (case insensitive).
> >
> > Um... since when are URLs generically case insensitive? The host
> > portion is - https://tools.ietf.org/html/r
> > https://tools.ietf.org/html/rfc3986#section-3.2.2c3986#section-3.2.2
> > and the scheme
> > portion is - https://tools.ietf.org/html/rfc3986#section-3.1 however
> > nothing about the PATH specifies that it should or must be (in section
> > 3.3)
I would agree with this we should not assume that URLs are case sensitive nor 
should
We assume they are ascii. I would be in favor of option 2 with utf8_bin as this 
support
Both Unicode and case sensitivity.
> 
> Heh, OK, I'm out of date.  When we first argued over this, Microsoft
> required case insensitive matching for the path component because IIS
> was doing lookups on vfat filesystems which are naturally case
> insensitive.  If that's been excised from the standard, I'm happy to
> keep it in the dustbin of history.
> 
> > While it's a little off topic, this is the 2nd time in a month it came
> > up, so I'd like to know if there is a reference for the case
> > insensitive pov.
> 
> I checked; it looks to be implementation specific.  So php, for
> instance, does case sensitive
> 
> /index.php != /Index.php
> 
> But drupal does case insensitive
> 
> /node/6 == /Node/6 == /NoDe/6
> 
> So all in all, a bit of a mess.
> 
> James
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[James Bottomley]

On Wed, 2016-02-24 at 11:40 -0500, Sean Dague wrote:
> On 02/24/2016 11:28 AM, James Bottomley wrote:
> > On Wed, 2016-02-24 at 07:48 -0500, Sean Dague wrote:
> > > We have a specific bug around aggregrate metadata setting in Nova
> > > which
> > > exposes a larger issue with our mysql schema.
> > > https://bugs.launchpad.net/nova/+bug/1538011
> > > 
> > > On mysql the following will explode with a 500:
> > > 
> > > > nova aggregate-create agg1
> > > > nova aggregate-set-metadata agg1 abc=1
> > > > nova aggregate-set-metadata agg1 ABC=2
> > > 
> > > mysql (by default) treats abc == ABC. However the python code
> > > does
> > > not.
> > > 
> > > We have a couple of options:
> > > 
> > > 1) make the API explicitly case fold
> > > 
> > > 2) update the mysql DB to use latin_bin collation for these
> > > columns
> > > 
> > > 3) make this a 400 error because duplicates were found
> > > 
> > > 
> > > Options 1 & 2 make all OpenStack environments consistent
> > > regardless
> > > of
> > > backend.
> > > 
> > > Option 2 is potentially expensive TABLE alter.
> > > 
> > > Option 3 gets rid of the 500 error, however at the risk that the
> > > behavior for this API is different depending on DB backend. Which
> > > is
> > > less than ideal.
> > > 
> > > 
> > > My preference is slightly towards #1. It's taken a long time for 
> > > someone to report this issue, so I think it's an edge case, and 
> > > people weren't think about this being case sensitive. It has the
> > > risk 
> > > of impacting someone on an odd db platform that has been using
> > > that
> > > feature.
> > > 
> > > There are going to be a few other APIs to clean up in a similar
> > > way. 
> > > I don't think this comes in under a microversion because of how
> > > deep 
> > > in the db api layer this is, and it's just not viable to keep
> > > both
> > > paths.
> > 
> > This is actually one of the curses wished on us by REST.  Since the
> > intent is to use web requests for the API, the API name must follow
> > the
> > case sensitivity rules for URL matching (case insensitive). 
> 
> Um... since when are URLs generically case insensitive? The host
> portion
> is - https://tools.ietf.org/html/r
> https://tools.ietf.org/html/rfc3986#section-3.2.2c3986#section-3.2.2 
> and the scheme
> portion is - https://tools.ietf.org/html/rfc3986#section-3.1 however
> nothing about the PATH specifies that it should or must be (in
> section 3.3)

Heh, OK, I'm out of date.  When we first argued over this, Microsoft
required case insensitive matching for the path component because IIS
was doing lookups on vfat filesystems which are naturally case
insensitive.  If that's been excised from the standard, I'm happy to
keep it in the dustbin of history.

> While it's a little off topic, this is the 2nd time in a month it 
> came up, so I'd like to know if there is a reference for the case
> insensitive pov.

I checked; it looks to be implementation specific.  So php, for
instance, does case sensitive

/index.php != /Index.php

But drupal does case insensitive

/node/6 == /Node/6 == /NoDe/6

So all in all, a bit of a mess.

James


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[Sean Dague]

On 02/24/2016 11:28 AM, James Bottomley wrote:
> On Wed, 2016-02-24 at 07:48 -0500, Sean Dague wrote:
>> We have a specific bug around aggregrate metadata setting in Nova
>> which
>> exposes a larger issue with our mysql schema.
>> https://bugs.launchpad.net/nova/+bug/1538011
>>
>> On mysql the following will explode with a 500:
>>
>>> nova aggregate-create agg1
>>> nova aggregate-set-metadata agg1 abc=1
>>> nova aggregate-set-metadata agg1 ABC=2
>>
>> mysql (by default) treats abc == ABC. However the python code does
>> not.
>>
>> We have a couple of options:
>>
>> 1) make the API explicitly case fold
>>
>> 2) update the mysql DB to use latin_bin collation for these columns
>>
>> 3) make this a 400 error because duplicates were found
>>
>>
>> Options 1 & 2 make all OpenStack environments consistent regardless
>> of
>> backend.
>>
>> Option 2 is potentially expensive TABLE alter.
>>
>> Option 3 gets rid of the 500 error, however at the risk that the
>> behavior for this API is different depending on DB backend. Which is
>> less than ideal.
>>
>>
>> My preference is slightly towards #1. It's taken a long time for 
>> someone to report this issue, so I think it's an edge case, and 
>> people weren't think about this being case sensitive. It has the risk 
>> of impacting someone on an odd db platform that has been using that
>> feature.
>>
>> There are going to be a few other APIs to clean up in a similar way. 
>> I don't think this comes in under a microversion because of how deep 
>> in the db api layer this is, and it's just not viable to keep both
>> paths.
> 
> This is actually one of the curses wished on us by REST.  Since the
> intent is to use web requests for the API, the API name must follow the
> case sensitivity rules for URL matching (case insensitive). 

Um... since when are URLs generically case insensitive? The host portion
is - https://tools.ietf.org/html/rfc3986#section-3.2.2 and the scheme
portion is - https://tools.ietf.org/html/rfc3986#section-3.1 however
nothing about the PATH specifies that it should or must be (in section 3.3)

While it's a little off topic, this is the 2nd time in a month it came
up, so I'd like to know if there is a reference for the case insensitive
pov.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[Chris Dent]

On Wed, 24 Feb 2016, James Bottomley wrote:


This is actually one of the curses wished on us by REST.  Since the
intent is to use web requests for the API, the API name must follow the
case sensitivity rules for URL matching (case insensitive).  However,
the rest of the parameters can be case sensitive.  That means that if
your column name maps to an API, it must be case insensitive, but if it
maps to a data input it may be case sensitive.


I'm not sure what you are going on about here. URLs are case
sensitive[1]. If Nova is using case insensitive rules for route matching
that's a Nova thing and has nothing to do with either REST or HTTP.

[1] https://tools.ietf.org/html/rfc7230#section-2.7.3

--
Chris Dent   (�s°□°)�s�喋擤ォ�http://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[James Bottomley]

On Wed, 2016-02-24 at 07:48 -0500, Sean Dague wrote:
> We have a specific bug around aggregrate metadata setting in Nova
> which
> exposes a larger issue with our mysql schema.
> https://bugs.launchpad.net/nova/+bug/1538011
> 
> On mysql the following will explode with a 500:
> 
> > nova aggregate-create agg1
> > nova aggregate-set-metadata agg1 abc=1
> > nova aggregate-set-metadata agg1 ABC=2
> 
> mysql (by default) treats abc == ABC. However the python code does
> not.
> 
> We have a couple of options:
> 
> 1) make the API explicitly case fold
> 
> 2) update the mysql DB to use latin_bin collation for these columns
> 
> 3) make this a 400 error because duplicates were found
> 
> 
> Options 1 & 2 make all OpenStack environments consistent regardless
> of
> backend.
> 
> Option 2 is potentially expensive TABLE alter.
> 
> Option 3 gets rid of the 500 error, however at the risk that the
> behavior for this API is different depending on DB backend. Which is
> less than ideal.
> 
> 
> My preference is slightly towards #1. It's taken a long time for 
> someone to report this issue, so I think it's an edge case, and 
> people weren't think about this being case sensitive. It has the risk 
> of impacting someone on an odd db platform that has been using that
> feature.
> 
> There are going to be a few other APIs to clean up in a similar way. 
> I don't think this comes in under a microversion because of how deep 
> in the db api layer this is, and it's just not viable to keep both
> paths.

This is actually one of the curses wished on us by REST.  Since the
intent is to use web requests for the API, the API name must follow the
case sensitivity rules for URL matching (case insensitive).  However,
the rest of the parameters can be case sensitive.  That means that if
your column name maps to an API, it must be case insensitive, but if it
maps to a data input it may be case sensitive.

I think option 1 is the best course, but someone will have to take a
look and make sure there are no APIs that suddenly have case
insensitivity rules for data inputs that aren't expressed currently.

James



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[Andrew Laski]

On Wed, Feb 24, 2016, at 07:48 AM, Sean Dague wrote:
> We have a specific bug around aggregrate metadata setting in Nova which
> exposes a larger issue with our mysql schema.
> https://bugs.launchpad.net/nova/+bug/1538011
> 
> On mysql the following will explode with a 500:
> 
> > nova aggregate-create agg1
> > nova aggregate-set-metadata agg1 abc=1
> > nova aggregate-set-metadata agg1 ABC=2
> 
> mysql (by default) treats abc == ABC. However the python code does not.
> 
> We have a couple of options:
> 
> 1) make the API explicitly case fold
> 
> 2) update the mysql DB to use latin_bin collation for these columns
> 
> 3) make this a 400 error because duplicates were found
> 
> 
> Options 1 & 2 make all OpenStack environments consistent regardless of
> backend.
> 
> Option 2 is potentially expensive TABLE alter.
> 
> Option 3 gets rid of the 500 error, however at the risk that the
> behavior for this API is different depending on DB backend. Which is
> less than ideal.
> 
> 
> My preference is slightly towards #1. It's taken a long time for someone
> to report this issue, so I think it's an edge case, and people weren't
> think about this being case sensitive. It has the risk of impacting
> someone on an odd db platform that has been using that feature.

#1 is my preference as well. Nova should be the only consumer of this
metadata and it does not need case sensitivity for this.

> 
> There are going to be a few other APIs to clean up in a similar way. I
> don't think this comes in under a microversion because of how deep in
> the db api layer this is, and it's just not viable to keep both paths.
> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] solving API case sensitivity issues

[Jay Pipes]

On 02/24/2016 07:48 AM, Sean Dague wrote:

We have a specific bug around aggregrate metadata setting in Nova which
exposes a larger issue with our mysql schema.
https://bugs.launchpad.net/nova/+bug/1538011

On mysql the following will explode with a 500:


nova aggregate-create agg1
nova aggregate-set-metadata agg1 abc=1
nova aggregate-set-metadata agg1 ABC=2


mysql (by default) treats abc == ABC. However the python code does not.

We have a couple of options:

1) make the API explicitly case fold

2) update the mysql DB to use latin_bin collation for these columns

3) make this a 400 error because duplicates were found


Options 1 & 2 make all OpenStack environments consistent regardless of
backend.

Option 2 is potentially expensive TABLE alter.

Option 3 gets rid of the 500 error, however at the risk that the
behavior for this API is different depending on DB backend. Which is
less than ideal.


My preference is slightly towards #1. It's taken a long time for someone
to report this issue, so I think it's an edge case, and people weren't
think about this being case sensitive. It has the risk of impacting
someone on an odd db platform that has been using that feature.


Agreed, #1 is my preference, even though I shake my fist at MySQL being 
so dumb in this regard.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] solving API case sensitivity issues

[Sean Dague]

We have a specific bug around aggregrate metadata setting in Nova which
exposes a larger issue with our mysql schema.
https://bugs.launchpad.net/nova/+bug/1538011

On mysql the following will explode with a 500:

> nova aggregate-create agg1
> nova aggregate-set-metadata agg1 abc=1
> nova aggregate-set-metadata agg1 ABC=2

mysql (by default) treats abc == ABC. However the python code does not.

We have a couple of options:

1) make the API explicitly case fold

2) update the mysql DB to use latin_bin collation for these columns

3) make this a 400 error because duplicates were found


Options 1 & 2 make all OpenStack environments consistent regardless of
backend.

Option 2 is potentially expensive TABLE alter.

Option 3 gets rid of the 500 error, however at the risk that the
behavior for this API is different depending on DB backend. Which is
less than ideal.


My preference is slightly towards #1. It's taken a long time for someone
to report this issue, so I think it's an edge case, and people weren't
think about this being case sensitive. It has the risk of impacting
someone on an odd db platform that has been using that feature.

There are going to be a few other APIs to clean up in a similar way. I
don't think this comes in under a microversion because of how deep in
the db api layer this is, and it's just not viable to keep both paths.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev