Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Morgan Fainberg]

On March 4, 2014 at 10:45:02, Vishvananda Ishaya (vishvana...@gmail.com) wrote:

On Mar 3, 2014, at 11:32 AM, Jay Pipes  wrote: 

> On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote: 
>> On Mar 3, 2014, at 6:48 AM, Jay Pipes  wrote: 
>> 
>>> On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote: 
 Having done some work with MySQL (specifically around similar data 
 sets) and discussing the changes with some former coworkers (MySQL 
 experts) I am inclined to believe the move from varchar to binary 
 absolutely would increase performance like this. 
 
 
 However, I would like to get some real benchmarks around it and if it 
 really makes a difference we should get a smart "UUID" type into the 
 common SQLlibs (can pgsql see a similar benefit? Db2?) I think this 
 conversation. Should be split off from the keystone one at hand - I 
 don't want valuable information / discussions to get lost. 
>>> 
>>> No disagreement on either point. However, this should be done after the 
>>> standardization to a UUID user_id in Keystone, as a separate performance 
>>> improvement patch. Agree? 
>>> 
>>> Best, 
>>> -jay 
>> 
>> -1 
>> 
>> The expectation in other projects has been that project_ids and user_ids are 
>> opaque strings. I need to see more clear benefit to enforcing stricter 
>> typing on these, because I think it might break a lot of things. 
> 
> What does Nova lose here? The proposal is to have Keystone's user_id 
> values be UUIDs all the time. There would be a migration or helper 
> script against Nova's database that would change all non-UUID user_id 
> values to the Keystone UUID values. 

So I don’t have a problem with keystone internally using uuids, but forcing 
a migration of user identifiers isn’t something that should be taken lightly. 
One example is external logging and billing systems which now have to be 
migrated. 

I’m not opposed to the migration in principle. We may have to do a similar 
thing for project_ids with hierarchical multitenancy, for example. I just 
think we need a really good reason to do it, and the performance arguments 
just don’t seem good enough without a little empirical data. 

Vish 
Any one of the proposals we’re planning on using will not affect any current 
IDs.  Since the user_id is a blob, if we start issuing a new “id” format, 
ideally it shouldn’t matter as long as old IDs continue to work. If we do make 
any kind of migration for issued IDs I would expect that to be very deliberate 
and outside of this change set. Specifically this change set would enable 
multiple LDAP backends (among other user_id uniqueness benefits for federation, 
etc). 

I am very concerned about the external tools that reference IDs we currently 
have.

—Morgan





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Vishvananda Ishaya]

On Mar 3, 2014, at 11:32 AM, Jay Pipes  wrote:

> On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote:
>> On Mar 3, 2014, at 6:48 AM, Jay Pipes  wrote:
>> 
>>> On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
 Having done some work with MySQL (specifically around similar data
 sets) and discussing the changes with some former coworkers (MySQL
 experts) I am inclined to believe the move from varchar  to binary
 absolutely would increase performance like this.
 
 
 However, I would like to get some real benchmarks around it and if it
 really makes a difference we should get a smart "UUID" type into the
 common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
 conversation. Should be split off from the keystone one at hand - I
 don't want valuable information / discussions to get lost.
>>> 
>>> No disagreement on either point. However, this should be done after the
>>> standardization to a UUID user_id in Keystone, as a separate performance
>>> improvement patch. Agree?
>>> 
>>> Best,
>>> -jay
>> 
>> -1
>> 
>> The expectation in other projects has been that project_ids and user_ids are 
>> opaque strings. I need to see more clear benefit to enforcing stricter 
>> typing on these, because I think it might break a lot of things.
> 
> What does Nova lose here? The proposal is to have Keystone's user_id
> values be UUIDs all the time. There would be a migration or helper
> script against Nova's database that would change all non-UUID user_id
> values to the Keystone UUID values.

So I don’t have a problem with keystone internally using uuids, but forcing
a migration of user identifiers isn’t something that should be taken lightly.
One example is external logging and billing systems which now have to be
migrated.

I’m not opposed to the migration in principle. We may have to do a similar
thing for project_ids with hierarchical multitenancy, for example. I just
think we need a really good reason to do it, and the performance arguments
just don’t seem good enough without a little empirical data.

Vish

> 
> If there's stuff in Nova that would break (which is doubtful,
> considering like you say, these are supposed to be "opaque values" and
> as such should not have any restrictions or validation on their value),
> then that is code in Nova that should be fixed.
> 
> Honestly, we shouldn't accept poor or loose code just because "stuff
> might break".
> 
> -jay
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Clint Byrum]

Excerpts from Adam Young's message of 2014-03-03 23:06:58 -0800:
> On 03/03/2014 02:32 PM, Jay Pipes wrote:
> > On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote:
> >> On Mar 3, 2014, at 6:48 AM, Jay Pipes  wrote:
> >>
> >>> On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
>  Having done some work with MySQL (specifically around similar data
>  sets) and discussing the changes with some former coworkers (MySQL
>  experts) I am inclined to believe the move from varchar  to binary
>  absolutely would increase performance like this.
> 
> 
>  However, I would like to get some real benchmarks around it and if it
>  really makes a difference we should get a smart "UUID" type into the
>  common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
>  conversation. Should be split off from the keystone one at hand - I
>  don't want valuable information / discussions to get lost.
> >>> No disagreement on either point. However, this should be done after the
> >>> standardization to a UUID user_id in Keystone, as a separate performance
> >>> improvement patch. Agree?
> >>>
> >>> Best,
> >>> -jay
> >> -1
> >>
> >> The expectation in other projects has been that project_ids and user_ids 
> >> are opaque strings. I need to see more clear benefit to enforcing stricter 
> >> typing on these, because I think it might break a lot of things.
> > What does Nova lose here? The proposal is to have Keystone's user_id
> > values be UUIDs all the time. There would be a migration or helper
> > script against Nova's database that would change all non-UUID user_id
> > values to the Keystone UUID values.
> >
> > If there's stuff in Nova that would break (which is doubtful,
> > considering like you say, these are supposed to be "opaque values" and
> > as such should not have any restrictions or validation on their value),
> > then that is code in Nova that should be fixed.
> >
> > Honestly, we shouldn't accept poor or loose code just because "stuff
> > might break".
> Why do you think that UUIDs would be some sort of magic bullet?
> 
> Thus far, we've stated that IDs are opaque strings, and that they will 
> be URL safe.  Adding additional Database lokups on top of LDAP calls is 
> going to make LDAP code paths even more convoluted than they are now.  
> I'd like to avoid that. And I don't see the benefit.
> 

Each row in each table throughout OpenStack that has to treat the user
id from Keystone as an opaque ID must then store the keystone id as a 64
byte string. Being able to make it a 16 byte binary value means a
significant saving at scale.

If you're using LDAP, you're only going to store that CN once per user,
and only going to look it up when said user tries to log in. After that,
the uuid will be passed around.

> 
> I'd be interested in hearing why CN is a bad default choice for LDAP; we 
> have to use a very high up theRFE chain object class by default and 
> didn't have that many attributes to chose from.
> 

CN is fine. External IDs of various shapes and sizes can be used. The
point is to make keystone get a little more opionated about its' id
strings in order to make the rest of OpenStack more scalable.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Adam Young]

On 03/03/2014 02:32 PM, Jay Pipes wrote:

On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote:

On Mar 3, 2014, at 6:48 AM, Jay Pipes  wrote:


On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:

Having done some work with MySQL (specifically around similar data
sets) and discussing the changes with some former coworkers (MySQL
experts) I am inclined to believe the move from varchar  to binary
absolutely would increase performance like this.


However, I would like to get some real benchmarks around it and if it
really makes a difference we should get a smart "UUID" type into the
common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
conversation. Should be split off from the keystone one at hand - I
don't want valuable information / discussions to get lost.

No disagreement on either point. However, this should be done after the
standardization to a UUID user_id in Keystone, as a separate performance
improvement patch. Agree?

Best,
-jay

-1

The expectation in other projects has been that project_ids and user_ids are 
opaque strings. I need to see more clear benefit to enforcing stricter typing 
on these, because I think it might break a lot of things.

What does Nova lose here? The proposal is to have Keystone's user_id
values be UUIDs all the time. There would be a migration or helper
script against Nova's database that would change all non-UUID user_id
values to the Keystone UUID values.

If there's stuff in Nova that would break (which is doubtful,
considering like you say, these are supposed to be "opaque values" and
as such should not have any restrictions or validation on their value),
then that is code in Nova that should be fixed.

Honestly, we shouldn't accept poor or loose code just because "stuff
might break".

Why do you think that UUIDs would be some sort of magic bullet?

Thus far, we've stated that IDs are opaque strings, and that they will 
be URL safe.  Adding additional Database lokups on top of LDAP calls is 
going to make LDAP code paths even more convoluted than they are now.  
I'd like to avoid that. And I don't see the benefit.



I'd be interested in hearing why CN is a bad default choice for LDAP; we 
have to use a very high up theRFE chain object class by default and 
didn't have that many attributes to chose from.








-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Jay Pipes]

On Mon, 2014-03-03 at 11:18 -0800, Mark Washenberger wrote:
> 
> 
> 
> On Sat, Mar 1, 2014 at 12:51 PM, Jay Pipes  wrote:
> On Fri, 2014-02-28 at 15:25 -0800, Mark Washenberger wrote:
> > I believe we have some agreement here. Other openstack
> services should
> > be able to use a strongly typed identifier for users. I just
> think if
> > we want to go that route, we probably need to create a new
> field to
> > act as the proper user uuid, rather than repurposing the
> existing
> > field. It sounds like many existing LDAP deployments would
> break if we
> > repurpose the existing field.
> 
> 
> Hi Mark,
> 
> Please see my earlier response on this thread. I am proposing
> putting
> external identifiers into a mapping table that would correlate
> a
> Keystone UUID user ID with external identifiers (of variable
> length).
> 
> 
> The thing you seem to be missing is that the current user-id attribute
> is an "external identifier" depending on the identity backend you're
> using today. For example in the LDAP driver it is the CN by default
> (which is ridiculous for a large number of reasons, but let's leave
> those aside.) So if you want to create a new, strongly typed internal
> uuid identifier that makes the db performance scream, more power to
> you. But it's going to have to be a new field.

No, it won't. The proposal is to create a new table that maps a UUID
value to the external identifier value.

The migration would essentially do this pseudocode:

recs = get_identity_records()
for rec in recs:
  if not is_like_uuid(rec.user_id):
  external_id = rec.user_id
  new_uuid = uuid.uuid4()
  insert_into_external_ids(new_uuid, external_id)
  rec.user_id = new_uuid
  save_identity_record(rec)

Cascading updates on the user_id field to its child relations will take
care of the changes in user_id column in foriegn keys.

A migration script in, say, Nova, would do something like:

UPDATE instances SET owner_id = ext.user_id
FROM instances JOIN keystone.external_id_mapping ext
ON instances.owner_id = ext.external_id;

Best,
-jay 



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Jay Pipes]

On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote:
> On Mar 3, 2014, at 6:48 AM, Jay Pipes  wrote:
> 
> > On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
> >> Having done some work with MySQL (specifically around similar data
> >> sets) and discussing the changes with some former coworkers (MySQL
> >> experts) I am inclined to believe the move from varchar  to binary
> >> absolutely would increase performance like this.
> >> 
> >> 
> >> However, I would like to get some real benchmarks around it and if it
> >> really makes a difference we should get a smart "UUID" type into the
> >> common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
> >> conversation. Should be split off from the keystone one at hand - I
> >> don't want valuable information / discussions to get lost.
> > 
> > No disagreement on either point. However, this should be done after the
> > standardization to a UUID user_id in Keystone, as a separate performance
> > improvement patch. Agree?
> > 
> > Best,
> > -jay
> 
> -1
> 
> The expectation in other projects has been that project_ids and user_ids are 
> opaque strings. I need to see more clear benefit to enforcing stricter typing 
> on these, because I think it might break a lot of things.

What does Nova lose here? The proposal is to have Keystone's user_id
values be UUIDs all the time. There would be a migration or helper
script against Nova's database that would change all non-UUID user_id
values to the Keystone UUID values.

If there's stuff in Nova that would break (which is doubtful,
considering like you say, these are supposed to be "opaque values" and
as such should not have any restrictions or validation on their value),
then that is code in Nova that should be fixed.

Honestly, we shouldn't accept poor or loose code just because "stuff
might break".

-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Mark Washenberger]

On Sat, Mar 1, 2014 at 12:51 PM, Jay Pipes  wrote:

> On Fri, 2014-02-28 at 15:25 -0800, Mark Washenberger wrote:
> > I believe we have some agreement here. Other openstack services should
> > be able to use a strongly typed identifier for users. I just think if
> > we want to go that route, we probably need to create a new field to
> > act as the proper user uuid, rather than repurposing the existing
> > field. It sounds like many existing LDAP deployments would break if we
> > repurpose the existing field.
>
> Hi Mark,
>
> Please see my earlier response on this thread. I am proposing putting
> external identifiers into a mapping table that would correlate a
> Keystone UUID user ID with external identifiers (of variable length).
>

The thing you seem to be missing is that the current user-id attribute is
an "external identifier" depending on the identity backend you're using
today. For example in the LDAP driver it is the CN by default (which is
ridiculous for a large number of reasons, but let's leave those aside.) So
if you want to create a new, strongly typed internal uuid identifier that
makes the db performance scream, more power to you. But it's going to have
to be a new field.



>
> Once authentication has occurred (with any auth backend including LDAP),
> Keystone would only communicate to the other OpenStack services the UUID
> user ID from Keystone. This would indeed require a migration to each
> non-Keystone service that stores the user IDs as-is from Keystone
> currently (such as Glance or Nova).
>
> Once the migrations are run, then only UUID values would be stored, and
> further migrations could be run that would streamline the columns that
> stores these user IDs to a more efficient CHAR(32) or BINARY(16)
> internal storage format.
>
> Hope that clears things up.
>
> Best,
> -jay
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Vishvananda Ishaya]

On Mar 3, 2014, at 6:48 AM, Jay Pipes  wrote:

> On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
>> Having done some work with MySQL (specifically around similar data
>> sets) and discussing the changes with some former coworkers (MySQL
>> experts) I am inclined to believe the move from varchar  to binary
>> absolutely would increase performance like this.
>> 
>> 
>> However, I would like to get some real benchmarks around it and if it
>> really makes a difference we should get a smart "UUID" type into the
>> common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
>> conversation. Should be split off from the keystone one at hand - I
>> don't want valuable information / discussions to get lost.
> 
> No disagreement on either point. However, this should be done after the
> standardization to a UUID user_id in Keystone, as a separate performance
> improvement patch. Agree?
> 
> Best,
> -jay

-1

The expectation in other projects has been that project_ids and user_ids are 
opaque strings. I need to see more clear benefit to enforcing stricter typing 
on these, because I think it might break a lot of things.

Vish



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Dolph Mathews]

On Mon, Mar 3, 2014 at 8:48 AM, Jay Pipes  wrote:

> On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
> > Having done some work with MySQL (specifically around similar data
> > sets) and discussing the changes with some former coworkers (MySQL
> > experts) I am inclined to believe the move from varchar  to binary
> > absolutely would increase performance like this.
> >
> >
> > However, I would like to get some real benchmarks around it and if it
> > really makes a difference we should get a smart "UUID" type into the
> > common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
> > conversation. Should be split off from the keystone one at hand - I
> > don't want valuable information / discussions to get lost.
>
> No disagreement on either point. However, this should be done after the
> standardization to a UUID user_id in Keystone, as a separate performance
> improvement patch. Agree?
>
> Best,
> -jay
>
>
>
++ https://blueprints.launchpad.net/keystone/+spec/sql-id-binary-16


>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Jay Pipes]

On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
> Having done some work with MySQL (specifically around similar data
> sets) and discussing the changes with some former coworkers (MySQL
> experts) I am inclined to believe the move from varchar  to binary
> absolutely would increase performance like this.
> 
> 
> However, I would like to get some real benchmarks around it and if it
> really makes a difference we should get a smart "UUID" type into the
> common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
> conversation. Should be split off from the keystone one at hand - I
> don't want valuable information / discussions to get lost.

No disagreement on either point. However, this should be done after the
standardization to a UUID user_id in Keystone, as a separate performance
improvement patch. Agree?

Best,
-jay



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Morgan Fainberg]

Having done some work with MySQL (specifically around similar data sets)
and discussing the changes with some former coworkers (MySQL experts) I am
inclined to believe the move from varchar  to binary absolutely would
increase performance like this.

However, I would like to get some real benchmarks around it and if it
really makes a difference we should get a smart "UUID" type into the common
SQLlibs (can pgsql see a similar benefit? Db2?) I think this conversation.
Should be split off from the keystone one at hand - I don't want valuable
information / discussions to get lost.

Cheers,
Morgan.

Sent via mobile


On Sunday, March 2, 2014, Sean Dague  wrote:

> On 03/01/2014 08:00 PM, Clint Byrum wrote:
> > Excerpts from Robert Collins's message of 2014-03-01 14:26:57 -0800:
> >> On 1 March 2014 13:28, Clint Byrum >
> wrote:
> >>
> >>> +1. A Keystone record belongs to Keystone, and it should have a
> Keystone
> >>> ID. External records that are linked should be linked separately.
> >>>
> >>> It may not be obvious to everyone, but MySQL uses B-trees for indexes.
> >>> B-trees cannot have variable-length keys.
> >>
> >> Hmm, B-Trees and B+-Trees both can have variable length keys. I'll
> >> accept an assertion that MySQL index B-trees cannot - but we should be
> >> precise here, because its not a global limitation.
> >>
> >
> > Sorry, I misspoke, _InnoDB's_ b-tree's cannot have variable length keys.
> > :-P
>
> On a previous project we did a transition from varchar based UUID to
> binary based UUID in MySQL. The micro benchmarks on joins got faster by
> a factor of 10,000 (yes 10k). Granted, MySQL has evolved since then, and
> this was a micro benchmark, however this is definitely work considering.
>
> -Sean
>
> --
> Sean Dague
> Samsung Research America
> s...@dague.net  / sean.da...@samsung.com 
> http://dague.net
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Sean Dague]

On 03/01/2014 08:00 PM, Clint Byrum wrote:
> Excerpts from Robert Collins's message of 2014-03-01 14:26:57 -0800:
>> On 1 March 2014 13:28, Clint Byrum  wrote:
>>
>>> +1. A Keystone record belongs to Keystone, and it should have a Keystone
>>> ID. External records that are linked should be linked separately.
>>>
>>> It may not be obvious to everyone, but MySQL uses B-trees for indexes.
>>> B-trees cannot have variable-length keys.
>>
>> Hmm, B-Trees and B+-Trees both can have variable length keys. I'll
>> accept an assertion that MySQL index B-trees cannot - but we should be
>> precise here, because its not a global limitation.
>>
> 
> Sorry, I misspoke, _InnoDB's_ b-tree's cannot have variable length keys.
> :-P

On a previous project we did a transition from varchar based UUID to
binary based UUID in MySQL. The micro benchmarks on joins got faster by
a factor of 10,000 (yes 10k). Granted, MySQL has evolved since then, and
this was a micro benchmark, however this is definitely work considering.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Clint Byrum]

Excerpts from Robert Collins's message of 2014-03-01 14:26:57 -0800:
> On 1 March 2014 13:28, Clint Byrum  wrote:
> 
> > +1. A Keystone record belongs to Keystone, and it should have a Keystone
> > ID. External records that are linked should be linked separately.
> >
> > It may not be obvious to everyone, but MySQL uses B-trees for indexes.
> > B-trees cannot have variable-length keys.
> 
> Hmm, B-Trees and B+-Trees both can have variable length keys. I'll
> accept an assertion that MySQL index B-trees cannot - but we should be
> precise here, because its not a global limitation.
> 

Sorry, I misspoke, _InnoDB's_ b-tree's cannot have variable length keys.
:-P

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Robert Collins]

On 1 March 2014 13:28, Clint Byrum  wrote:

> +1. A Keystone record belongs to Keystone, and it should have a Keystone
> ID. External records that are linked should be linked separately.
>
> It may not be obvious to everyone, but MySQL uses B-trees for indexes.
> B-trees cannot have variable-length keys.

Hmm, B-Trees and B+-Trees both can have variable length keys. I'll
accept an assertion that MySQL index B-trees cannot - but we should be
precise here, because its not a global limitation.

That said, our DB design basically needs to take into account the
union of limitations of all target DB platforms, so we do still need
to cater for this.
-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Jay Pipes]

On Fri, 2014-02-28 at 15:25 -0800, Mark Washenberger wrote:
> I believe we have some agreement here. Other openstack services should
> be able to use a strongly typed identifier for users. I just think if
> we want to go that route, we probably need to create a new field to
> act as the proper user uuid, rather than repurposing the existing
> field. It sounds like many existing LDAP deployments would break if we
> repurpose the existing field.

Hi Mark,

Please see my earlier response on this thread. I am proposing putting
external identifiers into a mapping table that would correlate a
Keystone UUID user ID with external identifiers (of variable length). 

Once authentication has occurred (with any auth backend including LDAP),
Keystone would only communicate to the other OpenStack services the UUID
user ID from Keystone. This would indeed require a migration to each
non-Keystone service that stores the user IDs as-is from Keystone
currently (such as Glance or Nova).

Once the migrations are run, then only UUID values would be stored, and
further migrations could be run that would streamline the columns that
stores these user IDs to a more efficient CHAR(32) or BINARY(16)
internal storage format.

Hope that clears things up.

Best,
-jay 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Clint Byrum]

Excerpts from Jay Pipes's message of 2014-02-28 14:26:26 -0800:
> On Fri, 2014-02-28 at 13:10 -0800, Mark Washenberger wrote:
> > 
> > On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash
> >  wrote:
> > Hi Mark,
> > 
> > 
> > So we would not modify any existing IDs, so no migration
> > required.
> > 
> > 
> > Okay, I just want to be painfully clear--we're not proposing changing
> > any of the current restrictions on the user-id field. We will not:
> >   - require it to be a uuid
> >   - encode it as binary instead of char
> >   - shrink its size below the current 64 characters
> 
> The first would be required for the real solution. The second and third
> are performance improvements.
> 
> > Any of those could require a migration for existing IDs depending on
> > how your identity driver functions.
> 
> Personally, I think to fix this issue permanently and properly,
> migrations for database schemas of Glance and Nova would indeed need to
> accompany a proposed patch that restricts the Keystone external user ID
> to only a UUID value.
> 
> I entirely disagree with allowing non-UUID values for the user ID value
> that is exposed outside of Keystone. All other solutions (including the
> proposals to continue using the user_id fields with non-UUID values) are
> just hacks IMO.

+1. A Keystone record belongs to Keystone, and it should have a Keystone
ID. External records that are linked should be linked separately.

It may not be obvious to everyone, but MySQL uses B-trees for indexes.
B-trees cannot have variable-length keys. So varchar(64) means 64-byte
index keys. If you aren't careful and let that column be stored as utf-8,
this actually means *192* byte index keys, because MySQL uses 3-byte
utf-8 and thus a 64 "character" column could have 192 bytes. This does
not scale well as you are doing index scans and range lookups, not to
mention just generally raising memory and I/O pressure on the server.

What Jay is suggesting is that we actually be opinionated and store
Keystone users with 16-byte binary UUID's, and only ever use the UUID (in
the 32-byte text notation where appropriate) when returning a keystone ID.

Then only the initial authentication step where the user presents
external identification requires access to anything larger, allowing
all other Keystone operations to perform much better and keeping the
keystone database footprint smaller.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Mark Washenberger]

On Fri, Feb 28, 2014 at 2:26 PM, Jay Pipes  wrote:

> On Fri, 2014-02-28 at 13:10 -0800, Mark Washenberger wrote:
> >
> > On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash
> >  wrote:
> > Hi Mark,
> >
> >
> > So we would not modify any existing IDs, so no migration
> > required.
> >
> >
> > Okay, I just want to be painfully clear--we're not proposing changing
> > any of the current restrictions on the user-id field. We will not:
> >   - require it to be a uuid
> >   - encode it as binary instead of char
> >   - shrink its size below the current 64 characters
>
> The first would be required for the real solution. The second and third
> are performance improvements.
>
> > Any of those could require a migration for existing IDs depending on
> > how your identity driver functions.
>
> Personally, I think to fix this issue permanently and properly,
> migrations for database schemas of Glance and Nova would indeed need to
> accompany a proposed patch that restricts the Keystone external user ID
> to only a UUID value.
>
> I entirely disagree with allowing non-UUID values for the user ID value
> that is exposed outside of Keystone. All other solutions (including the
> proposals to continue using the user_id fields with non-UUID values) are
> just hacks IMO.
>

I believe we have some agreement here. Other openstack services should be
able to use a strongly typed identifier for users. I just think if we want
to go that route, we probably need to create a new field to act as the
proper user uuid, rather than repurposing the existing field. It sounds
like many existing LDAP deployments would break if we repurpose the
existing field.


> Best,
> -jay
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Jay Pipes]

On Fri, 2014-02-28 at 13:10 -0800, Mark Washenberger wrote:
> 
> On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash
>  wrote:
> Hi Mark,
> 
> 
> So we would not modify any existing IDs, so no migration
> required.
> 
> 
> Okay, I just want to be painfully clear--we're not proposing changing
> any of the current restrictions on the user-id field. We will not:
>   - require it to be a uuid
>   - encode it as binary instead of char
>   - shrink its size below the current 64 characters

The first would be required for the real solution. The second and third
are performance improvements.

> Any of those could require a migration for existing IDs depending on
> how your identity driver functions.

Personally, I think to fix this issue permanently and properly,
migrations for database schemas of Glance and Nova would indeed need to
accompany a proposed patch that restricts the Keystone external user ID
to only a UUID value.

I entirely disagree with allowing non-UUID values for the user ID value
that is exposed outside of Keystone. All other solutions (including the
proposals to continue using the user_id fields with non-UUID values) are
just hacks IMO.

Best,
-jay



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Mark Washenberger]

On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash wrote:

> Hi Mark,
>
> So we would not modify any existing IDs, so no migration required.
>

Okay, I just want to be painfully clear--we're not proposing changing any
of the current restrictions on the user-id field. We will not:
  - require it to be a uuid
  - encode it as binary instead of char
  - shrink its size below the current 64 characters

Any of those could require a migration for existing IDs depending on how
your identity driver functions.

If I'm just being Chicken Little, please reassure me once more and I'll be
quiet :-)



>
> Henry
>
> On 28 Feb 2014, at 17:38, Mark Washenberger <
> mark.washenber...@markwash.net> wrote:
>
>
>
>
> On Wed, Feb 26, 2014 at 5:25 AM, Dolph Mathews wrote:
>
>>
>> On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes  wrote:
>>
>>> On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
>>> > For purposes of supporting multiple backends for Identity (multiple
>>> > LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
>>> > increase the maximum size of the USER_ID field from an upper limit of
>>> > 64 to an upper limit of 255. This change would not impact any
>>> > currently assigned USER_IDs (they would remain in the old simple UUID
>>> > format), however, new USER_IDs would be increased to include the IDP
>>> > identifier (e.g. USER_ID@@IDP_IDENTIFIER).
>>>
>>> -1
>>>
>>> I think a better solution would be to have a simple translation table
>>> only in Keystone that would store this longer identifier (for folks
>>> using federation and/or LDAP) along with the Keystone user UUID that is
>>> used in foreign key relations and other mapping tables through Keystone
>>> and other projects.
>>>
>>
>> Morgan and I talked this suggestion through last night and agreed it's
>> probably the best approach, and has the benefit of zero impact on other
>> services, which is something we're obviously trying to avoid. I imagine it
>> could be as simple as a user_id to domain_id lookup table. All we really
>> care about is "given a globally unique user ID, which identity backend is
>> the user from?"
>>
>> On the downside, it would likely become bloated with unused ephemeral
>> user IDs, so we'll need enough metadata about the mapping to implement a
>> purging behavior down the line.
>>
>
> Is this approach planning on reusing the existing user-id field, then? It
> seems like this creates a migration problem for folks who are currently
> using user-ids that are generated by their identity backends.
>
>
>>
>>
>>>
>>> The only identifiers that would ever be communicated to any non-Keystone
>>> OpenStack endpoint would be the UUID user and tenant IDs.
>>>
>>> > There is the obvious concern that projects are utilizing (and storing)
>>> > the user_id in a field that cannot accommodate the increased upper
>>> > limit. Before this change is merged in, it is important for the
>>> > Keystone team to understand if there are any places that would be
>>> > overflowed by the increased size.
>>>
>>> I would go so far as to say the user_id and tenant_id fields should be
>>> *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
>>> performance reasons. Lengthening commonly-used and frequently-joined
>>> identifier fields is not a good option, IMO.
>>>
>>> Best,
>>> -jay
>>>
>>> > The review that would implement this change in size
>>> > is https://review.openstack.org/#/c/74214 and is actively being worked
>>> > on/reviewed.
>>> >
>>> >
>>> > I have already spoken with the Nova team, and a single instance has
>>> > been identified that would require a migration (that will have a fix
>>> > proposed for the I3 timeline).
>>> >
>>> >
>>> > If there are any other known locations that would have issues with an
>>> > increased USER_ID size, or any concerns with this change to USER_ID
>>> > format, please respond so that the issues/concerns can be addressed.
>>> >  Again, the plan is not to change current USER_IDs but that new ones
>>> > could be up to 255 characters in length.
>>> >
>>> >
>>> > Cheers,
>>> > Morgan Fainberg
>>> > --
>>> > Morgan Fainberg
>>> > Principal Software Engineer
>>> > Core Developer, Keystone
>>> > m...@metacloud.com
>>> >
>>> >
>>> > ___
>>> > OpenStack-dev mailing list
>>> > OpenStack-dev@lists.openstack.org
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> 

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Henry Nash]

Hi Mark,

So we would not modify any existing IDs, so no migration required.

Henry
On 28 Feb 2014, at 17:38, Mark Washenberger  
wrote:

> 
> 
> 
> On Wed, Feb 26, 2014 at 5:25 AM, Dolph Mathews  
> wrote:
> 
> On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes  wrote:
> On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
> > For purposes of supporting multiple backends for Identity (multiple
> > LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
> > increase the maximum size of the USER_ID field from an upper limit of
> > 64 to an upper limit of 255. This change would not impact any
> > currently assigned USER_IDs (they would remain in the old simple UUID
> > format), however, new USER_IDs would be increased to include the IDP
> > identifier (e.g. USER_ID@@IDP_IDENTIFIER).
> 
> -1
> 
> I think a better solution would be to have a simple translation table
> only in Keystone that would store this longer identifier (for folks
> using federation and/or LDAP) along with the Keystone user UUID that is
> used in foreign key relations and other mapping tables through Keystone
> and other projects.
> 
> Morgan and I talked this suggestion through last night and agreed it's 
> probably the best approach, and has the benefit of zero impact on other 
> services, which is something we're obviously trying to avoid. I imagine it 
> could be as simple as a user_id to domain_id lookup table. All we really care 
> about is "given a globally unique user ID, which identity backend is the user 
> from?"
> 
> On the downside, it would likely become bloated with unused ephemeral user 
> IDs, so we'll need enough metadata about the mapping to implement a purging 
> behavior down the line.
> 
> Is this approach planning on reusing the existing user-id field, then? It 
> seems like this creates a migration problem for folks who are currently using 
> user-ids that are generated by their identity backends.
>  
>  
> 
> The only identifiers that would ever be communicated to any non-Keystone
> OpenStack endpoint would be the UUID user and tenant IDs.
> 
> > There is the obvious concern that projects are utilizing (and storing)
> > the user_id in a field that cannot accommodate the increased upper
> > limit. Before this change is merged in, it is important for the
> > Keystone team to understand if there are any places that would be
> > overflowed by the increased size.
> 
> I would go so far as to say the user_id and tenant_id fields should be
> *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
> performance reasons. Lengthening commonly-used and frequently-joined
> identifier fields is not a good option, IMO.
> 
> Best,
> -jay
> 
> > The review that would implement this change in size
> > is https://review.openstack.org/#/c/74214 and is actively being worked
> > on/reviewed.
> >
> >
> > I have already spoken with the Nova team, and a single instance has
> > been identified that would require a migration (that will have a fix
> > proposed for the I3 timeline).
> >
> >
> > If there are any other known locations that would have issues with an
> > increased USER_ID size, or any concerns with this change to USER_ID
> > format, please respond so that the issues/concerns can be addressed.
> >  Again, the plan is not to change current USER_IDs but that new ones
> > could be up to 255 characters in length.
> >
> >
> > Cheers,
> > Morgan Fainberg
> > —
> > Morgan Fainberg
> > Principal Software Engineer
> > Core Developer, Keystone
> > m...@metacloud.com
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Mark Washenberger]

On Wed, Feb 26, 2014 at 5:25 AM, Dolph Mathews wrote:

>
> On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes  wrote:
>
>> On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
>> > For purposes of supporting multiple backends for Identity (multiple
>> > LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
>> > increase the maximum size of the USER_ID field from an upper limit of
>> > 64 to an upper limit of 255. This change would not impact any
>> > currently assigned USER_IDs (they would remain in the old simple UUID
>> > format), however, new USER_IDs would be increased to include the IDP
>> > identifier (e.g. USER_ID@@IDP_IDENTIFIER).
>>
>> -1
>>
>> I think a better solution would be to have a simple translation table
>> only in Keystone that would store this longer identifier (for folks
>> using federation and/or LDAP) along with the Keystone user UUID that is
>> used in foreign key relations and other mapping tables through Keystone
>> and other projects.
>>
>
> Morgan and I talked this suggestion through last night and agreed it's
> probably the best approach, and has the benefit of zero impact on other
> services, which is something we're obviously trying to avoid. I imagine it
> could be as simple as a user_id to domain_id lookup table. All we really
> care about is "given a globally unique user ID, which identity backend is
> the user from?"
>
> On the downside, it would likely become bloated with unused ephemeral user
> IDs, so we'll need enough metadata about the mapping to implement a purging
> behavior down the line.
>

Is this approach planning on reusing the existing user-id field, then? It
seems like this creates a migration problem for folks who are currently
using user-ids that are generated by their identity backends.


>
>
>>
>> The only identifiers that would ever be communicated to any non-Keystone
>> OpenStack endpoint would be the UUID user and tenant IDs.
>>
>> > There is the obvious concern that projects are utilizing (and storing)
>> > the user_id in a field that cannot accommodate the increased upper
>> > limit. Before this change is merged in, it is important for the
>> > Keystone team to understand if there are any places that would be
>> > overflowed by the increased size.
>>
>> I would go so far as to say the user_id and tenant_id fields should be
>> *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
>> performance reasons. Lengthening commonly-used and frequently-joined
>> identifier fields is not a good option, IMO.
>>
>> Best,
>> -jay
>>
>> > The review that would implement this change in size
>> > is https://review.openstack.org/#/c/74214 and is actively being worked
>> > on/reviewed.
>> >
>> >
>> > I have already spoken with the Nova team, and a single instance has
>> > been identified that would require a migration (that will have a fix
>> > proposed for the I3 timeline).
>> >
>> >
>> > If there are any other known locations that would have issues with an
>> > increased USER_ID size, or any concerns with this change to USER_ID
>> > format, please respond so that the issues/concerns can be addressed.
>> >  Again, the plan is not to change current USER_IDs but that new ones
>> > could be up to 255 characters in length.
>> >
>> >
>> > Cheers,
>> > Morgan Fainberg
>> > --
>> > Morgan Fainberg
>> > Principal Software Engineer
>> > Core Developer, Keystone
>> > m...@metacloud.com
>> >
>> >
>> > ___
>> > OpenStack-dev mailing list
>> > OpenStack-dev@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Jay Pipes]

On Thu, 2014-02-27 at 17:02 -0500, Adam Young wrote:
> On 02/27/2014 12:52 PM, Jay Pipes wrote:
> > On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
> >> So a couple of things about this:
> >>
> >>
> >> 1) Today (and also true for Grizzly and Havana), the user can chose
> >> what LDAP attribute should be returned as the user or group ID.  So it
> >> is NOT a safe assumption today (ignoring any support for
> >> domain-specific LDAP support) that the format of a user or group ID is
> >> a 32 char UUID.  Quite often, I would think, that email address would
> >> be chosen by a cloud provider as the LDAP id field, by default we use
> >> the CN.  Since we really don't want to ever change the user or group
> >> ID we have given out from keystone for a particular entity, this means
> >> we need to update nova (or anything else) that has made a 32 char
> >> assumption.
> > I don't believe this is correct. Keystone is the service that deals with
> > authentication. As such, Keystone should be the one and only one service
> > that should have any need whatsoever to need to understand a non-UUID
> > value for a user ID. The only value that should ever be communicated
> > *from* Keystone should be the UUID value of the user.
> 
> We never said it was a UUID, its only been that in the "allocated by 
> Keystone" cases thus far.

Every user record in Keystone should have an "allocated by Keystone"
user ID that is a UUID value. Why? Because every single other OpenStack
service uses UUID values for identifiers (not names), and Keystone
should be no different.

> userids are blobswell, character blocks, and are not expected to be 
> interprested by the end services, but also not confined to be 32 
> characters long.

I would argue that they absolutely *should* be confined to 32 characters
(or technically, 16 bytes, which is the size of a UUID).

> Most LDAP use cases do not allow us to write data back the Directory. 
> Federation has the same use cases. 

Frankly, I don't care :) Let's say Super-Duper Identity Driver has 100
character IDs natively. Does that mean that these IDs need be sent all
over the OpenStack ecosystem? No. Have a translation table in Keystone
and spread the UUID user ID throughout OpenStack.

>  That means that we are going to end 
> up with a slew of ephemeral data in Keystone for no real value.

Not sure why you think this is ephemeral data... What is ephemeral about
a non-UUID identifier from another auth system (federated or not) that
is mapped to a Keystone user UUID? If you look at virtually every
flexible authentication mechanism in virtually all web frameworks out
there, you will see something very similar, where there is a table that
maps external IDs to the IDs that are native to some platform.

Best,
-jay

> >
> > If the Keystone service uses LDAP or federation for alternative
> > authentication schemes, then Keystone should have a mapping table that
> > translates those elongated and non-UUID identifiers values (email
> > addresses, LDAP CNs, etc) into the UUID value that is then communicated
> > to all other OpenStack services.
> The short of it is that we need to define a max userid length and stick 
> to it.   Thus far, we've had 64 characters in Keystone, and I think we 
> can stick with that.  Nova needs to expand its view to 64 chars.
> 
> The rest is Keystone internals to deal with.
> 
> >
> > Best,
> > -jay
> >
> >> 2) In oder to support the ability for service providers to be able to
> >> have the identity part of keystone be satisfied by a customer LDAP
> >> (i.e. for a given domain, have a specific LDAP), then, as has been
> >> stated, we need to subsequently, when handed an API call with just a
> >> user or group ID, be able to "route" this call to the correct LDAP.
> >>   Trying to keep true to the openstack design principles, we had
> >> planned to encode a domain identifier into the user or group ID - i.e.
> >> distribute the data to where it is needed, in ortherwords, the user
> >> and group ID provide all the info we need to route the call to the
> >> right place. Two implementations come to mind:
> >> 2a) Simply concatenate the user/group ID with the domain_id, plus some
> >> separator and make a composite public facing ID.  e.g.
> >> "user_entity_id@@UUID_of_domain".  This would have a technical maximum
> >> size of 64+2+64 (i.e. 130), although in reality since we control
> >> domain_id and we know it is always 32 char UUID - in fact the max size
> >> would be 98.  This has the problem of increasing the size of the
> >> public facing field beyond the existing 64.  This is what we had
> >> planned for IceHouse - and is currently in review.
> >> 2b) Use a similar concatenation idea as 2a), but limit the total size
> >> to the existing 64. Since we control domain_id, we could (internally
> >> and not visibly to the outside world), create a domain_index, that was
> >> used in place of domain_id in the publicly visible field, to minimize
> >> the number of chars it requ

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Adam Young]

On 02/27/2014 12:52 PM, Jay Pipes wrote:

On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:

So a couple of things about this:


1) Today (and also true for Grizzly and Havana), the user can chose
what LDAP attribute should be returned as the user or group ID.  So it
is NOT a safe assumption today (ignoring any support for
domain-specific LDAP support) that the format of a user or group ID is
a 32 char UUID.  Quite often, I would think, that email address would
be chosen by a cloud provider as the LDAP id field, by default we use
the CN.  Since we really don't want to ever change the user or group
ID we have given out from keystone for a particular entity, this means
we need to update nova (or anything else) that has made a 32 char
assumption.

I don't believe this is correct. Keystone is the service that deals with
authentication. As such, Keystone should be the one and only one service
that should have any need whatsoever to need to understand a non-UUID
value for a user ID. The only value that should ever be communicated
*from* Keystone should be the UUID value of the user.


We never said it was a UUID, its only been that in the "allocated by 
Keystone" cases thus far.


userids are blobswell, character blocks, and are not expected to be 
interprested by the end services, but also not confined to be 32 
characters long.


Most LDAP use cases do not allow us to write data back the Directory. 
Federation has the same use cases.  That means that we are going to end 
up with a slew of ephemeral data in Keystone for no real value.






If the Keystone service uses LDAP or federation for alternative
authentication schemes, then Keystone should have a mapping table that
translates those elongated and non-UUID identifiers values (email
addresses, LDAP CNs, etc) into the UUID value that is then communicated
to all other OpenStack services.
The short of it is that we need to define a max userid length and stick 
to it.   Thus far, we've had 64 characters in Keystone, and I think we 
can stick with that.  Nova needs to expand its view to 64 chars.


The rest is Keystone internals to deal with.



Best,
-jay


2) In oder to support the ability for service providers to be able to
have the identity part of keystone be satisfied by a customer LDAP
(i.e. for a given domain, have a specific LDAP), then, as has been
stated, we need to subsequently, when handed an API call with just a
user or group ID, be able to "route" this call to the correct LDAP.
  Trying to keep true to the openstack design principles, we had
planned to encode a domain identifier into the user or group ID - i.e.
distribute the data to where it is needed, in ortherwords, the user
and group ID provide all the info we need to route the call to the
right place. Two implementations come to mind:
2a) Simply concatenate the user/group ID with the domain_id, plus some
separator and make a composite public facing ID.  e.g.
"user_entity_id@@UUID_of_domain".  This would have a technical maximum
size of 64+2+64 (i.e. 130), although in reality since we control
domain_id and we know it is always 32 char UUID - in fact the max size
would be 98.  This has the problem of increasing the size of the
public facing field beyond the existing 64.  This is what we had
planned for IceHouse - and is currently in review.
2b) Use a similar concatenation idea as 2a), but limit the total size
to the existing 64. Since we control domain_id, we could (internally
and not visibly to the outside world), create a domain_index, that was
used in place of domain_id in the publicly visible field, to minimize
the number of chars it requires.  So the public facing composite ID
might be something like @@<8 chars of
domain_index>.  There is a chance, of course, that  the 54 char
restriction might be problematic for LDAP usersbut I doubt it.  We
would make that a restriction and if it really became a problem, we
could consider a field size increase at a later release
3) The alternative to 2a and 2b is to have, as had been suggested, an
internal mapping table that maps external facing entity_ids to a
domain plus local entity ID.  The problem with this idea is that:
- This could become a very big table (you will essentially have an
entry for every user in every corporate LDAP that has accessed a given
openstack)
- Since most LDAPs are RO, we will never see deletes...so we won't
know when (without some kind of garbage collection) to cull entries
- It obviously does not solve 1) - since existing LDAP support can
break the 32 char limit - and so it isn't true that this mapping table
causes all public facing entity IDs to be simple 32 char UUIDs


 From a delivery into IceHouse point of view any of the above are
possible, since the actual mapping used is relatively small part of
the patch.  I personally favor 2b), since it is simple, has "less
moving parts" and does not change any external facing requirements for
storage of user and group IDs (above and beyond what is true today).


He

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Miller, Mark M (EB SW Cloud - R&D - Corvallis)]

I agree about not needing extra identity information outside of the user's 
UUID, but what about the role/project/domain information stored in the PKI 
token? Does it remain or go away?

From: Morgan Fainberg [mailto:m...@metacloud.com]
Sent: Thursday, February 27, 2014 12:11 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum 
from 64 to 255



On Thursday, February 27, 2014, Dolph Mathews 
mailto:dolph.math...@gmail.com>> wrote:

On Thu, Feb 27, 2014 at 11:52 AM, Jay Pipes 
> wrote:
On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
> So a couple of things about this:
>
>
> 1) Today (and also true for Grizzly and Havana), the user can chose
> what LDAP attribute should be returned as the user or group ID.  So it
> is NOT a safe assumption today (ignoring any support for
> domain-specific LDAP support) that the format of a user or group ID is
> a 32 char UUID.  Quite often, I would think, that email address would
> be chosen by a cloud provider as the LDAP id field, by default we use
> the CN.  Since we really don't want to ever change the user or group
> ID we have given out from keystone for a particular entity, this means
> we need to update nova (or anything else) that has made a 32 char
> assumption.
I don't believe this is correct. Keystone is the service that deals with
authentication. As such, Keystone should be the one and only one service
that should have any need whatsoever to need to understand a non-UUID
value for a user ID. The only value that should ever be communicated
*from* Keystone should be the UUID value of the user.

+++


If the Keystone service uses LDAP or federation for alternative
authentication schemes, then Keystone should have a mapping table that
translates those elongated and non-UUID identifiers values (email
addresses, LDAP CNs, etc) into the UUID value that is then communicated
to all other OpenStack services.

I'd take it one step further and say that at some point, keystone should stop 
leaking identity details such as user name / ID into OpenStack (they shouldn't 
appear in tokens, and shouldn't be expected output of auth_token). The use 
cases that "require" them are weak and would be better served by pure 
multitenant RBAC, ABAC, etc. There are a lot of blockers to making this happen 
(including a few in keystone's own API), but still food for thought.

++ this would be a great change!

I am on the same page as Dolph when it comes to approving of the UUID being the 
only value communicated outside of keystone. There is just no good reason to 
send out extra identity information (it isn't needed and can help to reduce 
token bloat etc).

--Morgan

Sent via mobile
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Morgan Fainberg]

On Thursday, February 27, 2014, Dolph Mathews 
wrote:

>
> On Thu, Feb 27, 2014 at 11:52 AM, Jay Pipes 
> 
> > wrote:
>
>> On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
>> > So a couple of things about this:
>> >
>> >
>> > 1) Today (and also true for Grizzly and Havana), the user can chose
>> > what LDAP attribute should be returned as the user or group ID.  So it
>> > is NOT a safe assumption today (ignoring any support for
>> > domain-specific LDAP support) that the format of a user or group ID is
>> > a 32 char UUID.  Quite often, I would think, that email address would
>> > be chosen by a cloud provider as the LDAP id field, by default we use
>> > the CN.  Since we really don't want to ever change the user or group
>> > ID we have given out from keystone for a particular entity, this means
>> > we need to update nova (or anything else) that has made a 32 char
>> > assumption.
>>
>> I don't believe this is correct. Keystone is the service that deals with
>> authentication. As such, Keystone should be the one and only one service
>> that should have any need whatsoever to need to understand a non-UUID
>> value for a user ID. The only value that should ever be communicated
>> *from* Keystone should be the UUID value of the user.
>>
>
> +++
>
>
>>
>> If the Keystone service uses LDAP or federation for alternative
>> authentication schemes, then Keystone should have a mapping table that
>> translates those elongated and non-UUID identifiers values (email
>> addresses, LDAP CNs, etc) into the UUID value that is then communicated
>> to all other OpenStack services.
>>
>
> I'd take it one step further and say that at some point, keystone should
> stop leaking identity details such as user name / ID into OpenStack (they
> shouldn't appear in tokens, and shouldn't be expected output of
> auth_token). The use cases that "require" them are weak and would be better
> served by pure multitenant RBAC, ABAC, etc. There are a lot of blockers to
> making this happen (including a few in keystone's own API), but still food
> for thought.
>
>
++ this would be a great change!

I am on the same page as Dolph when it comes to approving of the UUID being
the only value communicated outside of keystone. There is just no good
reason to send out extra identity information (it isn't needed and can help
to reduce token bloat etc).

--Morgan

Sent via mobile
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Dolph Mathews]

On Thu, Feb 27, 2014 at 11:52 AM, Jay Pipes  wrote:

> On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
> > So a couple of things about this:
> >
> >
> > 1) Today (and also true for Grizzly and Havana), the user can chose
> > what LDAP attribute should be returned as the user or group ID.  So it
> > is NOT a safe assumption today (ignoring any support for
> > domain-specific LDAP support) that the format of a user or group ID is
> > a 32 char UUID.  Quite often, I would think, that email address would
> > be chosen by a cloud provider as the LDAP id field, by default we use
> > the CN.  Since we really don't want to ever change the user or group
> > ID we have given out from keystone for a particular entity, this means
> > we need to update nova (or anything else) that has made a 32 char
> > assumption.
>
> I don't believe this is correct. Keystone is the service that deals with
> authentication. As such, Keystone should be the one and only one service
> that should have any need whatsoever to need to understand a non-UUID
> value for a user ID. The only value that should ever be communicated
> *from* Keystone should be the UUID value of the user.
>

+++


>
> If the Keystone service uses LDAP or federation for alternative
> authentication schemes, then Keystone should have a mapping table that
> translates those elongated and non-UUID identifiers values (email
> addresses, LDAP CNs, etc) into the UUID value that is then communicated
> to all other OpenStack services.
>

I'd take it one step further and say that at some point, keystone should
stop leaking identity details such as user name / ID into OpenStack (they
shouldn't appear in tokens, and shouldn't be expected output of
auth_token). The use cases that "require" them are weak and would be better
served by pure multitenant RBAC, ABAC, etc. There are a lot of blockers to
making this happen (including a few in keystone's own API), but still food
for thought.


>
> Best,
> -jay
>
> > 2) In oder to support the ability for service providers to be able to
> > have the identity part of keystone be satisfied by a customer LDAP
> > (i.e. for a given domain, have a specific LDAP), then, as has been
> > stated, we need to subsequently, when handed an API call with just a
> > user or group ID, be able to "route" this call to the correct LDAP.
> >  Trying to keep true to the openstack design principles, we had
> > planned to encode a domain identifier into the user or group ID - i.e.
> > distribute the data to where it is needed, in ortherwords, the user
> > and group ID provide all the info we need to route the call to the
> > right place. Two implementations come to mind:
> > 2a) Simply concatenate the user/group ID with the domain_id, plus some
> > separator and make a composite public facing ID.  e.g.
> > "user_entity_id@@UUID_of_domain".  This would have a technical maximum
> > size of 64+2+64 (i.e. 130), although in reality since we control
> > domain_id and we know it is always 32 char UUID - in fact the max size
> > would be 98.  This has the problem of increasing the size of the
> > public facing field beyond the existing 64.  This is what we had
> > planned for IceHouse - and is currently in review.
> > 2b) Use a similar concatenation idea as 2a), but limit the total size
> > to the existing 64. Since we control domain_id, we could (internally
> > and not visibly to the outside world), create a domain_index, that was
> > used in place of domain_id in the publicly visible field, to minimize
> > the number of chars it requires.  So the public facing composite ID
> > might be something like @@<8 chars of
> > domain_index>.  There is a chance, of course, that  the 54 char
> > restriction might be problematic for LDAP usersbut I doubt it.  We
> > would make that a restriction and if it really became a problem, we
> > could consider a field size increase at a later release
> > 3) The alternative to 2a and 2b is to have, as had been suggested, an
> > internal mapping table that maps external facing entity_ids to a
> > domain plus local entity ID.  The problem with this idea is that:
> > - This could become a very big table (you will essentially have an
> > entry for every user in every corporate LDAP that has accessed a given
> > openstack)
> > - Since most LDAPs are RO, we will never see deletes...so we won't
> > know when (without some kind of garbage collection) to cull entries
> > - It obviously does not solve 1) - since existing LDAP support can
> > break the 32 char limit - and so it isn't true that this mapping table
> > causes all public facing entity IDs to be simple 32 char UUIDs
> >
> >
> > From a delivery into IceHouse point of view any of the above are
> > possible, since the actual mapping used is relatively small part of
> > the patch.  I personally favor 2b), since it is simple, has "less
> > moving parts" and does not change any external facing requirements for
> > storage of user and group IDs (above and beyond what is true today).
> 

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Henry Nash]

On 27 Feb 2014, at 17:52, Jay Pipes  wrote:

> On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
>> So a couple of things about this:
>> 
>> 
>> 1) Today (and also true for Grizzly and Havana), the user can chose
>> what LDAP attribute should be returned as the user or group ID.  So it
>> is NOT a safe assumption today (ignoring any support for
>> domain-specific LDAP support) that the format of a user or group ID is
>> a 32 char UUID.  Quite often, I would think, that email address would
>> be chosen by a cloud provider as the LDAP id field, by default we use
>> the CN.  Since we really don't want to ever change the user or group
>> ID we have given out from keystone for a particular entity, this means
>> we need to update nova (or anything else) that has made a 32 char
>> assumption.
> 
> I don't believe this is correct. Keystone is the service that deals with
> authentication. As such, Keystone should be the one and only one service
> that should have any need whatsoever to need to understand a non-UUID
> value for a user ID. The only value that should ever be communicated
> *from* Keystone should be the UUID value of the user.
> 
> If the Keystone service uses LDAP or federation for alternative
> authentication schemes, then Keystone should have a mapping table that
> translates those elongated and non-UUID identifiers values (email
> addresses, LDAP CNs, etc) into the UUID value that is then communicated
> to all other OpenStack services.
> 
> Best,
> -jay
> 
So I think that's a perfectly reasonable point of viewour challenge is that 
this isn't what
Keystone has done to datee.g. anyone using a RO LDAP today is probably 
exposing
non-UUID identifiers out into nova and other projects (and maybe outside of 
openstack
altogether).  We can't (without breaking them) just change the IDs for any 
existing LDAP
entities.  So they best we could do is to say something like, new entities (and 
perhaps only
those in domain-specific backends) would use such a mapping capability.

Henry
>> 2) In oder to support the ability for service providers to be able to
>> have the identity part of keystone be satisfied by a customer LDAP
>> (i.e. for a given domain, have a specific LDAP), then, as has been
>> stated, we need to subsequently, when handed an API call with just a
>> user or group ID, be able to "route" this call to the correct LDAP.
>> Trying to keep true to the openstack design principles, we had
>> planned to encode a domain identifier into the user or group ID - i.e.
>> distribute the data to where it is needed, in ortherwords, the user
>> and group ID provide all the info we need to route the call to the
>> right place. Two implementations come to mind:
>> 2a) Simply concatenate the user/group ID with the domain_id, plus some
>> separator and make a composite public facing ID.  e.g.
>> "user_entity_id@@UUID_of_domain".  This would have a technical maximum
>> size of 64+2+64 (i.e. 130), although in reality since we control
>> domain_id and we know it is always 32 char UUID - in fact the max size
>> would be 98.  This has the problem of increasing the size of the
>> public facing field beyond the existing 64.  This is what we had
>> planned for IceHouse - and is currently in review.
>> 2b) Use a similar concatenation idea as 2a), but limit the total size
>> to the existing 64. Since we control domain_id, we could (internally
>> and not visibly to the outside world), create a domain_index, that was
>> used in place of domain_id in the publicly visible field, to minimize
>> the number of chars it requires.  So the public facing composite ID
>> might be something like @@<8 chars of
>> domain_index>.  There is a chance, of course, that  the 54 char
>> restriction might be problematic for LDAP usersbut I doubt it.  We
>> would make that a restriction and if it really became a problem, we
>> could consider a field size increase at a later release
>> 3) The alternative to 2a and 2b is to have, as had been suggested, an
>> internal mapping table that maps external facing entity_ids to a
>> domain plus local entity ID.  The problem with this idea is that:
>> - This could become a very big table (you will essentially have an
>> entry for every user in every corporate LDAP that has accessed a given
>> openstack)
>> - Since most LDAPs are RO, we will never see deletes...so we won't
>> know when (without some kind of garbage collection) to cull entries
>> - It obviously does not solve 1) - since existing LDAP support can
>> break the 32 char limit - and so it isn't true that this mapping table
>> causes all public facing entity IDs to be simple 32 char UUIDs
>> 
>> 
>> From a delivery into IceHouse point of view any of the above are
>> possible, since the actual mapping used is relatively small part of
>> the patch.  I personally favor 2b), since it is simple, has "less
>> moving parts" and does not change any external facing requirements for
>> storage of user and group IDs (above and beyond what is true t

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Jay Pipes]

On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
> So a couple of things about this:
> 
> 
> 1) Today (and also true for Grizzly and Havana), the user can chose
> what LDAP attribute should be returned as the user or group ID.  So it
> is NOT a safe assumption today (ignoring any support for
> domain-specific LDAP support) that the format of a user or group ID is
> a 32 char UUID.  Quite often, I would think, that email address would
> be chosen by a cloud provider as the LDAP id field, by default we use
> the CN.  Since we really don't want to ever change the user or group
> ID we have given out from keystone for a particular entity, this means
> we need to update nova (or anything else) that has made a 32 char
> assumption.

I don't believe this is correct. Keystone is the service that deals with
authentication. As such, Keystone should be the one and only one service
that should have any need whatsoever to need to understand a non-UUID
value for a user ID. The only value that should ever be communicated
*from* Keystone should be the UUID value of the user.

If the Keystone service uses LDAP or federation for alternative
authentication schemes, then Keystone should have a mapping table that
translates those elongated and non-UUID identifiers values (email
addresses, LDAP CNs, etc) into the UUID value that is then communicated
to all other OpenStack services.

Best,
-jay

> 2) In oder to support the ability for service providers to be able to
> have the identity part of keystone be satisfied by a customer LDAP
> (i.e. for a given domain, have a specific LDAP), then, as has been
> stated, we need to subsequently, when handed an API call with just a
> user or group ID, be able to "route" this call to the correct LDAP.
>  Trying to keep true to the openstack design principles, we had
> planned to encode a domain identifier into the user or group ID - i.e.
> distribute the data to where it is needed, in ortherwords, the user
> and group ID provide all the info we need to route the call to the
> right place. Two implementations come to mind:
> 2a) Simply concatenate the user/group ID with the domain_id, plus some
> separator and make a composite public facing ID.  e.g.
> "user_entity_id@@UUID_of_domain".  This would have a technical maximum
> size of 64+2+64 (i.e. 130), although in reality since we control
> domain_id and we know it is always 32 char UUID - in fact the max size
> would be 98.  This has the problem of increasing the size of the
> public facing field beyond the existing 64.  This is what we had
> planned for IceHouse - and is currently in review.
> 2b) Use a similar concatenation idea as 2a), but limit the total size
> to the existing 64. Since we control domain_id, we could (internally
> and not visibly to the outside world), create a domain_index, that was
> used in place of domain_id in the publicly visible field, to minimize
> the number of chars it requires.  So the public facing composite ID
> might be something like @@<8 chars of
> domain_index>.  There is a chance, of course, that  the 54 char
> restriction might be problematic for LDAP usersbut I doubt it.  We
> would make that a restriction and if it really became a problem, we
> could consider a field size increase at a later release
> 3) The alternative to 2a and 2b is to have, as had been suggested, an
> internal mapping table that maps external facing entity_ids to a
> domain plus local entity ID.  The problem with this idea is that:
> - This could become a very big table (you will essentially have an
> entry for every user in every corporate LDAP that has accessed a given
> openstack)
> - Since most LDAPs are RO, we will never see deletes...so we won't
> know when (without some kind of garbage collection) to cull entries
> - It obviously does not solve 1) - since existing LDAP support can
> break the 32 char limit - and so it isn't true that this mapping table
> causes all public facing entity IDs to be simple 32 char UUIDs
> 
> 
> From a delivery into IceHouse point of view any of the above are
> possible, since the actual mapping used is relatively small part of
> the patch.  I personally favor 2b), since it is simple, has "less
> moving parts" and does not change any external facing requirements for
> storage of user and group IDs (above and beyond what is true today).
> 
> 
> Henry
> On 27 Feb 2014, at 03:46, Adam Young  wrote:
> 
> > On 02/26/2014 08:25 AM, Dolph Mathews wrote:
> > 
> > > 
> > > On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes 
> > > wrote:
> > > On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
> > > > For purposes of supporting multiple backends for
> > > Identity (multiple
> > > > LDAP, mix of LDAP and SQL, federation, etc) Keystone is
> > > planning to
> > > > increase the maximum size of the USER_ID field from an
> > > upper limit of
> > > > 64 to an upper limit of 255. This change would not
> > > impact any
> > > >

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Henry Nash]

So a couple of things about this:

1) Today (and also true for Grizzly and Havana), the user can chose what LDAP 
attribute should be returned as the user or group ID.  So it is NOT a safe 
assumption today (ignoring any support for domain-specific LDAP support) that 
the format of a user or group ID is a 32 char UUID.  Quite often, I would 
think, that email address would be chosen by a cloud provider as the LDAP id 
field, by default we use the CN.  Since we really don't want to ever change the 
user or group ID we have given out from keystone for a particular entity, this 
means we need to update nova (or anything else) that has made a 32 char 
assumption.
2) In oder to support the ability for service providers to be able to have the 
identity part of keystone be satisfied by a customer LDAP (i.e. for a given 
domain, have a specific LDAP), then, as has been stated, we need to 
subsequently, when handed an API call with just a user or group ID, be able to 
"route" this call to the correct LDAP.  Trying to keep true to the openstack 
design principles, we had planned to encode a domain identifier into the user 
or group ID - i.e. distribute the data to where it is needed, in ortherwords, 
the user and group ID provide all the info we need to route the call to the 
right place. Two implementations come to mind:
2a) Simply concatenate the user/group ID with the domain_id, plus some 
separator and make a composite public facing ID.  e.g. 
"user_entity_id@@UUID_of_domain".  This would have a technical maximum size of 
64+2+64 (i.e. 130), although in reality since we control domain_id and we know 
it is always 32 char UUID - in fact the max size would be 98.  This has the 
problem of increasing the size of the public facing field beyond the existing 
64.  This is what we had planned for IceHouse - and is currently in review.
2b) Use a similar concatenation idea as 2a), but limit the total size to the 
existing 64. Since we control domain_id, we could (internally and not visibly 
to the outside world), create a domain_index, that was used in place of 
domain_id in the publicly visible field, to minimize the number of chars it 
requires.  So the public facing composite ID might be something like @@<8 chars of domain_index>.  There is a chance, of course, 
that  the 54 char restriction might be problematic for LDAP usersbut I 
doubt it.  We would make that a restriction and if it really became a problem, 
we could consider a field size increase at a later release
3) The alternative to 2a and 2b is to have, as had been suggested, an internal 
mapping table that maps external facing entity_ids to a domain plus local 
entity ID.  The problem with this idea is that:
- This could become a very big table (you will essentially have an entry for 
every user in every corporate LDAP that has accessed a given openstack)
- Since most LDAPs are RO, we will never see deletes...so we won't know when 
(without some kind of garbage collection) to cull entries
- It obviously does not solve 1) - since existing LDAP support can break the 32 
char limit - and so it isn't true that this mapping table causes all public 
facing entity IDs to be simple 32 char UUIDs

From a delivery into IceHouse point of view any of the above are possible, 
since the actual mapping used is relatively small part of the patch.  I 
personally favor 2b), since it is simple, has "less moving parts" and does not 
change any external facing requirements for storage of user and group IDs 
(above and beyond what is true today).

Henry
On 27 Feb 2014, at 03:46, Adam Young  wrote:

> On 02/26/2014 08:25 AM, Dolph Mathews wrote:
>> 
>> On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes  wrote:
>> On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
>> > For purposes of supporting multiple backends for Identity (multiple
>> > LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
>> > increase the maximum size of the USER_ID field from an upper limit of
>> > 64 to an upper limit of 255. This change would not impact any
>> > currently assigned USER_IDs (they would remain in the old simple UUID
>> > format), however, new USER_IDs would be increased to include the IDP
>> > identifier (e.g. USER_ID@@IDP_IDENTIFIER).
>> 
>> -1
>> 
>> I think a better solution would be to have a simple translation table
>> only in Keystone that would store this longer identifier (for folks
>> using federation and/or LDAP) along with the Keystone user UUID that is
>> used in foreign key relations and other mapping tables through Keystone
>> and other projects.
>> 
>> Morgan and I talked this suggestion through last night and agreed it's 
>> probably the best approach, and has the benefit of zero impact on other 
>> services, which is something we're obviously trying to avoid. I imagine it 
>> could be as simple as a user_id to domain_id lookup table. All we really 
>> care about is "given a globally unique user ID, which identity backend is 
>> the user from?"
>> 
>> On the 

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Adam Young]

On 02/26/2014 08:25 AM, Dolph Mathews wrote:


On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes > wrote:


On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
> For purposes of supporting multiple backends for Identity (multiple
> LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
> increase the maximum size of the USER_ID field from an upper
limit of
> 64 to an upper limit of 255. This change would not impact any
> currently assigned USER_IDs (they would remain in the old simple
UUID
> format), however, new USER_IDs would be increased to include the IDP
> identifier (e.g. USER_ID@@IDP_IDENTIFIER).

-1

I think a better solution would be to have a simple translation table
only in Keystone that would store this longer identifier (for folks
using federation and/or LDAP) along with the Keystone user UUID
that is
used in foreign key relations and other mapping tables through
Keystone
and other projects.


Morgan and I talked this suggestion through last night and agreed it's 
probably the best approach, and has the benefit of zero impact on 
other services, which is something we're obviously trying to avoid. I 
imagine it could be as simple as a user_id to domain_id lookup table. 
All we really care about is "given a globally unique user ID, which 
identity backend is the user from?"


On the downside, it would likely become bloated with unused ephemeral 
user IDs, so we'll need enough metadata about the mapping to implement 
a purging behavior down the line.
UUIDs are 32 chars long.  Its really just uuid@@uuid that pushes us over 
the 64 character limit.
If we can shorten up the IDP_ID we can fit everything in 64 chars (which 
means only Nova needs to expand its column size)


What if we enumerated IDPs by index, from 1000 to  or 
something comparable, and then use the new domain_index (or prot domain 
id to not be a uuid).  Then the above scheme would work and no migration 
would be required.





The only identifiers that would ever be communicated to any
non-Keystone
OpenStack endpoint would be the UUID user and tenant IDs.

> There is the obvious concern that projects are utilizing (and
storing)
> the user_id in a field that cannot accommodate the increased upper
> limit. Before this change is merged in, it is important for the
> Keystone team to understand if there are any places that would be
> overflowed by the increased size.

I would go so far as to say the user_id and tenant_id fields should be
*reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
performance reasons. Lengthening commonly-used and frequently-joined
identifier fields is not a good option, IMO.

Best,
-jay

> The review that would implement this change in size
> is https://review.openstack.org/#/c/74214 and is actively being
worked
> on/reviewed.
>
>
> I have already spoken with the Nova team, and a single instance has
> been identified that would require a migration (that will have a fix
> proposed for the I3 timeline).
>
>
> If there are any other known locations that would have issues
with an
> increased USER_ID size, or any concerns with this change to USER_ID
> format, please respond so that the issues/concerns can be addressed.
>  Again, the plan is not to change current USER_IDs but that new ones
> could be up to 255 characters in length.
>
>
> Cheers,
> Morgan Fainberg
> ---
> Morgan Fainberg
> Principal Software Engineer
> Core Developer, Keystone
> m...@metacloud.com 
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org

> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Dolph Mathews]

On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes  wrote:

> On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
> > For purposes of supporting multiple backends for Identity (multiple
> > LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
> > increase the maximum size of the USER_ID field from an upper limit of
> > 64 to an upper limit of 255. This change would not impact any
> > currently assigned USER_IDs (they would remain in the old simple UUID
> > format), however, new USER_IDs would be increased to include the IDP
> > identifier (e.g. USER_ID@@IDP_IDENTIFIER).
>
> -1
>
> I think a better solution would be to have a simple translation table
> only in Keystone that would store this longer identifier (for folks
> using federation and/or LDAP) along with the Keystone user UUID that is
> used in foreign key relations and other mapping tables through Keystone
> and other projects.
>

Morgan and I talked this suggestion through last night and agreed it's
probably the best approach, and has the benefit of zero impact on other
services, which is something we're obviously trying to avoid. I imagine it
could be as simple as a user_id to domain_id lookup table. All we really
care about is "given a globally unique user ID, which identity backend is
the user from?"

On the downside, it would likely become bloated with unused ephemeral user
IDs, so we'll need enough metadata about the mapping to implement a purging
behavior down the line.


>
> The only identifiers that would ever be communicated to any non-Keystone
> OpenStack endpoint would be the UUID user and tenant IDs.
>
> > There is the obvious concern that projects are utilizing (and storing)
> > the user_id in a field that cannot accommodate the increased upper
> > limit. Before this change is merged in, it is important for the
> > Keystone team to understand if there are any places that would be
> > overflowed by the increased size.
>
> I would go so far as to say the user_id and tenant_id fields should be
> *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
> performance reasons. Lengthening commonly-used and frequently-joined
> identifier fields is not a good option, IMO.
>
> Best,
> -jay
>
> > The review that would implement this change in size
> > is https://review.openstack.org/#/c/74214 and is actively being worked
> > on/reviewed.
> >
> >
> > I have already spoken with the Nova team, and a single instance has
> > been identified that would require a migration (that will have a fix
> > proposed for the I3 timeline).
> >
> >
> > If there are any other known locations that would have issues with an
> > increased USER_ID size, or any concerns with this change to USER_ID
> > format, please respond so that the issues/concerns can be addressed.
> >  Again, the plan is not to change current USER_IDs but that new ones
> > could be up to 255 characters in length.
> >
> >
> > Cheers,
> > Morgan Fainberg
> > —
> > Morgan Fainberg
> > Principal Software Engineer
> > Core Developer, Keystone
> > m...@metacloud.com
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Dolph Mathews]

On Wed, Feb 26, 2014 at 4:23 AM, Marco Fargetta
wrote:

> Hi Morgan,
>
> On Tue, Feb 25, 2014 at 11:47:43AM -0800, Morgan Fainberg wrote:
> > For purposes of supporting multiple backends for Identity (multiple
> LDAP, mix
> > of LDAP and SQL, federation, etc) Keystone is planning to increase the
> maximum
> > size of the USER_ID field from an upper limit of 64 to an upper limit of
> 255.
> > This change would not impact any currently assigned USER_IDs (they would
> remain
> > in the old simple UUID format), however, new USER_IDs would be increased
> to
> > include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER).
> >
>
> in this case if a user would access with different systems (e.g. SAML with
> portal, LDAP with CLI) it is mapped to two different identities inside
> keystone.
> Is this correct? If so, is there any way to map an individual person with
> two

identities sharing resources?
>

That's correct - they'd result in different identities and keystone has no
means of presuming they are the same "person." But I think you answered it
yourself, in that they would effectively be sharing resources with
themselves, so you'd just have to ensure they had the same authorization on
the same projects using both identities.


>
> Cheers,
> Marco
>
>
> > There is the obvious concern that projects are utilizing (and storing)
> the
> > user_id in a field that cannot accommodate the increased upper limit.
> Before
> > this change is merged in, it is important for the Keystone team to
> understand
> > if there are any places that would be overflowed by the increased size.
> >
> > The review that would implement this change in size is https://
> > review.openstack.org/#/c/74214 and is actively being worked on/reviewed.
> >
> > I have already spoken with the Nova team, and a single instance has been
> > identified that would require a migration (that will have a fix proposed
> for
> > the I3 timeline).
> >
> > If there are any other known locations that would have issues with an
> increased
> > USER_ID size, or any concerns with this change to USER_ID format, please
> > respond so that the issues/concerns can be addressed.  Again, the plan
> is not
> > to change current USER_IDs but that new ones could be up to 255
> characters in
> > length.
> >
> > Cheers,
> > Morgan Fainberg
> >
> > —
> > Morgan Fainberg
> > Principal Software Engineer
> > Core Developer, Keystone
> > m...@metacloud.com
> >
>
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> --
> 
> Eng. Marco Fargetta, PhD
>
> Istituto Nazionale di Fisica Nucleare (INFN)
> Catania, Italy
>
> EMail: marco.farge...@ct.infn.it
> 
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Marco Fargetta]

Hi Morgan,

On Tue, Feb 25, 2014 at 11:47:43AM -0800, Morgan Fainberg wrote:
> For purposes of supporting multiple backends for Identity (multiple LDAP, mix
> of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum
> size of the USER_ID field from an upper limit of 64 to an upper limit of 255.
> This change would not impact any currently assigned USER_IDs (they would 
> remain
> in the old simple UUID format), however, new USER_IDs would be increased to
> include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). 
> 

in this case if a user would access with different systems (e.g. SAML with
portal, LDAP with CLI) it is mapped to two different identities inside keystone.
Is this correct? If so, is there any way to map an individual person with two
identities sharing resources?

Cheers,
Marco


> There is the obvious concern that projects are utilizing (and storing) the
> user_id in a field that cannot accommodate the increased upper limit. Before
> this change is merged in, it is important for the Keystone team to understand
> if there are any places that would be overflowed by the increased size.
> 
> The review that would implement this change in size is https://
> review.openstack.org/#/c/74214 and is actively being worked on/reviewed.
> 
> I have already spoken with the Nova team, and a single instance has been
> identified that would require a migration (that will have a fix proposed for
> the I3 timeline). 
> 
> If there are any other known locations that would have issues with an 
> increased
> USER_ID size, or any concerns with this change to USER_ID format, please
> respond so that the issues/concerns can be addressed.  Again, the plan is not
> to change current USER_IDs but that new ones could be up to 255 characters in
> length.
> 
> Cheers,
> Morgan Fainberg
> 
> —
> Morgan Fainberg
> Principal Software Engineer
> Core Developer, Keystone
> m...@metacloud.com
> 

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 

Eng. Marco Fargetta, PhD
 
Istituto Nazionale di Fisica Nucleare (INFN)
Catania, Italy

EMail: marco.farge...@ct.infn.it




smime.p7s
Description: S/MIME cryptographic signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Steven Hardy]

On Tue, Feb 25, 2014 at 11:47:43AM -0800, Morgan Fainberg wrote:
> For purposes of supporting multiple backends for Identity (multiple LDAP, mix 
> of LDAP and SQL, federation, etc) Keystone is planning to increase the 
> maximum size of the USER_ID field from an upper limit of 64 to an upper limit 
> of 255. This change would not impact any currently assigned USER_IDs (they 
> would remain in the old simple UUID format), however, new USER_IDs would be 
> increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). 

Hmm, my immediate reaction is there must be a better way than mangling both
bits of data into the ID field, considering pretty much everything
everywhere else in openstack uses uuids for user-visible object identifiers.

> There is the obvious concern that projects are utilizing (and storing) the 
> user_id in a field that cannot accommodate the increased upper limit. Before 
> this change is merged in, it is important for the Keystone team to understand 
> if there are any places that would be overflowed by the increased size.
> 
> The review that would implement this change in size is 
> https://review.openstack.org/#/c/74214 and is actively being worked 
> on/reviewed.
> 
> I have already spoken with the Nova team, and a single instance has been 
> identified that would require a migration (that will have a fix proposed for 
> the I3 timeline). 
> 
> If there are any other known locations that would have issues with an 
> increased USER_ID size, or any concerns with this change to USER_ID format, 
> please respond so that the issues/concerns can be addressed.  Again, the plan 
> is not to change current USER_IDs but that new ones could be up to 255 
> characters in length.

Yes, this will affect Heat in at least one place - we store the trustor
user ID when we create a trust between the stack owner and the heat service
user:

https://github.com/openstack/heat/blob/master/heat/db/sqlalchemy/migrate_repo/versions/027_user_creds_trusts.py#L23

IMHO this is coming pretty late in the cycle considering the potential
impact, but if this is definitely happening we can go ahead an update our
schema.

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Jay Pipes]

On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
> For purposes of supporting multiple backends for Identity (multiple
> LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
> increase the maximum size of the USER_ID field from an upper limit of
> 64 to an upper limit of 255. This change would not impact any
> currently assigned USER_IDs (they would remain in the old simple UUID
> format), however, new USER_IDs would be increased to include the IDP
> identifier (e.g. USER_ID@@IDP_IDENTIFIER).

-1

I think a better solution would be to have a simple translation table
only in Keystone that would store this longer identifier (for folks
using federation and/or LDAP) along with the Keystone user UUID that is
used in foreign key relations and other mapping tables through Keystone
and other projects.

The only identifiers that would ever be communicated to any non-Keystone
OpenStack endpoint would be the UUID user and tenant IDs.

> There is the obvious concern that projects are utilizing (and storing)
> the user_id in a field that cannot accommodate the increased upper
> limit. Before this change is merged in, it is important for the
> Keystone team to understand if there are any places that would be
> overflowed by the increased size.

I would go so far as to say the user_id and tenant_id fields should be
*reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
performance reasons. Lengthening commonly-used and frequently-joined
identifier fields is not a good option, IMO.

Best,
-jay

> The review that would implement this change in size
> is https://review.openstack.org/#/c/74214 and is actively being worked
> on/reviewed.
> 
> 
> I have already spoken with the Nova team, and a single instance has
> been identified that would require a migration (that will have a fix
> proposed for the I3 timeline). 
> 
> 
> If there are any other known locations that would have issues with an
> increased USER_ID size, or any concerns with this change to USER_ID
> format, please respond so that the issues/concerns can be addressed.
>  Again, the plan is not to change current USER_IDs but that new ones
> could be up to 255 characters in length.
> 
> 
> Cheers,
> Morgan Fainberg
> —
> Morgan Fainberg
> Principal Software Engineer
> Core Developer, Keystone
> m...@metacloud.com
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

[Morgan Fainberg]

For purposes of supporting multiple backends for Identity (multiple LDAP, mix 
of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum 
size of the USER_ID field from an upper limit of 64 to an upper limit of 255. 
This change would not impact any currently assigned USER_IDs (they would remain 
in the old simple UUID format), however, new USER_IDs would be increased to 
include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). 

There is the obvious concern that projects are utilizing (and storing) the 
user_id in a field that cannot accommodate the increased upper limit. Before 
this change is merged in, it is important for the Keystone team to understand 
if there are any places that would be overflowed by the increased size.

The review that would implement this change in size is 
https://review.openstack.org/#/c/74214 and is actively being worked on/reviewed.

I have already spoken with the Nova team, and a single instance has been 
identified that would require a migration (that will have a fix proposed for 
the I3 timeline). 

If there are any other known locations that would have issues with an increased 
USER_ID size, or any concerns with this change to USER_ID format, please 
respond so that the issues/concerns can be addressed.  Again, the plan is not 
to change current USER_IDs but that new ones could be up to 255 characters in 
length.

Cheers,
Morgan Fainberg
—
Morgan Fainberg
Principal Software Engineer
Core Developer, Keystone
m...@metacloud.com___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev