Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-04 Thread Vishvananda Ishaya

On Mar 3, 2014, at 11:32 AM, Jay Pipes jaypi...@gmail.com wrote:

 On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote:
 On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote:
 
 On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
 Having done some work with MySQL (specifically around similar data
 sets) and discussing the changes with some former coworkers (MySQL
 experts) I am inclined to believe the move from varchar  to binary
 absolutely would increase performance like this.
 
 
 However, I would like to get some real benchmarks around it and if it
 really makes a difference we should get a smart UUID type into the
 common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
 conversation. Should be split off from the keystone one at hand - I
 don't want valuable information / discussions to get lost.
 
 No disagreement on either point. However, this should be done after the
 standardization to a UUID user_id in Keystone, as a separate performance
 improvement patch. Agree?
 
 Best,
 -jay
 
 -1
 
 The expectation in other projects has been that project_ids and user_ids are 
 opaque strings. I need to see more clear benefit to enforcing stricter 
 typing on these, because I think it might break a lot of things.
 
 What does Nova lose here? The proposal is to have Keystone's user_id
 values be UUIDs all the time. There would be a migration or helper
 script against Nova's database that would change all non-UUID user_id
 values to the Keystone UUID values.

So I don’t have a problem with keystone internally using uuids, but forcing
a migration of user identifiers isn’t something that should be taken lightly.
One example is external logging and billing systems which now have to be
migrated.

I’m not opposed to the migration in principle. We may have to do a similar
thing for project_ids with hierarchical multitenancy, for example. I just
think we need a really good reason to do it, and the performance arguments
just don’t seem good enough without a little empirical data.

Vish

 
 If there's stuff in Nova that would break (which is doubtful,
 considering like you say, these are supposed to be opaque values and
 as such should not have any restrictions or validation on their value),
 then that is code in Nova that should be fixed.
 
 Honestly, we shouldn't accept poor or loose code just because stuff
 might break.
 
 -jay
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-04 Thread Morgan Fainberg
On March 4, 2014 at 10:45:02, Vishvananda Ishaya (vishvana...@gmail.com) wrote:

On Mar 3, 2014, at 11:32 AM, Jay Pipes jaypi...@gmail.com wrote: 

 On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote: 
 On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote: 
 
 On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote: 
 Having done some work with MySQL (specifically around similar data 
 sets) and discussing the changes with some former coworkers (MySQL 
 experts) I am inclined to believe the move from varchar to binary 
 absolutely would increase performance like this. 
 
 
 However, I would like to get some real benchmarks around it and if it 
 really makes a difference we should get a smart UUID type into the 
 common SQLlibs (can pgsql see a similar benefit? Db2?) I think this 
 conversation. Should be split off from the keystone one at hand - I 
 don't want valuable information / discussions to get lost. 
 
 No disagreement on either point. However, this should be done after the 
 standardization to a UUID user_id in Keystone, as a separate performance 
 improvement patch. Agree? 
 
 Best, 
 -jay 
 
 -1 
 
 The expectation in other projects has been that project_ids and user_ids are 
 opaque strings. I need to see more clear benefit to enforcing stricter 
 typing on these, because I think it might break a lot of things. 
 
 What does Nova lose here? The proposal is to have Keystone's user_id 
 values be UUIDs all the time. There would be a migration or helper 
 script against Nova's database that would change all non-UUID user_id 
 values to the Keystone UUID values. 

So I don’t have a problem with keystone internally using uuids, but forcing 
a migration of user identifiers isn’t something that should be taken lightly. 
One example is external logging and billing systems which now have to be 
migrated. 

I’m not opposed to the migration in principle. We may have to do a similar 
thing for project_ids with hierarchical multitenancy, for example. I just 
think we need a really good reason to do it, and the performance arguments 
just don’t seem good enough without a little empirical data. 

Vish 
Any one of the proposals we’re planning on using will not affect any current 
IDs.  Since the user_id is a blob, if we start issuing a new “id” format, 
ideally it shouldn’t matter as long as old IDs continue to work. If we do make 
any kind of migration for issued IDs I would expect that to be very deliberate 
and outside of this change set. Specifically this change set would enable 
multiple LDAP backends (among other user_id uniqueness benefits for federation, 
etc). 

I am very concerned about the external tools that reference IDs we currently 
have.

—Morgan





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-03 Thread Jay Pipes
On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
 Having done some work with MySQL (specifically around similar data
 sets) and discussing the changes with some former coworkers (MySQL
 experts) I am inclined to believe the move from varchar  to binary
 absolutely would increase performance like this.
 
 
 However, I would like to get some real benchmarks around it and if it
 really makes a difference we should get a smart UUID type into the
 common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
 conversation. Should be split off from the keystone one at hand - I
 don't want valuable information / discussions to get lost.

No disagreement on either point. However, this should be done after the
standardization to a UUID user_id in Keystone, as a separate performance
improvement patch. Agree?

Best,
-jay



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-03 Thread Dolph Mathews
On Mon, Mar 3, 2014 at 8:48 AM, Jay Pipes jaypi...@gmail.com wrote:

 On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
  Having done some work with MySQL (specifically around similar data
  sets) and discussing the changes with some former coworkers (MySQL
  experts) I am inclined to believe the move from varchar  to binary
  absolutely would increase performance like this.
 
 
  However, I would like to get some real benchmarks around it and if it
  really makes a difference we should get a smart UUID type into the
  common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
  conversation. Should be split off from the keystone one at hand - I
  don't want valuable information / discussions to get lost.

 No disagreement on either point. However, this should be done after the
 standardization to a UUID user_id in Keystone, as a separate performance
 improvement patch. Agree?

 Best,
 -jay



++ https://blueprints.launchpad.net/keystone/+spec/sql-id-binary-16



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-03 Thread Vishvananda Ishaya

On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote:

 On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
 Having done some work with MySQL (specifically around similar data
 sets) and discussing the changes with some former coworkers (MySQL
 experts) I am inclined to believe the move from varchar  to binary
 absolutely would increase performance like this.
 
 
 However, I would like to get some real benchmarks around it and if it
 really makes a difference we should get a smart UUID type into the
 common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
 conversation. Should be split off from the keystone one at hand - I
 don't want valuable information / discussions to get lost.
 
 No disagreement on either point. However, this should be done after the
 standardization to a UUID user_id in Keystone, as a separate performance
 improvement patch. Agree?
 
 Best,
 -jay

-1

The expectation in other projects has been that project_ids and user_ids are 
opaque strings. I need to see more clear benefit to enforcing stricter typing 
on these, because I think it might break a lot of things.

Vish



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-03 Thread Mark Washenberger
On Sat, Mar 1, 2014 at 12:51 PM, Jay Pipes jaypi...@gmail.com wrote:

 On Fri, 2014-02-28 at 15:25 -0800, Mark Washenberger wrote:
  I believe we have some agreement here. Other openstack services should
  be able to use a strongly typed identifier for users. I just think if
  we want to go that route, we probably need to create a new field to
  act as the proper user uuid, rather than repurposing the existing
  field. It sounds like many existing LDAP deployments would break if we
  repurpose the existing field.

 Hi Mark,

 Please see my earlier response on this thread. I am proposing putting
 external identifiers into a mapping table that would correlate a
 Keystone UUID user ID with external identifiers (of variable length).


The thing you seem to be missing is that the current user-id attribute is
an external identifier depending on the identity backend you're using
today. For example in the LDAP driver it is the CN by default (which is
ridiculous for a large number of reasons, but let's leave those aside.) So
if you want to create a new, strongly typed internal uuid identifier that
makes the db performance scream, more power to you. But it's going to have
to be a new field.




 Once authentication has occurred (with any auth backend including LDAP),
 Keystone would only communicate to the other OpenStack services the UUID
 user ID from Keystone. This would indeed require a migration to each
 non-Keystone service that stores the user IDs as-is from Keystone
 currently (such as Glance or Nova).

 Once the migrations are run, then only UUID values would be stored, and
 further migrations could be run that would streamline the columns that
 stores these user IDs to a more efficient CHAR(32) or BINARY(16)
 internal storage format.

 Hope that clears things up.

 Best,
 -jay


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-03 Thread Jay Pipes
On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote:
 On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote:
 
  On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
  Having done some work with MySQL (specifically around similar data
  sets) and discussing the changes with some former coworkers (MySQL
  experts) I am inclined to believe the move from varchar  to binary
  absolutely would increase performance like this.
  
  
  However, I would like to get some real benchmarks around it and if it
  really makes a difference we should get a smart UUID type into the
  common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
  conversation. Should be split off from the keystone one at hand - I
  don't want valuable information / discussions to get lost.
  
  No disagreement on either point. However, this should be done after the
  standardization to a UUID user_id in Keystone, as a separate performance
  improvement patch. Agree?
  
  Best,
  -jay
 
 -1
 
 The expectation in other projects has been that project_ids and user_ids are 
 opaque strings. I need to see more clear benefit to enforcing stricter typing 
 on these, because I think it might break a lot of things.

What does Nova lose here? The proposal is to have Keystone's user_id
values be UUIDs all the time. There would be a migration or helper
script against Nova's database that would change all non-UUID user_id
values to the Keystone UUID values.

If there's stuff in Nova that would break (which is doubtful,
considering like you say, these are supposed to be opaque values and
as such should not have any restrictions or validation on their value),
then that is code in Nova that should be fixed.

Honestly, we shouldn't accept poor or loose code just because stuff
might break.

-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-03 Thread Jay Pipes
On Mon, 2014-03-03 at 11:18 -0800, Mark Washenberger wrote:
 
 
 
 On Sat, Mar 1, 2014 at 12:51 PM, Jay Pipes jaypi...@gmail.com wrote:
 On Fri, 2014-02-28 at 15:25 -0800, Mark Washenberger wrote:
  I believe we have some agreement here. Other openstack
 services should
  be able to use a strongly typed identifier for users. I just
 think if
  we want to go that route, we probably need to create a new
 field to
  act as the proper user uuid, rather than repurposing the
 existing
  field. It sounds like many existing LDAP deployments would
 break if we
  repurpose the existing field.
 
 
 Hi Mark,
 
 Please see my earlier response on this thread. I am proposing
 putting
 external identifiers into a mapping table that would correlate
 a
 Keystone UUID user ID with external identifiers (of variable
 length).
 
 
 The thing you seem to be missing is that the current user-id attribute
 is an external identifier depending on the identity backend you're
 using today. For example in the LDAP driver it is the CN by default
 (which is ridiculous for a large number of reasons, but let's leave
 those aside.) So if you want to create a new, strongly typed internal
 uuid identifier that makes the db performance scream, more power to
 you. But it's going to have to be a new field.

No, it won't. The proposal is to create a new table that maps a UUID
value to the external identifier value.

The migration would essentially do this pseudocode:

recs = get_identity_records()
for rec in recs:
  if not is_like_uuid(rec.user_id):
  external_id = rec.user_id
  new_uuid = uuid.uuid4()
  insert_into_external_ids(new_uuid, external_id)
  rec.user_id = new_uuid
  save_identity_record(rec)

Cascading updates on the user_id field to its child relations will take
care of the changes in user_id column in foriegn keys.

A migration script in, say, Nova, would do something like:

UPDATE instances SET owner_id = ext.user_id
FROM instances JOIN keystone.external_id_mapping ext
ON instances.owner_id = ext.external_id;

Best,
-jay 



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-03 Thread Adam Young

On 03/03/2014 02:32 PM, Jay Pipes wrote:

On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote:

On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote:


On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:

Having done some work with MySQL (specifically around similar data
sets) and discussing the changes with some former coworkers (MySQL
experts) I am inclined to believe the move from varchar  to binary
absolutely would increase performance like this.


However, I would like to get some real benchmarks around it and if it
really makes a difference we should get a smart UUID type into the
common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
conversation. Should be split off from the keystone one at hand - I
don't want valuable information / discussions to get lost.

No disagreement on either point. However, this should be done after the
standardization to a UUID user_id in Keystone, as a separate performance
improvement patch. Agree?

Best,
-jay

-1

The expectation in other projects has been that project_ids and user_ids are 
opaque strings. I need to see more clear benefit to enforcing stricter typing 
on these, because I think it might break a lot of things.

What does Nova lose here? The proposal is to have Keystone's user_id
values be UUIDs all the time. There would be a migration or helper
script against Nova's database that would change all non-UUID user_id
values to the Keystone UUID values.

If there's stuff in Nova that would break (which is doubtful,
considering like you say, these are supposed to be opaque values and
as such should not have any restrictions or validation on their value),
then that is code in Nova that should be fixed.

Honestly, we shouldn't accept poor or loose code just because stuff
might break.

Why do you think that UUIDs would be some sort of magic bullet?

Thus far, we've stated that IDs are opaque strings, and that they will 
be URL safe.  Adding additional Database lokups on top of LDAP calls is 
going to make LDAP code paths even more convoluted than they are now.  
I'd like to avoid that. And I don't see the benefit.



I'd be interested in hearing why CN is a bad default choice for LDAP; we 
have to use a very high up theRFE chain object class by default and 
didn't have that many attributes to chose from.








-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-03 Thread Clint Byrum
Excerpts from Adam Young's message of 2014-03-03 23:06:58 -0800:
 On 03/03/2014 02:32 PM, Jay Pipes wrote:
  On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote:
  On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote:
 
  On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote:
  Having done some work with MySQL (specifically around similar data
  sets) and discussing the changes with some former coworkers (MySQL
  experts) I am inclined to believe the move from varchar  to binary
  absolutely would increase performance like this.
 
 
  However, I would like to get some real benchmarks around it and if it
  really makes a difference we should get a smart UUID type into the
  common SQLlibs (can pgsql see a similar benefit? Db2?) I think this
  conversation. Should be split off from the keystone one at hand - I
  don't want valuable information / discussions to get lost.
  No disagreement on either point. However, this should be done after the
  standardization to a UUID user_id in Keystone, as a separate performance
  improvement patch. Agree?
 
  Best,
  -jay
  -1
 
  The expectation in other projects has been that project_ids and user_ids 
  are opaque strings. I need to see more clear benefit to enforcing stricter 
  typing on these, because I think it might break a lot of things.
  What does Nova lose here? The proposal is to have Keystone's user_id
  values be UUIDs all the time. There would be a migration or helper
  script against Nova's database that would change all non-UUID user_id
  values to the Keystone UUID values.
 
  If there's stuff in Nova that would break (which is doubtful,
  considering like you say, these are supposed to be opaque values and
  as such should not have any restrictions or validation on their value),
  then that is code in Nova that should be fixed.
 
  Honestly, we shouldn't accept poor or loose code just because stuff
  might break.
 Why do you think that UUIDs would be some sort of magic bullet?
 
 Thus far, we've stated that IDs are opaque strings, and that they will 
 be URL safe.  Adding additional Database lokups on top of LDAP calls is 
 going to make LDAP code paths even more convoluted than they are now.  
 I'd like to avoid that. And I don't see the benefit.
 

Each row in each table throughout OpenStack that has to treat the user
id from Keystone as an opaque ID must then store the keystone id as a 64
byte string. Being able to make it a 16 byte binary value means a
significant saving at scale.

If you're using LDAP, you're only going to store that CN once per user,
and only going to look it up when said user tries to log in. After that,
the uuid will be passed around.

 
 I'd be interested in hearing why CN is a bad default choice for LDAP; we 
 have to use a very high up theRFE chain object class by default and 
 didn't have that many attributes to chose from.
 

CN is fine. External IDs of various shapes and sizes can be used. The
point is to make keystone get a little more opionated about its' id
strings in order to make the rest of OpenStack more scalable.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-02 Thread Sean Dague
On 03/01/2014 08:00 PM, Clint Byrum wrote:
 Excerpts from Robert Collins's message of 2014-03-01 14:26:57 -0800:
 On 1 March 2014 13:28, Clint Byrum cl...@fewbar.com wrote:

 +1. A Keystone record belongs to Keystone, and it should have a Keystone
 ID. External records that are linked should be linked separately.

 It may not be obvious to everyone, but MySQL uses B-trees for indexes.
 B-trees cannot have variable-length keys.

 Hmm, B-Trees and B+-Trees both can have variable length keys. I'll
 accept an assertion that MySQL index B-trees cannot - but we should be
 precise here, because its not a global limitation.

 
 Sorry, I misspoke, _InnoDB's_ b-tree's cannot have variable length keys.
 :-P

On a previous project we did a transition from varchar based UUID to
binary based UUID in MySQL. The micro benchmarks on joins got faster by
a factor of 10,000 (yes 10k). Granted, MySQL has evolved since then, and
this was a micro benchmark, however this is definitely work considering.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-02 Thread Morgan Fainberg
Having done some work with MySQL (specifically around similar data sets)
and discussing the changes with some former coworkers (MySQL experts) I am
inclined to believe the move from varchar  to binary absolutely would
increase performance like this.

However, I would like to get some real benchmarks around it and if it
really makes a difference we should get a smart UUID type into the common
SQLlibs (can pgsql see a similar benefit? Db2?) I think this conversation.
Should be split off from the keystone one at hand - I don't want valuable
information / discussions to get lost.

Cheers,
Morgan.

Sent via mobile


On Sunday, March 2, 2014, Sean Dague s...@dague.net wrote:

 On 03/01/2014 08:00 PM, Clint Byrum wrote:
  Excerpts from Robert Collins's message of 2014-03-01 14:26:57 -0800:
  On 1 March 2014 13:28, Clint Byrum cl...@fewbar.com javascript:;
 wrote:
 
  +1. A Keystone record belongs to Keystone, and it should have a
 Keystone
  ID. External records that are linked should be linked separately.
 
  It may not be obvious to everyone, but MySQL uses B-trees for indexes.
  B-trees cannot have variable-length keys.
 
  Hmm, B-Trees and B+-Trees both can have variable length keys. I'll
  accept an assertion that MySQL index B-trees cannot - but we should be
  precise here, because its not a global limitation.
 
 
  Sorry, I misspoke, _InnoDB's_ b-tree's cannot have variable length keys.
  :-P

 On a previous project we did a transition from varchar based UUID to
 binary based UUID in MySQL. The micro benchmarks on joins got faster by
 a factor of 10,000 (yes 10k). Granted, MySQL has evolved since then, and
 this was a micro benchmark, however this is definitely work considering.

 -Sean

 --
 Sean Dague
 Samsung Research America
 s...@dague.net javascript:; / sean.da...@samsung.com javascript:;
 http://dague.net


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-01 Thread Jay Pipes
On Fri, 2014-02-28 at 15:25 -0800, Mark Washenberger wrote:
 I believe we have some agreement here. Other openstack services should
 be able to use a strongly typed identifier for users. I just think if
 we want to go that route, we probably need to create a new field to
 act as the proper user uuid, rather than repurposing the existing
 field. It sounds like many existing LDAP deployments would break if we
 repurpose the existing field.

Hi Mark,

Please see my earlier response on this thread. I am proposing putting
external identifiers into a mapping table that would correlate a
Keystone UUID user ID with external identifiers (of variable length). 

Once authentication has occurred (with any auth backend including LDAP),
Keystone would only communicate to the other OpenStack services the UUID
user ID from Keystone. This would indeed require a migration to each
non-Keystone service that stores the user IDs as-is from Keystone
currently (such as Glance or Nova).

Once the migrations are run, then only UUID values would be stored, and
further migrations could be run that would streamline the columns that
stores these user IDs to a more efficient CHAR(32) or BINARY(16)
internal storage format.

Hope that clears things up.

Best,
-jay 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-01 Thread Robert Collins
On 1 March 2014 13:28, Clint Byrum cl...@fewbar.com wrote:

 +1. A Keystone record belongs to Keystone, and it should have a Keystone
 ID. External records that are linked should be linked separately.

 It may not be obvious to everyone, but MySQL uses B-trees for indexes.
 B-trees cannot have variable-length keys.

Hmm, B-Trees and B+-Trees both can have variable length keys. I'll
accept an assertion that MySQL index B-trees cannot - but we should be
precise here, because its not a global limitation.

That said, our DB design basically needs to take into account the
union of limitations of all target DB platforms, so we do still need
to cater for this.
-Rob

-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-03-01 Thread Clint Byrum
Excerpts from Robert Collins's message of 2014-03-01 14:26:57 -0800:
 On 1 March 2014 13:28, Clint Byrum cl...@fewbar.com wrote:
 
  +1. A Keystone record belongs to Keystone, and it should have a Keystone
  ID. External records that are linked should be linked separately.
 
  It may not be obvious to everyone, but MySQL uses B-trees for indexes.
  B-trees cannot have variable-length keys.
 
 Hmm, B-Trees and B+-Trees both can have variable length keys. I'll
 accept an assertion that MySQL index B-trees cannot - but we should be
 precise here, because its not a global limitation.
 

Sorry, I misspoke, _InnoDB's_ b-tree's cannot have variable length keys.
:-P

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-28 Thread Mark Washenberger
On Wed, Feb 26, 2014 at 5:25 AM, Dolph Mathews dolph.math...@gmail.comwrote:


 On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com wrote:

 On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
  For purposes of supporting multiple backends for Identity (multiple
  LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
  increase the maximum size of the USER_ID field from an upper limit of
  64 to an upper limit of 255. This change would not impact any
  currently assigned USER_IDs (they would remain in the old simple UUID
  format), however, new USER_IDs would be increased to include the IDP
  identifier (e.g. USER_ID@@IDP_IDENTIFIER).

 -1

 I think a better solution would be to have a simple translation table
 only in Keystone that would store this longer identifier (for folks
 using federation and/or LDAP) along with the Keystone user UUID that is
 used in foreign key relations and other mapping tables through Keystone
 and other projects.


 Morgan and I talked this suggestion through last night and agreed it's
 probably the best approach, and has the benefit of zero impact on other
 services, which is something we're obviously trying to avoid. I imagine it
 could be as simple as a user_id to domain_id lookup table. All we really
 care about is given a globally unique user ID, which identity backend is
 the user from?

 On the downside, it would likely become bloated with unused ephemeral user
 IDs, so we'll need enough metadata about the mapping to implement a purging
 behavior down the line.


Is this approach planning on reusing the existing user-id field, then? It
seems like this creates a migration problem for folks who are currently
using user-ids that are generated by their identity backends.





 The only identifiers that would ever be communicated to any non-Keystone
 OpenStack endpoint would be the UUID user and tenant IDs.

  There is the obvious concern that projects are utilizing (and storing)
  the user_id in a field that cannot accommodate the increased upper
  limit. Before this change is merged in, it is important for the
  Keystone team to understand if there are any places that would be
  overflowed by the increased size.

 I would go so far as to say the user_id and tenant_id fields should be
 *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
 performance reasons. Lengthening commonly-used and frequently-joined
 identifier fields is not a good option, IMO.

 Best,
 -jay

  The review that would implement this change in size
  is https://review.openstack.org/#/c/74214 and is actively being worked
  on/reviewed.
 
 
  I have already spoken with the Nova team, and a single instance has
  been identified that would require a migration (that will have a fix
  proposed for the I3 timeline).
 
 
  If there are any other known locations that would have issues with an
  increased USER_ID size, or any concerns with this change to USER_ID
  format, please respond so that the issues/concerns can be addressed.
   Again, the plan is not to change current USER_IDs but that new ones
  could be up to 255 characters in length.
 
 
  Cheers,
  Morgan Fainberg
  --
  Morgan Fainberg
  Principal Software Engineer
  Core Developer, Keystone
  m...@metacloud.com
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-28 Thread Henry Nash
Hi Mark,

So we would not modify any existing IDs, so no migration required.

Henry
On 28 Feb 2014, at 17:38, Mark Washenberger mark.washenber...@markwash.net 
wrote:

 
 
 
 On Wed, Feb 26, 2014 at 5:25 AM, Dolph Mathews dolph.math...@gmail.com 
 wrote:
 
 On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com wrote:
 On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
  For purposes of supporting multiple backends for Identity (multiple
  LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
  increase the maximum size of the USER_ID field from an upper limit of
  64 to an upper limit of 255. This change would not impact any
  currently assigned USER_IDs (they would remain in the old simple UUID
  format), however, new USER_IDs would be increased to include the IDP
  identifier (e.g. USER_ID@@IDP_IDENTIFIER).
 
 -1
 
 I think a better solution would be to have a simple translation table
 only in Keystone that would store this longer identifier (for folks
 using federation and/or LDAP) along with the Keystone user UUID that is
 used in foreign key relations and other mapping tables through Keystone
 and other projects.
 
 Morgan and I talked this suggestion through last night and agreed it's 
 probably the best approach, and has the benefit of zero impact on other 
 services, which is something we're obviously trying to avoid. I imagine it 
 could be as simple as a user_id to domain_id lookup table. All we really care 
 about is given a globally unique user ID, which identity backend is the user 
 from?
 
 On the downside, it would likely become bloated with unused ephemeral user 
 IDs, so we'll need enough metadata about the mapping to implement a purging 
 behavior down the line.
 
 Is this approach planning on reusing the existing user-id field, then? It 
 seems like this creates a migration problem for folks who are currently using 
 user-ids that are generated by their identity backends.
  
  
 
 The only identifiers that would ever be communicated to any non-Keystone
 OpenStack endpoint would be the UUID user and tenant IDs.
 
  There is the obvious concern that projects are utilizing (and storing)
  the user_id in a field that cannot accommodate the increased upper
  limit. Before this change is merged in, it is important for the
  Keystone team to understand if there are any places that would be
  overflowed by the increased size.
 
 I would go so far as to say the user_id and tenant_id fields should be
 *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
 performance reasons. Lengthening commonly-used and frequently-joined
 identifier fields is not a good option, IMO.
 
 Best,
 -jay
 
  The review that would implement this change in size
  is https://review.openstack.org/#/c/74214 and is actively being worked
  on/reviewed.
 
 
  I have already spoken with the Nova team, and a single instance has
  been identified that would require a migration (that will have a fix
  proposed for the I3 timeline).
 
 
  If there are any other known locations that would have issues with an
  increased USER_ID size, or any concerns with this change to USER_ID
  format, please respond so that the issues/concerns can be addressed.
   Again, the plan is not to change current USER_IDs but that new ones
  could be up to 255 characters in length.
 
 
  Cheers,
  Morgan Fainberg
  —
  Morgan Fainberg
  Principal Software Engineer
  Core Developer, Keystone
  m...@metacloud.com
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-28 Thread Mark Washenberger
On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash hen...@linux.vnet.ibm.comwrote:

 Hi Mark,

 So we would not modify any existing IDs, so no migration required.


Okay, I just want to be painfully clear--we're not proposing changing any
of the current restrictions on the user-id field. We will not:
  - require it to be a uuid
  - encode it as binary instead of char
  - shrink its size below the current 64 characters

Any of those could require a migration for existing IDs depending on how
your identity driver functions.

If I'm just being Chicken Little, please reassure me once more and I'll be
quiet :-)




 Henry

 On 28 Feb 2014, at 17:38, Mark Washenberger 
 mark.washenber...@markwash.net wrote:




 On Wed, Feb 26, 2014 at 5:25 AM, Dolph Mathews dolph.math...@gmail.comwrote:


 On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com wrote:

 On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
  For purposes of supporting multiple backends for Identity (multiple
  LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
  increase the maximum size of the USER_ID field from an upper limit of
  64 to an upper limit of 255. This change would not impact any
  currently assigned USER_IDs (they would remain in the old simple UUID
  format), however, new USER_IDs would be increased to include the IDP
  identifier (e.g. USER_ID@@IDP_IDENTIFIER).

 -1

 I think a better solution would be to have a simple translation table
 only in Keystone that would store this longer identifier (for folks
 using federation and/or LDAP) along with the Keystone user UUID that is
 used in foreign key relations and other mapping tables through Keystone
 and other projects.


 Morgan and I talked this suggestion through last night and agreed it's
 probably the best approach, and has the benefit of zero impact on other
 services, which is something we're obviously trying to avoid. I imagine it
 could be as simple as a user_id to domain_id lookup table. All we really
 care about is given a globally unique user ID, which identity backend is
 the user from?

 On the downside, it would likely become bloated with unused ephemeral
 user IDs, so we'll need enough metadata about the mapping to implement a
 purging behavior down the line.


 Is this approach planning on reusing the existing user-id field, then? It
 seems like this creates a migration problem for folks who are currently
 using user-ids that are generated by their identity backends.





 The only identifiers that would ever be communicated to any non-Keystone
 OpenStack endpoint would be the UUID user and tenant IDs.

  There is the obvious concern that projects are utilizing (and storing)
  the user_id in a field that cannot accommodate the increased upper
  limit. Before this change is merged in, it is important for the
  Keystone team to understand if there are any places that would be
  overflowed by the increased size.

 I would go so far as to say the user_id and tenant_id fields should be
 *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
 performance reasons. Lengthening commonly-used and frequently-joined
 identifier fields is not a good option, IMO.

 Best,
 -jay

  The review that would implement this change in size
  is https://review.openstack.org/#/c/74214 and is actively being worked
  on/reviewed.
 
 
  I have already spoken with the Nova team, and a single instance has
  been identified that would require a migration (that will have a fix
  proposed for the I3 timeline).
 
 
  If there are any other known locations that would have issues with an
  increased USER_ID size, or any concerns with this change to USER_ID
  format, please respond so that the issues/concerns can be addressed.
   Again, the plan is not to change current USER_IDs but that new ones
  could be up to 255 characters in length.
 
 
  Cheers,
  Morgan Fainberg
  --
  Morgan Fainberg
  Principal Software Engineer
  Core Developer, Keystone
  m...@metacloud.com
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-28 Thread Jay Pipes
On Fri, 2014-02-28 at 13:10 -0800, Mark Washenberger wrote:
 
 On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash
 hen...@linux.vnet.ibm.com wrote:
 Hi Mark,
 
 
 So we would not modify any existing IDs, so no migration
 required.
 
 
 Okay, I just want to be painfully clear--we're not proposing changing
 any of the current restrictions on the user-id field. We will not:
   - require it to be a uuid
   - encode it as binary instead of char
   - shrink its size below the current 64 characters

The first would be required for the real solution. The second and third
are performance improvements.

 Any of those could require a migration for existing IDs depending on
 how your identity driver functions.

Personally, I think to fix this issue permanently and properly,
migrations for database schemas of Glance and Nova would indeed need to
accompany a proposed patch that restricts the Keystone external user ID
to only a UUID value.

I entirely disagree with allowing non-UUID values for the user ID value
that is exposed outside of Keystone. All other solutions (including the
proposals to continue using the user_id fields with non-UUID values) are
just hacks IMO.

Best,
-jay



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-28 Thread Mark Washenberger
On Fri, Feb 28, 2014 at 2:26 PM, Jay Pipes jaypi...@gmail.com wrote:

 On Fri, 2014-02-28 at 13:10 -0800, Mark Washenberger wrote:
 
  On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash
  hen...@linux.vnet.ibm.com wrote:
  Hi Mark,
 
 
  So we would not modify any existing IDs, so no migration
  required.
 
 
  Okay, I just want to be painfully clear--we're not proposing changing
  any of the current restrictions on the user-id field. We will not:
- require it to be a uuid
- encode it as binary instead of char
- shrink its size below the current 64 characters

 The first would be required for the real solution. The second and third
 are performance improvements.

  Any of those could require a migration for existing IDs depending on
  how your identity driver functions.

 Personally, I think to fix this issue permanently and properly,
 migrations for database schemas of Glance and Nova would indeed need to
 accompany a proposed patch that restricts the Keystone external user ID
 to only a UUID value.

 I entirely disagree with allowing non-UUID values for the user ID value
 that is exposed outside of Keystone. All other solutions (including the
 proposals to continue using the user_id fields with non-UUID values) are
 just hacks IMO.


I believe we have some agreement here. Other openstack services should be
able to use a strongly typed identifier for users. I just think if we want
to go that route, we probably need to create a new field to act as the
proper user uuid, rather than repurposing the existing field. It sounds
like many existing LDAP deployments would break if we repurpose the
existing field.


 Best,
 -jay



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-28 Thread Clint Byrum
Excerpts from Jay Pipes's message of 2014-02-28 14:26:26 -0800:
 On Fri, 2014-02-28 at 13:10 -0800, Mark Washenberger wrote:
  
  On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash
  hen...@linux.vnet.ibm.com wrote:
  Hi Mark,
  
  
  So we would not modify any existing IDs, so no migration
  required.
  
  
  Okay, I just want to be painfully clear--we're not proposing changing
  any of the current restrictions on the user-id field. We will not:
- require it to be a uuid
- encode it as binary instead of char
- shrink its size below the current 64 characters
 
 The first would be required for the real solution. The second and third
 are performance improvements.
 
  Any of those could require a migration for existing IDs depending on
  how your identity driver functions.
 
 Personally, I think to fix this issue permanently and properly,
 migrations for database schemas of Glance and Nova would indeed need to
 accompany a proposed patch that restricts the Keystone external user ID
 to only a UUID value.
 
 I entirely disagree with allowing non-UUID values for the user ID value
 that is exposed outside of Keystone. All other solutions (including the
 proposals to continue using the user_id fields with non-UUID values) are
 just hacks IMO.

+1. A Keystone record belongs to Keystone, and it should have a Keystone
ID. External records that are linked should be linked separately.

It may not be obvious to everyone, but MySQL uses B-trees for indexes.
B-trees cannot have variable-length keys. So varchar(64) means 64-byte
index keys. If you aren't careful and let that column be stored as utf-8,
this actually means *192* byte index keys, because MySQL uses 3-byte
utf-8 and thus a 64 character column could have 192 bytes. This does
not scale well as you are doing index scans and range lookups, not to
mention just generally raising memory and I/O pressure on the server.

What Jay is suggesting is that we actually be opinionated and store
Keystone users with 16-byte binary UUID's, and only ever use the UUID (in
the 32-byte text notation where appropriate) when returning a keystone ID.

Then only the initial authentication step where the user presents
external identification requires access to anything larger, allowing
all other Keystone operations to perform much better and keeping the
keystone database footprint smaller.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-27 Thread Henry Nash
So a couple of things about this:

1) Today (and also true for Grizzly and Havana), the user can chose what LDAP 
attribute should be returned as the user or group ID.  So it is NOT a safe 
assumption today (ignoring any support for domain-specific LDAP support) that 
the format of a user or group ID is a 32 char UUID.  Quite often, I would 
think, that email address would be chosen by a cloud provider as the LDAP id 
field, by default we use the CN.  Since we really don't want to ever change the 
user or group ID we have given out from keystone for a particular entity, this 
means we need to update nova (or anything else) that has made a 32 char 
assumption.
2) In oder to support the ability for service providers to be able to have the 
identity part of keystone be satisfied by a customer LDAP (i.e. for a given 
domain, have a specific LDAP), then, as has been stated, we need to 
subsequently, when handed an API call with just a user or group ID, be able to 
route this call to the correct LDAP.  Trying to keep true to the openstack 
design principles, we had planned to encode a domain identifier into the user 
or group ID - i.e. distribute the data to where it is needed, in ortherwords, 
the user and group ID provide all the info we need to route the call to the 
right place. Two implementations come to mind:
2a) Simply concatenate the user/group ID with the domain_id, plus some 
separator and make a composite public facing ID.  e.g. 
user_entity_id@@UUID_of_domain.  This would have a technical maximum size of 
64+2+64 (i.e. 130), although in reality since we control domain_id and we know 
it is always 32 char UUID - in fact the max size would be 98.  This has the 
problem of increasing the size of the public facing field beyond the existing 
64.  This is what we had planned for IceHouse - and is currently in review.
2b) Use a similar concatenation idea as 2a), but limit the total size to the 
existing 64. Since we control domain_id, we could (internally and not visibly 
to the outside world), create a domain_index, that was used in place of 
domain_id in the publicly visible field, to minimize the number of chars it 
requires.  So the public facing composite ID might be something like up to 54 
chars of entity_id@@8 chars of domain_index.  There is a chance, of course, 
that  the 54 char restriction might be problematic for LDAP usersbut I 
doubt it.  We would make that a restriction and if it really became a problem, 
we could consider a field size increase at a later release
3) The alternative to 2a and 2b is to have, as had been suggested, an internal 
mapping table that maps external facing entity_ids to a domain plus local 
entity ID.  The problem with this idea is that:
- This could become a very big table (you will essentially have an entry for 
every user in every corporate LDAP that has accessed a given openstack)
- Since most LDAPs are RO, we will never see deletes...so we won't know when 
(without some kind of garbage collection) to cull entries
- It obviously does not solve 1) - since existing LDAP support can break the 32 
char limit - and so it isn't true that this mapping table causes all public 
facing entity IDs to be simple 32 char UUIDs

From a delivery into IceHouse point of view any of the above are possible, 
since the actual mapping used is relatively small part of the patch.  I 
personally favor 2b), since it is simple, has less moving parts and does not 
change any external facing requirements for storage of user and group IDs 
(above and beyond what is true today).

Henry
On 27 Feb 2014, at 03:46, Adam Young ayo...@redhat.com wrote:

 On 02/26/2014 08:25 AM, Dolph Mathews wrote:
 
 On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com wrote:
 On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
  For purposes of supporting multiple backends for Identity (multiple
  LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
  increase the maximum size of the USER_ID field from an upper limit of
  64 to an upper limit of 255. This change would not impact any
  currently assigned USER_IDs (they would remain in the old simple UUID
  format), however, new USER_IDs would be increased to include the IDP
  identifier (e.g. USER_ID@@IDP_IDENTIFIER).
 
 -1
 
 I think a better solution would be to have a simple translation table
 only in Keystone that would store this longer identifier (for folks
 using federation and/or LDAP) along with the Keystone user UUID that is
 used in foreign key relations and other mapping tables through Keystone
 and other projects.
 
 Morgan and I talked this suggestion through last night and agreed it's 
 probably the best approach, and has the benefit of zero impact on other 
 services, which is something we're obviously trying to avoid. I imagine it 
 could be as simple as a user_id to domain_id lookup table. All we really 
 care about is given a globally unique user ID, which identity backend is 
 the user from?
 
 On the 

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-27 Thread Jay Pipes
On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
 So a couple of things about this:
 
 
 1) Today (and also true for Grizzly and Havana), the user can chose
 what LDAP attribute should be returned as the user or group ID.  So it
 is NOT a safe assumption today (ignoring any support for
 domain-specific LDAP support) that the format of a user or group ID is
 a 32 char UUID.  Quite often, I would think, that email address would
 be chosen by a cloud provider as the LDAP id field, by default we use
 the CN.  Since we really don't want to ever change the user or group
 ID we have given out from keystone for a particular entity, this means
 we need to update nova (or anything else) that has made a 32 char
 assumption.

I don't believe this is correct. Keystone is the service that deals with
authentication. As such, Keystone should be the one and only one service
that should have any need whatsoever to need to understand a non-UUID
value for a user ID. The only value that should ever be communicated
*from* Keystone should be the UUID value of the user.

If the Keystone service uses LDAP or federation for alternative
authentication schemes, then Keystone should have a mapping table that
translates those elongated and non-UUID identifiers values (email
addresses, LDAP CNs, etc) into the UUID value that is then communicated
to all other OpenStack services.

Best,
-jay

 2) In oder to support the ability for service providers to be able to
 have the identity part of keystone be satisfied by a customer LDAP
 (i.e. for a given domain, have a specific LDAP), then, as has been
 stated, we need to subsequently, when handed an API call with just a
 user or group ID, be able to route this call to the correct LDAP.
  Trying to keep true to the openstack design principles, we had
 planned to encode a domain identifier into the user or group ID - i.e.
 distribute the data to where it is needed, in ortherwords, the user
 and group ID provide all the info we need to route the call to the
 right place. Two implementations come to mind:
 2a) Simply concatenate the user/group ID with the domain_id, plus some
 separator and make a composite public facing ID.  e.g.
 user_entity_id@@UUID_of_domain.  This would have a technical maximum
 size of 64+2+64 (i.e. 130), although in reality since we control
 domain_id and we know it is always 32 char UUID - in fact the max size
 would be 98.  This has the problem of increasing the size of the
 public facing field beyond the existing 64.  This is what we had
 planned for IceHouse - and is currently in review.
 2b) Use a similar concatenation idea as 2a), but limit the total size
 to the existing 64. Since we control domain_id, we could (internally
 and not visibly to the outside world), create a domain_index, that was
 used in place of domain_id in the publicly visible field, to minimize
 the number of chars it requires.  So the public facing composite ID
 might be something like up to 54 chars of entity_id@@8 chars of
 domain_index.  There is a chance, of course, that  the 54 char
 restriction might be problematic for LDAP usersbut I doubt it.  We
 would make that a restriction and if it really became a problem, we
 could consider a field size increase at a later release
 3) The alternative to 2a and 2b is to have, as had been suggested, an
 internal mapping table that maps external facing entity_ids to a
 domain plus local entity ID.  The problem with this idea is that:
 - This could become a very big table (you will essentially have an
 entry for every user in every corporate LDAP that has accessed a given
 openstack)
 - Since most LDAPs are RO, we will never see deletes...so we won't
 know when (without some kind of garbage collection) to cull entries
 - It obviously does not solve 1) - since existing LDAP support can
 break the 32 char limit - and so it isn't true that this mapping table
 causes all public facing entity IDs to be simple 32 char UUIDs
 
 
 From a delivery into IceHouse point of view any of the above are
 possible, since the actual mapping used is relatively small part of
 the patch.  I personally favor 2b), since it is simple, has less
 moving parts and does not change any external facing requirements for
 storage of user and group IDs (above and beyond what is true today).
 
 
 Henry
 On 27 Feb 2014, at 03:46, Adam Young ayo...@redhat.com wrote:
 
  On 02/26/2014 08:25 AM, Dolph Mathews wrote:
  
   
   On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com
   wrote:
   On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
For purposes of supporting multiple backends for
   Identity (multiple
LDAP, mix of LDAP and SQL, federation, etc) Keystone is
   planning to
increase the maximum size of the USER_ID field from an
   upper limit of
64 to an upper limit of 255. This change would not
   impact any
currently assigned USER_IDs (they would remain in the
   

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-27 Thread Henry Nash

On 27 Feb 2014, at 17:52, Jay Pipes jaypi...@gmail.com wrote:

 On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
 So a couple of things about this:
 
 
 1) Today (and also true for Grizzly and Havana), the user can chose
 what LDAP attribute should be returned as the user or group ID.  So it
 is NOT a safe assumption today (ignoring any support for
 domain-specific LDAP support) that the format of a user or group ID is
 a 32 char UUID.  Quite often, I would think, that email address would
 be chosen by a cloud provider as the LDAP id field, by default we use
 the CN.  Since we really don't want to ever change the user or group
 ID we have given out from keystone for a particular entity, this means
 we need to update nova (or anything else) that has made a 32 char
 assumption.
 
 I don't believe this is correct. Keystone is the service that deals with
 authentication. As such, Keystone should be the one and only one service
 that should have any need whatsoever to need to understand a non-UUID
 value for a user ID. The only value that should ever be communicated
 *from* Keystone should be the UUID value of the user.
 
 If the Keystone service uses LDAP or federation for alternative
 authentication schemes, then Keystone should have a mapping table that
 translates those elongated and non-UUID identifiers values (email
 addresses, LDAP CNs, etc) into the UUID value that is then communicated
 to all other OpenStack services.
 
 Best,
 -jay
 
So I think that's a perfectly reasonable point of viewour challenge is that 
this isn't what
Keystone has done to datee.g. anyone using a RO LDAP today is probably 
exposing
non-UUID identifiers out into nova and other projects (and maybe outside of 
openstack
altogether).  We can't (without breaking them) just change the IDs for any 
existing LDAP
entities.  So they best we could do is to say something like, new entities (and 
perhaps only
those in domain-specific backends) would use such a mapping capability.

Henry
 2) In oder to support the ability for service providers to be able to
 have the identity part of keystone be satisfied by a customer LDAP
 (i.e. for a given domain, have a specific LDAP), then, as has been
 stated, we need to subsequently, when handed an API call with just a
 user or group ID, be able to route this call to the correct LDAP.
 Trying to keep true to the openstack design principles, we had
 planned to encode a domain identifier into the user or group ID - i.e.
 distribute the data to where it is needed, in ortherwords, the user
 and group ID provide all the info we need to route the call to the
 right place. Two implementations come to mind:
 2a) Simply concatenate the user/group ID with the domain_id, plus some
 separator and make a composite public facing ID.  e.g.
 user_entity_id@@UUID_of_domain.  This would have a technical maximum
 size of 64+2+64 (i.e. 130), although in reality since we control
 domain_id and we know it is always 32 char UUID - in fact the max size
 would be 98.  This has the problem of increasing the size of the
 public facing field beyond the existing 64.  This is what we had
 planned for IceHouse - and is currently in review.
 2b) Use a similar concatenation idea as 2a), but limit the total size
 to the existing 64. Since we control domain_id, we could (internally
 and not visibly to the outside world), create a domain_index, that was
 used in place of domain_id in the publicly visible field, to minimize
 the number of chars it requires.  So the public facing composite ID
 might be something like up to 54 chars of entity_id@@8 chars of
 domain_index.  There is a chance, of course, that  the 54 char
 restriction might be problematic for LDAP usersbut I doubt it.  We
 would make that a restriction and if it really became a problem, we
 could consider a field size increase at a later release
 3) The alternative to 2a and 2b is to have, as had been suggested, an
 internal mapping table that maps external facing entity_ids to a
 domain plus local entity ID.  The problem with this idea is that:
 - This could become a very big table (you will essentially have an
 entry for every user in every corporate LDAP that has accessed a given
 openstack)
 - Since most LDAPs are RO, we will never see deletes...so we won't
 know when (without some kind of garbage collection) to cull entries
 - It obviously does not solve 1) - since existing LDAP support can
 break the 32 char limit - and so it isn't true that this mapping table
 causes all public facing entity IDs to be simple 32 char UUIDs
 
 
 From a delivery into IceHouse point of view any of the above are
 possible, since the actual mapping used is relatively small part of
 the patch.  I personally favor 2b), since it is simple, has less
 moving parts and does not change any external facing requirements for
 storage of user and group IDs (above and beyond what is true today).
 
 
 Henry
 On 27 Feb 2014, at 03:46, Adam Young ayo...@redhat.com wrote:
 
 On 02/26/2014 

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-27 Thread Dolph Mathews
On Thu, Feb 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.com wrote:

 On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
  So a couple of things about this:
 
 
  1) Today (and also true for Grizzly and Havana), the user can chose
  what LDAP attribute should be returned as the user or group ID.  So it
  is NOT a safe assumption today (ignoring any support for
  domain-specific LDAP support) that the format of a user or group ID is
  a 32 char UUID.  Quite often, I would think, that email address would
  be chosen by a cloud provider as the LDAP id field, by default we use
  the CN.  Since we really don't want to ever change the user or group
  ID we have given out from keystone for a particular entity, this means
  we need to update nova (or anything else) that has made a 32 char
  assumption.

 I don't believe this is correct. Keystone is the service that deals with
 authentication. As such, Keystone should be the one and only one service
 that should have any need whatsoever to need to understand a non-UUID
 value for a user ID. The only value that should ever be communicated
 *from* Keystone should be the UUID value of the user.


+++



 If the Keystone service uses LDAP or federation for alternative
 authentication schemes, then Keystone should have a mapping table that
 translates those elongated and non-UUID identifiers values (email
 addresses, LDAP CNs, etc) into the UUID value that is then communicated
 to all other OpenStack services.


I'd take it one step further and say that at some point, keystone should
stop leaking identity details such as user name / ID into OpenStack (they
shouldn't appear in tokens, and shouldn't be expected output of
auth_token). The use cases that require them are weak and would be better
served by pure multitenant RBAC, ABAC, etc. There are a lot of blockers to
making this happen (including a few in keystone's own API), but still food
for thought.



 Best,
 -jay

  2) In oder to support the ability for service providers to be able to
  have the identity part of keystone be satisfied by a customer LDAP
  (i.e. for a given domain, have a specific LDAP), then, as has been
  stated, we need to subsequently, when handed an API call with just a
  user or group ID, be able to route this call to the correct LDAP.
   Trying to keep true to the openstack design principles, we had
  planned to encode a domain identifier into the user or group ID - i.e.
  distribute the data to where it is needed, in ortherwords, the user
  and group ID provide all the info we need to route the call to the
  right place. Two implementations come to mind:
  2a) Simply concatenate the user/group ID with the domain_id, plus some
  separator and make a composite public facing ID.  e.g.
  user_entity_id@@UUID_of_domain.  This would have a technical maximum
  size of 64+2+64 (i.e. 130), although in reality since we control
  domain_id and we know it is always 32 char UUID - in fact the max size
  would be 98.  This has the problem of increasing the size of the
  public facing field beyond the existing 64.  This is what we had
  planned for IceHouse - and is currently in review.
  2b) Use a similar concatenation idea as 2a), but limit the total size
  to the existing 64. Since we control domain_id, we could (internally
  and not visibly to the outside world), create a domain_index, that was
  used in place of domain_id in the publicly visible field, to minimize
  the number of chars it requires.  So the public facing composite ID
  might be something like up to 54 chars of entity_id@@8 chars of
  domain_index.  There is a chance, of course, that  the 54 char
  restriction might be problematic for LDAP usersbut I doubt it.  We
  would make that a restriction and if it really became a problem, we
  could consider a field size increase at a later release
  3) The alternative to 2a and 2b is to have, as had been suggested, an
  internal mapping table that maps external facing entity_ids to a
  domain plus local entity ID.  The problem with this idea is that:
  - This could become a very big table (you will essentially have an
  entry for every user in every corporate LDAP that has accessed a given
  openstack)
  - Since most LDAPs are RO, we will never see deletes...so we won't
  know when (without some kind of garbage collection) to cull entries
  - It obviously does not solve 1) - since existing LDAP support can
  break the 32 char limit - and so it isn't true that this mapping table
  causes all public facing entity IDs to be simple 32 char UUIDs
 
 
  From a delivery into IceHouse point of view any of the above are
  possible, since the actual mapping used is relatively small part of
  the patch.  I personally favor 2b), since it is simple, has less
  moving parts and does not change any external facing requirements for
  storage of user and group IDs (above and beyond what is true today).
 
 
  Henry
  On 27 Feb 2014, at 03:46, Adam Young ayo...@redhat.com wrote:
 
   On 02/26/2014 08:25 

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-27 Thread Morgan Fainberg
On Thursday, February 27, 2014, Dolph Mathews dolph.math...@gmail.com
wrote:


 On Thu, Feb 27, 2014 at 11:52 AM, Jay Pipes 
 jaypi...@gmail.comjavascript:_e(%7B%7D,'cvml','jaypi...@gmail.com');
  wrote:

 On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
  So a couple of things about this:
 
 
  1) Today (and also true for Grizzly and Havana), the user can chose
  what LDAP attribute should be returned as the user or group ID.  So it
  is NOT a safe assumption today (ignoring any support for
  domain-specific LDAP support) that the format of a user or group ID is
  a 32 char UUID.  Quite often, I would think, that email address would
  be chosen by a cloud provider as the LDAP id field, by default we use
  the CN.  Since we really don't want to ever change the user or group
  ID we have given out from keystone for a particular entity, this means
  we need to update nova (or anything else) that has made a 32 char
  assumption.

 I don't believe this is correct. Keystone is the service that deals with
 authentication. As such, Keystone should be the one and only one service
 that should have any need whatsoever to need to understand a non-UUID
 value for a user ID. The only value that should ever be communicated
 *from* Keystone should be the UUID value of the user.


 +++



 If the Keystone service uses LDAP or federation for alternative
 authentication schemes, then Keystone should have a mapping table that
 translates those elongated and non-UUID identifiers values (email
 addresses, LDAP CNs, etc) into the UUID value that is then communicated
 to all other OpenStack services.


 I'd take it one step further and say that at some point, keystone should
 stop leaking identity details such as user name / ID into OpenStack (they
 shouldn't appear in tokens, and shouldn't be expected output of
 auth_token). The use cases that require them are weak and would be better
 served by pure multitenant RBAC, ABAC, etc. There are a lot of blockers to
 making this happen (including a few in keystone's own API), but still food
 for thought.


++ this would be a great change!

I am on the same page as Dolph when it comes to approving of the UUID being
the only value communicated outside of keystone. There is just no good
reason to send out extra identity information (it isn't needed and can help
to reduce token bloat etc).

--Morgan

Sent via mobile
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-27 Thread Miller, Mark M (EB SW Cloud - RD - Corvallis)
I agree about not needing extra identity information outside of the user's 
UUID, but what about the role/project/domain information stored in the PKI 
token? Does it remain or go away?

From: Morgan Fainberg [mailto:m...@metacloud.com]
Sent: Thursday, February 27, 2014 12:11 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum 
from 64 to 255



On Thursday, February 27, 2014, Dolph Mathews 
dolph.math...@gmail.commailto:dolph.math...@gmail.com wrote:

On Thu, Feb 27, 2014 at 11:52 AM, Jay Pipes 
jaypi...@gmail.comjavascript:_e(%7B%7D,'cvml','jaypi...@gmail.com'); wrote:
On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
 So a couple of things about this:


 1) Today (and also true for Grizzly and Havana), the user can chose
 what LDAP attribute should be returned as the user or group ID.  So it
 is NOT a safe assumption today (ignoring any support for
 domain-specific LDAP support) that the format of a user or group ID is
 a 32 char UUID.  Quite often, I would think, that email address would
 be chosen by a cloud provider as the LDAP id field, by default we use
 the CN.  Since we really don't want to ever change the user or group
 ID we have given out from keystone for a particular entity, this means
 we need to update nova (or anything else) that has made a 32 char
 assumption.
I don't believe this is correct. Keystone is the service that deals with
authentication. As such, Keystone should be the one and only one service
that should have any need whatsoever to need to understand a non-UUID
value for a user ID. The only value that should ever be communicated
*from* Keystone should be the UUID value of the user.

+++


If the Keystone service uses LDAP or federation for alternative
authentication schemes, then Keystone should have a mapping table that
translates those elongated and non-UUID identifiers values (email
addresses, LDAP CNs, etc) into the UUID value that is then communicated
to all other OpenStack services.

I'd take it one step further and say that at some point, keystone should stop 
leaking identity details such as user name / ID into OpenStack (they shouldn't 
appear in tokens, and shouldn't be expected output of auth_token). The use 
cases that require them are weak and would be better served by pure 
multitenant RBAC, ABAC, etc. There are a lot of blockers to making this happen 
(including a few in keystone's own API), but still food for thought.

++ this would be a great change!

I am on the same page as Dolph when it comes to approving of the UUID being the 
only value communicated outside of keystone. There is just no good reason to 
send out extra identity information (it isn't needed and can help to reduce 
token bloat etc).

--Morgan

Sent via mobile
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-27 Thread Adam Young

On 02/27/2014 12:52 PM, Jay Pipes wrote:

On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:

So a couple of things about this:


1) Today (and also true for Grizzly and Havana), the user can chose
what LDAP attribute should be returned as the user or group ID.  So it
is NOT a safe assumption today (ignoring any support for
domain-specific LDAP support) that the format of a user or group ID is
a 32 char UUID.  Quite often, I would think, that email address would
be chosen by a cloud provider as the LDAP id field, by default we use
the CN.  Since we really don't want to ever change the user or group
ID we have given out from keystone for a particular entity, this means
we need to update nova (or anything else) that has made a 32 char
assumption.

I don't believe this is correct. Keystone is the service that deals with
authentication. As such, Keystone should be the one and only one service
that should have any need whatsoever to need to understand a non-UUID
value for a user ID. The only value that should ever be communicated
*from* Keystone should be the UUID value of the user.


We never said it was a UUID, its only been that in the allocated by 
Keystone cases thus far.


userids are blobswell, character blocks, and are not expected to be 
interprested by the end services, but also not confined to be 32 
characters long.


Most LDAP use cases do not allow us to write data back the Directory. 
Federation has the same use cases.  That means that we are going to end 
up with a slew of ephemeral data in Keystone for no real value.






If the Keystone service uses LDAP or federation for alternative
authentication schemes, then Keystone should have a mapping table that
translates those elongated and non-UUID identifiers values (email
addresses, LDAP CNs, etc) into the UUID value that is then communicated
to all other OpenStack services.
The short of it is that we need to define a max userid length and stick 
to it.   Thus far, we've had 64 characters in Keystone, and I think we 
can stick with that.  Nova needs to expand its view to 64 chars.


The rest is Keystone internals to deal with.



Best,
-jay


2) In oder to support the ability for service providers to be able to
have the identity part of keystone be satisfied by a customer LDAP
(i.e. for a given domain, have a specific LDAP), then, as has been
stated, we need to subsequently, when handed an API call with just a
user or group ID, be able to route this call to the correct LDAP.
  Trying to keep true to the openstack design principles, we had
planned to encode a domain identifier into the user or group ID - i.e.
distribute the data to where it is needed, in ortherwords, the user
and group ID provide all the info we need to route the call to the
right place. Two implementations come to mind:
2a) Simply concatenate the user/group ID with the domain_id, plus some
separator and make a composite public facing ID.  e.g.
user_entity_id@@UUID_of_domain.  This would have a technical maximum
size of 64+2+64 (i.e. 130), although in reality since we control
domain_id and we know it is always 32 char UUID - in fact the max size
would be 98.  This has the problem of increasing the size of the
public facing field beyond the existing 64.  This is what we had
planned for IceHouse - and is currently in review.
2b) Use a similar concatenation idea as 2a), but limit the total size
to the existing 64. Since we control domain_id, we could (internally
and not visibly to the outside world), create a domain_index, that was
used in place of domain_id in the publicly visible field, to minimize
the number of chars it requires.  So the public facing composite ID
might be something like up to 54 chars of entity_id@@8 chars of
domain_index.  There is a chance, of course, that  the 54 char
restriction might be problematic for LDAP usersbut I doubt it.  We
would make that a restriction and if it really became a problem, we
could consider a field size increase at a later release
3) The alternative to 2a and 2b is to have, as had been suggested, an
internal mapping table that maps external facing entity_ids to a
domain plus local entity ID.  The problem with this idea is that:
- This could become a very big table (you will essentially have an
entry for every user in every corporate LDAP that has accessed a given
openstack)
- Since most LDAPs are RO, we will never see deletes...so we won't
know when (without some kind of garbage collection) to cull entries
- It obviously does not solve 1) - since existing LDAP support can
break the 32 char limit - and so it isn't true that this mapping table
causes all public facing entity IDs to be simple 32 char UUIDs


 From a delivery into IceHouse point of view any of the above are
possible, since the actual mapping used is relatively small part of
the patch.  I personally favor 2b), since it is simple, has less
moving parts and does not change any external facing requirements for
storage of user and group IDs (above and beyond what is 

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-27 Thread Jay Pipes
On Thu, 2014-02-27 at 17:02 -0500, Adam Young wrote:
 On 02/27/2014 12:52 PM, Jay Pipes wrote:
  On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote:
  So a couple of things about this:
 
 
  1) Today (and also true for Grizzly and Havana), the user can chose
  what LDAP attribute should be returned as the user or group ID.  So it
  is NOT a safe assumption today (ignoring any support for
  domain-specific LDAP support) that the format of a user or group ID is
  a 32 char UUID.  Quite often, I would think, that email address would
  be chosen by a cloud provider as the LDAP id field, by default we use
  the CN.  Since we really don't want to ever change the user or group
  ID we have given out from keystone for a particular entity, this means
  we need to update nova (or anything else) that has made a 32 char
  assumption.
  I don't believe this is correct. Keystone is the service that deals with
  authentication. As such, Keystone should be the one and only one service
  that should have any need whatsoever to need to understand a non-UUID
  value for a user ID. The only value that should ever be communicated
  *from* Keystone should be the UUID value of the user.
 
 We never said it was a UUID, its only been that in the allocated by 
 Keystone cases thus far.

Every user record in Keystone should have an allocated by Keystone
user ID that is a UUID value. Why? Because every single other OpenStack
service uses UUID values for identifiers (not names), and Keystone
should be no different.

 userids are blobswell, character blocks, and are not expected to be 
 interprested by the end services, but also not confined to be 32 
 characters long.

I would argue that they absolutely *should* be confined to 32 characters
(or technically, 16 bytes, which is the size of a UUID).

 Most LDAP use cases do not allow us to write data back the Directory. 
 Federation has the same use cases. 

Frankly, I don't care :) Let's say Super-Duper Identity Driver has 100
character IDs natively. Does that mean that these IDs need be sent all
over the OpenStack ecosystem? No. Have a translation table in Keystone
and spread the UUID user ID throughout OpenStack.

  That means that we are going to end 
 up with a slew of ephemeral data in Keystone for no real value.

Not sure why you think this is ephemeral data... What is ephemeral about
a non-UUID identifier from another auth system (federated or not) that
is mapped to a Keystone user UUID? If you look at virtually every
flexible authentication mechanism in virtually all web frameworks out
there, you will see something very similar, where there is a table that
maps external IDs to the IDs that are native to some platform.

Best,
-jay

 
  If the Keystone service uses LDAP or federation for alternative
  authentication schemes, then Keystone should have a mapping table that
  translates those elongated and non-UUID identifiers values (email
  addresses, LDAP CNs, etc) into the UUID value that is then communicated
  to all other OpenStack services.
 The short of it is that we need to define a max userid length and stick 
 to it.   Thus far, we've had 64 characters in Keystone, and I think we 
 can stick with that.  Nova needs to expand its view to 64 chars.
 
 The rest is Keystone internals to deal with.
 
 
  Best,
  -jay
 
  2) In oder to support the ability for service providers to be able to
  have the identity part of keystone be satisfied by a customer LDAP
  (i.e. for a given domain, have a specific LDAP), then, as has been
  stated, we need to subsequently, when handed an API call with just a
  user or group ID, be able to route this call to the correct LDAP.
Trying to keep true to the openstack design principles, we had
  planned to encode a domain identifier into the user or group ID - i.e.
  distribute the data to where it is needed, in ortherwords, the user
  and group ID provide all the info we need to route the call to the
  right place. Two implementations come to mind:
  2a) Simply concatenate the user/group ID with the domain_id, plus some
  separator and make a composite public facing ID.  e.g.
  user_entity_id@@UUID_of_domain.  This would have a technical maximum
  size of 64+2+64 (i.e. 130), although in reality since we control
  domain_id and we know it is always 32 char UUID - in fact the max size
  would be 98.  This has the problem of increasing the size of the
  public facing field beyond the existing 64.  This is what we had
  planned for IceHouse - and is currently in review.
  2b) Use a similar concatenation idea as 2a), but limit the total size
  to the existing 64. Since we control domain_id, we could (internally
  and not visibly to the outside world), create a domain_index, that was
  used in place of domain_id in the publicly visible field, to minimize
  the number of chars it requires.  So the public facing composite ID
  might be something like up to 54 chars of entity_id@@8 chars of
  domain_index.  There is a chance, of course, that  the 

Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-26 Thread Marco Fargetta
Hi Morgan,

On Tue, Feb 25, 2014 at 11:47:43AM -0800, Morgan Fainberg wrote:
 For purposes of supporting multiple backends for Identity (multiple LDAP, mix
 of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum
 size of the USER_ID field from an upper limit of 64 to an upper limit of 255.
 This change would not impact any currently assigned USER_IDs (they would 
 remain
 in the old simple UUID format), however, new USER_IDs would be increased to
 include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). 
 

in this case if a user would access with different systems (e.g. SAML with
portal, LDAP with CLI) it is mapped to two different identities inside keystone.
Is this correct? If so, is there any way to map an individual person with two
identities sharing resources?

Cheers,
Marco


 There is the obvious concern that projects are utilizing (and storing) the
 user_id in a field that cannot accommodate the increased upper limit. Before
 this change is merged in, it is important for the Keystone team to understand
 if there are any places that would be overflowed by the increased size.
 
 The review that would implement this change in size is https://
 review.openstack.org/#/c/74214 and is actively being worked on/reviewed.
 
 I have already spoken with the Nova team, and a single instance has been
 identified that would require a migration (that will have a fix proposed for
 the I3 timeline). 
 
 If there are any other known locations that would have issues with an 
 increased
 USER_ID size, or any concerns with this change to USER_ID format, please
 respond so that the issues/concerns can be addressed.  Again, the plan is not
 to change current USER_IDs but that new ones could be up to 255 characters in
 length.
 
 Cheers,
 Morgan Fainberg
 
 —
 Morgan Fainberg
 Principal Software Engineer
 Core Developer, Keystone
 m...@metacloud.com
 

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 

Eng. Marco Fargetta, PhD
 
Istituto Nazionale di Fisica Nucleare (INFN)
Catania, Italy

EMail: marco.farge...@ct.infn.it




smime.p7s
Description: S/MIME cryptographic signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-26 Thread Dolph Mathews
On Wed, Feb 26, 2014 at 4:23 AM, Marco Fargetta
marco.farge...@ct.infn.itwrote:

 Hi Morgan,

 On Tue, Feb 25, 2014 at 11:47:43AM -0800, Morgan Fainberg wrote:
  For purposes of supporting multiple backends for Identity (multiple
 LDAP, mix
  of LDAP and SQL, federation, etc) Keystone is planning to increase the
 maximum
  size of the USER_ID field from an upper limit of 64 to an upper limit of
 255.
  This change would not impact any currently assigned USER_IDs (they would
 remain
  in the old simple UUID format), however, new USER_IDs would be increased
 to
  include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER).
 

 in this case if a user would access with different systems (e.g. SAML with
 portal, LDAP with CLI) it is mapped to two different identities inside
 keystone.
 Is this correct? If so, is there any way to map an individual person with
 two

identities sharing resources?


That's correct - they'd result in different identities and keystone has no
means of presuming they are the same person. But I think you answered it
yourself, in that they would effectively be sharing resources with
themselves, so you'd just have to ensure they had the same authorization on
the same projects using both identities.



 Cheers,
 Marco


  There is the obvious concern that projects are utilizing (and storing)
 the
  user_id in a field that cannot accommodate the increased upper limit.
 Before
  this change is merged in, it is important for the Keystone team to
 understand
  if there are any places that would be overflowed by the increased size.
 
  The review that would implement this change in size is https://
  review.openstack.org/#/c/74214 and is actively being worked on/reviewed.
 
  I have already spoken with the Nova team, and a single instance has been
  identified that would require a migration (that will have a fix proposed
 for
  the I3 timeline).
 
  If there are any other known locations that would have issues with an
 increased
  USER_ID size, or any concerns with this change to USER_ID format, please
  respond so that the issues/concerns can be addressed.  Again, the plan
 is not
  to change current USER_IDs but that new ones could be up to 255
 characters in
  length.
 
  Cheers,
  Morgan Fainberg
 
  —
  Morgan Fainberg
  Principal Software Engineer
  Core Developer, Keystone
  m...@metacloud.com
 

  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 --
 
 Eng. Marco Fargetta, PhD

 Istituto Nazionale di Fisica Nucleare (INFN)
 Catania, Italy

 EMail: marco.farge...@ct.infn.it
 


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-26 Thread Dolph Mathews
On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com wrote:

 On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
  For purposes of supporting multiple backends for Identity (multiple
  LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
  increase the maximum size of the USER_ID field from an upper limit of
  64 to an upper limit of 255. This change would not impact any
  currently assigned USER_IDs (they would remain in the old simple UUID
  format), however, new USER_IDs would be increased to include the IDP
  identifier (e.g. USER_ID@@IDP_IDENTIFIER).

 -1

 I think a better solution would be to have a simple translation table
 only in Keystone that would store this longer identifier (for folks
 using federation and/or LDAP) along with the Keystone user UUID that is
 used in foreign key relations and other mapping tables through Keystone
 and other projects.


Morgan and I talked this suggestion through last night and agreed it's
probably the best approach, and has the benefit of zero impact on other
services, which is something we're obviously trying to avoid. I imagine it
could be as simple as a user_id to domain_id lookup table. All we really
care about is given a globally unique user ID, which identity backend is
the user from?

On the downside, it would likely become bloated with unused ephemeral user
IDs, so we'll need enough metadata about the mapping to implement a purging
behavior down the line.



 The only identifiers that would ever be communicated to any non-Keystone
 OpenStack endpoint would be the UUID user and tenant IDs.

  There is the obvious concern that projects are utilizing (and storing)
  the user_id in a field that cannot accommodate the increased upper
  limit. Before this change is merged in, it is important for the
  Keystone team to understand if there are any places that would be
  overflowed by the increased size.

 I would go so far as to say the user_id and tenant_id fields should be
 *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
 performance reasons. Lengthening commonly-used and frequently-joined
 identifier fields is not a good option, IMO.

 Best,
 -jay

  The review that would implement this change in size
  is https://review.openstack.org/#/c/74214 and is actively being worked
  on/reviewed.
 
 
  I have already spoken with the Nova team, and a single instance has
  been identified that would require a migration (that will have a fix
  proposed for the I3 timeline).
 
 
  If there are any other known locations that would have issues with an
  increased USER_ID size, or any concerns with this change to USER_ID
  format, please respond so that the issues/concerns can be addressed.
   Again, the plan is not to change current USER_IDs but that new ones
  could be up to 255 characters in length.
 
 
  Cheers,
  Morgan Fainberg
  —
  Morgan Fainberg
  Principal Software Engineer
  Core Developer, Keystone
  m...@metacloud.com
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-26 Thread Adam Young

On 02/26/2014 08:25 AM, Dolph Mathews wrote:


On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com 
mailto:jaypi...@gmail.com wrote:


On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
 For purposes of supporting multiple backends for Identity (multiple
 LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
 increase the maximum size of the USER_ID field from an upper
limit of
 64 to an upper limit of 255. This change would not impact any
 currently assigned USER_IDs (they would remain in the old simple
UUID
 format), however, new USER_IDs would be increased to include the IDP
 identifier (e.g. USER_ID@@IDP_IDENTIFIER).

-1

I think a better solution would be to have a simple translation table
only in Keystone that would store this longer identifier (for folks
using federation and/or LDAP) along with the Keystone user UUID
that is
used in foreign key relations and other mapping tables through
Keystone
and other projects.


Morgan and I talked this suggestion through last night and agreed it's 
probably the best approach, and has the benefit of zero impact on 
other services, which is something we're obviously trying to avoid. I 
imagine it could be as simple as a user_id to domain_id lookup table. 
All we really care about is given a globally unique user ID, which 
identity backend is the user from?


On the downside, it would likely become bloated with unused ephemeral 
user IDs, so we'll need enough metadata about the mapping to implement 
a purging behavior down the line.
UUIDs are 32 chars long.  Its really just uuid@@uuid that pushes us over 
the 64 character limit.
If we can shorten up the IDP_ID we can fit everything in 64 chars (which 
means only Nova needs to expand its column size)


What if we enumerated IDPs by index, from 1000 to  or 
something comparable, and then use the new domain_index (or prot domain 
id to not be a uuid).  Then the above scheme would work and no migration 
would be required.





The only identifiers that would ever be communicated to any
non-Keystone
OpenStack endpoint would be the UUID user and tenant IDs.

 There is the obvious concern that projects are utilizing (and
storing)
 the user_id in a field that cannot accommodate the increased upper
 limit. Before this change is merged in, it is important for the
 Keystone team to understand if there are any places that would be
 overflowed by the increased size.

I would go so far as to say the user_id and tenant_id fields should be
*reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
performance reasons. Lengthening commonly-used and frequently-joined
identifier fields is not a good option, IMO.

Best,
-jay

 The review that would implement this change in size
 is https://review.openstack.org/#/c/74214 and is actively being
worked
 on/reviewed.


 I have already spoken with the Nova team, and a single instance has
 been identified that would require a migration (that will have a fix
 proposed for the I3 timeline).


 If there are any other known locations that would have issues
with an
 increased USER_ID size, or any concerns with this change to USER_ID
 format, please respond so that the issues/concerns can be addressed.
  Again, the plan is not to change current USER_IDs but that new ones
 could be up to 255 characters in length.


 Cheers,
 Morgan Fainberg
 ---
 Morgan Fainberg
 Principal Software Engineer
 Core Developer, Keystone
 m...@metacloud.com mailto:m...@metacloud.com


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
mailto:OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
mailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-25 Thread Jay Pipes
On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote:
 For purposes of supporting multiple backends for Identity (multiple
 LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to
 increase the maximum size of the USER_ID field from an upper limit of
 64 to an upper limit of 255. This change would not impact any
 currently assigned USER_IDs (they would remain in the old simple UUID
 format), however, new USER_IDs would be increased to include the IDP
 identifier (e.g. USER_ID@@IDP_IDENTIFIER).

-1

I think a better solution would be to have a simple translation table
only in Keystone that would store this longer identifier (for folks
using federation and/or LDAP) along with the Keystone user UUID that is
used in foreign key relations and other mapping tables through Keystone
and other projects.

The only identifiers that would ever be communicated to any non-Keystone
OpenStack endpoint would be the UUID user and tenant IDs.

 There is the obvious concern that projects are utilizing (and storing)
 the user_id in a field that cannot accommodate the increased upper
 limit. Before this change is merged in, it is important for the
 Keystone team to understand if there are any places that would be
 overflowed by the increased size.

I would go so far as to say the user_id and tenant_id fields should be
*reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for
performance reasons. Lengthening commonly-used and frequently-joined
identifier fields is not a good option, IMO.

Best,
-jay

 The review that would implement this change in size
 is https://review.openstack.org/#/c/74214 and is actively being worked
 on/reviewed.
 
 
 I have already spoken with the Nova team, and a single instance has
 been identified that would require a migration (that will have a fix
 proposed for the I3 timeline). 
 
 
 If there are any other known locations that would have issues with an
 increased USER_ID size, or any concerns with this change to USER_ID
 format, please respond so that the issues/concerns can be addressed.
  Again, the plan is not to change current USER_IDs but that new ones
 could be up to 255 characters in length.
 
 
 Cheers,
 Morgan Fainberg
 —
 Morgan Fainberg
 Principal Software Engineer
 Core Developer, Keystone
 m...@metacloud.com
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255

2014-02-25 Thread Steven Hardy
On Tue, Feb 25, 2014 at 11:47:43AM -0800, Morgan Fainberg wrote:
 For purposes of supporting multiple backends for Identity (multiple LDAP, mix 
 of LDAP and SQL, federation, etc) Keystone is planning to increase the 
 maximum size of the USER_ID field from an upper limit of 64 to an upper limit 
 of 255. This change would not impact any currently assigned USER_IDs (they 
 would remain in the old simple UUID format), however, new USER_IDs would be 
 increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). 

Hmm, my immediate reaction is there must be a better way than mangling both
bits of data into the ID field, considering pretty much everything
everywhere else in openstack uses uuids for user-visible object identifiers.

 There is the obvious concern that projects are utilizing (and storing) the 
 user_id in a field that cannot accommodate the increased upper limit. Before 
 this change is merged in, it is important for the Keystone team to understand 
 if there are any places that would be overflowed by the increased size.
 
 The review that would implement this change in size is 
 https://review.openstack.org/#/c/74214 and is actively being worked 
 on/reviewed.
 
 I have already spoken with the Nova team, and a single instance has been 
 identified that would require a migration (that will have a fix proposed for 
 the I3 timeline). 
 
 If there are any other known locations that would have issues with an 
 increased USER_ID size, or any concerns with this change to USER_ID format, 
 please respond so that the issues/concerns can be addressed.  Again, the plan 
 is not to change current USER_IDs but that new ones could be up to 255 
 characters in length.

Yes, this will affect Heat in at least one place - we store the trustor
user ID when we create a trust between the stack owner and the heat service
user:

https://github.com/openstack/heat/blob/master/heat/db/sqlalchemy/migrate_repo/versions/027_user_creds_trusts.py#L23

IMHO this is coming pretty late in the cycle considering the potential
impact, but if this is definitely happening we can go ahead an update our
schema.

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev