Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Mar 3, 2014, at 11:32 AM, Jay Pipes jaypi...@gmail.com wrote: On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote: On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote: On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote: Having done some work with MySQL (specifically around similar data sets) and discussing the changes with some former coworkers (MySQL experts) I am inclined to believe the move from varchar to binary absolutely would increase performance like this. However, I would like to get some real benchmarks around it and if it really makes a difference we should get a smart UUID type into the common SQLlibs (can pgsql see a similar benefit? Db2?) I think this conversation. Should be split off from the keystone one at hand - I don't want valuable information / discussions to get lost. No disagreement on either point. However, this should be done after the standardization to a UUID user_id in Keystone, as a separate performance improvement patch. Agree? Best, -jay -1 The expectation in other projects has been that project_ids and user_ids are opaque strings. I need to see more clear benefit to enforcing stricter typing on these, because I think it might break a lot of things. What does Nova lose here? The proposal is to have Keystone's user_id values be UUIDs all the time. There would be a migration or helper script against Nova's database that would change all non-UUID user_id values to the Keystone UUID values. So I don’t have a problem with keystone internally using uuids, but forcing a migration of user identifiers isn’t something that should be taken lightly. One example is external logging and billing systems which now have to be migrated. I’m not opposed to the migration in principle. We may have to do a similar thing for project_ids with hierarchical multitenancy, for example. I just think we need a really good reason to do it, and the performance arguments just don’t seem good enough without a little empirical data. Vish If there's stuff in Nova that would break (which is doubtful, considering like you say, these are supposed to be opaque values and as such should not have any restrictions or validation on their value), then that is code in Nova that should be fixed. Honestly, we shouldn't accept poor or loose code just because stuff might break. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On March 4, 2014 at 10:45:02, Vishvananda Ishaya (vishvana...@gmail.com) wrote: On Mar 3, 2014, at 11:32 AM, Jay Pipes jaypi...@gmail.com wrote: On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote: On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote: On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote: Having done some work with MySQL (specifically around similar data sets) and discussing the changes with some former coworkers (MySQL experts) I am inclined to believe the move from varchar to binary absolutely would increase performance like this. However, I would like to get some real benchmarks around it and if it really makes a difference we should get a smart UUID type into the common SQLlibs (can pgsql see a similar benefit? Db2?) I think this conversation. Should be split off from the keystone one at hand - I don't want valuable information / discussions to get lost. No disagreement on either point. However, this should be done after the standardization to a UUID user_id in Keystone, as a separate performance improvement patch. Agree? Best, -jay -1 The expectation in other projects has been that project_ids and user_ids are opaque strings. I need to see more clear benefit to enforcing stricter typing on these, because I think it might break a lot of things. What does Nova lose here? The proposal is to have Keystone's user_id values be UUIDs all the time. There would be a migration or helper script against Nova's database that would change all non-UUID user_id values to the Keystone UUID values. So I don’t have a problem with keystone internally using uuids, but forcing a migration of user identifiers isn’t something that should be taken lightly. One example is external logging and billing systems which now have to be migrated. I’m not opposed to the migration in principle. We may have to do a similar thing for project_ids with hierarchical multitenancy, for example. I just think we need a really good reason to do it, and the performance arguments just don’t seem good enough without a little empirical data. Vish Any one of the proposals we’re planning on using will not affect any current IDs. Since the user_id is a blob, if we start issuing a new “id” format, ideally it shouldn’t matter as long as old IDs continue to work. If we do make any kind of migration for issued IDs I would expect that to be very deliberate and outside of this change set. Specifically this change set would enable multiple LDAP backends (among other user_id uniqueness benefits for federation, etc). I am very concerned about the external tools that reference IDs we currently have. —Morgan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Mon, Mar 3, 2014 at 8:48 AM, Jay Pipes jaypi...@gmail.com wrote: On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote: Having done some work with MySQL (specifically around similar data sets) and discussing the changes with some former coworkers (MySQL experts) I am inclined to believe the move from varchar to binary absolutely would increase performance like this. However, I would like to get some real benchmarks around it and if it really makes a difference we should get a smart UUID type into the common SQLlibs (can pgsql see a similar benefit? Db2?) I think this conversation. Should be split off from the keystone one at hand - I don't want valuable information / discussions to get lost. No disagreement on either point. However, this should be done after the standardization to a UUID user_id in Keystone, as a separate performance improvement patch. Agree? Best, -jay ++ https://blueprints.launchpad.net/keystone/+spec/sql-id-binary-16 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote: On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote: Having done some work with MySQL (specifically around similar data sets) and discussing the changes with some former coworkers (MySQL experts) I am inclined to believe the move from varchar to binary absolutely would increase performance like this. However, I would like to get some real benchmarks around it and if it really makes a difference we should get a smart UUID type into the common SQLlibs (can pgsql see a similar benefit? Db2?) I think this conversation. Should be split off from the keystone one at hand - I don't want valuable information / discussions to get lost. No disagreement on either point. However, this should be done after the standardization to a UUID user_id in Keystone, as a separate performance improvement patch. Agree? Best, -jay -1 The expectation in other projects has been that project_ids and user_ids are opaque strings. I need to see more clear benefit to enforcing stricter typing on these, because I think it might break a lot of things. Vish signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Sat, Mar 1, 2014 at 12:51 PM, Jay Pipes jaypi...@gmail.com wrote: On Fri, 2014-02-28 at 15:25 -0800, Mark Washenberger wrote: I believe we have some agreement here. Other openstack services should be able to use a strongly typed identifier for users. I just think if we want to go that route, we probably need to create a new field to act as the proper user uuid, rather than repurposing the existing field. It sounds like many existing LDAP deployments would break if we repurpose the existing field. Hi Mark, Please see my earlier response on this thread. I am proposing putting external identifiers into a mapping table that would correlate a Keystone UUID user ID with external identifiers (of variable length). The thing you seem to be missing is that the current user-id attribute is an external identifier depending on the identity backend you're using today. For example in the LDAP driver it is the CN by default (which is ridiculous for a large number of reasons, but let's leave those aside.) So if you want to create a new, strongly typed internal uuid identifier that makes the db performance scream, more power to you. But it's going to have to be a new field. Once authentication has occurred (with any auth backend including LDAP), Keystone would only communicate to the other OpenStack services the UUID user ID from Keystone. This would indeed require a migration to each non-Keystone service that stores the user IDs as-is from Keystone currently (such as Glance or Nova). Once the migrations are run, then only UUID values would be stored, and further migrations could be run that would streamline the columns that stores these user IDs to a more efficient CHAR(32) or BINARY(16) internal storage format. Hope that clears things up. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote: On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote: On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote: Having done some work with MySQL (specifically around similar data sets) and discussing the changes with some former coworkers (MySQL experts) I am inclined to believe the move from varchar to binary absolutely would increase performance like this. However, I would like to get some real benchmarks around it and if it really makes a difference we should get a smart UUID type into the common SQLlibs (can pgsql see a similar benefit? Db2?) I think this conversation. Should be split off from the keystone one at hand - I don't want valuable information / discussions to get lost. No disagreement on either point. However, this should be done after the standardization to a UUID user_id in Keystone, as a separate performance improvement patch. Agree? Best, -jay -1 The expectation in other projects has been that project_ids and user_ids are opaque strings. I need to see more clear benefit to enforcing stricter typing on these, because I think it might break a lot of things. What does Nova lose here? The proposal is to have Keystone's user_id values be UUIDs all the time. There would be a migration or helper script against Nova's database that would change all non-UUID user_id values to the Keystone UUID values. If there's stuff in Nova that would break (which is doubtful, considering like you say, these are supposed to be opaque values and as such should not have any restrictions or validation on their value), then that is code in Nova that should be fixed. Honestly, we shouldn't accept poor or loose code just because stuff might break. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Mon, 2014-03-03 at 11:18 -0800, Mark Washenberger wrote: On Sat, Mar 1, 2014 at 12:51 PM, Jay Pipes jaypi...@gmail.com wrote: On Fri, 2014-02-28 at 15:25 -0800, Mark Washenberger wrote: I believe we have some agreement here. Other openstack services should be able to use a strongly typed identifier for users. I just think if we want to go that route, we probably need to create a new field to act as the proper user uuid, rather than repurposing the existing field. It sounds like many existing LDAP deployments would break if we repurpose the existing field. Hi Mark, Please see my earlier response on this thread. I am proposing putting external identifiers into a mapping table that would correlate a Keystone UUID user ID with external identifiers (of variable length). The thing you seem to be missing is that the current user-id attribute is an external identifier depending on the identity backend you're using today. For example in the LDAP driver it is the CN by default (which is ridiculous for a large number of reasons, but let's leave those aside.) So if you want to create a new, strongly typed internal uuid identifier that makes the db performance scream, more power to you. But it's going to have to be a new field. No, it won't. The proposal is to create a new table that maps a UUID value to the external identifier value. The migration would essentially do this pseudocode: recs = get_identity_records() for rec in recs: if not is_like_uuid(rec.user_id): external_id = rec.user_id new_uuid = uuid.uuid4() insert_into_external_ids(new_uuid, external_id) rec.user_id = new_uuid save_identity_record(rec) Cascading updates on the user_id field to its child relations will take care of the changes in user_id column in foriegn keys. A migration script in, say, Nova, would do something like: UPDATE instances SET owner_id = ext.user_id FROM instances JOIN keystone.external_id_mapping ext ON instances.owner_id = ext.external_id; Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On 03/03/2014 02:32 PM, Jay Pipes wrote: On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote: On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote: On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote: Having done some work with MySQL (specifically around similar data sets) and discussing the changes with some former coworkers (MySQL experts) I am inclined to believe the move from varchar to binary absolutely would increase performance like this. However, I would like to get some real benchmarks around it and if it really makes a difference we should get a smart UUID type into the common SQLlibs (can pgsql see a similar benefit? Db2?) I think this conversation. Should be split off from the keystone one at hand - I don't want valuable information / discussions to get lost. No disagreement on either point. However, this should be done after the standardization to a UUID user_id in Keystone, as a separate performance improvement patch. Agree? Best, -jay -1 The expectation in other projects has been that project_ids and user_ids are opaque strings. I need to see more clear benefit to enforcing stricter typing on these, because I think it might break a lot of things. What does Nova lose here? The proposal is to have Keystone's user_id values be UUIDs all the time. There would be a migration or helper script against Nova's database that would change all non-UUID user_id values to the Keystone UUID values. If there's stuff in Nova that would break (which is doubtful, considering like you say, these are supposed to be opaque values and as such should not have any restrictions or validation on their value), then that is code in Nova that should be fixed. Honestly, we shouldn't accept poor or loose code just because stuff might break. Why do you think that UUIDs would be some sort of magic bullet? Thus far, we've stated that IDs are opaque strings, and that they will be URL safe. Adding additional Database lokups on top of LDAP calls is going to make LDAP code paths even more convoluted than they are now. I'd like to avoid that. And I don't see the benefit. I'd be interested in hearing why CN is a bad default choice for LDAP; we have to use a very high up theRFE chain object class by default and didn't have that many attributes to chose from. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
Excerpts from Adam Young's message of 2014-03-03 23:06:58 -0800: On 03/03/2014 02:32 PM, Jay Pipes wrote: On Mon, 2014-03-03 at 11:09 -0800, Vishvananda Ishaya wrote: On Mar 3, 2014, at 6:48 AM, Jay Pipes jaypi...@gmail.com wrote: On Sun, 2014-03-02 at 12:05 -0800, Morgan Fainberg wrote: Having done some work with MySQL (specifically around similar data sets) and discussing the changes with some former coworkers (MySQL experts) I am inclined to believe the move from varchar to binary absolutely would increase performance like this. However, I would like to get some real benchmarks around it and if it really makes a difference we should get a smart UUID type into the common SQLlibs (can pgsql see a similar benefit? Db2?) I think this conversation. Should be split off from the keystone one at hand - I don't want valuable information / discussions to get lost. No disagreement on either point. However, this should be done after the standardization to a UUID user_id in Keystone, as a separate performance improvement patch. Agree? Best, -jay -1 The expectation in other projects has been that project_ids and user_ids are opaque strings. I need to see more clear benefit to enforcing stricter typing on these, because I think it might break a lot of things. What does Nova lose here? The proposal is to have Keystone's user_id values be UUIDs all the time. There would be a migration or helper script against Nova's database that would change all non-UUID user_id values to the Keystone UUID values. If there's stuff in Nova that would break (which is doubtful, considering like you say, these are supposed to be opaque values and as such should not have any restrictions or validation on their value), then that is code in Nova that should be fixed. Honestly, we shouldn't accept poor or loose code just because stuff might break. Why do you think that UUIDs would be some sort of magic bullet? Thus far, we've stated that IDs are opaque strings, and that they will be URL safe. Adding additional Database lokups on top of LDAP calls is going to make LDAP code paths even more convoluted than they are now. I'd like to avoid that. And I don't see the benefit. Each row in each table throughout OpenStack that has to treat the user id from Keystone as an opaque ID must then store the keystone id as a 64 byte string. Being able to make it a 16 byte binary value means a significant saving at scale. If you're using LDAP, you're only going to store that CN once per user, and only going to look it up when said user tries to log in. After that, the uuid will be passed around. I'd be interested in hearing why CN is a bad default choice for LDAP; we have to use a very high up theRFE chain object class by default and didn't have that many attributes to chose from. CN is fine. External IDs of various shapes and sizes can be used. The point is to make keystone get a little more opionated about its' id strings in order to make the rest of OpenStack more scalable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On 03/01/2014 08:00 PM, Clint Byrum wrote: Excerpts from Robert Collins's message of 2014-03-01 14:26:57 -0800: On 1 March 2014 13:28, Clint Byrum cl...@fewbar.com wrote: +1. A Keystone record belongs to Keystone, and it should have a Keystone ID. External records that are linked should be linked separately. It may not be obvious to everyone, but MySQL uses B-trees for indexes. B-trees cannot have variable-length keys. Hmm, B-Trees and B+-Trees both can have variable length keys. I'll accept an assertion that MySQL index B-trees cannot - but we should be precise here, because its not a global limitation. Sorry, I misspoke, _InnoDB's_ b-tree's cannot have variable length keys. :-P On a previous project we did a transition from varchar based UUID to binary based UUID in MySQL. The micro benchmarks on joins got faster by a factor of 10,000 (yes 10k). Granted, MySQL has evolved since then, and this was a micro benchmark, however this is definitely work considering. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
Having done some work with MySQL (specifically around similar data sets) and discussing the changes with some former coworkers (MySQL experts) I am inclined to believe the move from varchar to binary absolutely would increase performance like this. However, I would like to get some real benchmarks around it and if it really makes a difference we should get a smart UUID type into the common SQLlibs (can pgsql see a similar benefit? Db2?) I think this conversation. Should be split off from the keystone one at hand - I don't want valuable information / discussions to get lost. Cheers, Morgan. Sent via mobile On Sunday, March 2, 2014, Sean Dague s...@dague.net wrote: On 03/01/2014 08:00 PM, Clint Byrum wrote: Excerpts from Robert Collins's message of 2014-03-01 14:26:57 -0800: On 1 March 2014 13:28, Clint Byrum cl...@fewbar.com javascript:; wrote: +1. A Keystone record belongs to Keystone, and it should have a Keystone ID. External records that are linked should be linked separately. It may not be obvious to everyone, but MySQL uses B-trees for indexes. B-trees cannot have variable-length keys. Hmm, B-Trees and B+-Trees both can have variable length keys. I'll accept an assertion that MySQL index B-trees cannot - but we should be precise here, because its not a global limitation. Sorry, I misspoke, _InnoDB's_ b-tree's cannot have variable length keys. :-P On a previous project we did a transition from varchar based UUID to binary based UUID in MySQL. The micro benchmarks on joins got faster by a factor of 10,000 (yes 10k). Granted, MySQL has evolved since then, and this was a micro benchmark, however this is definitely work considering. -Sean -- Sean Dague Samsung Research America s...@dague.net javascript:; / sean.da...@samsung.com javascript:; http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Fri, 2014-02-28 at 15:25 -0800, Mark Washenberger wrote: I believe we have some agreement here. Other openstack services should be able to use a strongly typed identifier for users. I just think if we want to go that route, we probably need to create a new field to act as the proper user uuid, rather than repurposing the existing field. It sounds like many existing LDAP deployments would break if we repurpose the existing field. Hi Mark, Please see my earlier response on this thread. I am proposing putting external identifiers into a mapping table that would correlate a Keystone UUID user ID with external identifiers (of variable length). Once authentication has occurred (with any auth backend including LDAP), Keystone would only communicate to the other OpenStack services the UUID user ID from Keystone. This would indeed require a migration to each non-Keystone service that stores the user IDs as-is from Keystone currently (such as Glance or Nova). Once the migrations are run, then only UUID values would be stored, and further migrations could be run that would streamline the columns that stores these user IDs to a more efficient CHAR(32) or BINARY(16) internal storage format. Hope that clears things up. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On 1 March 2014 13:28, Clint Byrum cl...@fewbar.com wrote: +1. A Keystone record belongs to Keystone, and it should have a Keystone ID. External records that are linked should be linked separately. It may not be obvious to everyone, but MySQL uses B-trees for indexes. B-trees cannot have variable-length keys. Hmm, B-Trees and B+-Trees both can have variable length keys. I'll accept an assertion that MySQL index B-trees cannot - but we should be precise here, because its not a global limitation. That said, our DB design basically needs to take into account the union of limitations of all target DB platforms, so we do still need to cater for this. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Wed, Feb 26, 2014 at 5:25 AM, Dolph Mathews dolph.math...@gmail.comwrote: On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com wrote: On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote: For purposes of supporting multiple backends for Identity (multiple LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum size of the USER_ID field from an upper limit of 64 to an upper limit of 255. This change would not impact any currently assigned USER_IDs (they would remain in the old simple UUID format), however, new USER_IDs would be increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). -1 I think a better solution would be to have a simple translation table only in Keystone that would store this longer identifier (for folks using federation and/or LDAP) along with the Keystone user UUID that is used in foreign key relations and other mapping tables through Keystone and other projects. Morgan and I talked this suggestion through last night and agreed it's probably the best approach, and has the benefit of zero impact on other services, which is something we're obviously trying to avoid. I imagine it could be as simple as a user_id to domain_id lookup table. All we really care about is given a globally unique user ID, which identity backend is the user from? On the downside, it would likely become bloated with unused ephemeral user IDs, so we'll need enough metadata about the mapping to implement a purging behavior down the line. Is this approach planning on reusing the existing user-id field, then? It seems like this creates a migration problem for folks who are currently using user-ids that are generated by their identity backends. The only identifiers that would ever be communicated to any non-Keystone OpenStack endpoint would be the UUID user and tenant IDs. There is the obvious concern that projects are utilizing (and storing) the user_id in a field that cannot accommodate the increased upper limit. Before this change is merged in, it is important for the Keystone team to understand if there are any places that would be overflowed by the increased size. I would go so far as to say the user_id and tenant_id fields should be *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for performance reasons. Lengthening commonly-used and frequently-joined identifier fields is not a good option, IMO. Best, -jay The review that would implement this change in size is https://review.openstack.org/#/c/74214 and is actively being worked on/reviewed. I have already spoken with the Nova team, and a single instance has been identified that would require a migration (that will have a fix proposed for the I3 timeline). If there are any other known locations that would have issues with an increased USER_ID size, or any concerns with this change to USER_ID format, please respond so that the issues/concerns can be addressed. Again, the plan is not to change current USER_IDs but that new ones could be up to 255 characters in length. Cheers, Morgan Fainberg -- Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
Hi Mark, So we would not modify any existing IDs, so no migration required. Henry On 28 Feb 2014, at 17:38, Mark Washenberger mark.washenber...@markwash.net wrote: On Wed, Feb 26, 2014 at 5:25 AM, Dolph Mathews dolph.math...@gmail.com wrote: On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com wrote: On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote: For purposes of supporting multiple backends for Identity (multiple LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum size of the USER_ID field from an upper limit of 64 to an upper limit of 255. This change would not impact any currently assigned USER_IDs (they would remain in the old simple UUID format), however, new USER_IDs would be increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). -1 I think a better solution would be to have a simple translation table only in Keystone that would store this longer identifier (for folks using federation and/or LDAP) along with the Keystone user UUID that is used in foreign key relations and other mapping tables through Keystone and other projects. Morgan and I talked this suggestion through last night and agreed it's probably the best approach, and has the benefit of zero impact on other services, which is something we're obviously trying to avoid. I imagine it could be as simple as a user_id to domain_id lookup table. All we really care about is given a globally unique user ID, which identity backend is the user from? On the downside, it would likely become bloated with unused ephemeral user IDs, so we'll need enough metadata about the mapping to implement a purging behavior down the line. Is this approach planning on reusing the existing user-id field, then? It seems like this creates a migration problem for folks who are currently using user-ids that are generated by their identity backends. The only identifiers that would ever be communicated to any non-Keystone OpenStack endpoint would be the UUID user and tenant IDs. There is the obvious concern that projects are utilizing (and storing) the user_id in a field that cannot accommodate the increased upper limit. Before this change is merged in, it is important for the Keystone team to understand if there are any places that would be overflowed by the increased size. I would go so far as to say the user_id and tenant_id fields should be *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for performance reasons. Lengthening commonly-used and frequently-joined identifier fields is not a good option, IMO. Best, -jay The review that would implement this change in size is https://review.openstack.org/#/c/74214 and is actively being worked on/reviewed. I have already spoken with the Nova team, and a single instance has been identified that would require a migration (that will have a fix proposed for the I3 timeline). If there are any other known locations that would have issues with an increased USER_ID size, or any concerns with this change to USER_ID format, please respond so that the issues/concerns can be addressed. Again, the plan is not to change current USER_IDs but that new ones could be up to 255 characters in length. Cheers, Morgan Fainberg — Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash hen...@linux.vnet.ibm.comwrote: Hi Mark, So we would not modify any existing IDs, so no migration required. Okay, I just want to be painfully clear--we're not proposing changing any of the current restrictions on the user-id field. We will not: - require it to be a uuid - encode it as binary instead of char - shrink its size below the current 64 characters Any of those could require a migration for existing IDs depending on how your identity driver functions. If I'm just being Chicken Little, please reassure me once more and I'll be quiet :-) Henry On 28 Feb 2014, at 17:38, Mark Washenberger mark.washenber...@markwash.net wrote: On Wed, Feb 26, 2014 at 5:25 AM, Dolph Mathews dolph.math...@gmail.comwrote: On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com wrote: On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote: For purposes of supporting multiple backends for Identity (multiple LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum size of the USER_ID field from an upper limit of 64 to an upper limit of 255. This change would not impact any currently assigned USER_IDs (they would remain in the old simple UUID format), however, new USER_IDs would be increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). -1 I think a better solution would be to have a simple translation table only in Keystone that would store this longer identifier (for folks using federation and/or LDAP) along with the Keystone user UUID that is used in foreign key relations and other mapping tables through Keystone and other projects. Morgan and I talked this suggestion through last night and agreed it's probably the best approach, and has the benefit of zero impact on other services, which is something we're obviously trying to avoid. I imagine it could be as simple as a user_id to domain_id lookup table. All we really care about is given a globally unique user ID, which identity backend is the user from? On the downside, it would likely become bloated with unused ephemeral user IDs, so we'll need enough metadata about the mapping to implement a purging behavior down the line. Is this approach planning on reusing the existing user-id field, then? It seems like this creates a migration problem for folks who are currently using user-ids that are generated by their identity backends. The only identifiers that would ever be communicated to any non-Keystone OpenStack endpoint would be the UUID user and tenant IDs. There is the obvious concern that projects are utilizing (and storing) the user_id in a field that cannot accommodate the increased upper limit. Before this change is merged in, it is important for the Keystone team to understand if there are any places that would be overflowed by the increased size. I would go so far as to say the user_id and tenant_id fields should be *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for performance reasons. Lengthening commonly-used and frequently-joined identifier fields is not a good option, IMO. Best, -jay The review that would implement this change in size is https://review.openstack.org/#/c/74214 and is actively being worked on/reviewed. I have already spoken with the Nova team, and a single instance has been identified that would require a migration (that will have a fix proposed for the I3 timeline). If there are any other known locations that would have issues with an increased USER_ID size, or any concerns with this change to USER_ID format, please respond so that the issues/concerns can be addressed. Again, the plan is not to change current USER_IDs but that new ones could be up to 255 characters in length. Cheers, Morgan Fainberg -- Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Fri, 2014-02-28 at 13:10 -0800, Mark Washenberger wrote: On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash hen...@linux.vnet.ibm.com wrote: Hi Mark, So we would not modify any existing IDs, so no migration required. Okay, I just want to be painfully clear--we're not proposing changing any of the current restrictions on the user-id field. We will not: - require it to be a uuid - encode it as binary instead of char - shrink its size below the current 64 characters The first would be required for the real solution. The second and third are performance improvements. Any of those could require a migration for existing IDs depending on how your identity driver functions. Personally, I think to fix this issue permanently and properly, migrations for database schemas of Glance and Nova would indeed need to accompany a proposed patch that restricts the Keystone external user ID to only a UUID value. I entirely disagree with allowing non-UUID values for the user ID value that is exposed outside of Keystone. All other solutions (including the proposals to continue using the user_id fields with non-UUID values) are just hacks IMO. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Fri, Feb 28, 2014 at 2:26 PM, Jay Pipes jaypi...@gmail.com wrote: On Fri, 2014-02-28 at 13:10 -0800, Mark Washenberger wrote: On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash hen...@linux.vnet.ibm.com wrote: Hi Mark, So we would not modify any existing IDs, so no migration required. Okay, I just want to be painfully clear--we're not proposing changing any of the current restrictions on the user-id field. We will not: - require it to be a uuid - encode it as binary instead of char - shrink its size below the current 64 characters The first would be required for the real solution. The second and third are performance improvements. Any of those could require a migration for existing IDs depending on how your identity driver functions. Personally, I think to fix this issue permanently and properly, migrations for database schemas of Glance and Nova would indeed need to accompany a proposed patch that restricts the Keystone external user ID to only a UUID value. I entirely disagree with allowing non-UUID values for the user ID value that is exposed outside of Keystone. All other solutions (including the proposals to continue using the user_id fields with non-UUID values) are just hacks IMO. I believe we have some agreement here. Other openstack services should be able to use a strongly typed identifier for users. I just think if we want to go that route, we probably need to create a new field to act as the proper user uuid, rather than repurposing the existing field. It sounds like many existing LDAP deployments would break if we repurpose the existing field. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
Excerpts from Jay Pipes's message of 2014-02-28 14:26:26 -0800: On Fri, 2014-02-28 at 13:10 -0800, Mark Washenberger wrote: On Fri, Feb 28, 2014 at 10:39 AM, Henry Nash hen...@linux.vnet.ibm.com wrote: Hi Mark, So we would not modify any existing IDs, so no migration required. Okay, I just want to be painfully clear--we're not proposing changing any of the current restrictions on the user-id field. We will not: - require it to be a uuid - encode it as binary instead of char - shrink its size below the current 64 characters The first would be required for the real solution. The second and third are performance improvements. Any of those could require a migration for existing IDs depending on how your identity driver functions. Personally, I think to fix this issue permanently and properly, migrations for database schemas of Glance and Nova would indeed need to accompany a proposed patch that restricts the Keystone external user ID to only a UUID value. I entirely disagree with allowing non-UUID values for the user ID value that is exposed outside of Keystone. All other solutions (including the proposals to continue using the user_id fields with non-UUID values) are just hacks IMO. +1. A Keystone record belongs to Keystone, and it should have a Keystone ID. External records that are linked should be linked separately. It may not be obvious to everyone, but MySQL uses B-trees for indexes. B-trees cannot have variable-length keys. So varchar(64) means 64-byte index keys. If you aren't careful and let that column be stored as utf-8, this actually means *192* byte index keys, because MySQL uses 3-byte utf-8 and thus a 64 character column could have 192 bytes. This does not scale well as you are doing index scans and range lookups, not to mention just generally raising memory and I/O pressure on the server. What Jay is suggesting is that we actually be opinionated and store Keystone users with 16-byte binary UUID's, and only ever use the UUID (in the 32-byte text notation where appropriate) when returning a keystone ID. Then only the initial authentication step where the user presents external identification requires access to anything larger, allowing all other Keystone operations to perform much better and keeping the keystone database footprint smaller. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
So a couple of things about this: 1) Today (and also true for Grizzly and Havana), the user can chose what LDAP attribute should be returned as the user or group ID. So it is NOT a safe assumption today (ignoring any support for domain-specific LDAP support) that the format of a user or group ID is a 32 char UUID. Quite often, I would think, that email address would be chosen by a cloud provider as the LDAP id field, by default we use the CN. Since we really don't want to ever change the user or group ID we have given out from keystone for a particular entity, this means we need to update nova (or anything else) that has made a 32 char assumption. 2) In oder to support the ability for service providers to be able to have the identity part of keystone be satisfied by a customer LDAP (i.e. for a given domain, have a specific LDAP), then, as has been stated, we need to subsequently, when handed an API call with just a user or group ID, be able to route this call to the correct LDAP. Trying to keep true to the openstack design principles, we had planned to encode a domain identifier into the user or group ID - i.e. distribute the data to where it is needed, in ortherwords, the user and group ID provide all the info we need to route the call to the right place. Two implementations come to mind: 2a) Simply concatenate the user/group ID with the domain_id, plus some separator and make a composite public facing ID. e.g. user_entity_id@@UUID_of_domain. This would have a technical maximum size of 64+2+64 (i.e. 130), although in reality since we control domain_id and we know it is always 32 char UUID - in fact the max size would be 98. This has the problem of increasing the size of the public facing field beyond the existing 64. This is what we had planned for IceHouse - and is currently in review. 2b) Use a similar concatenation idea as 2a), but limit the total size to the existing 64. Since we control domain_id, we could (internally and not visibly to the outside world), create a domain_index, that was used in place of domain_id in the publicly visible field, to minimize the number of chars it requires. So the public facing composite ID might be something like up to 54 chars of entity_id@@8 chars of domain_index. There is a chance, of course, that the 54 char restriction might be problematic for LDAP usersbut I doubt it. We would make that a restriction and if it really became a problem, we could consider a field size increase at a later release 3) The alternative to 2a and 2b is to have, as had been suggested, an internal mapping table that maps external facing entity_ids to a domain plus local entity ID. The problem with this idea is that: - This could become a very big table (you will essentially have an entry for every user in every corporate LDAP that has accessed a given openstack) - Since most LDAPs are RO, we will never see deletes...so we won't know when (without some kind of garbage collection) to cull entries - It obviously does not solve 1) - since existing LDAP support can break the 32 char limit - and so it isn't true that this mapping table causes all public facing entity IDs to be simple 32 char UUIDs From a delivery into IceHouse point of view any of the above are possible, since the actual mapping used is relatively small part of the patch. I personally favor 2b), since it is simple, has less moving parts and does not change any external facing requirements for storage of user and group IDs (above and beyond what is true today). Henry On 27 Feb 2014, at 03:46, Adam Young ayo...@redhat.com wrote: On 02/26/2014 08:25 AM, Dolph Mathews wrote: On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com wrote: On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote: For purposes of supporting multiple backends for Identity (multiple LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum size of the USER_ID field from an upper limit of 64 to an upper limit of 255. This change would not impact any currently assigned USER_IDs (they would remain in the old simple UUID format), however, new USER_IDs would be increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). -1 I think a better solution would be to have a simple translation table only in Keystone that would store this longer identifier (for folks using federation and/or LDAP) along with the Keystone user UUID that is used in foreign key relations and other mapping tables through Keystone and other projects. Morgan and I talked this suggestion through last night and agreed it's probably the best approach, and has the benefit of zero impact on other services, which is something we're obviously trying to avoid. I imagine it could be as simple as a user_id to domain_id lookup table. All we really care about is given a globally unique user ID, which identity backend is the user from? On the
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On 27 Feb 2014, at 17:52, Jay Pipes jaypi...@gmail.com wrote: On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote: So a couple of things about this: 1) Today (and also true for Grizzly and Havana), the user can chose what LDAP attribute should be returned as the user or group ID. So it is NOT a safe assumption today (ignoring any support for domain-specific LDAP support) that the format of a user or group ID is a 32 char UUID. Quite often, I would think, that email address would be chosen by a cloud provider as the LDAP id field, by default we use the CN. Since we really don't want to ever change the user or group ID we have given out from keystone for a particular entity, this means we need to update nova (or anything else) that has made a 32 char assumption. I don't believe this is correct. Keystone is the service that deals with authentication. As such, Keystone should be the one and only one service that should have any need whatsoever to need to understand a non-UUID value for a user ID. The only value that should ever be communicated *from* Keystone should be the UUID value of the user. If the Keystone service uses LDAP or federation for alternative authentication schemes, then Keystone should have a mapping table that translates those elongated and non-UUID identifiers values (email addresses, LDAP CNs, etc) into the UUID value that is then communicated to all other OpenStack services. Best, -jay So I think that's a perfectly reasonable point of viewour challenge is that this isn't what Keystone has done to datee.g. anyone using a RO LDAP today is probably exposing non-UUID identifiers out into nova and other projects (and maybe outside of openstack altogether). We can't (without breaking them) just change the IDs for any existing LDAP entities. So they best we could do is to say something like, new entities (and perhaps only those in domain-specific backends) would use such a mapping capability. Henry 2) In oder to support the ability for service providers to be able to have the identity part of keystone be satisfied by a customer LDAP (i.e. for a given domain, have a specific LDAP), then, as has been stated, we need to subsequently, when handed an API call with just a user or group ID, be able to route this call to the correct LDAP. Trying to keep true to the openstack design principles, we had planned to encode a domain identifier into the user or group ID - i.e. distribute the data to where it is needed, in ortherwords, the user and group ID provide all the info we need to route the call to the right place. Two implementations come to mind: 2a) Simply concatenate the user/group ID with the domain_id, plus some separator and make a composite public facing ID. e.g. user_entity_id@@UUID_of_domain. This would have a technical maximum size of 64+2+64 (i.e. 130), although in reality since we control domain_id and we know it is always 32 char UUID - in fact the max size would be 98. This has the problem of increasing the size of the public facing field beyond the existing 64. This is what we had planned for IceHouse - and is currently in review. 2b) Use a similar concatenation idea as 2a), but limit the total size to the existing 64. Since we control domain_id, we could (internally and not visibly to the outside world), create a domain_index, that was used in place of domain_id in the publicly visible field, to minimize the number of chars it requires. So the public facing composite ID might be something like up to 54 chars of entity_id@@8 chars of domain_index. There is a chance, of course, that the 54 char restriction might be problematic for LDAP usersbut I doubt it. We would make that a restriction and if it really became a problem, we could consider a field size increase at a later release 3) The alternative to 2a and 2b is to have, as had been suggested, an internal mapping table that maps external facing entity_ids to a domain plus local entity ID. The problem with this idea is that: - This could become a very big table (you will essentially have an entry for every user in every corporate LDAP that has accessed a given openstack) - Since most LDAPs are RO, we will never see deletes...so we won't know when (without some kind of garbage collection) to cull entries - It obviously does not solve 1) - since existing LDAP support can break the 32 char limit - and so it isn't true that this mapping table causes all public facing entity IDs to be simple 32 char UUIDs From a delivery into IceHouse point of view any of the above are possible, since the actual mapping used is relatively small part of the patch. I personally favor 2b), since it is simple, has less moving parts and does not change any external facing requirements for storage of user and group IDs (above and beyond what is true today). Henry On 27 Feb 2014, at 03:46, Adam Young ayo...@redhat.com wrote: On 02/26/2014
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Thu, Feb 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.com wrote: On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote: So a couple of things about this: 1) Today (and also true for Grizzly and Havana), the user can chose what LDAP attribute should be returned as the user or group ID. So it is NOT a safe assumption today (ignoring any support for domain-specific LDAP support) that the format of a user or group ID is a 32 char UUID. Quite often, I would think, that email address would be chosen by a cloud provider as the LDAP id field, by default we use the CN. Since we really don't want to ever change the user or group ID we have given out from keystone for a particular entity, this means we need to update nova (or anything else) that has made a 32 char assumption. I don't believe this is correct. Keystone is the service that deals with authentication. As such, Keystone should be the one and only one service that should have any need whatsoever to need to understand a non-UUID value for a user ID. The only value that should ever be communicated *from* Keystone should be the UUID value of the user. +++ If the Keystone service uses LDAP or federation for alternative authentication schemes, then Keystone should have a mapping table that translates those elongated and non-UUID identifiers values (email addresses, LDAP CNs, etc) into the UUID value that is then communicated to all other OpenStack services. I'd take it one step further and say that at some point, keystone should stop leaking identity details such as user name / ID into OpenStack (they shouldn't appear in tokens, and shouldn't be expected output of auth_token). The use cases that require them are weak and would be better served by pure multitenant RBAC, ABAC, etc. There are a lot of blockers to making this happen (including a few in keystone's own API), but still food for thought. Best, -jay 2) In oder to support the ability for service providers to be able to have the identity part of keystone be satisfied by a customer LDAP (i.e. for a given domain, have a specific LDAP), then, as has been stated, we need to subsequently, when handed an API call with just a user or group ID, be able to route this call to the correct LDAP. Trying to keep true to the openstack design principles, we had planned to encode a domain identifier into the user or group ID - i.e. distribute the data to where it is needed, in ortherwords, the user and group ID provide all the info we need to route the call to the right place. Two implementations come to mind: 2a) Simply concatenate the user/group ID with the domain_id, plus some separator and make a composite public facing ID. e.g. user_entity_id@@UUID_of_domain. This would have a technical maximum size of 64+2+64 (i.e. 130), although in reality since we control domain_id and we know it is always 32 char UUID - in fact the max size would be 98. This has the problem of increasing the size of the public facing field beyond the existing 64. This is what we had planned for IceHouse - and is currently in review. 2b) Use a similar concatenation idea as 2a), but limit the total size to the existing 64. Since we control domain_id, we could (internally and not visibly to the outside world), create a domain_index, that was used in place of domain_id in the publicly visible field, to minimize the number of chars it requires. So the public facing composite ID might be something like up to 54 chars of entity_id@@8 chars of domain_index. There is a chance, of course, that the 54 char restriction might be problematic for LDAP usersbut I doubt it. We would make that a restriction and if it really became a problem, we could consider a field size increase at a later release 3) The alternative to 2a and 2b is to have, as had been suggested, an internal mapping table that maps external facing entity_ids to a domain plus local entity ID. The problem with this idea is that: - This could become a very big table (you will essentially have an entry for every user in every corporate LDAP that has accessed a given openstack) - Since most LDAPs are RO, we will never see deletes...so we won't know when (without some kind of garbage collection) to cull entries - It obviously does not solve 1) - since existing LDAP support can break the 32 char limit - and so it isn't true that this mapping table causes all public facing entity IDs to be simple 32 char UUIDs From a delivery into IceHouse point of view any of the above are possible, since the actual mapping used is relatively small part of the patch. I personally favor 2b), since it is simple, has less moving parts and does not change any external facing requirements for storage of user and group IDs (above and beyond what is true today). Henry On 27 Feb 2014, at 03:46, Adam Young ayo...@redhat.com wrote: On 02/26/2014 08:25
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Thursday, February 27, 2014, Dolph Mathews dolph.math...@gmail.com wrote: On Thu, Feb 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.comjavascript:_e(%7B%7D,'cvml','jaypi...@gmail.com'); wrote: On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote: So a couple of things about this: 1) Today (and also true for Grizzly and Havana), the user can chose what LDAP attribute should be returned as the user or group ID. So it is NOT a safe assumption today (ignoring any support for domain-specific LDAP support) that the format of a user or group ID is a 32 char UUID. Quite often, I would think, that email address would be chosen by a cloud provider as the LDAP id field, by default we use the CN. Since we really don't want to ever change the user or group ID we have given out from keystone for a particular entity, this means we need to update nova (or anything else) that has made a 32 char assumption. I don't believe this is correct. Keystone is the service that deals with authentication. As such, Keystone should be the one and only one service that should have any need whatsoever to need to understand a non-UUID value for a user ID. The only value that should ever be communicated *from* Keystone should be the UUID value of the user. +++ If the Keystone service uses LDAP or federation for alternative authentication schemes, then Keystone should have a mapping table that translates those elongated and non-UUID identifiers values (email addresses, LDAP CNs, etc) into the UUID value that is then communicated to all other OpenStack services. I'd take it one step further and say that at some point, keystone should stop leaking identity details such as user name / ID into OpenStack (they shouldn't appear in tokens, and shouldn't be expected output of auth_token). The use cases that require them are weak and would be better served by pure multitenant RBAC, ABAC, etc. There are a lot of blockers to making this happen (including a few in keystone's own API), but still food for thought. ++ this would be a great change! I am on the same page as Dolph when it comes to approving of the UUID being the only value communicated outside of keystone. There is just no good reason to send out extra identity information (it isn't needed and can help to reduce token bloat etc). --Morgan Sent via mobile ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
I agree about not needing extra identity information outside of the user's UUID, but what about the role/project/domain information stored in the PKI token? Does it remain or go away? From: Morgan Fainberg [mailto:m...@metacloud.com] Sent: Thursday, February 27, 2014 12:11 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255 On Thursday, February 27, 2014, Dolph Mathews dolph.math...@gmail.commailto:dolph.math...@gmail.com wrote: On Thu, Feb 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.comjavascript:_e(%7B%7D,'cvml','jaypi...@gmail.com'); wrote: On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote: So a couple of things about this: 1) Today (and also true for Grizzly and Havana), the user can chose what LDAP attribute should be returned as the user or group ID. So it is NOT a safe assumption today (ignoring any support for domain-specific LDAP support) that the format of a user or group ID is a 32 char UUID. Quite often, I would think, that email address would be chosen by a cloud provider as the LDAP id field, by default we use the CN. Since we really don't want to ever change the user or group ID we have given out from keystone for a particular entity, this means we need to update nova (or anything else) that has made a 32 char assumption. I don't believe this is correct. Keystone is the service that deals with authentication. As such, Keystone should be the one and only one service that should have any need whatsoever to need to understand a non-UUID value for a user ID. The only value that should ever be communicated *from* Keystone should be the UUID value of the user. +++ If the Keystone service uses LDAP or federation for alternative authentication schemes, then Keystone should have a mapping table that translates those elongated and non-UUID identifiers values (email addresses, LDAP CNs, etc) into the UUID value that is then communicated to all other OpenStack services. I'd take it one step further and say that at some point, keystone should stop leaking identity details such as user name / ID into OpenStack (they shouldn't appear in tokens, and shouldn't be expected output of auth_token). The use cases that require them are weak and would be better served by pure multitenant RBAC, ABAC, etc. There are a lot of blockers to making this happen (including a few in keystone's own API), but still food for thought. ++ this would be a great change! I am on the same page as Dolph when it comes to approving of the UUID being the only value communicated outside of keystone. There is just no good reason to send out extra identity information (it isn't needed and can help to reduce token bloat etc). --Morgan Sent via mobile ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On 02/27/2014 12:52 PM, Jay Pipes wrote: On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote: So a couple of things about this: 1) Today (and also true for Grizzly and Havana), the user can chose what LDAP attribute should be returned as the user or group ID. So it is NOT a safe assumption today (ignoring any support for domain-specific LDAP support) that the format of a user or group ID is a 32 char UUID. Quite often, I would think, that email address would be chosen by a cloud provider as the LDAP id field, by default we use the CN. Since we really don't want to ever change the user or group ID we have given out from keystone for a particular entity, this means we need to update nova (or anything else) that has made a 32 char assumption. I don't believe this is correct. Keystone is the service that deals with authentication. As such, Keystone should be the one and only one service that should have any need whatsoever to need to understand a non-UUID value for a user ID. The only value that should ever be communicated *from* Keystone should be the UUID value of the user. We never said it was a UUID, its only been that in the allocated by Keystone cases thus far. userids are blobswell, character blocks, and are not expected to be interprested by the end services, but also not confined to be 32 characters long. Most LDAP use cases do not allow us to write data back the Directory. Federation has the same use cases. That means that we are going to end up with a slew of ephemeral data in Keystone for no real value. If the Keystone service uses LDAP or federation for alternative authentication schemes, then Keystone should have a mapping table that translates those elongated and non-UUID identifiers values (email addresses, LDAP CNs, etc) into the UUID value that is then communicated to all other OpenStack services. The short of it is that we need to define a max userid length and stick to it. Thus far, we've had 64 characters in Keystone, and I think we can stick with that. Nova needs to expand its view to 64 chars. The rest is Keystone internals to deal with. Best, -jay 2) In oder to support the ability for service providers to be able to have the identity part of keystone be satisfied by a customer LDAP (i.e. for a given domain, have a specific LDAP), then, as has been stated, we need to subsequently, when handed an API call with just a user or group ID, be able to route this call to the correct LDAP. Trying to keep true to the openstack design principles, we had planned to encode a domain identifier into the user or group ID - i.e. distribute the data to where it is needed, in ortherwords, the user and group ID provide all the info we need to route the call to the right place. Two implementations come to mind: 2a) Simply concatenate the user/group ID with the domain_id, plus some separator and make a composite public facing ID. e.g. user_entity_id@@UUID_of_domain. This would have a technical maximum size of 64+2+64 (i.e. 130), although in reality since we control domain_id and we know it is always 32 char UUID - in fact the max size would be 98. This has the problem of increasing the size of the public facing field beyond the existing 64. This is what we had planned for IceHouse - and is currently in review. 2b) Use a similar concatenation idea as 2a), but limit the total size to the existing 64. Since we control domain_id, we could (internally and not visibly to the outside world), create a domain_index, that was used in place of domain_id in the publicly visible field, to minimize the number of chars it requires. So the public facing composite ID might be something like up to 54 chars of entity_id@@8 chars of domain_index. There is a chance, of course, that the 54 char restriction might be problematic for LDAP usersbut I doubt it. We would make that a restriction and if it really became a problem, we could consider a field size increase at a later release 3) The alternative to 2a and 2b is to have, as had been suggested, an internal mapping table that maps external facing entity_ids to a domain plus local entity ID. The problem with this idea is that: - This could become a very big table (you will essentially have an entry for every user in every corporate LDAP that has accessed a given openstack) - Since most LDAPs are RO, we will never see deletes...so we won't know when (without some kind of garbage collection) to cull entries - It obviously does not solve 1) - since existing LDAP support can break the 32 char limit - and so it isn't true that this mapping table causes all public facing entity IDs to be simple 32 char UUIDs From a delivery into IceHouse point of view any of the above are possible, since the actual mapping used is relatively small part of the patch. I personally favor 2b), since it is simple, has less moving parts and does not change any external facing requirements for storage of user and group IDs (above and beyond what is
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Thu, 2014-02-27 at 17:02 -0500, Adam Young wrote: On 02/27/2014 12:52 PM, Jay Pipes wrote: On Thu, 2014-02-27 at 16:13 +, Henry Nash wrote: So a couple of things about this: 1) Today (and also true for Grizzly and Havana), the user can chose what LDAP attribute should be returned as the user or group ID. So it is NOT a safe assumption today (ignoring any support for domain-specific LDAP support) that the format of a user or group ID is a 32 char UUID. Quite often, I would think, that email address would be chosen by a cloud provider as the LDAP id field, by default we use the CN. Since we really don't want to ever change the user or group ID we have given out from keystone for a particular entity, this means we need to update nova (or anything else) that has made a 32 char assumption. I don't believe this is correct. Keystone is the service that deals with authentication. As such, Keystone should be the one and only one service that should have any need whatsoever to need to understand a non-UUID value for a user ID. The only value that should ever be communicated *from* Keystone should be the UUID value of the user. We never said it was a UUID, its only been that in the allocated by Keystone cases thus far. Every user record in Keystone should have an allocated by Keystone user ID that is a UUID value. Why? Because every single other OpenStack service uses UUID values for identifiers (not names), and Keystone should be no different. userids are blobswell, character blocks, and are not expected to be interprested by the end services, but also not confined to be 32 characters long. I would argue that they absolutely *should* be confined to 32 characters (or technically, 16 bytes, which is the size of a UUID). Most LDAP use cases do not allow us to write data back the Directory. Federation has the same use cases. Frankly, I don't care :) Let's say Super-Duper Identity Driver has 100 character IDs natively. Does that mean that these IDs need be sent all over the OpenStack ecosystem? No. Have a translation table in Keystone and spread the UUID user ID throughout OpenStack. That means that we are going to end up with a slew of ephemeral data in Keystone for no real value. Not sure why you think this is ephemeral data... What is ephemeral about a non-UUID identifier from another auth system (federated or not) that is mapped to a Keystone user UUID? If you look at virtually every flexible authentication mechanism in virtually all web frameworks out there, you will see something very similar, where there is a table that maps external IDs to the IDs that are native to some platform. Best, -jay If the Keystone service uses LDAP or federation for alternative authentication schemes, then Keystone should have a mapping table that translates those elongated and non-UUID identifiers values (email addresses, LDAP CNs, etc) into the UUID value that is then communicated to all other OpenStack services. The short of it is that we need to define a max userid length and stick to it. Thus far, we've had 64 characters in Keystone, and I think we can stick with that. Nova needs to expand its view to 64 chars. The rest is Keystone internals to deal with. Best, -jay 2) In oder to support the ability for service providers to be able to have the identity part of keystone be satisfied by a customer LDAP (i.e. for a given domain, have a specific LDAP), then, as has been stated, we need to subsequently, when handed an API call with just a user or group ID, be able to route this call to the correct LDAP. Trying to keep true to the openstack design principles, we had planned to encode a domain identifier into the user or group ID - i.e. distribute the data to where it is needed, in ortherwords, the user and group ID provide all the info we need to route the call to the right place. Two implementations come to mind: 2a) Simply concatenate the user/group ID with the domain_id, plus some separator and make a composite public facing ID. e.g. user_entity_id@@UUID_of_domain. This would have a technical maximum size of 64+2+64 (i.e. 130), although in reality since we control domain_id and we know it is always 32 char UUID - in fact the max size would be 98. This has the problem of increasing the size of the public facing field beyond the existing 64. This is what we had planned for IceHouse - and is currently in review. 2b) Use a similar concatenation idea as 2a), but limit the total size to the existing 64. Since we control domain_id, we could (internally and not visibly to the outside world), create a domain_index, that was used in place of domain_id in the publicly visible field, to minimize the number of chars it requires. So the public facing composite ID might be something like up to 54 chars of entity_id@@8 chars of domain_index. There is a chance, of course, that the
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
Hi Morgan, On Tue, Feb 25, 2014 at 11:47:43AM -0800, Morgan Fainberg wrote: For purposes of supporting multiple backends for Identity (multiple LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum size of the USER_ID field from an upper limit of 64 to an upper limit of 255. This change would not impact any currently assigned USER_IDs (they would remain in the old simple UUID format), however, new USER_IDs would be increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). in this case if a user would access with different systems (e.g. SAML with portal, LDAP with CLI) it is mapped to two different identities inside keystone. Is this correct? If so, is there any way to map an individual person with two identities sharing resources? Cheers, Marco There is the obvious concern that projects are utilizing (and storing) the user_id in a field that cannot accommodate the increased upper limit. Before this change is merged in, it is important for the Keystone team to understand if there are any places that would be overflowed by the increased size. The review that would implement this change in size is https:// review.openstack.org/#/c/74214 and is actively being worked on/reviewed. I have already spoken with the Nova team, and a single instance has been identified that would require a migration (that will have a fix proposed for the I3 timeline). If there are any other known locations that would have issues with an increased USER_ID size, or any concerns with this change to USER_ID format, please respond so that the issues/concerns can be addressed. Again, the plan is not to change current USER_IDs but that new ones could be up to 255 characters in length. Cheers, Morgan Fainberg — Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Eng. Marco Fargetta, PhD Istituto Nazionale di Fisica Nucleare (INFN) Catania, Italy EMail: marco.farge...@ct.infn.it smime.p7s Description: S/MIME cryptographic signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Wed, Feb 26, 2014 at 4:23 AM, Marco Fargetta marco.farge...@ct.infn.itwrote: Hi Morgan, On Tue, Feb 25, 2014 at 11:47:43AM -0800, Morgan Fainberg wrote: For purposes of supporting multiple backends for Identity (multiple LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum size of the USER_ID field from an upper limit of 64 to an upper limit of 255. This change would not impact any currently assigned USER_IDs (they would remain in the old simple UUID format), however, new USER_IDs would be increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). in this case if a user would access with different systems (e.g. SAML with portal, LDAP with CLI) it is mapped to two different identities inside keystone. Is this correct? If so, is there any way to map an individual person with two identities sharing resources? That's correct - they'd result in different identities and keystone has no means of presuming they are the same person. But I think you answered it yourself, in that they would effectively be sharing resources with themselves, so you'd just have to ensure they had the same authorization on the same projects using both identities. Cheers, Marco There is the obvious concern that projects are utilizing (and storing) the user_id in a field that cannot accommodate the increased upper limit. Before this change is merged in, it is important for the Keystone team to understand if there are any places that would be overflowed by the increased size. The review that would implement this change in size is https:// review.openstack.org/#/c/74214 and is actively being worked on/reviewed. I have already spoken with the Nova team, and a single instance has been identified that would require a migration (that will have a fix proposed for the I3 timeline). If there are any other known locations that would have issues with an increased USER_ID size, or any concerns with this change to USER_ID format, please respond so that the issues/concerns can be addressed. Again, the plan is not to change current USER_IDs but that new ones could be up to 255 characters in length. Cheers, Morgan Fainberg — Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Eng. Marco Fargetta, PhD Istituto Nazionale di Fisica Nucleare (INFN) Catania, Italy EMail: marco.farge...@ct.infn.it ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com wrote: On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote: For purposes of supporting multiple backends for Identity (multiple LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum size of the USER_ID field from an upper limit of 64 to an upper limit of 255. This change would not impact any currently assigned USER_IDs (they would remain in the old simple UUID format), however, new USER_IDs would be increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). -1 I think a better solution would be to have a simple translation table only in Keystone that would store this longer identifier (for folks using federation and/or LDAP) along with the Keystone user UUID that is used in foreign key relations and other mapping tables through Keystone and other projects. Morgan and I talked this suggestion through last night and agreed it's probably the best approach, and has the benefit of zero impact on other services, which is something we're obviously trying to avoid. I imagine it could be as simple as a user_id to domain_id lookup table. All we really care about is given a globally unique user ID, which identity backend is the user from? On the downside, it would likely become bloated with unused ephemeral user IDs, so we'll need enough metadata about the mapping to implement a purging behavior down the line. The only identifiers that would ever be communicated to any non-Keystone OpenStack endpoint would be the UUID user and tenant IDs. There is the obvious concern that projects are utilizing (and storing) the user_id in a field that cannot accommodate the increased upper limit. Before this change is merged in, it is important for the Keystone team to understand if there are any places that would be overflowed by the increased size. I would go so far as to say the user_id and tenant_id fields should be *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for performance reasons. Lengthening commonly-used and frequently-joined identifier fields is not a good option, IMO. Best, -jay The review that would implement this change in size is https://review.openstack.org/#/c/74214 and is actively being worked on/reviewed. I have already spoken with the Nova team, and a single instance has been identified that would require a migration (that will have a fix proposed for the I3 timeline). If there are any other known locations that would have issues with an increased USER_ID size, or any concerns with this change to USER_ID format, please respond so that the issues/concerns can be addressed. Again, the plan is not to change current USER_IDs but that new ones could be up to 255 characters in length. Cheers, Morgan Fainberg — Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On 02/26/2014 08:25 AM, Dolph Mathews wrote: On Tue, Feb 25, 2014 at 2:38 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote: For purposes of supporting multiple backends for Identity (multiple LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum size of the USER_ID field from an upper limit of 64 to an upper limit of 255. This change would not impact any currently assigned USER_IDs (they would remain in the old simple UUID format), however, new USER_IDs would be increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). -1 I think a better solution would be to have a simple translation table only in Keystone that would store this longer identifier (for folks using federation and/or LDAP) along with the Keystone user UUID that is used in foreign key relations and other mapping tables through Keystone and other projects. Morgan and I talked this suggestion through last night and agreed it's probably the best approach, and has the benefit of zero impact on other services, which is something we're obviously trying to avoid. I imagine it could be as simple as a user_id to domain_id lookup table. All we really care about is given a globally unique user ID, which identity backend is the user from? On the downside, it would likely become bloated with unused ephemeral user IDs, so we'll need enough metadata about the mapping to implement a purging behavior down the line. UUIDs are 32 chars long. Its really just uuid@@uuid that pushes us over the 64 character limit. If we can shorten up the IDP_ID we can fit everything in 64 chars (which means only Nova needs to expand its column size) What if we enumerated IDPs by index, from 1000 to or something comparable, and then use the new domain_index (or prot domain id to not be a uuid). Then the above scheme would work and no migration would be required. The only identifiers that would ever be communicated to any non-Keystone OpenStack endpoint would be the UUID user and tenant IDs. There is the obvious concern that projects are utilizing (and storing) the user_id in a field that cannot accommodate the increased upper limit. Before this change is merged in, it is important for the Keystone team to understand if there are any places that would be overflowed by the increased size. I would go so far as to say the user_id and tenant_id fields should be *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for performance reasons. Lengthening commonly-used and frequently-joined identifier fields is not a good option, IMO. Best, -jay The review that would implement this change in size is https://review.openstack.org/#/c/74214 and is actively being worked on/reviewed. I have already spoken with the Nova team, and a single instance has been identified that would require a migration (that will have a fix proposed for the I3 timeline). If there are any other known locations that would have issues with an increased USER_ID size, or any concerns with this change to USER_ID format, please respond so that the issues/concerns can be addressed. Again, the plan is not to change current USER_IDs but that new ones could be up to 255 characters in length. Cheers, Morgan Fainberg --- Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com mailto:m...@metacloud.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Tue, 2014-02-25 at 11:47 -0800, Morgan Fainberg wrote: For purposes of supporting multiple backends for Identity (multiple LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum size of the USER_ID field from an upper limit of 64 to an upper limit of 255. This change would not impact any currently assigned USER_IDs (they would remain in the old simple UUID format), however, new USER_IDs would be increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). -1 I think a better solution would be to have a simple translation table only in Keystone that would store this longer identifier (for folks using federation and/or LDAP) along with the Keystone user UUID that is used in foreign key relations and other mapping tables through Keystone and other projects. The only identifiers that would ever be communicated to any non-Keystone OpenStack endpoint would be the UUID user and tenant IDs. There is the obvious concern that projects are utilizing (and storing) the user_id in a field that cannot accommodate the increased upper limit. Before this change is merged in, it is important for the Keystone team to understand if there are any places that would be overflowed by the increased size. I would go so far as to say the user_id and tenant_id fields should be *reduced* in size to a fixed 16-char BINARY or 32-char CHAR field for performance reasons. Lengthening commonly-used and frequently-joined identifier fields is not a good option, IMO. Best, -jay The review that would implement this change in size is https://review.openstack.org/#/c/74214 and is actively being worked on/reviewed. I have already spoken with the Nova team, and a single instance has been identified that would require a migration (that will have a fix proposed for the I3 timeline). If there are any other known locations that would have issues with an increased USER_ID size, or any concerns with this change to USER_ID format, please respond so that the issues/concerns can be addressed. Again, the plan is not to change current USER_IDs but that new ones could be up to 255 characters in length. Cheers, Morgan Fainberg — Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][keystone] Increase of USER_ID length maximum from 64 to 255
On Tue, Feb 25, 2014 at 11:47:43AM -0800, Morgan Fainberg wrote: For purposes of supporting multiple backends for Identity (multiple LDAP, mix of LDAP and SQL, federation, etc) Keystone is planning to increase the maximum size of the USER_ID field from an upper limit of 64 to an upper limit of 255. This change would not impact any currently assigned USER_IDs (they would remain in the old simple UUID format), however, new USER_IDs would be increased to include the IDP identifier (e.g. USER_ID@@IDP_IDENTIFIER). Hmm, my immediate reaction is there must be a better way than mangling both bits of data into the ID field, considering pretty much everything everywhere else in openstack uses uuids for user-visible object identifiers. There is the obvious concern that projects are utilizing (and storing) the user_id in a field that cannot accommodate the increased upper limit. Before this change is merged in, it is important for the Keystone team to understand if there are any places that would be overflowed by the increased size. The review that would implement this change in size is https://review.openstack.org/#/c/74214 and is actively being worked on/reviewed. I have already spoken with the Nova team, and a single instance has been identified that would require a migration (that will have a fix proposed for the I3 timeline). If there are any other known locations that would have issues with an increased USER_ID size, or any concerns with this change to USER_ID format, please respond so that the issues/concerns can be addressed. Again, the plan is not to change current USER_IDs but that new ones could be up to 255 characters in length. Yes, this will affect Heat in at least one place - we store the trustor user ID when we create a trust between the stack owner and the heat service user: https://github.com/openstack/heat/blob/master/heat/db/sqlalchemy/migrate_repo/versions/027_user_creds_trusts.py#L23 IMHO this is coming pretty late in the cycle considering the potential impact, but if this is definitely happening we can go ahead an update our schema. Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
