Re: [openstack-dev] [Keystone] Alternative federation mapping
We started an etherpad to gather requirements etc: https://etherpad.openstack.org/p/mapping-engine-enhancements ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Alternative federation mapping
On 11/04/2014 02:46 AM, David Chadwick wrote: Hi John Good morning David. I hope you're enjoying Paris and the summit is both productive and enjoyable. Wish I was there :-) Its seems like your objective is somewhat different to what was intended with the current mapping rules. You seem to want a general purpose mapping engine that can map from any set of attribute values into any other set of values, whereas the primary objective of the current mapping rules is to map from any set of attribute values into a Keystone group, so that we can use the existing Keystone functions to assign roles (and hence permissions) to the group members. So the current mapping rules provide a means of identifying and then authorising a potentially random set of external users, who logically form a coherent group of users for authorisation purposes. O.K. group assignment is the final goal in Keystone. I suppose the relevant question then is the functionality in the current Keystone mapper sufficiently rich such that you can present to it an arbitrary set of values and yield a group assessment? It's been a while since I looked at the mapper, things might have changed, but it seemed to me it had a lot of baked in assumptions about the data (assertion) it would receive. As long as those assumptions held true all is good. My concern arose from real world experience where I saw a lot of messy data (plus I had a task that had some other requirements). There is often little consistency over how data is formatted and what data is included when you receive it from a foreign source. Now combine the messy data with complex rules dictated by management and you have an admin with a headache who is asked to make sure the rules are implemented. An admin might have to implement something like this: If the user is a member of domain D and has authenticated with mechanisms X,Y or Z and has IdP attribute A but does not have suffix S in their username and is not in a blacklist then assign them group G and transform their username by stripping the suffix, replacing all hyphens with underscores and lowercase it. I'll grant you this example is perhaps a bit contrived but it's not too far afield from the questions I've seen admins ask when trying to manage actual RADIUS deployments. BTW, where is that domain information coming from in the example? Usually it has to be extracted from the username in any one of a number of formats. It's things like this that motivate me towards a more general purpose mechanism because at the end of the day the real world isn't pretty :-) FWIW FreeRADIUS didn't start out with a policy language with capabilities like this, it grew one out of necessity. I'm definitely not trying to say Keystone needs to switch mappers, instead what I'm offering is one approach you might want to consider before the current mapping syntax becomes entrenched and ugly real world problems begin to crop up. I don't have any illusions this solution is ideal, these things are difficult to spec out and write. One advantage is it's easy to extend in a backwards compatible manner with minimal effort (basically it's just adding a new function you can call). FWIW the ideal mapper in my mind is something written in a general purpose scripting language where you have virtually no limitations on how you transform values and enforce conditions, but as I indicated in my other document managing a script interpreter for this task has its own problems. Which is the lesser of two evils, a script interpreter or a custom policy language? I don't think I have the answer to that but came down on the side of the custom policy language as being the most palatable and portable. Am I right in assuming that you will also want this functionality after your general purpose mapping has taken place? The mapper I designed does give you a lot of flexibility to assign a foreign identity to a group (or multiple groups) with no additional steps so I'm not sure I follow the above comment. There is no need for any extra or multiple steps, it should be self contained. -- John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Alternative federation mapping
On Tue, Nov 4, 2014 at 10:30 AM, John Dennis jden...@redhat.com wrote: O.K. group assignment is the final goal in Keystone. I suppose the relevant question then is the functionality in the current Keystone mapper sufficiently rich such that you can present to it an arbitrary set of values and yield a group assessment? It's been a while since I looked at the mapper, things might have changed, but it seemed to me it had a lot of baked in assumptions about the data (assertion) it would receive. As long as those assumptions held true all is good. There are probably a few other assumptions, but the main one is that the mapper expects the incoming data to be a dictionary where the value is a string. If there are multiple values we expect them to be delimited with a semicolon in the string. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek www: http://dstanek.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Alternative federation mapping
On 11/04/2014 04:19 PM, David Stanek wrote: There are probably a few other assumptions, but the main one is that the mapper expects the incoming data to be a dictionary where the value is a string. If there are multiple values we expect them to be delimited with a semicolon in the string. and ... any value with a colon will be split regardless of the value's semantic meaning. In other words the colon character is illegal in any context other than as a list separator. And you had better hope that any value whose semantic meaning is a list will use the colon separator and not space, tab, comma, etc. as the separator. -- John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Alternative federation mapping
Hi John there is another way of tackling this problem, based on the following assumption We know the person who should be authorised, but we dont know what random set of attributes the IDP will provide for him (or if we do know, they are as horrible as you indicated in your example below). We will demonstrate this solution this week at the keystone design summit. It is based on the administrator sending the user (out of band) the name of the group he is invited to join (specified as a Virtual Organisation name and role) plus a secret. The user logs in with his IDP, the IDP sends the unknown/horrible set of attributes to keystone, the user asks to join the group and presents the secret, and our code automatically adds a mapping rule specifically for this user to put him in the group. The user can either be automatically added to the group, or put in a pending queue for the administrator to OK at a later time. Users who try to hack in are blacklisted if they present a wrong secret for 3 times. Would this solve your problem? regards David On 04/11/2014 15:30, John Dennis wrote: On 11/04/2014 02:46 AM, David Chadwick wrote: Hi John Good morning David. I hope you're enjoying Paris and the summit is both productive and enjoyable. Wish I was there :-) Its seems like your objective is somewhat different to what was intended with the current mapping rules. You seem to want a general purpose mapping engine that can map from any set of attribute values into any other set of values, whereas the primary objective of the current mapping rules is to map from any set of attribute values into a Keystone group, so that we can use the existing Keystone functions to assign roles (and hence permissions) to the group members. So the current mapping rules provide a means of identifying and then authorising a potentially random set of external users, who logically form a coherent group of users for authorisation purposes. O.K. group assignment is the final goal in Keystone. I suppose the relevant question then is the functionality in the current Keystone mapper sufficiently rich such that you can present to it an arbitrary set of values and yield a group assessment? It's been a while since I looked at the mapper, things might have changed, but it seemed to me it had a lot of baked in assumptions about the data (assertion) it would receive. As long as those assumptions held true all is good. My concern arose from real world experience where I saw a lot of messy data (plus I had a task that had some other requirements). There is often little consistency over how data is formatted and what data is included when you receive it from a foreign source. Now combine the messy data with complex rules dictated by management and you have an admin with a headache who is asked to make sure the rules are implemented. An admin might have to implement something like this: If the user is a member of domain D and has authenticated with mechanisms X,Y or Z and has IdP attribute A but does not have suffix S in their username and is not in a blacklist then assign them group G and transform their username by stripping the suffix, replacing all hyphens with underscores and lowercase it. I'll grant you this example is perhaps a bit contrived but it's not too far afield from the questions I've seen admins ask when trying to manage actual RADIUS deployments. BTW, where is that domain information coming from in the example? Usually it has to be extracted from the username in any one of a number of formats. It's things like this that motivate me towards a more general purpose mechanism because at the end of the day the real world isn't pretty :-) FWIW FreeRADIUS didn't start out with a policy language with capabilities like this, it grew one out of necessity. I'm definitely not trying to say Keystone needs to switch mappers, instead what I'm offering is one approach you might want to consider before the current mapping syntax becomes entrenched and ugly real world problems begin to crop up. I don't have any illusions this solution is ideal, these things are difficult to spec out and write. One advantage is it's easy to extend in a backwards compatible manner with minimal effort (basically it's just adding a new function you can call). FWIW the ideal mapper in my mind is something written in a general purpose scripting language where you have virtually no limitations on how you transform values and enforce conditions, but as I indicated in my other document managing a script interpreter for this task has its own problems. Which is the lesser of two evils, a script interpreter or a custom policy language? I don't think I have the answer to that but came down on the side of the custom policy language as being the most palatable and portable. Am I right in assuming that you will also want this functionality after your general purpose mapping has taken place? The mapper
Re: [openstack-dev] [Keystone] Alternative federation mapping
On Nov 2, 2014, at 22:21, Dolph Mathews dolph.math...@gmail.com wrote: On Sunday, November 2, 2014, John Dennis jden...@redhat.com wrote: It was hoped we could simply borrow the Keystone mapping implementation but it was found to be too limiting and not sufficiently expressive. We could not find another alternative so we designed a new mapper which is described in this PDF. In what way was it too limited? Did you consider extending the existing grammar and extending the existing mapping engine? I am very interested in knowing the limitations you ran into. I am fairly certain we are willing to update the engine to meet the needs of the deployers, but knowing what those limitations are and what this new proposed engine provides that we don't (for this use case) is important. --Morgan Sent via mobile___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Alternative federation mapping
I agree with Morgan. We designed the current mapping functionality to cover all the use cases we were aware of, but if there are more, then we would love to hear about them and make the fixes that are necessary. Attribute mapping is a critical component of federation, and it should be fit for purpose regards David On 03/11/2014 09:08, Morgan Fainberg wrote: On Nov 2, 2014, at 22:21, Dolph Mathews dolph.math...@gmail.com mailto:dolph.math...@gmail.com wrote: On Sunday, November 2, 2014, John Dennis jden...@redhat.com mailto:jden...@redhat.com wrote: It was hoped we could simply borrow the Keystone mapping implementation but it was found to be too limiting and not sufficiently expressive. We could not find another alternative so we designed a new mapper which is described in this PDF. In what way was it too limited? Did you consider extending the existing grammar and extending the existing mapping engine? I am very interested in knowing the limitations you ran into. I am fairly certain we are willing to update the engine to meet the needs of the deployers, but knowing what those limitations are and what this new proposed engine provides that we don't (for this use case) is important. --Morgan Sent via mobile ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Alternative federation mapping
On 11/03/2014 08:50 AM, John Dennis wrote:
I had a bunch of notes but I can't find them at the moment
so I'm going from memory here (always a bit risky).
I found my notes, attached is an reStructured text document that
summarizes the issues I found with the current Keystone mapping, the
basic requirements for mapping based on my prior experience and reasons
for selecting the approach I did. I hope this explains the motivations
and rationale to put things into context.
--
John
Federation Mapping
==
Introduction
With federated authentication a trusted external IdP (Identity
Provider) authenticates an entity and provides attributes associated
with the authenticated entity such as full name, organization,
location, group membership, roles, etc. to the local authorization
system. The remote identity attributes usually do not directly
correlate to the local identity attributes as such they must be mapped
or transformed to be consistent with the local identity attributes.
Problems with existing contributed OpenStack federation mapping
---
A federated mapping implementation was contributed to OpenStack which
is based on static rules. However that mapping system lacks basic
features required in real world deployments. Examples of the
deficiencies are:
* Replacement substitutions are specified by an index, e.g. {0} making
them difficult to use.
* If you edit a rule all the indices shift and you have carefully
adjust multiple locations, this is tedious and error prone.
* Numeric replacements are not friendly, it's difficult to remember what
index maps to which value. Likewise its difficult to read a rule
with a indexed substitutions and understand what is being
replaced, a number provides no context for the reader.
* Impossible to specify how strings containing lists of values are to
be split.
* Any string containing a colon is automatically split irregardless of
semantics of the string.
* Impossible to perform tests and perform conditional logic. For
example you cannot test if the user is a member of a group and if so
add a role based on the group membership. You cannot test if a list
is empty, has a certain number of members, if a value starts with a
prefix or ends with a suffix, etc.
* No mechanism to handle case sensitivity.
* Cannot split direct map value into multiple values, e.g. user@domain
cannot be returned as ['user', 'domain'], i.e. {0}, {1}. This
requires more robust regular expression support than is provided.
* Cannot operate on substrings or do replacements. Common operations
such as replacing hyphens with underscores or stripping off prefixes
or suffixes are impossible.
* Direct map values cannot be reassigned by later rules.
e.g. email might be in assertion so it would be direct map,
but if it was absent one can't synthesize it from other values
in the assertion, i.e. user + @ + domain
* Difficult to build values by string interpolation or concatenation.
* The local array elements are dicts whose keys can contain 'user' or
'group' or both, but you can't have more that one user or group in an
element and elements that define a user more than once produce an
error.
* Logical OR requires cut-n-paste copies of many rules with only a minor
difference in each rule, rules quickly become unreadable and difficult
to ascertain the logic.
* ``not_any_of`` does not work when regex is True, enabling regex
effectively changes the condition to ``any_one_of``. (This is a bug
that can be fixed)
Examples of real work tasks current mapping cannot handle
-
I've worked with RADIUS for years. RADIUS is often configured to
operate in a federated mode where the attributes supplied to the
RADIUS server have to be manipulated to match local conventions and
policies. Below are some of the most common issues I've seen admins
have to tackle.
* Split the realm from the username. Assign the username and realm
independently in the result. This is by far the most common issue
admins raise.
* Strip prefixes and suffixes and/or take a prefix or suffix and map
it to a new independent value. Believe it or not many organizations
embed group/role information in their usernames. Usernames such as
johndoe_staff where the username is johndoe and role is staff
are depressingly common.
* Behave differently depending on the realm, the IdP, a DNS name or a
network address.
* Test for membership in a collection. Examples are whitelists,
blacklists, groups, etc. The result of the membership test modifies
how the user is ultimately mapped and the privileges they receive.
* Search for and/or extract substrings, usually demands regular
expression support. The result of the regular expression search may
alter the flow of control.
* Filter certain characters.
* Convert to lower case or
Re: [openstack-dev] [Keystone] Alternative federation mapping
Hi John Its seems like your objective is somewhat different to what was intended with the current mapping rules. You seem to want a general purpose mapping engine that can map from any set of attribute values into any other set of values, whereas the primary objective of the current mapping rules is to map from any set of attribute values into a Keystone group, so that we can use the existing Keystone functions to assign roles (and hence permissions) to the group members. So the current mapping rules provide a means of identifying and then authorising a potentially random set of external users, who logically form a coherent group of users for authorisation purposes. Am I right in assuming that you will also want this functionality after your general purpose mapping has taken place? regards David On 03/11/2014 20:58, John Dennis wrote: On 11/03/2014 08:50 AM, John Dennis wrote: I had a bunch of notes but I can't find them at the moment so I'm going from memory here (always a bit risky). I found my notes, attached is an reStructured text document that summarizes the issues I found with the current Keystone mapping, the basic requirements for mapping based on my prior experience and reasons for selecting the approach I did. I hope this explains the motivations and rationale to put things into context. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Alternative federation mapping
Hi John, It indeed looks interesting and enhancing the mapping engine is on ours to-do list for a long time. I'd be happy to talk this through during the summit. Do you think you will be able to come for a Keystone websso/federation Design Session on Wednesday at 16.30? Thanks, Marek On 02.11.2014 18:29, John Dennis wrote: While working on federated authentication for a different project (OpenDaylight) we discovered we needed to map from the assertion provided by an external federated IdP to local values. This is essentially the same requirement which exists in Keystone's federated support. It was hoped we could simply borrow the Keystone mapping implementation but it was found to be too limiting and not sufficiently expressive. We could not find another alternative so we designed a new mapper which is described in this PDF. https://jdennis.fedorapeople.org/doc/mapping.pdf The mapper as described in the document has implementations in both Java and Python. The Java implementation is currently in use in OpenDaylight (a Java based project). For those interested I can provide a pointer to OpenDaylight specific documentation on how this mapper is used in conjunction with the Apache web server providing authentication and SSSD providing identity attributes to a Java servlet container. My goal here is to make Keystone developers aware of an alternative mapper which may provide needed mapping features not currently available and for which different language implementations already exist. Note, the mapper is easily extended should a need arise. Source code and documentation can be found here by cloning this git repo: git clone git://fedorapeople.org/~jdennis/federated-mapping.git Note, I put this git repo together quickly by pulling together things from a variety of sources, as such there may be things needing to be cleaned up in the repo, at the moment it's really just meant to browse. Over the next few days I'll make sure everything builds and executes cleanly. Posting this now in case folks want to have conversations at the Paris Summit. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Alternative federation mapping
On 11/02/2014 03:55 PM, Marek Denis wrote: Hi John, It indeed looks interesting and enhancing the mapping engine is on ours to-do list for a long time. I'd be happy to talk this through during the summit. Do you think you will be able to come for a Keystone websso/federation Design Session on Wednesday at 16.30? Thank you Marek. I'd love to discuss this in person however I'm not attending this summit. However, my manager Nathan Kinder who has been following this work will be in Paris as well as Adam Young who is also in our group. Of course I'm happy to discuss this electronically, turn it into a blueprint or whatever is deemed appropriate. -- John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Alternative federation mapping
On Sunday, November 2, 2014, John Dennis jden...@redhat.com wrote: It was hoped we could simply borrow the Keystone mapping implementation but it was found to be too limiting and not sufficiently expressive. We could not find another alternative so we designed a new mapper which is described in this PDF. In what way was it too limited? Did you consider extending the existing grammar and extending the existing mapping engine? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
