Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/14/2013 09:01 AM, Simo Sorce wrote: On Wed, 2013-02-13 at 21:55 -0500, Dmitri Pal wrote: On 02/13/2013 09:48 PM, Nathan Kinder wrote: On 02/13/2013 06:18 PM, Dmitri Pal wrote: On 02/13/2013 02:08 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 13:30 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. I think he means have a persistent search to watch for new entries and then act upon them. I think a 2 step approach is misguided, so I've written it off from the start. Simo. This all thread smells like rewriting winsync by using internal plugin and IPA framework. Is there any chance we can use existing winsync solution to do what we need but sync from staging instance rather than AD? Then the flow will be: HR system - staging DS instance - dirsync - IPA Couple assumptions: a) We are satisfied with the security of the existing winsync solution and authentication used there. I do not see why it should be different here as it is a very similar use case. b) I expect we can sync from 389 to 389 with minor config changes and effectively no code changes This is not the case. Winsync uses the AD Dirsync control, which 389 does not support. The Winsync code also expects the AD schema on the entries coming in from the Dirsync response. Even if 389 supported the Dirsync control, there would be a fair amount of code changes to deal with whatever schema we would need to expect. c) The users created via winsync now are created in a proper way so IPA accepts then as IPA users This solves the problem of add and delete of the users. I know that winsync already supports group add/delete but we never qualified it in IPA. So may be it is time, at least for this use case. I would really think that this might be a lower cost solution than writing yet another custom plugin. Let us try to reuse what we already have. I think a custom plug-in would be easier (though I do have concerns about the performance impact of the JSON approach). I really think that we rather have an external solution and not the one hot wired into the IPA internals via the proposed plugin. Something like an external proxy or gateway rather than the approach Simo proposed even if it would be more work. This work would be reusable. We already have several use cases for the LDAP proxy. This can be one more. A bit more and there will be enough need to build one anyways. The main value is that it is optional and external and acts as an ecosystem level solution that can be developed and tested on a separate schedule rather than being internal to IPA. Penrose. Simo. Neah... Count this plant dead. -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/14/2013 12:33 PM, Endi Sukma Dewata wrote: On 2/14/2013 8:06 AM, Simo Sorce wrote: On Thu, 2013-02-14 at 14:26 +0100, Petr Spacek wrote: In my Fedora 17 I found package python-ldaptor. It seems to offer nice support for writing own event-based LDAP servers. For simple LDAP proxy it could be enough. $ yum install python-ldaptor $ python import ldaptor.protocols.ldap.ldapserver help(ldaptor.protocols.ldap.ldapserver) No. LDAP proxies are *not* simple. Ask Endi he's worked on a meta-directory for years. Simo. It depends on what you want to do with the proxy. If it's only a thin layer which converts the LDAP ADD to IPA user-add it might not be that complicated. Penrose virtual directory consists of a frontend LDAP interface, a transformation engine, and backends which may include an LDAP server as well. The front-end LDAP interface is the proxy we're talking about here, it's only used to receive LDAP requests and pass them to the transformation engine. The transformation engine is where the complexity occurs. In IPA this is already handled by the framework. In Penrose it's quite complex because it aims to provide a generic way to map an LDAP request to multiple backends which involves dealing with different types of backends, joining the backends, transforming the DN attributes back and forth, etc. So I'd say implementing an LDAP frontend for IPA using Python is something worth exploring. That way it can run in the same process so there's no concern about JSON performance/stability. +1 I would rather go this way. Simo, Your point about how things should have been bight be righteous but Petr confusion about the role framework shows the reality. Currently all framework is developed as if it is the primary interface for creation/modification. LDAP is exposed but can be consumed by external clients as read only. This is how we decided to implement things. I clearly remember a lengthy discussion about this 4 years ago when the framework architecture was discussed. The main reason is plugability. Since we need to provide extensible plugability framework it is the only supportable option. For good or for bad it is how things are now and Petr is correct because this is what was decided and this is what he was told. It might have been a wrong decision (the interaction with the DS team was not at the same level as it is now) but under circumstances it was the one we made. And we now need to live with it. I do not think we have time to reevaluate things and move them into DS plugins is we find the framework abusing its power. I generally disagree with the notion that we can and should view add-user as the only special operation. Because of the framework extensibility any operation can be special. So I think that external LDAP proxy makes more sense. It is a bit more complex than what you propose but has several significant benefits: * We need LDAP proxy for other purposes so there will be a reuse of code and framework * We already have expertise * It allows us to keep what we already have done without any conceptual changes and continue with the approach we have been developing framework for last 4 years * It can be developed independently and incrementally without affecting the core IPA and its release schedule * It does not require and knowledge of the internals of IPA framework or DS plugins and can be developed by an external contributor I agree that virtual entries and staging areas are not the right approach. I also do not think that penrose at least in its current incarnation is the right approach either. Leveraging a python based LDAP proxy sounds like a good starting point. With that I suggest to move this conversation from what to who and how. Do we have an RFE logged? -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Fri, 2013-02-15 at 17:28 -0500, Dmitri Pal wrote: On 02/14/2013 12:33 PM, Endi Sukma Dewata wrote: On 2/14/2013 8:06 AM, Simo Sorce wrote: On Thu, 2013-02-14 at 14:26 +0100, Petr Spacek wrote: In my Fedora 17 I found package python-ldaptor. It seems to offer nice support for writing own event-based LDAP servers. For simple LDAP proxy it could be enough. $ yum install python-ldaptor $ python import ldaptor.protocols.ldap.ldapserver help(ldaptor.protocols.ldap.ldapserver) No. LDAP proxies are *not* simple. Ask Endi he's worked on a meta-directory for years. Simo. It depends on what you want to do with the proxy. If it's only a thin layer which converts the LDAP ADD to IPA user-add it might not be that complicated. Penrose virtual directory consists of a frontend LDAP interface, a transformation engine, and backends which may include an LDAP server as well. The front-end LDAP interface is the proxy we're talking about here, it's only used to receive LDAP requests and pass them to the transformation engine. The transformation engine is where the complexity occurs. In IPA this is already handled by the framework. In Penrose it's quite complex because it aims to provide a generic way to map an LDAP request to multiple backends which involves dealing with different types of backends, joining the backends, transforming the DN attributes back and forth, etc. So I'd say implementing an LDAP frontend for IPA using Python is something worth exploring. That way it can run in the same process so there's no concern about JSON performance/stability. +1 I would rather go this way. Simo, Your point about how things should have been bight be righteous but Petr confusion about the role framework shows the reality. Currently all framework is developed as if it is the primary interface for creation/modification. LDAP is exposed but can be consumed by external clients as read only. This is how we decided to implement things. I clearly remember a lengthy discussion about this 4 years ago when the framework architecture was discussed. The main reason is plugability. Since we need to provide extensible plugability framework it is the only supportable option. For good or for bad it is how things are now and Petr is correct because this is what was decided and this is what he was told. It might have been a wrong decision (the interaction with the DS team was not at the same level as it is now) but under circumstances it was the one we made. And we now need to live with it. I do not think we have time to reevaluate things and move them into DS plugins is we find the framework abusing its power. I generally disagree with the notion that we can and should view add-user as the only special operation. Because of the framework extensibility any operation can be special. I do not have time to explain why I do not agree with your analysis here. I just don't, I think I will re-raise this later. So I think that external LDAP proxy makes more sense. It is a bit more complex than what you propose but has several significant benefits: * We need LDAP proxy for other purposes so there will be a reuse of code and framework What other purposes ? * We already have expertise * It allows us to keep what we already have done without any conceptual changes and continue with the approach we have been developing framework for last 4 years I do not think there is any need to make substantial changes to the framework with either proposal, and both would cause similar changes anyway. * It can be developed independently and incrementally without affecting the core IPA and its release schedule Same for both approaches. * It does not require and knowledge of the internals of IPA framework or DS plugins and can be developed by an external contributor Same level of knowledge is required for both approaches. I agree that virtual entries and staging areas are not the right approach. I also do not think that penrose at least in its current incarnation is the right approach either. Leveraging a python based LDAP proxy sounds like a good starting point. I think it is a bad idea. With that I suggest to move this conversation from what to who and how. Do we have an RFE logged? I suggest we do nothing, I think there is a substantial under-estimation of what it means to introduce an LDAP proxy on top of the framework, or on the side, or whatever, it is not even clear at what level this proxy would operate and with what semantics, and it is a substantially larger project than the plugin I proposed. It would divert substantial resources, and take a long time to be made useful. NAK. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 07:11 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 10:57 -0700, Rich Megginson wrote: Rich, is there potential from deadlocking here due to the new transaction stuff ? Or can we single out this plugin to run before *any* transaction is started ? If you do this in a regular pre-op, not a betxn pre-op, then it should be fine. Ok in this case we should be able to create a regular pre-op plugin to intercept the ldap add call and then use the following flow: client --(LDAP)-- 389DS --(HTTP/json)-- framework --(LDAP)-- add So no deadlocks will happen, the remaining issue is how to make sure we do not loop by mistake in the second add. One way could be to have loop detection so that if more then two (1. original, 2. framework) adds for the same DN come in we just return errors. Another way is to use a special objectclass as I proposed in the thread and make sure the framework explictly blacklists it so that it can never try to send an add with the special oc even by mistake or user misconfiguration. And a third way is a separate LDAP tree. I'm not familiar with what DS plugins can do. Can we craft one that intercepts all read operations on cn=HR tree,$SUFFIX and does them on cn=users,cn=accounts,$SUFFIX instead (using that tree's permissions), and forwards all write operations on cn=HR tree to IPA via JSON? A fourth way is a proxy/gateway, essentially the same but with a separate server. -- Petr³ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 14.2.2013 03:55, Dmitri Pal wrote: On 02/13/2013 09:48 PM, Nathan Kinder wrote: On 02/13/2013 06:18 PM, Dmitri Pal wrote: On 02/13/2013 02:08 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 13:30 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. I think he means have a persistent search to watch for new entries and then act upon them. I think a 2 step approach is misguided, so I've written it off from the start. Simo. This all thread smells like rewriting winsync by using internal plugin and IPA framework. Is there any chance we can use existing winsync solution to do what we need but sync from staging instance rather than AD? Then the flow will be: HR system - staging DS instance - dirsync - IPA Couple assumptions: a) We are satisfied with the security of the existing winsync solution and authentication used there. I do not see why it should be different here as it is a very similar use case. b) I expect we can sync from 389 to 389 with minor config changes and effectively no code changes This is not the case. Winsync uses the AD Dirsync control, which 389 does not support. The Winsync code also expects the AD schema on the entries coming in from the Dirsync response. Even if 389 supported the Dirsync control, there would be a fair amount of code changes to deal with whatever schema we would need to expect. c) The users created via winsync now are created in a proper way so IPA accepts then as IPA users This solves the problem of add and delete of the users. I know that winsync already supports group add/delete but we never qualified it in IPA. So may be it is time, at least for this use case. I would really think that this might be a lower cost solution than writing yet another custom plugin. Let us try to reuse what we already have. I think a custom plug-in would be easier (though I do have concerns about the performance impact of the JSON approach). I really think that we rather have an external solution and not the one hot wired into the IPA internals via the proposed plugin. Something like an external proxy or gateway rather than the approach Simo proposed even if it would be more work. This work would be reusable. We already have several use cases for the LDAP proxy. This can be one more. A bit more and there will be enough need to build one anyways. The main value is that it is optional and external and acts as an ecosystem level solution that can be developed and tested on a separate schedule rather than being internal to IPA. In my Fedora 17 I found package python-ldaptor. It seems to offer nice support for writing own event-based LDAP servers. For simple LDAP proxy it could be enough. $ yum install python-ldaptor $ python import ldaptor.protocols.ldap.ldapserver help(ldaptor.protocols.ldap.ldapserver) -- Petr^2 Spacek ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 21:55 -0500, Dmitri Pal wrote: On 02/13/2013 09:48 PM, Nathan Kinder wrote: On 02/13/2013 06:18 PM, Dmitri Pal wrote: On 02/13/2013 02:08 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 13:30 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. I think he means have a persistent search to watch for new entries and then act upon them. I think a 2 step approach is misguided, so I've written it off from the start. Simo. This all thread smells like rewriting winsync by using internal plugin and IPA framework. Is there any chance we can use existing winsync solution to do what we need but sync from staging instance rather than AD? Then the flow will be: HR system - staging DS instance - dirsync - IPA Couple assumptions: a) We are satisfied with the security of the existing winsync solution and authentication used there. I do not see why it should be different here as it is a very similar use case. b) I expect we can sync from 389 to 389 with minor config changes and effectively no code changes This is not the case. Winsync uses the AD Dirsync control, which 389 does not support. The Winsync code also expects the AD schema on the entries coming in from the Dirsync response. Even if 389 supported the Dirsync control, there would be a fair amount of code changes to deal with whatever schema we would need to expect. c) The users created via winsync now are created in a proper way so IPA accepts then as IPA users This solves the problem of add and delete of the users. I know that winsync already supports group add/delete but we never qualified it in IPA. So may be it is time, at least for this use case. I would really think that this might be a lower cost solution than writing yet another custom plugin. Let us try to reuse what we already have. I think a custom plug-in would be easier (though I do have concerns about the performance impact of the JSON approach). I really think that we rather have an external solution and not the one hot wired into the IPA internals via the proposed plugin. Something like an external proxy or gateway rather than the approach Simo proposed even if it would be more work. This work would be reusable. We already have several use cases for the LDAP proxy. This can be one more. A bit more and there will be enough need to build one anyways. The main value is that it is optional and external and acts as an ecosystem level solution that can be developed and tested on a separate schedule rather than being internal to IPA. Penrose. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Thu, 2013-02-14 at 00:34 -0500, John Dennis wrote: On 02/14/2013 12:16 AM, Nathan Kinder wrote: On 02/13/2013 08:30 PM, John Dennis wrote: On 02/13/2013 10:40 PM, Nathan Kinder wrote: With the DS plug-in approach that calls into the IPA framework with a 'mock add' to form the resulting entry at the pre-op stage, we could take care of the initial ADD operation of the user entry. We would still need to have a way to trigger the additional write operations that the IPA framework performs. The proposed DS plug-in could do an internal search operation after the add, then it could perform the additional write operations that are needed. This logic would need to be in the DS plug-in, or we would need another way to get the details from the IPA framework via JSON. The approach begins to get a bit messy. Messy is one word for it :-) Herein lies the problem. user-add is actually a simple case where in this particular version of our code we know the LDAP operations that occur and in what order. But there is nothing in our coding guidelines that guarantees this, it just happens to be true today. Pre and post callbacks are free to do as they please. The logic in our ldap module can change (in fact it's undergoing a rewrite as we speak). We've committed to supporting an extension mechanism (plugins) that ties into the framework operations and who knows what might occur in that code. At the moment it's not used, but hopefully it will be. Then comes the question, where do we draw the line? Do we say only user-add with no additional options is supported because we know what occurs during it's processing and hence we feel safe emulating user-add outside the framework? We all know this is a slippery slope, as soon as you support user-add this way we'll be getting requests to support other commands or other options. And after we make framework changes (which we do frequently) are we going to test these out-of-band operations continue to emulate the ever changing framework? It's a slippery slope that can expose us to problems we don't need. Jenny, do you want to test an entirely different mechanism or do you want to limit it to our defined API? We have a defined API, I really think that's the only thing we should support. Backdoor shortcuts that sidestep our framework should be off the table IMHO, it's just inviting too many problems. This is why I think that the most sane approach is to put something in front of 'ipa user-add'. This is not without it's own set of difficulties, but if we need a basic LDAP interface that can be used to create IPA users, it should end up calling into the same code that 'ipa user-add' uses. This would ensure that any extensions (plugins) for the framework are called. I actually see this as very similar to the LDAP migration case. For migration, we basically take entries in LDIF format, extract the relevant data, then feed it into 'ipa user-add', right? That's more or less what we need to do here for HR systems that provision users, but the user addition needs to be pushed to IPA via an LDAP client instead of IPA pulling from an LDAP server. Yes, this is exactly what I'm saying and we concur. So as I proposed we can call ipa user-add from LDAP from a non-transactional pre-op plugin. We just need to be careful about when we allow that to avoid loops, but besides that problem it seem relatively easy and does not require crazy things like playgrounds or even full LDAP proxies. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Thu, 2013-02-14 at 14:26 +0100, Petr Spacek wrote: In my Fedora 17 I found package python-ldaptor. It seems to offer nice support for writing own event-based LDAP servers. For simple LDAP proxy it could be enough. $ yum install python-ldaptor $ python import ldaptor.protocols.ldap.ldapserver help(ldaptor.protocols.ldap.ldapserver) No. LDAP proxies are *not* simple. Ask Endi he's worked on a meta-directory for years. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
John Dennis wrote: On 02/14/2013 12:16 AM, Nathan Kinder wrote: On 02/13/2013 08:30 PM, John Dennis wrote: On 02/13/2013 10:40 PM, Nathan Kinder wrote: With the DS plug-in approach that calls into the IPA framework with a 'mock add' to form the resulting entry at the pre-op stage, we could take care of the initial ADD operation of the user entry. We would still need to have a way to trigger the additional write operations that the IPA framework performs. The proposed DS plug-in could do an internal search operation after the add, then it could perform the additional write operations that are needed. This logic would need to be in the DS plug-in, or we would need another way to get the details from the IPA framework via JSON. The approach begins to get a bit messy. Messy is one word for it :-) Herein lies the problem. user-add is actually a simple case where in this particular version of our code we know the LDAP operations that occur and in what order. But there is nothing in our coding guidelines that guarantees this, it just happens to be true today. Pre and post callbacks are free to do as they please. The logic in our ldap module can change (in fact it's undergoing a rewrite as we speak). We've committed to supporting an extension mechanism (plugins) that ties into the framework operations and who knows what might occur in that code. At the moment it's not used, but hopefully it will be. Then comes the question, where do we draw the line? Do we say only user-add with no additional options is supported because we know what occurs during it's processing and hence we feel safe emulating user-add outside the framework? We all know this is a slippery slope, as soon as you support user-add this way we'll be getting requests to support other commands or other options. And after we make framework changes (which we do frequently) are we going to test these out-of-band operations continue to emulate the ever changing framework? It's a slippery slope that can expose us to problems we don't need. Jenny, do you want to test an entirely different mechanism or do you want to limit it to our defined API? We have a defined API, I really think that's the only thing we should support. Backdoor shortcuts that sidestep our framework should be off the table IMHO, it's just inviting too many problems. This is why I think that the most sane approach is to put something in front of 'ipa user-add'. This is not without it's own set of difficulties, but if we need a basic LDAP interface that can be used to create IPA users, it should end up calling into the same code that 'ipa user-add' uses. This would ensure that any extensions (plugins) for the framework are called. I actually see this as very similar to the LDAP migration case. For migration, we basically take entries in LDIF format, extract the relevant data, then feed it into 'ipa user-add', right? That's more or less what we need to do here for HR systems that provision users, but the user addition needs to be pushed to IPA via an LDAP client instead of IPA pulling from an LDAP server. Yes, this is exactly what I'm saying and we concur. In reality we don't actually pass migrated users to user-add for performance reasons. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/14/2013 09:05 AM, Simo Sorce wrote: So as I proposed we can call ipa user-add from LDAP from a non-transactional pre-op plugin. We just need to be careful about when we allow that to avoid loops, but besides that problem it seem relatively easy and does not require crazy things like playgrounds or even full LDAP proxies. I think I need a clarification because perhaps I didn't fully understand your proposal. Is the idea with a non-transactional pre-op plugin it invokes user-add and then the pre-op returns *without* having modified ldap? In effect it acts as a trigger? That still implies there has to be a separate tree where the foreign entity writes (and the pre-op plugin watches) because otherwise how could the pre-op plugin distinguish between framework writes and foreign writes? If there is a separate tree where is the looping issue? You still haven't explained this. Also, under the scenario that a foreign entity writes something into LDAP (somewhere) and it triggers us to call user-add via some mechanism then what happens when errors occur? The foreign entity will not know we rejected the operation nor why. Also, don't forget they want to delete users, remove group membership, add group membership, add groups, remove groups etc. Some of these operations are dependent upon logic in our framework. I don't see how some of these operations can be reliably managed by a foreign entity simultaneously performing LDAP operations. -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/14/2013 09:29 AM, Rob Crittenden wrote: In reality we don't actually pass migrated users to user-add for performance reasons. What parts of user-add are we bypassing? -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
John Dennis wrote: On 02/14/2013 09:29 AM, Rob Crittenden wrote: In reality we don't actually pass migrated users to user-add for performance reasons. What parts of user-add are we bypassing? Everything. It doesn't call into it at all. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Thu, 2013-02-14 at 09:33 -0500, John Dennis wrote: On 02/14/2013 09:05 AM, Simo Sorce wrote: So as I proposed we can call ipa user-add from LDAP from a non-transactional pre-op plugin. We just need to be careful about when we allow that to avoid loops, but besides that problem it seem relatively easy and does not require crazy things like playgrounds or even full LDAP proxies. I think I need a clarification because perhaps I didn't fully understand your proposal. Is the idea with a non-transactional pre-op plugin it invokes user-add and then the pre-op returns *without* having modified ldap? In effect it acts as a trigger? Yes. That still implies there has to be a separate tree where the foreign entity writes (and the pre-op plugin watches) No. because otherwise how could the pre-op plugin distinguish between framework writes and foreign writes? The idea I flaunted around is that the pluggin triggers only if a special objectclass is used. The framewrok would never try to create a user with this special objectclass so loops are avoided. If there is a separate tree where is the looping issue? You still haven't explained this. No separate tree, the plugin intercepts the Add operations, and calls the framework. The framework does the real user creation and then returns. At that point the pre-op plugin returns immediately with the error code from the framework discarding the actual add operation. Also, under the scenario that a foreign entity writes something into LDAP (somewhere) and it triggers us to call user-add via some mechanism then what happens when errors occur? We return back an error. The foreign entity will not know we rejected the operation nor why. This is the problem if you use a staged approach, and one of the reasons I do not consider it as viable, and therefore reject any proposal that imply staged writes and separate operations. Also, don't forget they want to delete users, remove group membership, add group membership, add groups, remove groups etc. Some of these operations are dependent upon logic in our framework. Which ones. Only user creation is really special. Deleting a user should not depend on the framework, nor adding group members. Adding a user is a special case, and should be limited to that. I don't see how some of these operations can be reliably managed by a foreign entity simultaneously performing LDAP operations. I do not see what your concerns are. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Thu, 2013-02-14 at 09:47 -0500, Rob Crittenden wrote: John Dennis wrote: On 02/14/2013 09:05 AM, Simo Sorce wrote: So as I proposed we can call ipa user-add from LDAP from a non-transactional pre-op plugin. We just need to be careful about when we allow that to avoid loops, but besides that problem it seem relatively easy and does not require crazy things like playgrounds or even full LDAP proxies. I think I need a clarification because perhaps I didn't fully understand your proposal. Is the idea with a non-transactional pre-op plugin it invokes user-add and then the pre-op returns *without* having modified ldap? In effect it acts as a trigger? That still implies there has to be a separate tree where the foreign entity writes (and the pre-op plugin watches) because otherwise how could the pre-op plugin distinguish between framework writes and foreign writes? The proposal is that we write a 389-ds plugin that intercepts LDAP ADD operations. I don't think Simo proposed where in the tree this would be written, or whether we would be able to distinguish between a normal ADD and this special case (loop 1). I did, please see my other emails in the thread. For the purposes of modification you probably want it to write into the current user container. It will need to be able to push MOD requests for names, addresses, etc. Most LDAP apps take a single basedn against which to do things. We can't expect to be able to provide one set for adds and one set for mods. We do not intercept mods, period. The only *real* issue is creating users, because the framework adds a lot of stuff to properly format users, but modifications from LDAP clients is fair game. We cannot consider the framework as the only gatekeeper beyond object creation, if this is not clear we need to discuss. And of course this leads into deletes. We want users gone when they leave the company. Feature creep already. Do we do anything special in the framework for user deletes now ? I am not aware of any special ops, and I would be against making a delete operation a special operation. If there is a separate tree where is the looping issue? You still haven't explained this. The looping is in telling whether the add is from the external HR source or a normal ADD from IPA. If we get it wrong, ever, then it will ADD, then call ipa user-add, ADD, call ipa user-add, ADD, ... Yes, this is the only concern, I made one proposal to avoid the issue, there are also other ways, for example the framework could use a special control to tell our server 'I am the framework, do not loop'. We just need to decide what's the best way to do it. For example we already need to avoid any interference with replication, but that is easy to do, as we know if an operation is coming via replication. The other is winsync, but I would argue that winsync maybe should actually use this plugin as it would make it more consistent with the framework. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/14/2013 01:59 AM, Petr Viktorin wrote: On 02/13/2013 07:11 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 10:57 -0700, Rich Megginson wrote: Rich, is there potential from deadlocking here due to the new transaction stuff ? Or can we single out this plugin to run before *any* transaction is started ? If you do this in a regular pre-op, not a betxn pre-op, then it should be fine. Ok in this case we should be able to create a regular pre-op plugin to intercept the ldap add call and then use the following flow: client --(LDAP)-- 389DS --(HTTP/json)-- framework --(LDAP)-- add So no deadlocks will happen, the remaining issue is how to make sure we do not loop by mistake in the second add. One way could be to have loop detection so that if more then two (1. original, 2. framework) adds for the same DN come in we just return errors. Another way is to use a special objectclass as I proposed in the thread and make sure the framework explictly blacklists it so that it can never try to send an add with the special oc even by mistake or user misconfiguration. And a third way is a separate LDAP tree. I'm not familiar with what DS plugins can do. Can we craft one that intercepts all read operations on cn=HR tree,$SUFFIX and does them on cn=users,cn=accounts,$SUFFIX instead (using that tree's permissions), and forwards all write operations on cn=HR tree to IPA via JSON? Yes. A fourth way is a proxy/gateway, essentially the same but with a separate server. ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
Simo Sorce wrote: On Thu, 2013-02-14 at 09:47 -0500, Rob Crittenden wrote: John Dennis wrote: On 02/14/2013 09:05 AM, Simo Sorce wrote: So as I proposed we can call ipa user-add from LDAP from a non-transactional pre-op plugin. We just need to be careful about when we allow that to avoid loops, but besides that problem it seem relatively easy and does not require crazy things like playgrounds or even full LDAP proxies. I think I need a clarification because perhaps I didn't fully understand your proposal. Is the idea with a non-transactional pre-op plugin it invokes user-add and then the pre-op returns *without* having modified ldap? In effect it acts as a trigger? That still implies there has to be a separate tree where the foreign entity writes (and the pre-op plugin watches) because otherwise how could the pre-op plugin distinguish between framework writes and foreign writes? The proposal is that we write a 389-ds plugin that intercepts LDAP ADD operations. I don't think Simo proposed where in the tree this would be written, or whether we would be able to distinguish between a normal ADD and this special case (loop 1). I did, please see my other emails in the thread. For the purposes of modification you probably want it to write into the current user container. It will need to be able to push MOD requests for names, addresses, etc. Most LDAP apps take a single basedn against which to do things. We can't expect to be able to provide one set for adds and one set for mods. We do not intercept mods, period. The only *real* issue is creating users, because the framework adds a lot of stuff to properly format users, but modifications from LDAP clients is fair game. We are in agreement here. My point is that if we try to write to a special part of the tree then move the entry then MODS will always fail. We cannot consider the framework as the only gatekeeper beyond object creation, if this is not clear we need to discuss. That was actually the point of the framework from the beginning. And of course this leads into deletes. We want users gone when they leave the company. Feature creep already. Do we do anything special in the framework for user deletes now ? I am not aware of any special ops, and I would be against making a delete operation a special operation. Any operation can have a pre/post operation. One RFE we have is to add triggers so that shell scripts (or whatever) can be executed at times. This is is the sort of thing that would be missed. If there is a separate tree where is the looping issue? You still haven't explained this. The looping is in telling whether the add is from the external HR source or a normal ADD from IPA. If we get it wrong, ever, then it will ADD, then call ipa user-add, ADD, call ipa user-add, ADD, ... Yes, this is the only concern, I made one proposal to avoid the issue, there are also other ways, for example the framework could use a special control to tell our server 'I am the framework, do not loop'. We just need to decide what's the best way to do it. Right. I don't know enough about how much flexibility these external systems have when creating LDAP objects so they may not be able to add additional objectclasses. A control may be the way to go. For example we already need to avoid any interference with replication, but that is easy to do, as we know if an operation is coming via replication. The other is winsync, but I would argue that winsync maybe should actually use this plugin as it would make it more consistent with the framework. Yes, if we could solve this without too much performance issues that should work. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/14/2013 04:22 PM, Rich Megginson wrote: On 02/14/2013 01:59 AM, Petr Viktorin wrote: On 02/13/2013 07:11 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 10:57 -0700, Rich Megginson wrote: Rich, is there potential from deadlocking here due to the new transaction stuff ? Or can we single out this plugin to run before *any* transaction is started ? If you do this in a regular pre-op, not a betxn pre-op, then it should be fine. Ok in this case we should be able to create a regular pre-op plugin to intercept the ldap add call and then use the following flow: client --(LDAP)-- 389DS --(HTTP/json)-- framework --(LDAP)-- add So no deadlocks will happen, the remaining issue is how to make sure we do not loop by mistake in the second add. One way could be to have loop detection so that if more then two (1. original, 2. framework) adds for the same DN come in we just return errors. Another way is to use a special objectclass as I proposed in the thread and make sure the framework explictly blacklists it so that it can never try to send an add with the special oc even by mistake or user misconfiguration. And a third way is a separate LDAP tree. I'm not familiar with what DS plugins can do. Can we craft one that intercepts all read operations on cn=HR tree,$SUFFIX and does them on cn=users,cn=accounts,$SUFFIX instead (using that tree's permissions), and forwards all write operations on cn=HR tree to IPA via JSON? Yes. Then I recommend doing this. It avoids problems with delegation (the real tree permissions are used) and looping (plugin and IPA write to separate trees). Other operations (deletes, mods) can be either similarly delegated to the real tree, or passed through IPA if we want to do that in the future. Problems with replication can be solved by just not replicating the separate tree. It also doesn't pollute core IPA with special cases, which is what worries me. -- Petr³ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Thu, 2013-02-14 at 16:29 +0100, Petr Viktorin wrote: Then I recommend doing this. It avoids problems with delegation (the real tree permissions are used) and looping (plugin and IPA write to separate trees). Virtual objects are not free of issues, you are just trading some issues for others here, and I do not think you are properly evaluating the trade here. I am dead set against using staging areas, and I want to see proof they are a good idea, not handwaving 'fears' of using json in the server to forward operations directly to the framework. Other operations (deletes, mods) can be either similarly delegated to the real tree, And *this* is a slippery slope. you are trading delegating one single operation to the framework directly, with proper error propagation back to the client, with now implementing full virtualized operations for mod and delete, and bind ? and search and then ? You are now basically paving the way for a virtual directory in tree. Sorry, but no. or passed through IPA if we want to do that in the future. Problems with replication can be solved by just not replicating the separate tree. More and more hacks, for this supposedly 'cleaner' or 'simpler' solution ... sorry if I don't bite. It also doesn't pollute core IPA with special cases, which is what worries me. What does this mean ? We *have* a special case, and we are discussing how to handle it. The situation here (I do not want to call it a 'problem') is that we decided to put business logic around user creation in the framework because we thought that would be easier to prototype and develop in python code rather than in a plugin. However this special handling *must* be limited. LDAP is our main storage, and we must allow LDAP operations against it. Special cases needs to be limited to things that are really harder done in plugins rather then python code. For example if we need triggers for some operations in LDAP, they *must* be done through a 389ds plugin. Otherwise LDAP quickly becomes a read-only interface and interoperability quickly goes out of the window. I always treated the framework as a *management interface* on top. We can do things that are 'convenient', and 'help' admins avoid mistakes, but we cannot move core functionality in the framework, it would be a grave mistake. User creation *is* a special case, but should remain one of very few special exceptions. This very thread and the need for the interface proposed in this thread is a clear example of why we need to be extremely careful not to be too liberal with what business logic we move in the framework. LDAP keeps us honest, so we need to limit what we do in the framework, otherwise we'll keep taking shortcuts and soon enough it goes out of control and we loose interoperability with anything that is not purpose-built to talk to our framework. This should be an unacceptable scenario because it is like putting ourselves in a getto. We trade greater interoperability and acceptance for small conveniences here and there. We must thread *very* carefully along this line. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/14/2013 09:01 AM, Simo Sorce wrote: On Thu, 2013-02-14 at 16:29 +0100, Petr Viktorin wrote: Then I recommend doing this. It avoids problems with delegation (the real tree permissions are used) and looping (plugin and IPA write to separate trees). Virtual objects are not free of issues, you are just trading some issues for others here, and I do not think you are properly evaluating the trade here. I am dead set against using staging areas, and I want to see proof they are a good idea, not handwaving 'fears' of using json in the server to forward operations directly to the framework. Other operations (deletes, mods) can be either similarly delegated to the real tree, And *this* is a slippery slope. you are trading delegating one single operation to the framework directly, with proper error propagation back to the client, with now implementing full virtualized operations for mod and delete, and bind ? and search and then ? You are now basically paving the way for a virtual directory in tree. Sorry, but no. or passed through IPA if we want to do that in the future. Problems with replication can be solved by just not replicating the separate tree. More and more hacks, for this supposedly 'cleaner' or 'simpler' solution ... sorry if I don't bite. It also doesn't pollute core IPA with special cases, which is what worries me. What does this mean ? We *have* a special case, and we are discussing how to handle it. The situation here (I do not want to call it a 'problem') is that we decided to put business logic around user creation in the framework because we thought that would be easier to prototype and develop in python code rather than in a plugin. However this special handling *must* be limited. LDAP is our main storage, and we must allow LDAP operations against it. Special cases needs to be limited to things that are really harder done in plugins rather then python code. For example if we need triggers for some operations in LDAP, they *must* be done through a 389ds plugin. Otherwise LDAP quickly becomes a read-only interface and interoperability quickly goes out of the window. I always treated the framework as a *management interface* on top. We can do things that are 'convenient', and 'help' admins avoid mistakes, but we cannot move core functionality in the framework, it would be a grave mistake. User creation *is* a special case, but should remain one of very few special exceptions. This very thread and the need for the interface proposed in this thread is a clear example of why we need to be extremely careful not to be too liberal with what business logic we move in the framework. LDAP keeps us honest, so we need to limit what we do in the framework, otherwise we'll keep taking shortcuts and soon enough it goes out of control and we loose interoperability with anything that is not purpose-built to talk to our framework. This should be an unacceptable scenario because it is like putting ourselves in a getto. We trade greater interoperability and acceptance for small conveniences here and there. We must thread *very* carefully along this line. +1 - virtual trees usually end up being rat holes with no end What is the problem we're trying to solve? To be able to call python code in response to an LDAP operation? What if we added a python interpreter to 389, like OpenLDAP back-python? Simo. ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Thu, 2013-02-14 at 17:31 +0100, Petr Viktorin wrote: On 02/14/2013 05:01 PM, Simo Sorce wrote: On Thu, 2013-02-14 at 16:29 +0100, Petr Viktorin wrote: Then I recommend doing this. It avoids problems with delegation (the real tree permissions are used) and looping (plugin and IPA write to separate trees). Virtual objects are not free of issues, you are just trading some issues for others here, and I do not think you are properly evaluating the trade here. I am dead set against using staging areas, and I want to see proof they are a good idea, not handwaving 'fears' of using json in the server to forward operations directly to the framework. Other operations (deletes, mods) can be either similarly delegated to the real tree, And *this* is a slippery slope. you are trading delegating one single operation to the framework directly, with proper error propagation back to the client, with now implementing full virtualized operations for mod and delete, and bind ? and search and then ? You are now basically paving the way for a virtual directory in tree. Sorry, but no. or passed through IPA if we want to do that in the future. Problems with replication can be solved by just not replicating the separate tree. More and more hacks, for this supposedly 'cleaner' or 'simpler' solution ... sorry if I don't bite. It also doesn't pollute core IPA with special cases, which is what worries me. What does this mean ? We *have* a special case, and we are discussing how to handle it. The situation here (I do not want to call it a 'problem') is that we decided to put business logic around user creation in the framework because we thought that would be easier to prototype and develop in python code rather than in a plugin. However this special handling *must* be limited. LDAP is our main storage, and we must allow LDAP operations against it. Special cases needs to be limited to things that are really harder done in plugins rather then python code. For example if we need triggers for some operations in LDAP, they *must* be done through a 389ds plugin. Otherwise LDAP quickly becomes a read-only interface and interoperability quickly goes out of the window. I always treated the framework as a *management interface* on top. We can do things that are 'convenient', and 'help' admins avoid mistakes, but we cannot move core functionality in the framework, it would be a grave mistake. User creation *is* a special case, but should remain one of very few special exceptions. This very thread and the need for the interface proposed in this thread is a clear example of why we need to be extremely careful not to be too liberal with what business logic we move in the framework. LDAP keeps us honest, so we need to limit what we do in the framework, otherwise we'll keep taking shortcuts and soon enough it goes out of control and we loose interoperability with anything that is not purpose-built to talk to our framework. This should be an unacceptable scenario because it is like putting ourselves in a getto. We trade greater interoperability and acceptance for small conveniences here and there. We must thread *very* carefully along this line. Simo. Ah! This is a point we need to stress more, I didn't get it. All this time I thought of it the other way -- IPA being the main interface, with LDAP being more of an implementation detail -- the storage, and a read-only interface for LDAP consumers. Thanks for explaining, your solution makes perfect sense then. However, if there are more people like me, then there's code in IPA that assumes IPA is the only interface modifications, and raw LDAP operations are hacks that happen to work for now. IPA's command plugins also seem misguided now, as you pointed out. I don't like this view much but, well, I can adapt. Allowing writes by tools that aren't purpose-built to talk to our framework, (for example ones that don't honor our schema), still seems like something to be wary of. It's a balancing act. The schema is always honored, that's why there are MUST and MAY attributes. If the schema is build with the wrong specification of MUST and MAYs that's a bug on our part. Also, there are some parts of the tree that are more sensitive than others. Some things are pretty IPA specific, and for those cases relying a bit more on the framework is ok for now. But in general LDAP is a primary interface, it's where replication happens, and where all components converge. So it is the center of all things. Think about triggers for example. One of the triggers that gets often requested is to be create home directories when users are created. Now, besides the point of whether we want or can do that, if you were to implement it you would have to do it at the 389ds level. You cannot do it in the framework at all for the simple reason
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/14/2013 05:01 PM, Simo Sorce wrote: On Thu, 2013-02-14 at 16:29 +0100, Petr Viktorin wrote: Then I recommend doing this. It avoids problems with delegation (the real tree permissions are used) and looping (plugin and IPA write to separate trees). Virtual objects are not free of issues, you are just trading some issues for others here, and I do not think you are properly evaluating the trade here. I am dead set against using staging areas, and I want to see proof they are a good idea, not handwaving 'fears' of using json in the server to forward operations directly to the framework. Other operations (deletes, mods) can be either similarly delegated to the real tree, And *this* is a slippery slope. you are trading delegating one single operation to the framework directly, with proper error propagation back to the client, with now implementing full virtualized operations for mod and delete, and bind ? and search and then ? You are now basically paving the way for a virtual directory in tree. Sorry, but no. or passed through IPA if we want to do that in the future. Problems with replication can be solved by just not replicating the separate tree. More and more hacks, for this supposedly 'cleaner' or 'simpler' solution ... sorry if I don't bite. It also doesn't pollute core IPA with special cases, which is what worries me. What does this mean ? We *have* a special case, and we are discussing how to handle it. The situation here (I do not want to call it a 'problem') is that we decided to put business logic around user creation in the framework because we thought that would be easier to prototype and develop in python code rather than in a plugin. However this special handling *must* be limited. LDAP is our main storage, and we must allow LDAP operations against it. Special cases needs to be limited to things that are really harder done in plugins rather then python code. For example if we need triggers for some operations in LDAP, they *must* be done through a 389ds plugin. Otherwise LDAP quickly becomes a read-only interface and interoperability quickly goes out of the window. I always treated the framework as a *management interface* on top. We can do things that are 'convenient', and 'help' admins avoid mistakes, but we cannot move core functionality in the framework, it would be a grave mistake. User creation *is* a special case, but should remain one of very few special exceptions. This very thread and the need for the interface proposed in this thread is a clear example of why we need to be extremely careful not to be too liberal with what business logic we move in the framework. LDAP keeps us honest, so we need to limit what we do in the framework, otherwise we'll keep taking shortcuts and soon enough it goes out of control and we loose interoperability with anything that is not purpose-built to talk to our framework. This should be an unacceptable scenario because it is like putting ourselves in a getto. We trade greater interoperability and acceptance for small conveniences here and there. We must thread *very* carefully along this line. Simo. Ah! This is a point we need to stress more, I didn't get it. All this time I thought of it the other way -- IPA being the main interface, with LDAP being more of an implementation detail -- the storage, and a read-only interface for LDAP consumers. Thanks for explaining, your solution makes perfect sense then. However, if there are more people like me, then there's code in IPA that assumes IPA is the only interface modifications, and raw LDAP operations are hacks that happen to work for now. IPA's command plugins also seem misguided now, as you pointed out. I don't like this view much but, well, I can adapt. Allowing writes by tools that aren't purpose-built to talk to our framework, (for example ones that don't honor our schema), still seems like something to be wary of. -- Petr³ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 2/14/2013 8:06 AM, Simo Sorce wrote: On Thu, 2013-02-14 at 14:26 +0100, Petr Spacek wrote: In my Fedora 17 I found package python-ldaptor. It seems to offer nice support for writing own event-based LDAP servers. For simple LDAP proxy it could be enough. $ yum install python-ldaptor $ python import ldaptor.protocols.ldap.ldapserver help(ldaptor.protocols.ldap.ldapserver) No. LDAP proxies are *not* simple. Ask Endi he's worked on a meta-directory for years. Simo. It depends on what you want to do with the proxy. If it's only a thin layer which converts the LDAP ADD to IPA user-add it might not be that complicated. Penrose virtual directory consists of a frontend LDAP interface, a transformation engine, and backends which may include an LDAP server as well. The front-end LDAP interface is the proxy we're talking about here, it's only used to receive LDAP requests and pass them to the transformation engine. The transformation engine is where the complexity occurs. In IPA this is already handled by the framework. In Penrose it's quite complex because it aims to provide a generic way to map an LDAP request to multiple backends which involves dealing with different types of backends, joining the backends, transforming the DN attributes back and forth, etc. So I'd say implementing an LDAP frontend for IPA using Python is something worth exploring. That way it can run in the same process so there's no concern about JSON performance/stability. -- Endi S. Dewata ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 07:53 AM, Simo Sorce wrote: Hello list, with recently seen a few requests to add FreeIPA users via LDAP directly. This is a common method supported by many meta-directory/HR systems, however so far we cannot really recommend it because we add quite a number of attributes automatically in our framework code when we create users, and those functions can change in future versions. However these external tools are usually not very flexible and supporting them as they are would make for a much better experience for integrators. I had a brief discussion with Rob on IRC on how to address this situation. If we limit ourselves to users we could probably resolve this problem with a relatively simple 389ds plugin that intercept add operations that try to add a user. The idea is that the remote system would be allowed to set a minimum of attributes (even incomplete according to our schema). But as long as a specific set of objectclasses is set (say person and posixaccount) the operation would be recognized as an attempt to create a user account. In this case the plugin would take over the operation and perform a call against our framework using json. So the 389 plugin would make an http/json call to the framework? The call would send a reformatted request using the data we got in input so that any custom objectclass /attribute can be preserved. The call would also add a special flag so the framework knows this is coming from 389ds itself. The framework would treat this requests in a slightly special way, it would use all the common code currently used to properly format a user entry adding all the ancillary data we need, but instead of trying to ldapadd the entry, it would instead return it back to the caller. 389ds at this point gets back a json reply, convert it into an ldap add operation and proceeds with this new 'augmented' operation instead of the original one. What do people think about this option ? I think it would be extremely valuable for admins, as it would allow them to drive user 'synchronization' in a very simple way. It could also be used to properly augment winsync users so we can allow full creation when syncing from AD with all the proper attributes created through the json request. So I see a lot of optential here. The only doubt is the json back and forth communication. What do people on the framework side think ? Is there going to be any big problem in adapting the framework so we can use common code to prepare the object but then change the last stp and return a json reply instead of perfroming an ldap add operation ? Simo. ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 03:53 PM, Simo Sorce wrote: Hello list, with recently seen a few requests to add FreeIPA users via LDAP directly. This is a common method supported by many meta-directory/HR systems, however so far we cannot really recommend it because we add quite a number of attributes automatically in our framework code when we create users, and those functions can change in future versions. However these external tools are usually not very flexible and supporting them as they are would make for a much better experience for integrators. I had a brief discussion with Rob on IRC on how to address this situation. If we limit ourselves to users we could probably resolve this problem with a relatively simple 389ds plugin that intercept add operations that try to add a user. The idea is that the remote system would be allowed to set a minimum of attributes (even incomplete according to our schema). But as long as a specific set of objectclasses is set (say person and posixaccount) the operation would be recognized as an attempt to create a user account. In this case the plugin would take over the operation and perform a call against our framework using json. The call would send a reformatted request using the data we got in input so that any custom objectclass /attribute can be preserved. The call would also add a special flag so the framework knows this is coming from 389ds itself. The framework would treat this requests in a slightly special way, it would use all the common code currently used to properly format a user entry adding all the ancillary data we need, but instead of trying to ldapadd the entry, it would instead return it back to the caller. Why return the data as JSON, instead of writing it directly to LDAP? Our own post-callback assumes the user is already in LDAP, and who knows what user-supplied callbacks will do. Keep in mind IPA is plugable; at least for outside plugins' sake (if not our own sanity's) we should keep the number of code paths to a minimum. 389ds at this point gets back a json reply, convert it into an ldap add operation and proceeds with this new 'augmented' operation instead of the original one. What do people think about this option ? I think it would be extremely valuable for admins, as it would allow them to drive user 'synchronization' in a very simple way. It could also be used to properly augment winsync users so we can allow full creation when syncing from AD with all the proper attributes created through the json request. So I see a lot of optential here. The only doubt is the json back and forth communication. What do people on the framework side think ? Is there going to be any big problem in adapting the framework so we can use common code to prepare the object but then change the last stp and return a json reply instead of perfroming an ldap add operation ? Simo. -- Petr³ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 13.2.2013 15:53, Simo Sorce wrote: Hello list, with recently seen a few requests to add FreeIPA users via LDAP directly. This is a common method supported by many meta-directory/HR systems, however so far we cannot really recommend it because we add quite a number of attributes automatically in our framework code when we create users, and those functions can change in future versions. However these external tools are usually not very flexible and supporting them as they are would make for a much better experience for integrators. I had a brief discussion with Rob on IRC on how to address this situation. If we limit ourselves to users we could probably resolve this problem with a relatively simple 389ds plugin that intercept add operations that try to add a user. The idea is that the remote system would be allowed to set a minimum of attributes (even incomplete according to our schema). But as long as a specific set of objectclasses is set (say person and posixaccount) the operation would be recognized as an attempt to create a user account. In this case the plugin would take over the operation and perform a call against our framework using json. The call would send a reformatted request using the data we got in input so that any custom objectclass /attribute can be preserved. The call would also add a special flag so the framework knows this is coming from 389ds itself. The framework would treat this requests in a slightly special way, it would use all the common code currently used to properly format a user entry adding all the ancillary data we need, but instead of trying to ldapadd the entry, it would instead return it back to the caller. 389ds at this point gets back a json reply, convert it into an ldap add operation and proceeds with this new 'augmented' operation instead of the original one. What do people think about this option ? I think it would be extremely valuable for admins, as it would allow them to drive user 'synchronization' in a very simple way. It could also be used to properly augment winsync users so we can allow full creation when syncing from AD with all the proper attributes created through the json request. So I see a lot of optential here. The only doubt is the json back and forth communication. What do people on the framework side think ? Is there going to be any big problem in adapting the framework so we can use common code to prepare the object but then change the last stp and return a json reply instead of perfroming an ldap add operation ? Personally, I don't like the idea of JSON requests response parsing *inside* 389. I see a lot of opportunities for crash or security hole. What about small (optional) separate daemon? A) Variant with separate sub-tree 1. create some new subtree, e.g. cn=useradd-playground,dc=example,dc=com 2. watch this sub-tree with persistent search 3. catch new objects and run IPA commands as necessary 4. replace old objects with referrals to IPA objects (or do something different) I'm not sure that various meta-directory systems can handle situation where user disappeared/object is moved/object is morphed to referal. That can be a problem. B) More crazy variant with LDAP proxy (Now I'm on very thin ice, I don't know how complex it would be. I only found some examples like [1].) Prepare fake (non-IPA) LDAP server with same LDAP structure and let separate daemon watch server's content and run IPA commands as necessary. It would be best if the small daemon can pretend to be LDAP server and catch LDAP operations directly, as shown in [1]. [1] http://tonthon.blogspot.cz/2011/02/ldaptor-ldap-with-twisted-server-side.html -- Petr^2 Spacek ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 04:15 PM, Petr Spacek wrote: On 13.2.2013 15:53, Simo Sorce wrote: Hello list, with recently seen a few requests to add FreeIPA users via LDAP directly. This is a common method supported by many meta-directory/HR systems, however so far we cannot really recommend it because we add quite a number of attributes automatically in our framework code when we create users, and those functions can change in future versions. However these external tools are usually not very flexible and supporting them as they are would make for a much better experience for integrators. I had a brief discussion with Rob on IRC on how to address this situation. If we limit ourselves to users we could probably resolve this problem with a relatively simple 389ds plugin that intercept add operations that try to add a user. The idea is that the remote system would be allowed to set a minimum of attributes (even incomplete according to our schema). But as long as a specific set of objectclasses is set (say person and posixaccount) the operation would be recognized as an attempt to create a user account. In this case the plugin would take over the operation and perform a call against our framework using json. The call would send a reformatted request using the data we got in input so that any custom objectclass /attribute can be preserved. The call would also add a special flag so the framework knows this is coming from 389ds itself. The framework would treat this requests in a slightly special way, it would use all the common code currently used to properly format a user entry adding all the ancillary data we need, but instead of trying to ldapadd the entry, it would instead return it back to the caller. 389ds at this point gets back a json reply, convert it into an ldap add operation and proceeds with this new 'augmented' operation instead of the original one. What do people think about this option ? I think it would be extremely valuable for admins, as it would allow them to drive user 'synchronization' in a very simple way. It could also be used to properly augment winsync users so we can allow full creation when syncing from AD with all the proper attributes created through the json request. So I see a lot of optential here. The only doubt is the json back and forth communication. What do people on the framework side think ? Is there going to be any big problem in adapting the framework so we can use common code to prepare the object but then change the last stp and return a json reply instead of perfroming an ldap add operation ? Personally, I don't like the idea of JSON requests response parsing *inside* 389. I see a lot of opportunities for crash or security hole. It seems Petr is reading my mind here. I was thinking about these solutions too; they seem cleaner but I don't know if they're feasible. What about small (optional) separate daemon? A) Variant with separate sub-tree 1. create some new subtree, e.g. cn=useradd-playground,dc=example,dc=com 2. watch this sub-tree with persistent search 3. catch new objects and run IPA commands as necessary 4. replace old objects with referrals to IPA objects (or do something different) I'm not sure that various meta-directory systems can handle situation where user disappeared/object is moved/object is morphed to referal. That can be a problem. A separate tree would mean we're not committed to supporting this particular kind of magic forever. We could have different compatibility hacks in the future, each with its own semantics. B) More crazy variant with LDAP proxy (Now I'm on very thin ice, I don't know how complex it would be. I only found some examples like [1].) Prepare fake (non-IPA) LDAP server with same LDAP structure and let separate daemon watch server's content and run IPA commands as necessary. It would be best if the small daemon can pretend to be LDAP server and catch LDAP operations directly, as shown in [1]. An adaptor would certainly be the proper solution when we need to interop with an inflexible foreign interface. The question is if building one is worth it. In both cases rather than core IPA functionality this would be something external, something the users have to explicitly install and use, something that doesn't necessarily have to work right with user-supplied plugins, something limited (say, to users only), and something that'd always use existing code paths. I'm really worried about scope creep here; after a few years of adding features like this IPA would become unmaintainable. Better to have a focused core and add on top. -- Petr³ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 16:33 +0100, Petr Viktorin wrote: On 02/13/2013 04:15 PM, Petr Spacek wrote: On 13.2.2013 15:53, Simo Sorce wrote: Hello list, with recently seen a few requests to add FreeIPA users via LDAP directly. This is a common method supported by many meta-directory/HR systems, however so far we cannot really recommend it because we add quite a number of attributes automatically in our framework code when we create users, and those functions can change in future versions. However these external tools are usually not very flexible and supporting them as they are would make for a much better experience for integrators. I had a brief discussion with Rob on IRC on how to address this situation. If we limit ourselves to users we could probably resolve this problem with a relatively simple 389ds plugin that intercept add operations that try to add a user. The idea is that the remote system would be allowed to set a minimum of attributes (even incomplete according to our schema). But as long as a specific set of objectclasses is set (say person and posixaccount) the operation would be recognized as an attempt to create a user account. In this case the plugin would take over the operation and perform a call against our framework using json. The call would send a reformatted request using the data we got in input so that any custom objectclass /attribute can be preserved. The call would also add a special flag so the framework knows this is coming from 389ds itself. The framework would treat this requests in a slightly special way, it would use all the common code currently used to properly format a user entry adding all the ancillary data we need, but instead of trying to ldapadd the entry, it would instead return it back to the caller. 389ds at this point gets back a json reply, convert it into an ldap add operation and proceeds with this new 'augmented' operation instead of the original one. What do people think about this option ? I think it would be extremely valuable for admins, as it would allow them to drive user 'synchronization' in a very simple way. It could also be used to properly augment winsync users so we can allow full creation when syncing from AD with all the proper attributes created through the json request. So I see a lot of optential here. The only doubt is the json back and forth communication. What do people on the framework side think ? Is there going to be any big problem in adapting the framework so we can use common code to prepare the object but then change the last stp and return a json reply instead of perfroming an ldap add operation ? Personally, I don't like the idea of JSON requests response parsing *inside* 389. I see a lot of opportunities for crash or security hole. Petr, p[lease substantiate what is your worry, how is using json in 389ds a security issue ? It is well confined, and the only server you contact is the http server on localhost, not any random external server, so what makes you think of security holes ? What would cause crashes ? Please be precise and substantiate a threat model, I currently see nothing special in this scenario. It seems Petr is reading my mind here. I was thinking about these solutions too; they seem cleaner but I don't know if they're feasible. I am sorry, but 'cleaner' is really the last word I'd use, 'hack' is what comes to mind here to be honest. What about small (optional) separate daemon? One more moving part one additional process, that does very little, it looks to me as a big hammer to yield. A) Variant with separate sub-tree 1. create some new subtree, e.g. cn=useradd-playground,dc=example,dc=com This has more consequences than you may think. I do not like the separate field idea because you need to treat it in a special way. We would probably need special ACIs to anoty allow any other client to see this subtree otherwise they may see incomplete objects. Yet we need to allow some user to write to it. We need to decide case by case which plugins to enable, not all DS plugins can use filters or exclude subtrees so we may have issues with some plugins if we do not want them to operate on these special entries and so on. 2. watch this sub-tree with persistent search Persistent search are scarce, and user creation is rare, it would be over blown, plus it would be a race condition if the daemon dies for whatever reason. 3. catch new objects and run IPA commands as necessary With what user identity ? Keep in mind that one of the reasons we use delegation is to make sure that ACIs are meaningful *and* auditing reflects the actual user that did operations. With an external daemon you have to cases: 1) you use a fixed identity for the external daemon to create these users, that means the daemon now have to do additional checks to see if the user used to create the entry is authorized or not
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 16:12 +0100, Petr Viktorin wrote: Our own post-callback assumes the user is already in LDAP, and who knows what user-supplied callbacks will do. Keep in mind IPA is plugable; at least for outside plugins' sake (if not our own sanity's) we should keep the number of code paths to a minimum. True which is why my proposal is to not use the standard user-add RPC call, but have a separate one. This separate call would only call the core business logic to create the user account add operation, but none of the external plumbing. Ideally we spit the framework flow like this: Normal user - Real user-add --- . . . . . . . . . --- LDAP add \ / -- common logic -- / \ 389ds plugin - Mock user-add -- . . . . . . . . . --- json reply custom plugins should be called in the custom logic an operate on the object before the ADD is attempted. If we do it this way then most of the code path will be in common which is what we want, and only the mechanical operation of adding the actual object to ldap will be different. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 11:27 -0500, Simo Sorce wrote: This is why I proposed a plugin that is limited to users and calls the framework so we can use common code. The *simpler* way would be to simply replicate the core framework login in the 389ds plugin or even *move* it there. But we want to keep the logic in the framework as it is more flexible and easier to work with and extend, so I proposed a 389ds plugin that just *asks* the framwrok for the data. This keeps the busienss loginc in the python framewrok, yet it allows an LDAP driver to add users properly in IPA just using LDAP calls. I do not see this as a slippery slope, as it would be limited to user creation by definition. Btw as a bonus you get an API that user can use to 'test' user creation w/o actually creating users, I think this may come useful as well for people experimenting with creating plugins, as they can tweak their plugin configuration rapidly w/o actually touching LDAP. And only use the real creation path once they are happy with the results returned by the new API. It's not a reason in itself to add this stuff but it is a nice byproduct. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 08:08 -0700, Rich Megginson wrote: On 02/13/2013 07:53 AM, Simo Sorce wrote: Hello list, with recently seen a few requests to add FreeIPA users via LDAP directly. This is a common method supported by many meta-directory/HR systems, however so far we cannot really recommend it because we add quite a number of attributes automatically in our framework code when we create users, and those functions can change in future versions. However these external tools are usually not very flexible and supporting them as they are would make for a much better experience for integrators. I had a brief discussion with Rob on IRC on how to address this situation. If we limit ourselves to users we could probably resolve this problem with a relatively simple 389ds plugin that intercept add operations that try to add a user. The idea is that the remote system would be allowed to set a minimum of attributes (even incomplete according to our schema). But as long as a specific set of objectclasses is set (say person and posixaccount) the operation would be recognized as an attempt to create a user account. In this case the plugin would take over the operation and perform a call against our framework using json. So the 389 plugin would make an http/json call to the framework? This i the idea. 1. add operation comes in For example: dn: uid=foo,cn=users,cn=accounts,dc=freeipa,dc=org changetype: add objectclass: ipa-add-user objectclass: custom-stuff uid: foo custom-attr: bar ... (other attributes we require absolutely) 2. we match on special objectclass 'IPA-ADD-USER' 3. plugin converts incoming entry into json request and calls framework 4. framework does magic and creates a full ldap request 5. framework converts it back into a json object 6. plugin converts json object into an add request and replaces the original add with the new data 7. plugins tries to commit the new data as the original user. 8. if there are any errors they are sent back to the client. This plugin would need to operate before any other so that all other plugins will see the data as modified by the call to the framework, not the original request. Also the special objectclass is just an idea, I am open to other suggestions. The only important thing is that we have a way to distinguish between an complete add operation that should just go thorough and a an add operation that needs to be sent to the framework for adjustment. We could also decide to always intercept, but it seem to be a bit wasteful for regular framework add operations as it would go through it twice: http - framewrok - ldap add - framework (recheck) - add Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 11:44 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 16:12 +0100, Petr Viktorin wrote: Our own post-callback assumes the user is already in LDAP, and who knows what user-supplied callbacks will do. Keep in mind IPA is plugable; at least for outside plugins' sake (if not our own sanity's) we should keep the number of code paths to a minimum. True which is why my proposal is to not use the standard user-add RPC call, but have a separate one. This separate call would only call the core business logic to create the user account add operation, but none of the external plumbing. Ideally we spit the framework flow like this: Normal user - Real user-add --- . . . . . . . . . --- LDAP add \ / -- common logic -- / \ 389ds plugin - Mock user-add -- . . . . . . . . . --- json reply custom plugins should be called in the custom logic an operate on the object before the ADD is attempted. If we do it this way then most of the code path will be in common which is what we want, and only the mechanical operation of adding the actual object to ldap will be different. Simo. There is one missing a few steps. A plugin execution looks like: Normal user - Real user-add --- pre-op call(s) --- execute (LPAP add record) --- post-op call(s) which may do additional add/modify It is the postop calls that would be the problem. They assume that the entry has already been written (so, for example, it has a valid UID/GID/ipaUniqueId, etc). Why are they done after the add ? It seem dangerous. What happens id the first ldap add succeed and the post op fails ? We should exceute the ldap call after the post ops are perfomed imho. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 05:27 PM, Simo Sorce wrote: [...] I am sorry, but 'cleaner' is really the last word I'd use, 'hack' is what comes to mind here to be honest. Then I'm missing something. Thanks for your explanations. What about small (optional) separate daemon? One more moving part one additional process, that does very little, it looks to me as a big hammer to yield. A) Variant with separate sub-tree 1. create some new subtree, e.g. cn=useradd-playground,dc=example,dc=com This has more consequences than you may think. I do not like the separate field idea because you need to treat it in a special way. We would probably need special ACIs to anoty allow any other client to see this subtree otherwise they may see incomplete objects. Yet we need to allow some user to write to it. We need to decide case by case which plugins to enable, not all DS plugins can use filters or exclude subtrees so we may have issues with some plugins if we do not want them to operate on these special entries and so on. Is it possible to use read ACIs of the original tree? 2. watch this sub-tree with persistent search Persistent search are scarce, and user creation is rare, it would be over blown, plus it would be a race condition if the daemon dies for whatever reason. 3. catch new objects and run IPA commands as necessary With what user identity ? Keep in mind that one of the reasons we use delegation is to make sure that ACIs are meaningful *and* auditing reflects the actual user that did operations. Thanks. Now your proposal starts making sense to me. [...] In both cases rather than core IPA functionality this would be something external, something the users have to explicitly install and use, something that doesn't necessarily have to work right with user-supplied plugins, something limited (say, to users only), and something that'd always use existing code paths. I'm really worried about scope creep here; after a few years of adding features like this IPA would become unmaintainable. Better to have a focused core and add on top. This is why I proposed a plugin that is limited to users and calls the framework so we can use common code. The *simpler* way would be to simply replicate the core framework login in the 389ds plugin or even *move* it there. But we want to keep the logic in the framework as it is more flexible and easier to work with and extend, so I proposed a 389ds plugin that just *asks* the framwrok for the data. This keeps the busienss loginc in the python framewrok, yet it allows an LDAP driver to add users properly in IPA just using LDAP calls. The problem is that the framework may do more than LDAP operations. It supports user-written plugins that can literally do anything. We'd need to limit what IPA plugins can do, or how they do it, otherwise it's impossible to just return the result. So I can see a DS plugin calling IPA, but IPA really has to write the results itself. I do not see this as a slippery slope, as it would be limited to user creation by definition. I'd be extremely surprised if all of these inflexible external HR systems happened to limit themselves to user creation. -- Petr³ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 09:57 AM, Simo Sorce wrote: On Wed, 2013-02-13 at 11:44 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 16:12 +0100, Petr Viktorin wrote: Our own post-callback assumes the user is already in LDAP, and who knows what user-supplied callbacks will do. Keep in mind IPA is plugable; at least for outside plugins' sake (if not our own sanity's) we should keep the number of code paths to a minimum. True which is why my proposal is to not use the standard user-add RPC call, but have a separate one. This separate call would only call the core business logic to create the user account add operation, but none of the external plumbing. Ideally we spit the framework flow like this: Normal user - Real user-add --- . . . . . . . . . --- LDAP add \ / -- common logic -- / \ 389ds plugin - Mock user-add -- . . . . . . . . . --- json reply custom plugins should be called in the custom logic an operate on the object before the ADD is attempted. If we do it this way then most of the code path will be in common which is what we want, and only the mechanical operation of adding the actual object to ldap will be different. Simo. There is one missing a few steps. A plugin execution looks like: Normal user - Real user-add --- pre-op call(s) --- execute (LPAP add record) --- post-op call(s) which may do additional add/modify It is the postop calls that would be the problem. They assume that the entry has already been written (so, for example, it has a valid UID/GID/ipaUniqueId, etc). Why are they done after the add ? It seem dangerous. What happens id the first ldap add succeed and the post op fails ? Are you talking about 389 plugins? If so, then in 1.3.0 and later, if you do all of the pre-op/post-op as betxn plugins, then they all take place in the same transaction, and they all succeed or all fail. We should exceute the ldap call after the post ops are perfomed imho. Simo. ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 05:57 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 11:44 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 16:12 +0100, Petr Viktorin wrote: Our own post-callback assumes the user is already in LDAP, and who knows what user-supplied callbacks will do. Keep in mind IPA is plugable; at least for outside plugins' sake (if not our own sanity's) we should keep the number of code paths to a minimum. True which is why my proposal is to not use the standard user-add RPC call, but have a separate one. This separate call would only call the core business logic to create the user account add operation, but none of the external plumbing. Ideally we spit the framework flow like this: Normal user - Real user-add --- . . . . . . . . . --- LDAP add \ / -- common logic -- / \ 389ds plugin - Mock user-add -- . . . . . . . . . --- json reply custom plugins should be called in the custom logic an operate on the object before the ADD is attempted. If we do it this way then most of the code path will be in common which is what we want, and only the mechanical operation of adding the actual object to ldap will be different. Simo. There is one missing a few steps. A plugin execution looks like: Normal user - Real user-add --- pre-op call(s) --- execute (LPAP add record) --- post-op call(s) which may do additional add/modify It is the postop calls that would be the problem. They assume that the entry has already been written (so, for example, it has a valid UID/GID/ipaUniqueId, etc). Why are they done after the add ? It seem dangerous. This comment from the user-add post-op plugin should explain: # Fetch the entry again to update memberof, mep data, etc updated # at the end of the transaction. Some processing happens in DS plugins, we can't get the data if the object isn't added yet. What happens id the first ldap add succeed and the post op fails ? We should exceute the ldap call after the post ops are perfomed imho. -- Petr³ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
Simo Sorce wrote: On Wed, 2013-02-13 at 16:12 +0100, Petr Viktorin wrote: Our own post-callback assumes the user is already in LDAP, and who knows what user-supplied callbacks will do. Keep in mind IPA is plugable; at least for outside plugins' sake (if not our own sanity's) we should keep the number of code paths to a minimum. True which is why my proposal is to not use the standard user-add RPC call, but have a separate one. This separate call would only call the core business logic to create the user account add operation, but none of the external plumbing. Ideally we spit the framework flow like this: Normal user - Real user-add --- . . . . . . . . . --- LDAP add \ / -- common logic -- / \ 389ds plugin - Mock user-add -- . . . . . . . . . --- json reply custom plugins should be called in the custom logic an operate on the object before the ADD is attempted. If we do it this way then most of the code path will be in common which is what we want, and only the mechanical operation of adding the actual object to ldap will be different. Simo. There is one missing a few steps. A plugin execution looks like: Normal user - Real user-add --- pre-op call(s) --- execute (LPAP add record) --- post-op call(s) which may do additional add/modify It is the postop calls that would be the problem. They assume that the entry has already been written (so, for example, it has a valid UID/GID/ipaUniqueId, etc). rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 18:16 +0100, Petr Viktorin wrote: On 02/13/2013 05:57 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 11:44 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 16:12 +0100, Petr Viktorin wrote: Our own post-callback assumes the user is already in LDAP, and who knows what user-supplied callbacks will do. Keep in mind IPA is plugable; at least for outside plugins' sake (if not our own sanity's) we should keep the number of code paths to a minimum. True which is why my proposal is to not use the standard user-add RPC call, but have a separate one. This separate call would only call the core business logic to create the user account add operation, but none of the external plumbing. Ideally we spit the framework flow like this: Normal user - Real user-add --- . . . . . . . . . --- LDAP add \ / -- common logic -- / \ 389ds plugin - Mock user-add -- . . . . . . . . . --- json reply custom plugins should be called in the custom logic an operate on the object before the ADD is attempted. If we do it this way then most of the code path will be in common which is what we want, and only the mechanical operation of adding the actual object to ldap will be different. Simo. There is one missing a few steps. A plugin execution looks like: Normal user - Real user-add --- pre-op call(s) --- execute (LPAP add record) --- post-op call(s) which may do additional add/modify It is the postop calls that would be the problem. They assume that the entry has already been written (so, for example, it has a valid UID/GID/ipaUniqueId, etc). Why are they done after the add ? It seem dangerous. This comment from the user-add post-op plugin should explain: # Fetch the entry again to update memberof, mep data, etc updated # at the end of the transaction. Some processing happens in DS plugins, we can't get the data if the object isn't added yet. Is this data actually needed for the post-op plugins ? Also what happens if a post-op plugin fails ? Please don't tell me the framework deletes the user object that would be really bad. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
I appreciate Simo's concern for authorization and audit in this process, we must solve that problem. If I understand the proposal correctly it's akin to recording a macro that can be replayed. The framework executes as normal but instead of issuing the LDAP modify commands we record them. Then after the entire command completes we send the recorded operations back to 389DS in some form Did I understand this correctly? If so I'm very much against the idea of sending JSON back to 389DS to execute the totality of the operation. Why? It either breaks or has the potential to break our entire processing model, pre and post operations, validity checks (e.g. querying the current state) user supplied plugins, etc. I could see this working in some limited cases which might give you the illusion it would work. But the only robust general solution I think we can sign up for supporting is to use the API commands we designed, period. Anything else just seems like a nightmare scenario of corner cases. Therefore I think the proposal of watching something (yet to be determined), calling our API commands, and then cleaning up the watched entity afterwards is the best approach. Figuring out how to authenticate/authorize/audit this is the primary challenge, a challenge far more manageable then trying to subvert the framework with every known and unknown risk that introduces. It's hard enough as it is assuring our documented API works correctly. Our API is the only thing I think we can realistically commit to supporting. -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 18:11 +0100, Petr Viktorin wrote: 1. create some new subtree, e.g. cn=useradd-playground,dc=example,dc=com This has more consequences than you may think. I do not like the separate field idea because you need to treat it in a special way. We would probably need special ACIs to anoty allow any other client to see this subtree otherwise they may see incomplete objects. Yet we need to allow some user to write to it. We need to decide case by case which plugins to enable, not all DS plugins can use filters or exclude subtrees so we may have issues with some plugins if we do not want them to operate on these special entries and so on. Is it possible to use read ACIs of the original tree? Not sure what's the question here. Care to elaborate ? The problem is that the framework may do more than LDAP operations. It supports user-written plugins that can literally do anything. We'd need to limit what IPA plugins can do, or how they do it, otherwise it's impossible to just return the result. Ok, then maybe we can have a 'core' common logic part that has some more limitations. So I can see a DS plugin calling IPA, but IPA really has to write the results itself. This is harder, but could be done too. The way it would work would be to make 389ds able to perform s4u2self for the case a user bound to LDAP with a simple bind, and then s4u2proxy with the evidence ticket to get HTTP/server ticket, then we can connect to the framework using the user's identity. In turn the framework will perform an additional s4u2proxy delegation to get a ticket back to ldap/server to perform the add operation. This means the flow would be: client --(LDAP)-- 389DS --(HTTP/json)-- framework --(LDAP)-- add The problem with this solution is that there is a risk of loops, so we need a mechanism that makes it impossible for the 389ds plugin to recurse into the framework. Also we need to make sure the add operation is not blocking add operations by keeping a transaction open causing a deadlock. And we cannot do that by immediately returning to the client because we need to wait for the framework reply to know if the operation was successful or not. I see some more low level risks doing it this way which is why I made my proposal to avoid this potential loop. Rich, is there potential from deadlocking here due to the new transaction stuff ? Or can we single out this plugin to run before *any* transaction is started ? I do not see this as a slippery slope, as it would be limited to user creation by definition. I'd be extremely surprised if all of these inflexible external HR systems happened to limit themselves to user creation. HR systems only deal with hiring and firing employees, they *may* deal with adding users to groups, but that doesn't require special operations for us, it is just a matter of adding a 'member' attribute to a group object. I do not think we need to offer anything else here. We already have proof this is sufficient, because we have experience with the AD winsync plugin which has a similar function, the only differnce is that the driver is an AD server instead of an HR system directly. And with that plugin we also only create users by default, and we haven't heard many requests to do anything more. The only request we really had was to make sure we could sync also posix attrs from AD, but that's it, still just basic user info. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 12:40 -0500, John Dennis wrote: I appreciate Simo's concern for authorization and audit in this process, we must solve that problem. If I understand the proposal correctly it's akin to recording a macro that can be replayed. The framework executes as normal but instead of issuing the LDAP modify commands we record them. Then after the entire command completes we send the recorded operations back to 389DS in some form Did I understand this correctly? If so I'm very much against the idea of sending JSON back to 389DS to execute the totality of the operation. Why? It either breaks or has the potential to break our entire processing model, pre and post operations, validity checks (e.g. querying the current state) user supplied plugins, etc. I could see this working in some limited cases which might give you the illusion it would work. But the only robust general solution I think we can sign up for supporting is to use the API commands we designed, period. Anything else just seems like a nightmare scenario of corner cases. Therefore I think the proposal of watching something (yet to be determined), calling our API commands, and then cleaning up the watched entity afterwards is the best approach. Figuring out how to authenticate/authorize/audit this is the primary challenge, a challenge far more manageable then trying to subvert the framework with every known and unknown risk that introduces. It's hard enough as it is assuring our documented API works correctly. Our API is the only thing I think we can realistically commit to supporting. See my reply to Petr, it can be done the way you ask (ie framework does the actual ldap add), the only concern I have is looping and deadlocks. If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 10:50 AM, Simo Sorce wrote: On Wed, 2013-02-13 at 18:11 +0100, Petr Viktorin wrote: 1. create some new subtree, e.g. cn=useradd-playground,dc=example,dc=com This has more consequences than you may think. I do not like the separate field idea because you need to treat it in a special way. We would probably need special ACIs to anoty allow any other client to see this subtree otherwise they may see incomplete objects. Yet we need to allow some user to write to it. We need to decide case by case which plugins to enable, not all DS plugins can use filters or exclude subtrees so we may have issues with some plugins if we do not want them to operate on these special entries and so on. Is it possible to use read ACIs of the original tree? Not sure what's the question here. Care to elaborate ? The problem is that the framework may do more than LDAP operations. It supports user-written plugins that can literally do anything. We'd need to limit what IPA plugins can do, or how they do it, otherwise it's impossible to just return the result. Ok, then maybe we can have a 'core' common logic part that has some more limitations. So I can see a DS plugin calling IPA, but IPA really has to write the results itself. This is harder, but could be done too. The way it would work would be to make 389ds able to perform s4u2self for the case a user bound to LDAP with a simple bind, and then s4u2proxy with the evidence ticket to get HTTP/server ticket, then we can connect to the framework using the user's identity. In turn the framework will perform an additional s4u2proxy delegation to get a ticket back to ldap/server to perform the add operation. This means the flow would be: client --(LDAP)-- 389DS --(HTTP/json)-- framework --(LDAP)-- add The problem with this solution is that there is a risk of loops, so we need a mechanism that makes it impossible for the 389ds plugin to recurse into the framework. Also we need to make sure the add operation is not blocking add operations by keeping a transaction open causing a deadlock. And we cannot do that by immediately returning to the client because we need to wait for the framework reply to know if the operation was successful or not. I see some more low level risks doing it this way which is why I made my proposal to avoid this potential loop. Rich, is there potential from deadlocking here due to the new transaction stuff ? Or can we single out this plugin to run before *any* transaction is started ? If you do this in a regular pre-op, not a betxn pre-op, then it should be fine. I do not see this as a slippery slope, as it would be limited to user creation by definition. I'd be extremely surprised if all of these inflexible external HR systems happened to limit themselves to user creation. HR systems only deal with hiring and firing employees, they *may* deal with adding users to groups, but that doesn't require special operations for us, it is just a matter of adding a 'member' attribute to a group object. I do not think we need to offer anything else here. We already have proof this is sufficient, because we have experience with the AD winsync plugin which has a similar function, the only differnce is that the driver is an AD server instead of an HR system directly. And with that plugin we also only create users by default, and we haven't heard many requests to do anything more. The only request we really had was to make sure we could sync also posix attrs from AD, but that's it, still just basic user info. Simo. ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 10:57 -0700, Rich Megginson wrote: Rich, is there potential from deadlocking here due to the new transaction stuff ? Or can we single out this plugin to run before *any* transaction is started ? If you do this in a regular pre-op, not a betxn pre-op, then it should be fine. Ok in this case we should be able to create a regular pre-op plugin to intercept the ldap add call and then use the following flow: client --(LDAP)-- 389DS --(HTTP/json)-- framework --(LDAP)-- add So no deadlocks will happen, the remaining issue is how to make sure we do not loop by mistake in the second add. One way could be to have loop detection so that if more then two (1. original, 2. framework) adds for the same DN come in we just return errors. Another way is to use a special objectclass as I proposed in the thread and make sure the framework explictly blacklists it so that it can never try to send an add with the special oc even by mistake or user misconfiguration. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
Simo Sorce wrote: On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. I think he means have a persistent search to watch for new entries and then act upon them. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 01:30 PM, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. I think he means have a persistent search to watch for new entries and then act upon them. Yes, it could either be a persistent search or an external (cron) process that periodically polls LDAP. In each case there is an LDAP tree used as a staging area. The staging area is completely independent of IPA's LDAP area. If something shows up in the staging area the contents of the staging area are used to drive our IPA commands. The staging area is then cleaned of all the entries which succeeded. Under this scenario what are the looping and deadlock concerns? -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
John Dennis wrote: On 02/13/2013 01:30 PM, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. I think he means have a persistent search to watch for new entries and then act upon them. Yes, it could either be a persistent search or an external (cron) process that periodically polls LDAP. In each case there is an LDAP tree used as a staging area. The staging area is completely independent of IPA's LDAP area. If something shows up in the staging area the contents of the staging area are used to drive our IPA commands. The staging area is then cleaned of all the entries which succeeded. Under this scenario what are the looping and deadlock concerns? I don't think any. Simo's concern was doing an ADD within an ADD via his proposed 389-ds method. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On Wed, 2013-02-13 at 13:30 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. I think he means have a persistent search to watch for new entries and then act upon them. I think a 2 step approach is misguided, so I've written it off from the start. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 02:08 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 13:30 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. I think he means have a persistent search to watch for new entries and then act upon them. I think a 2 step approach is misguided, so I've written it off from the start. Simo. This all thread smells like rewriting winsync by using internal plugin and IPA framework. Is there any chance we can use existing winsync solution to do what we need but sync from staging instance rather than AD? Then the flow will be: HR system - staging DS instance - dirsync - IPA Couple assumptions: a) We are satisfied with the security of the existing winsync solution and authentication used there. I do not see why it should be different here as it is a very similar use case. b) I expect we can sync from 389 to 389 with minor config changes and effectively no code changes c) The users created via winsync now are created in a proper way so IPA accepts then as IPA users This solves the problem of add and delete of the users. I know that winsync already supports group add/delete but we never qualified it in IPA. So may be it is time, at least for this use case. I would really think that this might be a lower cost solution than writing yet another custom plugin. Let us try to reuse what we already have. -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 06:18 PM, Dmitri Pal wrote: On 02/13/2013 02:08 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 13:30 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. I think he means have a persistent search to watch for new entries and then act upon them. I think a 2 step approach is misguided, so I've written it off from the start. Simo. This all thread smells like rewriting winsync by using internal plugin and IPA framework. Is there any chance we can use existing winsync solution to do what we need but sync from staging instance rather than AD? Then the flow will be: HR system - staging DS instance - dirsync - IPA Couple assumptions: a) We are satisfied with the security of the existing winsync solution and authentication used there. I do not see why it should be different here as it is a very similar use case. b) I expect we can sync from 389 to 389 with minor config changes and effectively no code changes This is not the case. Winsync uses the AD Dirsync control, which 389 does not support. The Winsync code also expects the AD schema on the entries coming in from the Dirsync response. Even if 389 supported the Dirsync control, there would be a fair amount of code changes to deal with whatever schema we would need to expect. c) The users created via winsync now are created in a proper way so IPA accepts then as IPA users This solves the problem of add and delete of the users. I know that winsync already supports group add/delete but we never qualified it in IPA. So may be it is time, at least for this use case. I would really think that this might be a lower cost solution than writing yet another custom plugin. Let us try to reuse what we already have. I think a custom plug-in would be easier (though I do have concerns about the performance impact of the JSON approach). -NGK ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 09:48 PM, Nathan Kinder wrote: On 02/13/2013 06:18 PM, Dmitri Pal wrote: On 02/13/2013 02:08 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 13:30 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 12:59 -0500, John Dennis wrote: On 02/13/2013 12:53 PM, Simo Sorce wrote: If we can solve the looping and potential deadlocking concerns I think we can avoid the json reply and let the framework do the actual final ldap add. Could you elaborate on your looping and deadlock concerns? I don't see where they would arise if what we're watching is entirely independent of our LDAP tree. I do not understand what you are 'watching' ? Simo. I think he means have a persistent search to watch for new entries and then act upon them. I think a 2 step approach is misguided, so I've written it off from the start. Simo. This all thread smells like rewriting winsync by using internal plugin and IPA framework. Is there any chance we can use existing winsync solution to do what we need but sync from staging instance rather than AD? Then the flow will be: HR system - staging DS instance - dirsync - IPA Couple assumptions: a) We are satisfied with the security of the existing winsync solution and authentication used there. I do not see why it should be different here as it is a very similar use case. b) I expect we can sync from 389 to 389 with minor config changes and effectively no code changes This is not the case. Winsync uses the AD Dirsync control, which 389 does not support. The Winsync code also expects the AD schema on the entries coming in from the Dirsync response. Even if 389 supported the Dirsync control, there would be a fair amount of code changes to deal with whatever schema we would need to expect. c) The users created via winsync now are created in a proper way so IPA accepts then as IPA users This solves the problem of add and delete of the users. I know that winsync already supports group add/delete but we never qualified it in IPA. So may be it is time, at least for this use case. I would really think that this might be a lower cost solution than writing yet another custom plugin. Let us try to reuse what we already have. I think a custom plug-in would be easier (though I do have concerns about the performance impact of the JSON approach). I really think that we rather have an external solution and not the one hot wired into the IPA internals via the proposed plugin. Something like an external proxy or gateway rather than the approach Simo proposed even if it would be more work. This work would be reusable. We already have several use cases for the LDAP proxy. This can be one more. A bit more and there will be enough need to build one anyways. The main value is that it is optional and external and acts as an ecosystem level solution that can be developed and tested on a separate schedule rather than being internal to IPA. -NGK ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 09:16 AM, Petr Viktorin wrote: On 02/13/2013 05:57 PM, Simo Sorce wrote: On Wed, 2013-02-13 at 11:44 -0500, Rob Crittenden wrote: Simo Sorce wrote: On Wed, 2013-02-13 at 16:12 +0100, Petr Viktorin wrote: Our own post-callback assumes the user is already in LDAP, and who knows what user-supplied callbacks will do. Keep in mind IPA is plugable; at least for outside plugins' sake (if not our own sanity's) we should keep the number of code paths to a minimum. True which is why my proposal is to not use the standard user-add RPC call, but have a separate one. This separate call would only call the core business logic to create the user account add operation, but none of the external plumbing. Ideally we spit the framework flow like this: Normal user - Real user-add --- . . . . . . . . . --- LDAP add \ / -- common logic -- / \ 389ds plugin - Mock user-add -- . . . . . . . . . --- json reply custom plugins should be called in the custom logic an operate on the object before the ADD is attempted. If we do it this way then most of the code path will be in common which is what we want, and only the mechanical operation of adding the actual object to ldap will be different. Simo. There is one missing a few steps. A plugin execution looks like: Normal user - Real user-add --- pre-op call(s) --- execute (LPAP add record) --- post-op call(s) which may do additional add/modify It is the postop calls that would be the problem. They assume that the entry has already been written (so, for example, it has a valid UID/GID/ipaUniqueId, etc). Why are they done after the add ? It seem dangerous. This comment from the user-add post-op plugin should explain: # Fetch the entry again to update memberof, mep data, etc updated # at the end of the transaction. Some processing happens in DS plugins, we can't get the data if the object isn't added yet. You are referring to the fact that an 'ipa user-add' is actually multiple LDAP operations performed by the framework in IPA, right? I know at a minimum, the IPA framework performs 3 operations against DS when adding a user: - ADD operation to create the user entry (triggers a bunch of stuff in DS plug-ins) - SRCH operation to read the user entry with all of the dynamic stuff that was performed by the DS plug-ins - MOD operation to 'ipausers' group to add the new user as a group member With 389-ds-base 1.3.0, the pre/post-op DS plug-ins that IPA is concerned with are now betxn plug-ins, so that at least guarantees that the search operation will find everything it needs. The search could be eliminated if we add support for the 'Post-Read' control in DS as described in RFC 4527, which might be a good idea in the future (though I don't think it would really help us here). That still leaves the problem of an 'ipa user-add' being composed of multiple write operations against LDAP. With the DS plug-in approach that calls into the IPA framework with a 'mock add' to form the resulting entry at the pre-op stage, we could take care of the initial ADD operation of the user entry. We would still need to have a way to trigger the additional write operations that the IPA framework performs. The proposed DS plug-in could do an internal search operation after the add, then it could perform the additional write operations that are needed. This logic would need to be in the DS plug-in, or we would need another way to get the details from the IPA framework via JSON. The approach begins to get a bit messy. What happens id the first ldap add succeed and the post op fails ? We should exceute the ldap call after the post ops are perfomed imho. ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 10:40 PM, Nathan Kinder wrote: With the DS plug-in approach that calls into the IPA framework with a 'mock add' to form the resulting entry at the pre-op stage, we could take care of the initial ADD operation of the user entry. We would still need to have a way to trigger the additional write operations that the IPA framework performs. The proposed DS plug-in could do an internal search operation after the add, then it could perform the additional write operations that are needed. This logic would need to be in the DS plug-in, or we would need another way to get the details from the IPA framework via JSON. The approach begins to get a bit messy. Messy is one word for it :-) Herein lies the problem. user-add is actually a simple case where in this particular version of our code we know the LDAP operations that occur and in what order. But there is nothing in our coding guidelines that guarantees this, it just happens to be true today. Pre and post callbacks are free to do as they please. The logic in our ldap module can change (in fact it's undergoing a rewrite as we speak). We've committed to supporting an extension mechanism (plugins) that ties into the framework operations and who knows what might occur in that code. At the moment it's not used, but hopefully it will be. Then comes the question, where do we draw the line? Do we say only user-add with no additional options is supported because we know what occurs during it's processing and hence we feel safe emulating user-add outside the framework? We all know this is a slippery slope, as soon as you support user-add this way we'll be getting requests to support other commands or other options. And after we make framework changes (which we do frequently) are we going to test these out-of-band operations continue to emulate the ever changing framework? It's a slippery slope that can expose us to problems we don't need. Jenny, do you want to test an entirely different mechanism or do you want to limit it to our defined API? We have a defined API, I really think that's the only thing we should support. Backdoor shortcuts that sidestep our framework should be off the table IMHO, it's just inviting too many problems. -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/13/2013 08:30 PM, John Dennis wrote: On 02/13/2013 10:40 PM, Nathan Kinder wrote: With the DS plug-in approach that calls into the IPA framework with a 'mock add' to form the resulting entry at the pre-op stage, we could take care of the initial ADD operation of the user entry. We would still need to have a way to trigger the additional write operations that the IPA framework performs. The proposed DS plug-in could do an internal search operation after the add, then it could perform the additional write operations that are needed. This logic would need to be in the DS plug-in, or we would need another way to get the details from the IPA framework via JSON. The approach begins to get a bit messy. Messy is one word for it :-) Herein lies the problem. user-add is actually a simple case where in this particular version of our code we know the LDAP operations that occur and in what order. But there is nothing in our coding guidelines that guarantees this, it just happens to be true today. Pre and post callbacks are free to do as they please. The logic in our ldap module can change (in fact it's undergoing a rewrite as we speak). We've committed to supporting an extension mechanism (plugins) that ties into the framework operations and who knows what might occur in that code. At the moment it's not used, but hopefully it will be. Then comes the question, where do we draw the line? Do we say only user-add with no additional options is supported because we know what occurs during it's processing and hence we feel safe emulating user-add outside the framework? We all know this is a slippery slope, as soon as you support user-add this way we'll be getting requests to support other commands or other options. And after we make framework changes (which we do frequently) are we going to test these out-of-band operations continue to emulate the ever changing framework? It's a slippery slope that can expose us to problems we don't need. Jenny, do you want to test an entirely different mechanism or do you want to limit it to our defined API? We have a defined API, I really think that's the only thing we should support. Backdoor shortcuts that sidestep our framework should be off the table IMHO, it's just inviting too many problems. This is why I think that the most sane approach is to put something in front of 'ipa user-add'. This is not without it's own set of difficulties, but if we need a basic LDAP interface that can be used to create IPA users, it should end up calling into the same code that 'ipa user-add' uses. This would ensure that any extensions (plugins) for the framework are called. I actually see this as very similar to the LDAP migration case. For migration, we basically take entries in LDIF format, extract the relevant data, then feed it into 'ipa user-add', right? That's more or less what we need to do here for HR systems that provision users, but the user addition needs to be pushed to IPA via an LDAP client instead of IPA pulling from an LDAP server. ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFC] Creating a new plugin to make it simpler to add users via LDAP
On 02/14/2013 12:16 AM, Nathan Kinder wrote: On 02/13/2013 08:30 PM, John Dennis wrote: On 02/13/2013 10:40 PM, Nathan Kinder wrote: With the DS plug-in approach that calls into the IPA framework with a 'mock add' to form the resulting entry at the pre-op stage, we could take care of the initial ADD operation of the user entry. We would still need to have a way to trigger the additional write operations that the IPA framework performs. The proposed DS plug-in could do an internal search operation after the add, then it could perform the additional write operations that are needed. This logic would need to be in the DS plug-in, or we would need another way to get the details from the IPA framework via JSON. The approach begins to get a bit messy. Messy is one word for it :-) Herein lies the problem. user-add is actually a simple case where in this particular version of our code we know the LDAP operations that occur and in what order. But there is nothing in our coding guidelines that guarantees this, it just happens to be true today. Pre and post callbacks are free to do as they please. The logic in our ldap module can change (in fact it's undergoing a rewrite as we speak). We've committed to supporting an extension mechanism (plugins) that ties into the framework operations and who knows what might occur in that code. At the moment it's not used, but hopefully it will be. Then comes the question, where do we draw the line? Do we say only user-add with no additional options is supported because we know what occurs during it's processing and hence we feel safe emulating user-add outside the framework? We all know this is a slippery slope, as soon as you support user-add this way we'll be getting requests to support other commands or other options. And after we make framework changes (which we do frequently) are we going to test these out-of-band operations continue to emulate the ever changing framework? It's a slippery slope that can expose us to problems we don't need. Jenny, do you want to test an entirely different mechanism or do you want to limit it to our defined API? We have a defined API, I really think that's the only thing we should support. Backdoor shortcuts that sidestep our framework should be off the table IMHO, it's just inviting too many problems. This is why I think that the most sane approach is to put something in front of 'ipa user-add'. This is not without it's own set of difficulties, but if we need a basic LDAP interface that can be used to create IPA users, it should end up calling into the same code that 'ipa user-add' uses. This would ensure that any extensions (plugins) for the framework are called. I actually see this as very similar to the LDAP migration case. For migration, we basically take entries in LDIF format, extract the relevant data, then feed it into 'ipa user-add', right? That's more or less what we need to do here for HR systems that provision users, but the user addition needs to be pushed to IPA via an LDAP client instead of IPA pulling from an LDAP server. Yes, this is exactly what I'm saying and we concur. -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
