Re: [Freeipa-devel] ds-migrate feature enhancements
On 12/05/2014 12:32 AM, Alan Evans wrote:
Hello Martin,
I have spent some time looking into how to address
https://fedorahosted.org/freeipa/ticket/3709. I thought I would look at
--setattr --addattr from user-add and others to see if I could reuse code
for --user-attr-default. At least for parsing the commandline arguments
anyway. It does not look like it would be easy to re-purpose the code in
_convert_2_dict() from ipalib/plugins/baseldap.py (without blatantly
copying it). What are your thoughts about creating a Dict paramater type
in ipalib/parameters.py to to user for --setaddr --addattr --delattr and to
use for --user-attr-default?
Ah, I did not expect any integration with the standard baseldap.py --*attr
functions at all. Though reusing _convert_2_dict sounds as a good idea!
Also I was looking at string.Template() vs string.Formatter(). I think
string.Template is probably sufficient. But I don't follow PEPs so I don't
know if that's Python3 future proof or not. Thoughts?
You do not need to worry with Python3 at this moment too much, string.Template
is already used by FreeIPA anyway.
# Pseudocode
(attr, default) = args['userdefaultattr'][0].split('=',2)
entry_attrs[attr] = default.safe_substitute(entry_attrs.toDict())
In looking at ipalib/plugins/baseldap.py, I was wondering if It might make
sense to make migrate more in line with the --*attr paramaters.
Consider:
ipa migrate-ds --user-delattr ou
Instead of:
ipa-migrate-ds --user-ignore-attribute ou
Hm, I would be afraid that you would get into a too much complexity if you try
to integrate the new functionality which was supposed to be rather simple with
*attr parameters processed process_attr_options that even do LDAP search do
process the merge the *attr options with existing LDAP entry. You do not need
to worry about that, you always work only on top of the migrated user entry
that you have available. That simplifies the processing a lot.
Then we could also add user-{setattr,addattr,delattr,attrdefault} like:
ipa migrate-ds --user-attrdefault mail=$u...@example.com --user-setattr
cn=$givenname $sn --user-addattr description=Migrated $(date)
--user-delattr ou
Which would add a mail address if one wasn't present, replace the CN with
the first and last name, add a description attribute and get rid of the ou
attribute.
This functionality may make sense as an extension, eventually. Currently, we
have nothing so even having
--user-attr-default
and
--user-attr-set
would go a long way.
Another thought I had was about the behaivor of the search on the source
directory. Currently one provides --{user,group}-continer and
--{user,group}-objectclass and the scope is statically defined in the
script.
--{user,group}-filter
--
What about adding --{user,group}-scope ONE|SUB?
This is actually a very good idea and a valid option to add! We even have a
ticket in the FreeIPA Trac that no one brave enough took so far.
https://fedorahosted.org/freeipa/ticket/2547
If you decide to contribute this one, I would recommend doing in a separate
patch from your template work as it is parallel effort and would make the
review easier.
My directory for example is like:
dn: uid=me,ou=ops,ou=people,dc=example,dc=com
dn: uid=joe,ou=eng,ou=people,dc=example,dc=com
So currently I have to use the --continue switch and change my
--user-container to get all of my users. If the scope weren't limited to
ONELEVEL then I could get everyone in one pass.
ipa migrate-ds --user-container ou=People,dc=example,dc=com --user-scope SUB
This makes a lot of sense, +1 for adding the switches (should be very easy to
implement)
Lastly I was thinking that --user-objectclass and --exclude-users could be
made more powerful with --user-filter
ipa migrate-ds --user-container ou=People,dc=example,dc=com
--user-objectclass posixAccount --user-filter '(!(loginShell=/bin/false))'
or we could fire all of sales --user-filter '(!(ou=Sales))'
Which gets concatenated internally in the search to
((objectClass=posixAccount)(!(loginShell=/bin/false)))
Another great idea. Currently we only create automated filter in
def construct_filter(template, oc_list):
oc_subfilter = ''.join([ '(objectclass=%s)' % oc for oc in oc_list])
return template % oc_subfilter
based on objectclass. Your proposal would make it much more flexible. This one
is also completely parallel so it should be in a separate patch compared to the
template work.
I am really looking forward to your patches! :-)
On Thu, Dec 4, 2014 at 4:59 AM, Martin Kosek mko...@redhat.com wrote:
On 11/26/2014 11:52 AM, Martin Kosek wrote:
On 11/24/2014 11:24 PM, Alan Evans wrote:
I am in the midst of preparing for a migration from OpenLDAP to FreeIPA.
ds-migrate wasn't going to fill all of my needs so I thought I would
use it
for most and then make up some LDIF's and massage them to do the last
bit
of migration.
I have instead
Re: [Freeipa-devel] ds-migrate feature enhancements
On 11/26/2014 11:52 AM, Martin Kosek wrote:
On 11/24/2014 11:24 PM, Alan Evans wrote:
I am in the midst of preparing for a migration from OpenLDAP to FreeIPA.
ds-migrate wasn't going to fill all of my needs so I thought I would use it
for most and then make up some LDIF's and massage them to do the last bit
of migration.
I have instead decided to extend ds-migrate and I think that my features
might be of use to others so I would like to contribute them.
Great idea! :-) IMO, enhancing FreeIPA migration capability and making it more
even more pleasant experience for user users is a good goal.
Before I get
too for I wanted to get some input from the community.
Here are MY original goals:
* Migrate ssh public keys
The openssh-lpk schema is used in my tree so objectClass: ldapPublicKey
attribute: sshPublicKey
* Migrate disabled accounts as disabled
We 'disable' usere by setting their shadowExpire to a date in the past
and setting their shell to /bin/false
I realized that the ssh-public key problem is more generally an attribute
mapping problem and dealing with disabled users could be more generalized
too.
Here are instead the new features I would provide.
* Attribute mapping
Feature should check the new syntax exists and is the same as the old
syntax (perhaps further check for compatible syntax)
--user-attribute-map=oldAttribute=newAttribute
--group-attribute-map=foo=bar
Should I drop user/group and just make it --attribute-map and apply it to
both?
Should certain attributes be mapped by default, i.e.
sshPublicKey=ipaSSHPubKey (this means we also need to ignore the
objectClass ldapPublicKey by default) Maybe make a separate switch
--with-ssh-keys that automatically adds a map and an ignore?
If the mapping sshPublicKey - ipaSSHPubKey is clear, I would do it by default
and maybe add a switch to skip these advanced migrations. Just make sure the
expected format matches (ipa requires all 3 pieces of the public key, not just
the blob)
I think it would be very useful to combine user attribute mapping with the
ideas from
https://fedorahosted.org/freeipa/ticket/3709
i.e. filling attribute with values from original entry or a default. This may
lead to following syntax:
--user-attribute-default sn=notdefined --user-attribute-default
description=User with cn $cn
which would only use the pattern if attribute is not defined in original entry
and
--user-attribute-map sn=$cn
which would fill overwrite the attribute even if it was defined in the
original
entry.
* Handling disabled users
1. How to identify disabled users?
a. shadowExpire now()
--use-disable-shadow-expire
How would you like to disable users then? With nsAccountLock attribute?
Wouldn't it be better to instead map shadowExpire to krbPrincipalExpiration
attribute which should have sematically the same meaning?
b. loginShell is one of configurable shells
--use-disable-login-shell
--disabled-shell=/bin/false --disabled-shell=/sbin/nologin (these
Is this really a general mechanism? I think Kerberos principal expiration is
the way to go:
# ipa user-mod fbar --principal-expiration 2014010100Z
# ssh fbar@`hostname`
f...@vm-067.idm.lab.bos.redhat.com's password:
Permission denied, please try again.
# tail /var/log/krb5kdc.log
...
Nov 26 04:56:51 vm-067.idm.lab.bos.redhat.com krb5kdc[19615](info): AS_REQ (6
etypes {18 17 16 23 25 26}) 10.16.78.67: CLIENT EXPIRED:
f...@idm.lab.bos.redhat.com for
krbtgt/idm.lab.bos.redhat@idm.lab.bos.redhat.com, Client's entry in
database has expired
It is respected by Kerberos authentication and SSSD logins, at minimum.
two would be the defaults)
c. nsAccountLocked (though that would be straight copied by the
migrator anyway
+1 for copying.
d. From Open DJ the attribute ds-pwp-account-disabled can be used to
identify disabled users
(are there others?)
2. What do do with disabled users (in my case migrate and disable)
a. Migrate them and don't touch nsAccountLocked
b. Migrate them and set nsAccountLocked = true
--disable-users
c. Do not migrate them
--skip-disabled-users
d. Which is the default? Migrate and disable? If so which are the
default methods for identifying them? All methods?
I may be missing something, but to me following would make sense:
- properly migrate and fill krbPrincipalExpiration and nsACcountLock
attributes
- optionally implement --skip-disabled-users. Though it may be challenging to
implement the is_user_disabled function right so that it works on all sort
of
DS schemes.
So is there anything I'm missing? Any suggestions on the switches? I'm not
entirely sure I like them the way they are.
I have code to cover about 60% of the above already. The user-attr-map
feature is working and the --disabled-users and disabled-shells options are
working.
Regards,
Re: [Freeipa-devel] ds-migrate feature enhancements
Hello Martin,
I have spent some time looking into how to address
https://fedorahosted.org/freeipa/ticket/3709. I thought I would look at
--setattr --addattr from user-add and others to see if I could reuse code
for --user-attr-default. At least for parsing the commandline arguments
anyway. It does not look like it would be easy to re-purpose the code in
_convert_2_dict() from ipalib/plugins/baseldap.py (without blatantly
copying it). What are your thoughts about creating a Dict paramater type
in ipalib/parameters.py to to user for --setaddr --addattr --delattr and to
use for --user-attr-default?
Also I was looking at string.Template() vs string.Formatter(). I think
string.Template is probably sufficient. But I don't follow PEPs so I don't
know if that's Python3 future proof or not. Thoughts?
# Pseudocode
(attr, default) = args['userdefaultattr'][0].split('=',2)
entry_attrs[attr] = default.safe_substitute(entry_attrs.toDict())
In looking at ipalib/plugins/baseldap.py, I was wondering if It might make
sense to make migrate more in line with the --*attr paramaters.
Consider:
ipa migrate-ds --user-delattr ou
Instead of:
ipa-migrate-ds --user-ignore-attribute ou
Then we could also add user-{setattr,addattr,delattr,attrdefault} like:
ipa migrate-ds --user-attrdefault mail=$u...@example.com --user-setattr
cn=$givenname $sn --user-addattr description=Migrated $(date)
--user-delattr ou
Which would add a mail address if one wasn't present, replace the CN with
the first and last name, add a description attribute and get rid of the ou
attribute.
Another thought I had was about the behaivor of the search on the source
directory. Currently one provides --{user,group}-continer and
--{user,group}-objectclass and the scope is statically defined in the
script.
--{user,group}-filter
--
What about adding --{user,group}-scope ONE|SUB?
My directory for example is like:
dn: uid=me,ou=ops,ou=people,dc=example,dc=com
dn: uid=joe,ou=eng,ou=people,dc=example,dc=com
So currently I have to use the --continue switch and change my
--user-container to get all of my users. If the scope weren't limited to
ONELEVEL then I could get everyone in one pass.
ipa migrate-ds --user-container ou=People,dc=example,dc=com --user-scope SUB
Lastly I was thinking that --user-objectclass and --exclude-users could be
made more powerful with --user-filter
ipa migrate-ds --user-container ou=People,dc=example,dc=com
--user-objectclass posixAccount --user-filter '(!(loginShell=/bin/false))'
or we could fire all of sales --user-filter '(!(ou=Sales))'
Which gets concatenated internally in the search to
((objectClass=posixAccount)(!(loginShell=/bin/false)))
On Thu, Dec 4, 2014 at 4:59 AM, Martin Kosek mko...@redhat.com wrote:
On 11/26/2014 11:52 AM, Martin Kosek wrote:
On 11/24/2014 11:24 PM, Alan Evans wrote:
I am in the midst of preparing for a migration from OpenLDAP to FreeIPA.
ds-migrate wasn't going to fill all of my needs so I thought I would
use it
for most and then make up some LDIF's and massage them to do the last
bit
of migration.
I have instead decided to extend ds-migrate and I think that my features
might be of use to others so I would like to contribute them.
Great idea! :-) IMO, enhancing FreeIPA migration capability and making
it more
even more pleasant experience for user users is a good goal.
Before I get
too for I wanted to get some input from the community.
Here are MY original goals:
* Migrate ssh public keys
The openssh-lpk schema is used in my tree so objectClass:
ldapPublicKey
attribute: sshPublicKey
* Migrate disabled accounts as disabled
We 'disable' usere by setting their shadowExpire to a date in the past
and setting their shell to /bin/false
I realized that the ssh-public key problem is more generally an
attribute
mapping problem and dealing with disabled users could be more
generalized
too.
Here are instead the new features I would provide.
* Attribute mapping
Feature should check the new syntax exists and is the same as the old
syntax (perhaps further check for compatible syntax)
--user-attribute-map=oldAttribute=newAttribute
--group-attribute-map=foo=bar
Should I drop user/group and just make it --attribute-map and apply
it to
both?
Should certain attributes be mapped by default, i.e.
sshPublicKey=ipaSSHPubKey (this means we also need to ignore the
objectClass ldapPublicKey by default) Maybe make a separate switch
--with-ssh-keys that automatically adds a map and an ignore?
If the mapping sshPublicKey - ipaSSHPubKey is clear, I would do it by
default
and maybe add a switch to skip these advanced migrations. Just make sure
the
expected format matches (ipa requires all 3 pieces of the public key,
not just
the blob)
I think it would be very useful to combine user attribute mapping with
the
ideas from
https://fedorahosted.org/freeipa/ticket/3709
i.e.
Re: [Freeipa-devel] ds-migrate feature enhancements
On 11/24/2014 11:24 PM, Alan Evans wrote:
I am in the midst of preparing for a migration from OpenLDAP to FreeIPA.
ds-migrate wasn't going to fill all of my needs so I thought I would use it
for most and then make up some LDIF's and massage them to do the last bit
of migration.
I have instead decided to extend ds-migrate and I think that my features
might be of use to others so I would like to contribute them.
Great idea! :-) IMO, enhancing FreeIPA migration capability and making it more
even more pleasant experience for user users is a good goal.
Before I get
too for I wanted to get some input from the community.
Here are MY original goals:
* Migrate ssh public keys
The openssh-lpk schema is used in my tree so objectClass: ldapPublicKey
attribute: sshPublicKey
* Migrate disabled accounts as disabled
We 'disable' usere by setting their shadowExpire to a date in the past
and setting their shell to /bin/false
I realized that the ssh-public key problem is more generally an attribute
mapping problem and dealing with disabled users could be more generalized
too.
Here are instead the new features I would provide.
* Attribute mapping
Feature should check the new syntax exists and is the same as the old
syntax (perhaps further check for compatible syntax)
--user-attribute-map=oldAttribute=newAttribute
--group-attribute-map=foo=bar
Should I drop user/group and just make it --attribute-map and apply it to
both?
Should certain attributes be mapped by default, i.e.
sshPublicKey=ipaSSHPubKey (this means we also need to ignore the
objectClass ldapPublicKey by default) Maybe make a separate switch
--with-ssh-keys that automatically adds a map and an ignore?
If the mapping sshPublicKey - ipaSSHPubKey is clear, I would do it by default
and maybe add a switch to skip these advanced migrations. Just make sure the
expected format matches (ipa requires all 3 pieces of the public key, not just
the blob)
I think it would be very useful to combine user attribute mapping with the
ideas from
https://fedorahosted.org/freeipa/ticket/3709
i.e. filling attribute with values from original entry or a default. This may
lead to following syntax:
--user-attribute-default sn=notdefined --user-attribute-default
description=User with cn $cn
which would only use the pattern if attribute is not defined in original entry
and
--user-attribute-map sn=$cn
which would fill overwrite the attribute even if it was defined in the original
entry.
* Handling disabled users
1. How to identify disabled users?
a. shadowExpire now()
--use-disable-shadow-expire
How would you like to disable users then? With nsAccountLock attribute?
Wouldn't it be better to instead map shadowExpire to krbPrincipalExpiration
attribute which should have sematically the same meaning?
b. loginShell is one of configurable shells
--use-disable-login-shell
--disabled-shell=/bin/false --disabled-shell=/sbin/nologin (these
Is this really a general mechanism? I think Kerberos principal expiration is
the way to go:
# ipa user-mod fbar --principal-expiration 2014010100Z
# ssh fbar@`hostname`
f...@vm-067.idm.lab.bos.redhat.com's password:
Permission denied, please try again.
# tail /var/log/krb5kdc.log
...
Nov 26 04:56:51 vm-067.idm.lab.bos.redhat.com krb5kdc[19615](info): AS_REQ (6
etypes {18 17 16 23 25 26}) 10.16.78.67: CLIENT EXPIRED:
f...@idm.lab.bos.redhat.com for
krbtgt/idm.lab.bos.redhat@idm.lab.bos.redhat.com, Client's entry in
database has expired
It is respected by Kerberos authentication and SSSD logins, at minimum.
two would be the defaults)
c. nsAccountLocked (though that would be straight copied by the
migrator anyway
+1 for copying.
d. From Open DJ the attribute ds-pwp-account-disabled can be used to
identify disabled users
(are there others?)
2. What do do with disabled users (in my case migrate and disable)
a. Migrate them and don't touch nsAccountLocked
b. Migrate them and set nsAccountLocked = true
--disable-users
c. Do not migrate them
--skip-disabled-users
d. Which is the default? Migrate and disable? If so which are the
default methods for identifying them? All methods?
I may be missing something, but to me following would make sense:
- properly migrate and fill krbPrincipalExpiration and nsACcountLock attributes
- optionally implement --skip-disabled-users. Though it may be challenging to
implement the is_user_disabled function right so that it works on all sort of
DS schemes.
So is there anything I'm missing? Any suggestions on the switches? I'm not
entirely sure I like them the way they are.
I have code to cover about 60% of the above already. The user-attr-map
feature is working and the --disabled-users and disabled-shells options are
working.
Regards,
-Alan
HTH,
Martin
___
Freeipa-devel mailing list
[Freeipa-devel] ds-migrate feature enhancements
I am in the midst of preparing for a migration from OpenLDAP to FreeIPA. ds-migrate wasn't going to fill all of my needs so I thought I would use it for most and then make up some LDIF's and massage them to do the last bit of migration. I have instead decided to extend ds-migrate and I think that my features might be of use to others so I would like to contribute them. Before I get too for I wanted to get some input from the community. Here are MY original goals: * Migrate ssh public keys The openssh-lpk schema is used in my tree so objectClass: ldapPublicKey attribute: sshPublicKey * Migrate disabled accounts as disabled We 'disable' usere by setting their shadowExpire to a date in the past and setting their shell to /bin/false I realized that the ssh-public key problem is more generally an attribute mapping problem and dealing with disabled users could be more generalized too. Here are instead the new features I would provide. * Attribute mapping Feature should check the new syntax exists and is the same as the old syntax (perhaps further check for compatible syntax) --user-attribute-map=oldAttribute=newAttribute --group-attribute-map=foo=bar Should I drop user/group and just make it --attribute-map and apply it to both? Should certain attributes be mapped by default, i.e. sshPublicKey=ipaSSHPubKey (this means we also need to ignore the objectClass ldapPublicKey by default) Maybe make a separate switch --with-ssh-keys that automatically adds a map and an ignore? * Handling disabled users 1. How to identify disabled users? a. shadowExpire now() --use-disable-shadow-expire b. loginShell is one of configurable shells --use-disable-login-shell --disabled-shell=/bin/false --disabled-shell=/sbin/nologin (these two would be the defaults) c. nsAccountLocked (though that would be straight copied by the migrator anyway d. From Open DJ the attribute ds-pwp-account-disabled can be used to identify disabled users (are there others?) 2. What do do with disabled users (in my case migrate and disable) a. Migrate them and don't touch nsAccountLocked b. Migrate them and set nsAccountLocked = true --disable-users c. Do not migrate them --skip-disabled-users d. Which is the default? Migrate and disable? If so which are the default methods for identifying them? All methods? So is there anything I'm missing? Any suggestions on the switches? I'm not entirely sure I like them the way they are. I have code to cover about 60% of the above already. The user-attr-map feature is working and the --disabled-users and disabled-shells options are working. Regards, -Alan ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
