Re: [Freeipa-devel] [PATCH] 134 Improve handling of GIDs when migrating groups
Martin Kosek wrote: On Thu, 2011-10-06 at 21:31 -0400, Rob Crittenden wrote: Martin Kosek wrote: On Wed, 2011-10-05 at 13:44 -0400, Rob Crittenden wrote: Martin Kosek wrote: Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866 For argument's sake, what is the user going to see the first time they run this? I assume they won't think about these duplicate groups and just do the migration. This means that the result may be some users pointing to non-existent GIDs. At first I was thinking about making the GID the default behavior and just add flag --dont-overwrite-gid. But I was afraid this could do some damage and change GIDs where it is not required. However, I made some improvements in this area, please see below. If they re-run the migration with this option will it then fix everything up? Yep. I'm wondering if we need a --test argument so people can run the migration w/o writing entries to look for problems like this. rob If we want to do this, we would have to add a lot of LDAP query checks since mostly try doing the LDAP write and write failures in case of an exception. However, I updated the patch so that user is notified about existence of --group-overwrite-gid option better. If a migration of a group with a GID number fails because of DuplicateError, a notice about GID is displayed. This should make him check this situation and either use group-mod --gidnumber=... or re-run the migration with --group-overwrite-gid. I also updated the Password option not to ask user for LDAP password twice, because it makes me really mad :-) Martin # ipa migrate-ds ldap://panther.greyoak.com --user-container=cn=users,cn=accounts --group-container=cn=groups,cn=accounts --user-ignore-objectclass=radiusprofile Password: ipa: ERROR: an internal error has occurred [Thu Oct 06 21:28:49 2011] [error] ipa: ERROR: non-public: TypeError: _post_migrate_user() got an unexpected keyword argument 'options' [Thu Oct 06 21:28:49 2011] [error] Traceback (most recent call last): [Thu Oct 06 21:28:49 2011] [error] File /usr/lib/python2.7/site-packages/ipaserver/rpcserver.py, line 223, in wsgi_execute [Thu Oct 06 21:28:49 2011] [error] result = self.Command[name](*args, **options) [Thu Oct 06 21:28:49 2011] [error] File /usr/lib/python2.7/site-packages/ipalib/frontend.py, line 432, in __call__ [Thu Oct 06 21:28:49 2011] [error] ret = self.run(*args, **options) [Thu Oct 06 21:28:49 2011] [error] File /usr/lib/python2.7/site-packages/ipalib/frontend.py, line 738, in run [Thu Oct 06 21:28:49 2011] [error] return self.execute(*args, **options) [Thu Oct 06 21:28:49 2011] [error] File /usr/lib/python2.7/site-packages/ipalib/plugins/migration.py, line 633, in execute [Thu Oct 06 21:28:49 2011] [error] ldap, config, ds_ldap, ds_base_dn, options [Thu Oct 06 21:28:49 2011] [error] File /usr/lib/python2.7/site-packages/ipalib/plugins/migration.py, line 602, in migrate [Thu Oct 06 21:28:49 2011] [error] options = options, [Thu Oct 06 21:28:49 2011] [error] TypeError: _post_migrate_user() got an unexpected keyword argument 'options' rob Ouch. This one must have come from some previous tries. And since the users were already migrated in my testing, it was left undiscovered. I wonder why pylint was quiet. Sending a fixed version, it should work fine now. Martin ack, pushed to master and ipa-2-1 ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] 134 Improve handling of GIDs when migrating groups
On Thu, 2011-10-06 at 21:31 -0400, Rob Crittenden wrote:
Martin Kosek wrote:
On Wed, 2011-10-05 at 13:44 -0400, Rob Crittenden wrote:
Martin Kosek wrote:
Since IPA v2 server already contain predefined groups that may collide
with groups in migrated (IPA v1) server (for example admins, ipausers),
users having colliding group as their primary group may happen to belong
to an unknown group on new IPA v2 server.
Implement --group-overwrite-gid option to overwrite GID of already
existing groups to prevent this issue.
https://fedorahosted.org/freeipa/ticket/1866
For argument's sake, what is the user going to see the first time they
run this? I assume they won't think about these duplicate groups and
just do the migration. This means that the result may be some users
pointing to non-existent GIDs.
At first I was thinking about making the GID the default behavior and
just add flag --dont-overwrite-gid. But I was afraid this could do some
damage and change GIDs where it is not required. However, I made some
improvements in this area, please see below.
If they re-run the migration with this option will it then fix
everything up?
Yep.
I'm wondering if we need a --test argument so people can run the
migration w/o writing entries to look for problems like this.
rob
If we want to do this, we would have to add a lot of LDAP query checks
since mostly try doing the LDAP write and write failures in case of an
exception.
However, I updated the patch so that user is notified about existence of
--group-overwrite-gid option better. If a migration of a group with a
GID number fails because of DuplicateError, a notice about GID is
displayed. This should make him check this situation and either use
group-mod --gidnumber=... or re-run the migration with
--group-overwrite-gid.
I also updated the Password option not to ask user for LDAP password
twice, because it makes me really mad :-)
Martin
# ipa migrate-ds ldap://panther.greyoak.com
--user-container=cn=users,cn=accounts
--group-container=cn=groups,cn=accounts
--user-ignore-objectclass=radiusprofile
Password:
ipa: ERROR: an internal error has occurred
[Thu Oct 06 21:28:49 2011] [error] ipa: ERROR: non-public: TypeError:
_post_migrate_user() got an unexpected keyword argument 'options'
[Thu Oct 06 21:28:49 2011] [error] Traceback (most recent call last):
[Thu Oct 06 21:28:49 2011] [error] File
/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py, line 223, in
wsgi_execute
[Thu Oct 06 21:28:49 2011] [error] result =
self.Command[name](*args, **options)
[Thu Oct 06 21:28:49 2011] [error] File
/usr/lib/python2.7/site-packages/ipalib/frontend.py, line 432, in __call__
[Thu Oct 06 21:28:49 2011] [error] ret = self.run(*args, **options)
[Thu Oct 06 21:28:49 2011] [error] File
/usr/lib/python2.7/site-packages/ipalib/frontend.py, line 738, in run
[Thu Oct 06 21:28:49 2011] [error] return self.execute(*args, **options)
[Thu Oct 06 21:28:49 2011] [error] File
/usr/lib/python2.7/site-packages/ipalib/plugins/migration.py, line
633, in execute
[Thu Oct 06 21:28:49 2011] [error] ldap, config, ds_ldap,
ds_base_dn, options
[Thu Oct 06 21:28:49 2011] [error] File
/usr/lib/python2.7/site-packages/ipalib/plugins/migration.py, line
602, in migrate
[Thu Oct 06 21:28:49 2011] [error] options = options,
[Thu Oct 06 21:28:49 2011] [error] TypeError: _post_migrate_user() got
an unexpected keyword argument 'options'
rob
Ouch. This one must have come from some previous tries. And since the
users were already migrated in my testing, it was left undiscovered. I
wonder why pylint was quiet.
Sending a fixed version, it should work fine now.
Martin
From e0903eb55c2873b8985755c4dd7e88df424d4323 Mon Sep 17 00:00:00 2001
From: Martin Kosek mko...@redhat.com
Date: Mon, 3 Oct 2011 16:01:01 +0200
Subject: [PATCH] Improve handling of GIDs when migrating groups
Since IPA v2 server already contain predefined groups that may collide
with groups in migrated (IPA v1) server (for example admins, ipausers),
users having colliding group as their primary group may happen to belong
to an unknown group on new IPA v2 server.
Implement --group-overwrite-gid option to overwrite GID of already
existing groups to prevent this issue.
https://fedorahosted.org/freeipa/ticket/1866
---
API.txt |7 ++--
VERSION |2 +-
ipalib/plugins/migration.py | 73 --
3 files changed, 67 insertions(+), 15 deletions(-)
diff --git a/API.txt b/API.txt
index 10b3f86a8f33d53944423af46b42da1e7f377232..77ff362f943ded64a0b18443e3557f2cedf0b873 100644
--- a/API.txt
+++ b/API.txt
@@ -1701,9 +1701,9 @@ output: Output('summary', (type 'unicode', type 'NoneType'), 'User-friendly
output: Entry('result', type 'dict', Gettext('A dictionary representing an LDAP entry', domain='ipa',
Re: [Freeipa-devel] [PATCH] 134 Improve handling of GIDs when migrating groups
Martin Kosek wrote: On Wed, 2011-10-05 at 13:44 -0400, Rob Crittenden wrote: Martin Kosek wrote: Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866 For argument's sake, what is the user going to see the first time they run this? I assume they won't think about these duplicate groups and just do the migration. This means that the result may be some users pointing to non-existent GIDs. At first I was thinking about making the GID the default behavior and just add flag --dont-overwrite-gid. But I was afraid this could do some damage and change GIDs where it is not required. However, I made some improvements in this area, please see below. If they re-run the migration with this option will it then fix everything up? Yep. I'm wondering if we need a --test argument so people can run the migration w/o writing entries to look for problems like this. rob If we want to do this, we would have to add a lot of LDAP query checks since mostly try doing the LDAP write and write failures in case of an exception. However, I updated the patch so that user is notified about existence of --group-overwrite-gid option better. If a migration of a group with a GID number fails because of DuplicateError, a notice about GID is displayed. This should make him check this situation and either use group-mod --gidnumber=... or re-run the migration with --group-overwrite-gid. I also updated the Password option not to ask user for LDAP password twice, because it makes me really mad :-) Martin # ipa migrate-ds ldap://panther.greyoak.com --user-container=cn=users,cn=accounts --group-container=cn=groups,cn=accounts --user-ignore-objectclass=radiusprofile Password: ipa: ERROR: an internal error has occurred [Thu Oct 06 21:28:49 2011] [error] ipa: ERROR: non-public: TypeError: _post_migrate_user() got an unexpected keyword argument 'options' [Thu Oct 06 21:28:49 2011] [error] Traceback (most recent call last): [Thu Oct 06 21:28:49 2011] [error] File /usr/lib/python2.7/site-packages/ipaserver/rpcserver.py, line 223, in wsgi_execute [Thu Oct 06 21:28:49 2011] [error] result = self.Command[name](*args, **options) [Thu Oct 06 21:28:49 2011] [error] File /usr/lib/python2.7/site-packages/ipalib/frontend.py, line 432, in __call__ [Thu Oct 06 21:28:49 2011] [error] ret = self.run(*args, **options) [Thu Oct 06 21:28:49 2011] [error] File /usr/lib/python2.7/site-packages/ipalib/frontend.py, line 738, in run [Thu Oct 06 21:28:49 2011] [error] return self.execute(*args, **options) [Thu Oct 06 21:28:49 2011] [error] File /usr/lib/python2.7/site-packages/ipalib/plugins/migration.py, line 633, in execute [Thu Oct 06 21:28:49 2011] [error] ldap, config, ds_ldap, ds_base_dn, options [Thu Oct 06 21:28:49 2011] [error] File /usr/lib/python2.7/site-packages/ipalib/plugins/migration.py, line 602, in migrate [Thu Oct 06 21:28:49 2011] [error] options = options, [Thu Oct 06 21:28:49 2011] [error] TypeError: _post_migrate_user() got an unexpected keyword argument 'options' rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] 134 Improve handling of GIDs when migrating groups
Martin Kosek wrote: Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866 For argument's sake, what is the user going to see the first time they run this? I assume they won't think about these duplicate groups and just do the migration. This means that the result may be some users pointing to non-existent GIDs. If they re-run the migration with this option will it then fix everything up? I'm wondering if we need a --test argument so people can run the migration w/o writing entries to look for problems like this. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
[Freeipa-devel] [PATCH] 134 Improve handling of GIDs when migrating groups
Since IPA v2 server already contain predefined groups that may collide
with groups in migrated (IPA v1) server (for example admins, ipausers),
users having colliding group as their primary group may happen to belong
to an unknown group on new IPA v2 server.
Implement --group-overwrite-gid option to overwrite GID of already
existing groups to prevent this issue.
https://fedorahosted.org/freeipa/ticket/1866
From 7be44806e8f5f2862d1033a7c1726bbb0d7573ae Mon Sep 17 00:00:00 2001
From: Martin Kosek mko...@redhat.com
Date: Mon, 3 Oct 2011 16:01:01 +0200
Subject: [PATCH] Improve handling of GIDs when migrating groups
Since IPA v2 server already contain predefined groups that may collide
with groups in migrated (IPA v1) server (for example admins, ipausers),
users having colliding group as their primary group may happen to belong
to an unknown group on new IPA v2 server.
Implement --group-overwrite-gid option to overwrite GID of already
existing groups to prevent this issue.
https://fedorahosted.org/freeipa/ticket/1866
---
ipalib/plugins/migration.py | 65 +++---
1 files changed, 54 insertions(+), 11 deletions(-)
diff --git a/ipalib/plugins/migration.py b/ipalib/plugins/migration.py
index 93ac114d8f30ec9f97dcb4bc59ee9ac39f50f4c4..3321171062cf766a4629dc736b4965d997c7c6e1 100644
--- a/ipalib/plugins/migration.py
+++ b/ipalib/plugins/migration.py
@@ -71,6 +71,15 @@ EXAMPLES:
Specify the user and group container. This can be used to migrate user and
group data from an IPA v1 server:
ipa migrate-ds --user-container='cn=users,cn=accounts' --group-container='cn=groups,cn=accounts' ldap://ds.example.com:389
+
+ Since IPA v2 server already contain predefined groups that may collide with
+ groups in migrated (IPA v1) server (for example admins, ipausers), users having
+ colliding group as their primary group may happen to belong to an unknown group
+ on new IPA v2 server.
+ Use --group-overwrite-gid option to overwrite GID of already existing groups
+ to prevent this issue:
+ipa migrate-ds --group-overwrite-gid --user-container='cn=users,cn=accounts' --group-container='cn=groups,cn=accounts' ldap://ds.example.com:389
+
)
# USER MIGRATION CALLBACKS AND VARS
@@ -228,6 +237,21 @@ def _pre_migrate_group(ldap, pkey, dn, entry_attrs, failed, config, ctx, **kwarg
return dn
+def _group_exc_callback(ldap, dn, entry_attrs, exc, options):
+if isinstance(exc, errors.DuplicateEntry):
+if options.get('groupoverwritegid', False) and \
+ entry_attrs.get('gidnumber') is not None:
+try:
+new_entry_attrs = {'gidnumber':entry_attrs['gidnumber']}
+ldap.update_entry(dn, new_entry_attrs)
+except errors.EmptyModlist:
+# no change to the GID
+pass
+# mark as success
+return
+
+raise exc
+
# DS MIGRATION PLUGIN
def construct_filter(template, oc_list):
@@ -252,6 +276,7 @@ class migrate_ds(Command):
# pre_callback - is called for each object just after it was
#retrieved from DS and before being added to IPA
# post_callback - is called for each object after it was added to IPA
+# exc_callback - is called when adding entry to IPA raises an exception
#
# {pre, post}_callback parameters:
# ldap - ldap2 instance connected to IPA
@@ -270,7 +295,8 @@ class migrate_ds(Command):
'oc_blacklist_option' : 'userignoreobjectclass',
'attr_blacklist_option' : 'userignoreattribute',
'pre_callback' : _pre_migrate_user,
-'post_callback' : _post_migrate_user
+'post_callback' : _post_migrate_user,
+'exc_callback' : None
},
'group': {
'filter_template' : '((|%s)(cn=*))',
@@ -278,7 +304,8 @@ class migrate_ds(Command):
'oc_blacklist_option' : 'groupignoreobjectclass',
'attr_blacklist_option' : 'groupignoreattribute',
'pre_callback' : _pre_migrate_group,
-'post_callback' : None
+'post_callback' : None,
+'exc_callback' : _group_exc_callback,
},
}
migrate_order = ('user', 'group')
@@ -359,6 +386,12 @@ class migrate_ds(Command):
default=tuple(),
autofill=True,
),
+Flag('groupoverwritegid',
+cli_name='group_overwrite_gid',
+label=_('Overwrite GID'),
+doc=_('When migrating a group already existing in IPA domain overwrite the '\
+ 'group GID and report as success'),
+),
StrEnum('schema?',
cli_name='schema',
label=_('LDAP schema'),
@@ -539,16 +572,26 @@ can use their Kerberos accounts.''')
try:
ldap.add_entry(dn, entry_attrs)
except errors.ExecutionError, e:
-failed[ldap_obj_name][pkey] =
