Re: [Freeipa-devel] [PATCH] Make the migration plugin more configurable
Jakub Hrozek wrote: On Wed, Nov 24, 2010 at 04:54:19PM -0500, Rob Crittenden wrote: Jakub Hrozek wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/22/2010 04:21 PM, Jakub Hrozek wrote: On 11/22/2010 04:16 PM, Jakub Hrozek wrote: The code handles it (I just ran a quick test with --schema=RFC2307bis). It just iterates through all members of a group -- be it user member of group member, it's just a DN for the plugin. Jakub Sorry, I found another bug in the plugin. I'll send a new patch shortly, so please don't waste time reviewing this one. New patch is attached. It fixes two more bugs of the original plugin - determines whether a group member is a user or a nested group by checking the DN, not just the RDN attribute name and does not hardcode primary keys. Will this blow up in convert_members_rfc2307bis() if a member isn't contained in the users and groups containers? Should there be a failsafe to skip over things that don't match (along with appropriate reporting)? It wouldn't blow up but add the original DN into the member attribute which is probably worse. Thanks for catching this. I modified the patch to log all migrated users and groups with info() and skip those that don't match any of the containers while logging these entries with error(). Or if one of users or groups search bases isn't provided? If one of them isn't provided, a default would be used. It definitely doesn't like this: # ipa migrate-ds --user-container='' --group-container='cn=groups,cn=accounts' ldap://ds.example.com:389 When passed the right set of options it does seem to do the right thing. Sorry, but I don't quite understand the --user-container='' switch. Does it mean the users are rooted at the Base DN? Can you post the error or relevant log info? Please note that the default objectclass is person. The empty user-container isn't related to this patch so ACK, pushed to master. The error I'm seeing in the Apache error log is: [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] Traceback (most recent call last): [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/share/ipa/wsgi.py, line 27, in application [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] return api.Backend.session(environ, start_response) [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipaserver/rpcserver.py, line 142, in __call__ [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] return self.route(environ, start_response) [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipaserver/rpcserver.py, line 154, in route [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] return app(environ, start_response) [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipaserver/rpcserver.py, line 234, in __call__ [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] response = self.wsgi_execute(environ) [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipaserver/rpcserver.py, line 211, in wsgi_execute [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] result = self.Command[name](*args, **options) [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipalib/frontend.py, line 417, in __call__ [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] ret = self.run(*args, **options) [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipalib/frontend.py, line 690, in run [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] return self.execute(*args, **options) [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipalib/plugins/migration.py, line 380, in execute [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] ldap, config, ds_ldap, ds_base_dn, options [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipalib/plugins/migration.py, line 300, in migrate [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] search_filter, ['*'], search_base, ds_ldap.SCOPE_ONELEVEL#, [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipalib/encoder.py, line 188, in new_f [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] return f(*new_args, **kwargs) [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipalib/encoder.py, line 199, in new_f [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] return args[0].decode(f(*args, **kwargs)) [Tue Dec 07 10:38:10 2010] [error] [client 192.168.166.32] File /usr/lib/python2.6/site-packages/ipaserver/plugins/ldap2.py, line 516, in find_entries [Tue Dec 07 10:38:10 2010] [error]
Re: [Freeipa-devel] [PATCH] Make the migration plugin more configurable
On Wed, Nov 24, 2010 at 04:54:19PM -0500, Rob Crittenden wrote:
Jakub Hrozek wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 11/22/2010 04:21 PM, Jakub Hrozek wrote:
On 11/22/2010 04:16 PM, Jakub Hrozek wrote:
The code handles it (I just ran a quick test with --schema=RFC2307bis).
It just iterates through all members of a group -- be it user member of
group member, it's just a DN for the plugin.
Jakub
Sorry, I found another bug in the plugin. I'll send a new patch shortly,
so please don't waste time reviewing this one.
New patch is attached. It fixes two more bugs of the original plugin -
determines whether a group member is a user or a nested group by
checking the DN, not just the RDN attribute name and does not hardcode
primary keys.
Will this blow up in convert_members_rfc2307bis() if a member isn't
contained in the users and groups containers? Should there be a
failsafe to skip over things that don't match (along with
appropriate reporting)?
It wouldn't blow up but add the original DN into the member attribute
which is probably worse. Thanks for catching this. I modified the patch
to log all migrated users and groups with info() and skip those that
don't match any of the containers while logging these entries with
error().
Or if one of users or groups search bases
isn't provided?
If one of them isn't provided, a default would be used.
It definitely doesn't like this:
# ipa migrate-ds --user-container=''
--group-container='cn=groups,cn=accounts' ldap://ds.example.com:389
When passed the right set of options it does seem to do the right thing.
Sorry, but I don't quite understand the --user-container='' switch.
Does it mean the users are rooted at the Base DN? Can you post the error
or relevant log info? Please note that the default objectclass is
person.
From 1b0f43c4449bd26ffe6c57a594f3eaf367cda2c4 Mon Sep 17 00:00:00 2001
From: Jakub Hrozek jhro...@redhat.com
Date: Tue, 26 Oct 2010 16:10:42 -0400
Subject: [PATCH] Make the migration plugin more configurable
This patch adds new options to the migration plugin:
* the option to fine-tune the objectclass of users or groups being imported
* the option to select the LDAP schema (RFC2307 or RFC2307bis)
Also makes the logic that decides whether an entry is a nested group or user
(for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys
for migrated entries.
https://fedorahosted.org/freeipa/ticket/429
---
ipalib/plugins/migration.py | 136 ++-
1 files changed, 108 insertions(+), 28 deletions(-)
diff --git a/ipalib/plugins/migration.py b/ipalib/plugins/migration.py
index 6dc9934..213c0ee 100644
--- a/ipalib/plugins/migration.py
+++ b/ipalib/plugins/migration.py
@@ -26,9 +26,10 @@ Example: Migrate users and groups from DS to IPA
import logging
import re
+import ldap as _ldap
from ipalib import api, errors, output
-from ipalib import Command, List, Password, Str, Flag
+from ipalib import Command, List, Password, Str, Flag, StrEnum
from ipalib.cli import to_cli
if api.env.in_server and api.env.context in ['lite', 'server']:
try:
@@ -44,8 +45,10 @@ from ipalib.text import Gettext # FIXME: remove once the
other Gettext FIXME is
_krb_err_msg = _('Kerberos principal %s already exists. Use \'ipa user-mod\'
to set it manually.')
_grp_err_msg = _('Failed to add user to the default group. Use \'ipa
group-add-member\' to add manually.')
+_supported_schemas = (u'RFC2307bis', u'RFC2307')
-def _pre_migrate_user(ldap, pkey, dn, entry_attrs, failed, config, ctx):
+
+def _pre_migrate_user(ldap, pkey, dn, entry_attrs, failed, config, ctx,
**kwargs):
# get default primary group for new users
if 'def_group_dn' not in ctx:
def_group = config.get('ipadefaultprimarygroup')
@@ -90,37 +93,80 @@ def _post_migrate_user(ldap, pkey, dn, entry_attrs, failed,
config, ctx):
# GROUP MIGRATION CALLBACKS AND VARS
-def _pre_migrate_group(ldap, pkey, dn, entry_attrs, failed, config, ctx):
-def convert_members(member_attr, overwrite=False):
+def _pre_migrate_group(ldap, pkey, dn, entry_attrs, failed, config, ctx,
**kwargs):
+def convert_members_rfc2307bis(member_attr, search_bases, overwrite=False):
Convert DNs in member attributes to work in IPA.
new_members = []
entry_attrs.setdefault(member_attr, [])
for m in entry_attrs[member_attr]:
-col = m.find(',')
-if col == -1:
+try:
+# what str2dn returns looks like [[('cn', 'foo', 4)], [('dc',
'example', 1)], [('dc', 'com', 1)]]
+rdn = _ldap.dn.str2dn(m ,flags=_ldap.DN_FORMAT_LDAPV3)[0]
+rdnval = rdn[0][1]
+except IndexError:
+api.log.error('Malformed DN %s has no RDN?' % m)
+continue
+
+if m.lower().endswith(search_bases['user']):
+api.log.info('migrating user
Re: [Freeipa-devel] [PATCH] Make the migration plugin more configurable
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 11/22/2010 04:21 PM, Jakub Hrozek wrote:
On 11/22/2010 04:16 PM, Jakub Hrozek wrote:
The code handles it (I just ran a quick test with --schema=RFC2307bis).
It just iterates through all members of a group -- be it user member of
group member, it's just a DN for the plugin.
Jakub
Sorry, I found another bug in the plugin. I'll send a new patch shortly,
so please don't waste time reviewing this one.
New patch is attached. It fixes two more bugs of the original plugin -
determines whether a group member is a user or a nested group by
checking the DN, not just the RDN attribute name and does not hardcode
primary keys.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAkztK7sACgkQHsardTLnvCUdewCdECJneAALtFoe80bWgZqMUHJ2
FjIAn05ld9VSNwe8Xmhi7Y8R3g3Af/av
=ZInN
-END PGP SIGNATURE-
From 7a12a608e5fee82713af6c9c318dbf6b3e98ef83 Mon Sep 17 00:00:00 2001
From: Jakub Hrozek jhro...@redhat.com
Date: Tue, 26 Oct 2010 16:10:42 -0400
Subject: [PATCH] Make the migration plugin more configurable
This patch adds new options to the migration plugin:
* the option to fine-tune the objectclass of users or groups being imported
* the option to select the LDAP schema (RFC2307 or RFC2307bis)
Also makes the logic that decides whether an entry is a nested group or user
(for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys
for migrated entries.
https://fedorahosted.org/freeipa/ticket/429
---
ipalib/plugins/migration.py | 131 +--
1 files changed, 102 insertions(+), 29 deletions(-)
diff --git a/ipalib/plugins/migration.py b/ipalib/plugins/migration.py
index 6dc9934..48c8fa2 100644
--- a/ipalib/plugins/migration.py
+++ b/ipalib/plugins/migration.py
@@ -26,9 +26,10 @@ Example: Migrate users and groups from DS to IPA
import logging
import re
+import ldap as _ldap
from ipalib import api, errors, output
-from ipalib import Command, List, Password, Str, Flag
+from ipalib import Command, List, Password, Str, Flag, StrEnum
from ipalib.cli import to_cli
if api.env.in_server and api.env.context in ['lite', 'server']:
try:
@@ -44,8 +45,10 @@ from ipalib.text import Gettext # FIXME: remove once the other Gettext FIXME is
_krb_err_msg = _('Kerberos principal %s already exists. Use \'ipa user-mod\' to set it manually.')
_grp_err_msg = _('Failed to add user to the default group. Use \'ipa group-add-member\' to add manually.')
+_supported_schemas = (u'RFC2307bis', u'RFC2307')
-def _pre_migrate_user(ldap, pkey, dn, entry_attrs, failed, config, ctx):
+
+def _pre_migrate_user(ldap, pkey, dn, entry_attrs, failed, config, ctx, **kwargs):
# get default primary group for new users
if 'def_group_dn' not in ctx:
def_group = config.get('ipadefaultprimarygroup')
@@ -90,21 +93,30 @@ def _post_migrate_user(ldap, pkey, dn, entry_attrs, failed, config, ctx):
# GROUP MIGRATION CALLBACKS AND VARS
-def _pre_migrate_group(ldap, pkey, dn, entry_attrs, failed, config, ctx):
-def convert_members(member_attr, overwrite=False):
+def _pre_migrate_group(ldap, pkey, dn, entry_attrs, failed, config, ctx, **kwargs):
+def convert_members_rfc2307bis(member_attr, search_bases, overwrite=False):
Convert DNs in member attributes to work in IPA.
new_members = []
entry_attrs.setdefault(member_attr, [])
for m in entry_attrs[member_attr]:
-col = m.find(',')
-if col == -1:
-continue
-if m.startswith('uid'):
-m = '%s,%s' % (m[0:col], api.env.container_user)
-elif m.startswith('cn'):
-m = '%s,%s' % (m[0:col], api.env.container_group)
+try:
+# what str2dn returns looks like [[('cn', 'foo', 4)], [('dc', 'example', 1)], [('dc', 'com', 1)]]
+rdn = _ldap.dn.str2dn(m ,flags=_ldap.DN_FORMAT_LDAPV3)[0]
+rdnval = rdn[0][1]
+except IndexError:
+raise ValueError('Malformed DN %s has no RDN?' % m)
+
+if m.lower().endswith(search_bases['user']):
+m = '%s=%s,%s' % (api.Object.user.primary_key.name,
+ rdnval,
+ api.env.container_user)
+elif m.lower().endswith(search_bases['group']):
+m = '%s=%s,%s' % (api.Object.group.primary_key.name,
+ rdnval,
+ api.env.container_group)
+
m = ldap.normalize_dn(m)
new_members.append(m)
del entry_attrs[member_attr]
@@ -112,15 +124,42 @@ def _pre_migrate_group(ldap, pkey, dn, entry_attrs, failed, config, ctx):
entry_attrs['member'] = []
entry_attrs['member'] += new_members
+def
Re: [Freeipa-devel] [PATCH] Make the migration plugin more configurable
Jakub Hrozek wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/22/2010 04:21 PM, Jakub Hrozek wrote: On 11/22/2010 04:16 PM, Jakub Hrozek wrote: The code handles it (I just ran a quick test with --schema=RFC2307bis). It just iterates through all members of a group -- be it user member of group member, it's just a DN for the plugin. Jakub Sorry, I found another bug in the plugin. I'll send a new patch shortly, so please don't waste time reviewing this one. New patch is attached. It fixes two more bugs of the original plugin - determines whether a group member is a user or a nested group by checking the DN, not just the RDN attribute name and does not hardcode primary keys. Will this blow up in convert_members_rfc2307bis() if a member isn't contained in the users and groups containers? Should there be a failsafe to skip over things that don't match (along with appropriate reporting)? Or if one of users or groups search bases isn't provided? It definitely doesn't like this: # ipa migrate-ds --user-container='' --group-container='cn=groups,cn=accounts' ldap://ds.example.com:389 When passed the right set of options it does seem to do the right thing. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] Make the migration plugin more configurable
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/19/2010 08:17 PM, Rob Crittenden wrote: Jakub Hrozek wrote: This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being * imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) https://fedorahosted.org/freeipa/ticket/429 I don't see where the RFC 2307bis case handles nested groups. This should be supported, right? rob The code handles it (I just ran a quick test with --schema=RFC2307bis). It just iterates through all members of a group -- be it user member of group member, it's just a DN for the plugin. Jakub -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAkzqiVMACgkQHsardTLnvCWLFwCcD4GUp4RVUVoTzElVuHqayJOw Vq8An0tRSltlWh3Y2V92eWhyLsQwVwip =RITJ -END PGP SIGNATURE- ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCH] Make the migration plugin more configurable
Rob Crittenden wrote: Jakub Hrozek wrote: This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being * imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) https://fedorahosted.org/freeipa/ticket/429 I don't see where the RFC 2307bis case handles nested groups. This should be supported, right? I think Jakub refers to memberUid vs. member attribute. rob ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
[Freeipa-devel] [PATCH] Make the migration plugin more configurable
This patch adds new options to the migration plugin:
* the option to fine-tune the objectclass of users or groups being
* imported
* the option to select the LDAP schema (RFC2307 or RFC2307bis)
https://fedorahosted.org/freeipa/ticket/429
From 88165ff6ea2d889150e8bdc882a7b6bec1ab0519 Mon Sep 17 00:00:00 2001
From: Jakub Hrozek jhro...@redhat.com
Date: Tue, 26 Oct 2010 16:10:42 -0400
Subject: [PATCH] Make the migration plugin more configurable
This patch adds new options to the migration plugin:
* the option to fine-tune the objectclass of users or groups being imported
* the option to select the LDAP schema (RFC2307 or RFC2307bis)
https://fedorahosted.org/freeipa/ticket/429
---
ipalib/plugins/migration.py | 86 ++-
1 files changed, 68 insertions(+), 18 deletions(-)
diff --git a/ipalib/plugins/migration.py b/ipalib/plugins/migration.py
index 6dc9934..81e35e2 100644
--- a/ipalib/plugins/migration.py
+++ b/ipalib/plugins/migration.py
@@ -28,7 +28,7 @@ import logging
import re
from ipalib import api, errors, output
-from ipalib import Command, List, Password, Str, Flag
+from ipalib import Command, List, Password, Str, Flag, StrEnum
from ipalib.cli import to_cli
if api.env.in_server and api.env.context in ['lite', 'server']:
try:
@@ -44,8 +44,10 @@ from ipalib.text import Gettext # FIXME: remove once the
other Gettext FIXME is
_krb_err_msg = _('Kerberos principal %s already exists. Use \'ipa user-mod\'
to set it manually.')
_grp_err_msg = _('Failed to add user to the default group. Use \'ipa
group-add-member\' to add manually.')
+_supported_schemas = (u'RFC2307bis', u'RFC2307')
-def _pre_migrate_user(ldap, pkey, dn, entry_attrs, failed, config, ctx):
+
+def _pre_migrate_user(ldap, pkey, dn, entry_attrs, failed, config, ctx,
**kwargs):
# get default primary group for new users
if 'def_group_dn' not in ctx:
def_group = config.get('ipadefaultprimarygroup')
@@ -90,8 +92,8 @@ def _post_migrate_user(ldap, pkey, dn, entry_attrs, failed,
config, ctx):
# GROUP MIGRATION CALLBACKS AND VARS
-def _pre_migrate_group(ldap, pkey, dn, entry_attrs, failed, config, ctx):
-def convert_members(member_attr, overwrite=False):
+def _pre_migrate_group(ldap, pkey, dn, entry_attrs, failed, config, ctx,
**kwargs):
+def convert_members_rfc2307bis(member_attr, overwrite=False):
Convert DNs in member attributes to work in IPA.
@@ -112,15 +114,36 @@ def _pre_migrate_group(ldap, pkey, dn, entry_attrs,
failed, config, ctx):
entry_attrs['member'] = []
entry_attrs['member'] += new_members
+def convert_members_rfc2307(member_attr):
+
+Convert usernames in member attributes to work in IPA.
+
+new_members = []
+entry_attrs.setdefault(member_attr, [])
+for m in entry_attrs[member_attr]:
+memberdn = 'uid=%s,%s' % (m, api.env.container_user)
+new_members.append(ldap.normalize_dn(memberdn))
+entry_attrs['member'] = new_members
+
+schema = kwargs.get('schema', None)
entry_attrs['ipauniqueid'] = 'autogenerate'
-convert_members('member', overwrite=True)
-convert_members('uniquemember')
+if schema == 'RFC2307bis':
+convert_members_rfc2307bis('member', overwrite=True)
+convert_members_rfc2307bis('uniquemember')
+elif schema == 'RFC2307':
+convert_members_rfc2307('memberuid')
+else:
+raise ValueError('Schema %s not supported' % schema)
return dn
# DS MIGRATION PLUGIN
+def construct_filter(template, oc_list):
+oc_subfilter = ''.join([ '(objectclass=%s)' % oc for oc in oc_list])
+return template % oc_subfilter
+
def validate_ldapuri(ugettext, ldapuri):
m = re.match('^ldaps?://[-\w\.]+(:\d+)?$', ldapuri)
if not m:
@@ -152,14 +175,18 @@ class migrate_ds(Command):
#
# If pre_callback return value evaluates to False, migration
# of the current object is aborted.
-'user': (
-'((objectClass=person)(uid=*))',
-_pre_migrate_user, _post_migrate_user
-),
-'group': (
-
'((|(objectClass=groupOfUniqueNames)(objectClass=groupOfNames))(cn=*))',
-_pre_migrate_group, None
-),
+'user': {
+'filter_template' : '((|%s)(uid=*))',
+'oc_option' : 'userobjectclass',
+'pre_callback' : _pre_migrate_user,
+'post_callback' : _post_migrate_user
+},
+'group': {
+'filter_template' : '((|%s)(cn=*))',
+'oc_option' : 'groupobjectclass',
+'pre_callback' : _pre_migrate_group,
+'post_callback' : None
+},
}
migrate_order = ('user', 'group')
@@ -196,6 +223,28 @@ class migrate_ds(Command):
default=u'ou=groups',
autofill=True,
),
+List('userobjectclass?',
+
