Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available

[Sigbjorn Lie]

On 01/27/2012 07:42 PM, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On 01/27/2012 06:15 PM, Sigbjorn Lie wrote:
On 01/27/2012 03:55 PM, Rob Crittenden wrote:
Sigbjorn Lie wrote:

On Fri, January 27, 2012 15:37, Rob Crittenden wrote:
Stephen Gallagher wrote:

On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote:

Hi


The first naming context returned from the LDAP server is always
chosen
when using migrate-ds. This makes my import fail when I attempt
to import users and groups from
a previous LDAP server having more than 1 naming contexts 
available.

The migrate-ds script should accept an option to specify what
base_dn I
would like to import from.

Is there such an option today? I cannot find it...


Not currently. I noticed this earlier in the week and opened a
ticket on
it, https://fedorahosted.org/freeipa/ticket/2314


Just to add to this request, if the original LDAP server has a
defaultNamingContext attribute, it should be honored for
auto-detecting which base to migrate.

I'll update the 2314 to ensure we don't forget about this. 389-ds 
just
added support for defaultNamingContext.


Ok, thank you.

Anything I can do to work around this issue today? I suppose there
is just a file that need to be
hacked to set a set a value instead of the auto-detected value... ?


/usr/lib/python*/site-packages/ipalib/plugins/migration.py

~line 620 you'll see a block starting with the comment "retrieve DS
base DN".

Comment out the next 8 lines by prefixing them with # (these query to
get the namingContext then pull the first value out).

Add:

ds_base_dn = 'dc=yourbasedn,dc=com'

Alternatively you could always just add the above line to override
what is detected. Commenting out just saves an LDAP lookup.

Restart Apache.


I already found that file and did that earlier today, however I was
restarting tomcat6, not httpd... my bad. :)

I have to specify --group-objectclass=posixGroup to get groups
imported, that's fine. But I only get a few users imported. I see that
by default it seem to be looking for objectclass=person. Only a few
user accounts have that objectclass associated, so I add
--user-objectclass=posixAccount as all users have this objectclass
associated with their account.

$ ipa migrate-ds --user-container='ou=people'
--group-container='ou=group' --bind-dn='cn=directory manager'
--user-objectclass=account --group-objectclass=posixGroup
--schema=RFC2307 --continue ldap://ldapserver:399
ipa: ERROR: an internal error has occurred

Not good. I look in the /var/log/httpd/error_log file, and I find:

[Fri Jan 27 18:12:51 2012] [error] ipa: INFO: admin@NONE: ping(): 
SUCCESS
[Fri Jan 27 18:12:52 2012] [error] ipa: ERROR: non-public:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1:
invalid continuation byte
[Fri Jan 27 18:12:52 2012] [error] Traceback (most recent call last):
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 228,
in wsgi_execute
[Fri Jan 27 18:12:52 2012] [error] result = self.Command[name](*args,
**options)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 432, in
__call__
[Fri Jan 27 18:12:52 2012] [error] ret = self.run(*args, **options)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 738, in run
[Fri Jan 27 18:12:52 2012] [error] return self.execute(*args, 
**options)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line
634, in execute
[Fri Jan 27 18:12:52 2012] [error] ldap, config, ds_ldap, ds_base_dn,
options
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line
513, in migrate
[Fri Jan 27 18:12:52 2012] [error] search_refs=True # migrated DS may
contain search references
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 188, in 
new_f
[Fri Jan 27 18:12:52 2012] [error] return f(*new_args, **kwargs)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 199, in 
new_f
[Fri Jan 27 18:12:52 2012] [error] return args[0].decode(f(*args,
**kwargs))
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in 
decode
[Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
in var)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
<genexpr>
[Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
in var)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in 
decode
[Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var]
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in 
decode
[Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
in var)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
<genexpr>
[Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
in var)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 157, in 
decode
[Fri Jan 27 18:12:52 2012] [error] dct[k] = self._decode_dict_val(k, v)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 64, in
_decode_dict_val
[Fri Jan 27 18:12:52 2012] [error] return self.decode(val)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in 
decode
[Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var]
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 132, in 
decode
[Fri Jan 27 18:12:52 2012] [error]
var.decode(self.encoder_settings.decode_from)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode
[Fri Jan 27 18:12:52 2012] [error] return codecs.utf_8_decode(input,
errors, True)
[Fri Jan 27 18:12:52 2012] [error] UnicodeDecodeError: 'utf8' codec
can't decode byte 0xe5 in position 1: invalid continuation byte
[Fri Jan 27 18:12:52 2012] [error] ipa: INFO: admin@NONE:
migrate_ds(u'ldap://svg-p-idm02.none:389', u'********',
binddn=u'cn=directory manager', usercontainer=u'ou=people',
groupcontainer=u'ou=group', userobjectclass=(u'account',),
groupobjectclass=(u'posixGroup',), userignoreobjectclass=None,
userignoreattribute=None, groupignoreobjectclass=None,
groupignoreattribute=None, groupoverwritegid=False, schema=u'RFC2307',
continue=False, exclude_groups=None, exclude_users=None):
UnicodeDecodeError


Any suggestions?


Oh yes and of course I've already looked for accounts with any non-utf8
chars in any of the output of an ldapsearch of the same ldap tree I'm
trying to import from...

This came up yesterday internally too. I don't believe a bug or ticket 
has been filed yet.

My best guess on what is happening, based on what I saw with our own 
case, is this:

A migrated attribute is coming in that IPA doesn't know about. We 
default to treating all attributes that don't require special 
treatment (like binary values) as utf-8. I'm guessing that an unknown 
binary attribute is being migrated, we try to utf-8 encode it and 
kablooey.

There is no easy workaround for the above problem right now because 
that happens before the --ignore* options are considered.

One idea that has come to mind but is untested is to grab the missing 
schema in the form of a discrete 389-ds-compatible schema file. Stop 
dirsrv, drop the file into /etc/dirsrv/slapd-YOURINSTANCE/schema, 
start dirsrv

Then retry the migration.

Use --ignore* if you don't want/need these attributes, then you stop 
dirsrv, remove the schema file, start dirsrv.


I did use the ignore flags, ended up with: 
--user-ignore-attribute=memberOf,mail,telephoneNumber,userPassword 
--user-ignore-objectclass=organizationalPerson,posixAccount,shadowAccount,inetOrgPerson,organizationalPerson

Still no luck, same error.

According to your testing: is there still some value that should have 
been ignored I've overlooked ?

On the other hand, I've only been looking at entries within the ou's 
that contains users and groups. Will this issue also occur if there is 
non-UTF8 chars in other parts of the directory?


Regards,
Siggi


_______________________________________________
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users