On 01/16/2015 04:48 PM, Quayle, Bill wrote:
Thanks for looking into this!

I was finally able to import all 11811 user records into IPA, but even now, 
when I re-run the migrate, I get the same failure.

How did you do it in the end? Simply by running migrate-ds command multiple times or did you succeeded with the limits?


I enabled debug in the default.cfg, and this is the tail of the httpd error_log:

.
.
.
  [Fri Jan 16 09:28:29.046991 2015] [:error] [pid 14924] ipa: WARNING: GID 
number 11 of migrated user andy does not point to a known group.
[Fri Jan 16 09:28:29.051353 2015] [:error] [pid 14924] ipa: INFO: 
ad...@idmtest.example.com: migrate_ds(u'ldap://10.x.x.x:389', u'********', 
binddn=u'uid=me,ou=people,ou=agroup,dc=example,dc=com', 
usercontainer=u'ou=people', groupcontainer=u'ou=groups', 
userobjectclass=(u'person',), groupobjectclass=(u'groupOfUniqueNames', 
u'groupOfNames'), userignoreobjectclass=None, userignoreattribute=None, 
groupignoreobjectclass=None, groupignoreattribute=None, 
groupoverwritegid=False, schema=u'RFC2307bis', continue=True, 
basedn=u'ou=agroup,dc=example,dc=com', compat=False, version=u'2.65', 
exclude_groups=None, exclude_users=None): NetworkError
[Fri Jan 16 09:28:29.051428 2015] [:error] [pid 14924] ipa: DEBUG: response: 
NetworkError: cannot connect to 'ldap://10.x.x.x:389':
[Fri Jan 16 09:28:29.054057 2015] [:error] [pid 14924] ipa: DEBUG: no session 
id in request, generating empty session data with 
id=c0d2c8b3803593b30684e15ff1f57e0e
[Fri Jan 16 09:28:29.054173 2015] [:error] [pid 14924] ipa: DEBUG: store 
session: session_id=c0d2c8b3803593b30684e15ff1f57e0e 
start_timestamp=2015-01-16T09:28:29 access_timestamp=2015-01-16T09:28:29 
expiration_timestamp=1969-12-31T18:00:00
[Fri Jan 16 09:28:29.054395 2015] [:error] [pid 14924] ipa: DEBUG: finalize_kerberos_acquisition: 
xmlserver ccache_name="FILE:/run/httpd/krbcache/krb5cc_apache_zTGsku" 
session_id="c0d2c8b3803593b30684e15ff1f57e0e"
[Fri Jan 16 09:28:29.054463 2015] [:error] [pid 14924] ipa: DEBUG: reading ccache data 
from file "/run/httpd/krbcache/krb5cc_apache_zTGsku"
[Fri Jan 16 09:28:29.054851 2015] [:error] [pid 14924] ipa: DEBUG: 
get_credential_times: 
principal=HTTP/myipatestserver.example....@idmtest.example.com, 
authtime=01/15/15 16:44:10, starttime=01/15/15 16:44:17, endtime=01/16/15 
16:44:04, renew_till=12/31/69 18:00:00
[Fri Jan 16 09:28:29.055014 2015] [:error] [pid 14924] ipa: DEBUG: KRB5_CCache 
FILE:/run/httpd/krbcache/krb5cc_apache_zTGsku endtime=1421448244 (01/16/15 
16:44:04)
[Fri Jan 16 09:28:29.055109 2015] [:error] [pid 14924] ipa: DEBUG: 
set_session_expiration_time: duration_type=inactivity_timeout duration=1200 
max_age=1421447944 expiration=1421423309.06 (2015-01-16T09:48:29)
[Fri Jan 16 09:28:29.055217 2015] [:error] [pid 14924] ipa: DEBUG: store 
session: session_id=c0d2c8b3803593b30684e15ff1f57e0e 
start_timestamp=2015-01-16T09:28:29 access_timestamp=2015-01-16T09:28:29 
expiration_timestamp=2015-01-16T09:48:29
[Fri Jan 16 09:28:29.055806 2015] [:error] [pid 14924] ipa: DEBUG: Destroyed 
connection context.ldap2_140392345753040
[Fri Jan 16 09:28:29.056471 2015] [:error] [pid 14924] ipa: DEBUG: Destroyed 
connection context.ldap2

One thing that is also confusing me, is that I am getting this error:
[Fri Jan 16 09:28:29.007575 2015] [:error] [pid 14924] ipa: WARNING: GID number 
11 of migrated user anyone does not point to a known group.

migrate-ds command runs a search against the migrated OpenLDAP database and tries to find a group with gidNumber 11. When it fails to locate it, it reports this error. Do you have all the groups in DN "ou=people,ou=agroup,dc=example,dc=com"?

And it never migrates my groups.  The ou=Groups is used in my source openLDAP 
tree, so I'm not sure why it wouldn't migrate.

If i crashes during user migration, it won't even continue with groups. I know this is not a proper fix, but you could make sure the user migration part does not find anything (e.g. with --user-objectclass=foo) and using --continue option. Then it will jump directly to group migration.

I am still thinking it would make sense to also check the migrated OpenLDAP logs and see if there is anything interesting when the migration breaks.

HTH,
Martin

Bill
-----Original Message-----
From: Martin Kosek [mailto:mko...@redhat.com]
Sent: Friday, January 16, 2015 2:25 AM
To: Ludwig Krispenz
Cc: Quayle, Bill; 'freeipa-users@redhat.com'
Subject: Re: [Freeipa-users] migrate-ds aborts

On 01/16/2015 09:14 AM, Ludwig Krispenz wrote:

On 01/16/2015 08:43 AM, Martin Kosek wrote:
On 01/15/2015 06:31 PM, Quayle, Bill wrote:
I am migrating an openLDAP tree into ipa, and when I run ipa
migrate-ds, the migration aborts after roughly 36 seconds with:

ipa: ERROR: cannot connect to 'ldap://10.x.x.x:389':

It has transferred 9762 records, but seems to hit a timeout that
causes it to stop.

I've run it in debug mode, which only provides this:

ipa: DEBUG: Starting external process

ipa: DEBUG: args=keyctl pupdate 774698354

ipa: DEBUG: Process finished, return code=0

ipa: DEBUG: stdout=

ipa: DEBUG: stderr=

ipa: DEBUG: Caught fault 907 from server
https://foo.example.com/ipa/session/xml: cannot connect to
'ldap://10.x.x.x:389':

ipa: DEBUG: Destroyed connection context.xmlclient

ipa: ERROR: cannot connect to 'ldap://10.x.x.x:389':

Initially, it had transferred 2000 records and stopped, until I set
nsslapd-sizelimit in cn=config:

nsslapd-sizelimit: 20000

I then re-ran the migration a dozen times, each time it would
transfer more records, but would always time out at around the 36
second mark.  Now that I'm at 9762 records, it seems to have reached a peak.

I suspect this is another tunable, but haven't been able to find it,
any document that mentions it, or anyone else hitting this issue.

RHEL 7.0 server

idM ipa-server-3.3.3-28

source is RHEL 6.5 running openldap-2.4.23-34

command used to migrate:

ipa migrate-ds --continue --bind-dn="uid=me,ou=people,ou=foo,dc=example,dc=com"
--base-dn="ou=foo,dc=example,dc=com" ldap://10.x.x.x:389

*Cheers,*

*-Bill*

Ludwig, do you know? I am just thinking it may be also caused by some
form of timelimit, as mentioned in

https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Serve
r/8.2/html/Administration_Guide/User_Account_Management-Setting_Resou
rce_Limits_Based_on_the_Bind_DN.html


(those apply both for bind DNs and global cn=config). Maybe
nsslapd-timelimit could be increased? Although I saw the default is
3600, I assume it means 1 hour, i.e. not being the root cause.
we need the access and error logs from DS, if it is a DS limit it
should be seen in the err code.

+1

Could it be that migrate-ds has it's own limit waiting for a repsponse from DS ?

The search itself in migrate-ds is limit-less:

              try:
                  entries, truncated = ds_ldap.find_entries(
                      search_filter, ['*'], search_bases[ldap_obj_name],
                      ds_ldap.SCOPE_ONELEVEL,
                      time_limit=0, size_limit=-1,
                      search_refs=True    # migrated DS may contain search
references
                  )
               except...

Bill, I am wondering, could you add debug=True to /etc/ipa/default.conf on your 
server, reload the httpd process and re-run the migration? It should print 
additional debugging information that may help us.

Martin

________________________________


CONFIDENTIALITY AND SECURITY NOTICE

The contents of this message and any attachments may be confidential and 
proprietary. If you are not an intended recipient, please inform the sender of 
the transmission error and delete this message immediately without reading, 
distributing or copying the contents.


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Reply via email to