On 12/22/15 08:09, Petr Vobornik wrote:
On 12/22/2015 10:24 AM, thierry bordaz wrote:
On 12/21/2015 05:55 PM, Daryl Fonseca-Holt wrote:
Hi all,

Environment: RHEL6 with IPA 3.0 at current RedHat level. 64-core
256-GB RAM Oracle x4470 M2.

During our migration from NIS on Solaris 140,000+ accounts will be
added. After tuning per the guides dbmon.sh shows no roevicts and we
get high cache hit ratios.

Per a previous discussion with the list the input is broken down into
batches of less than 1,000 users and the default IPA group is changed
before each batch. This helped greatly.

Adding all the users takes many hours. Initially ipa user-add takes an
average 2.3 seconds per user but degrades by the time there are
140,000 users to an average 6.7 seconds per user.

In tracing it appears that a significant portion of the time ipa
user-add takes is not the add itself, it is the query at the end that
displays the resulting user account. Is there any legit way to prevent
this query?

The length of time it takes to migrate is not a big concern. The
concern is the start of the fall school term when we typically add
approximately 1,300 accounts per hour during the registration period
with our current system.

All suggestions will be appreciated.

Regards, Daryl

Hi Daryl,

I can reproduce similar trend of user-add becoming slower and slower.

Now in my tests (etime=7s) the time was spent half by authentication and
half by ADD and MOD (update of ipausers group). I agree there are many
direct SRCH (~10) but they all seems to be rapid.

I know that the vast majority of the time is spent in DS schema-compat
plugin. Disabling it, during provisioning, reduce the duration by ~3.
Now I do not know if it is a valid option to disable this plugin during


We must also distinguish performance on IPA 3.x (RHEL 6.x) and FreeIPA 4.2/4.3

FreeIPA 4.2 got some performance improvements mostly related to group membership handling.

Improving user-add is one of primary goals for 4.4 release:
* https://fedorahosted.org/freeipa/ticket/5448

There are couple issues tracked about plugins output (also planned to be fixed in 4.4):
* https://fedorahosted.org/freeipa/ticket/5281
* https://fedorahosted.org/freeipa/ticket/5282
* https://fedorahosted.org/freeipa/ticket/4995

You can try to call user-add with --raw options but it won't help much because some parts ignore it. Other than that, there is not clean workaround.

When user is added, the user_add typically:
* adds the user to ipadefaultprimarygroup
* converts manger dn to human friedly value (should not cause perf. issues) * set description to magic value to cause generation of user private group (calls user-mod) * fetches password attributes and resolves ssh pub keys (does couple of searches)

An unsupported possibility - do on your own risk - is to remove these operations from user_add post_callback in ipalib/plugins/user.py (around line 535 on 6.7).

Other thing which might help is to remove 'memberof' and 'memberofindirect' values from default_attributes in user.py (~line 216). Note that it also affects other user-* commands.

All should be tried in testing environment.

Another performance improvement is to call IPA API directly and use batch command - with that it is possible to add e.g. 100 users with one call and save some network calls.

Example could be seen in this ugly script: https://pvoborni.fedorapeople.org/scripts/ipa-generate-users.sh
I did test with RHEL7 and IPA 4.2 and the performance was much better. We don't have RHEL7 deployed yet so there are some issues with change management but we may have to revisit that.

I looked at changing the code but we can't end up with an unsupported product. We depend on our Red Hat support heavily for production. So I could speed up the migration from the NIS server but I won't be able to keep the optimization during day-to-day operation.

Thanks for the script. Ugly scripts are good learning examples because it's easy to see the real purpose without the distraction of frills.

Regards, Daryl

 Daryl Fonseca-Holt
 IST/CNS/Unix Server Team
 University of Manitoba

Manage your subscription for the Freeipa-users mailing list:
Go to http://freeipa.org for more info on the project

Reply via email to