On 11/04/2015 04:07 PM, Rob Crittenden wrote:
Daryl Fonseca-Holt wrote:
Hi All,

I am testing migration from NIS with a custom MySQL backend to IPA. In
our testing ipa user-add starts out at around 12 seconds per user but
slows down as more users are add. By 5000+ users it is taking 90+
seconds. We have 120,000+ users. I'm looking at 155 days to load all the
users :(

Per some performance tuning documentation I've increased
nsslapd-cachememsize to 35,651,584 and am currently getting pretty high
hit ratios (see below).

Use dbmon.sh instead of db_stat - it will give you the entry cache information as well as a summary of the db cache information (instead of the quite verbose db_stat output).

However, one thread of ns-slapd pegs out core at
100% and I can't get get it to add users any faster. I'm not seeing any
I/O or memory swapping.
The problem is most likely the default IPA users group. As it gets
humongous adding new members slows it down.

So could he disable the automember and memberof plugins? Then have those work offline, after the users are added?


Suggestions would be appreciated. Multi-master will probably help but
with that many accounts it would take a lot of masters to get additions
done to a resonable (45 seconds or less?) time. Is there any guideline
for number of users per master?
IPA is multi-master. All users are on all masters.

If anything I'd expect that running imports on different masters would
slow things down as changes on multiple masters would need to get merged
together, particularly the default group.

Right.


Certainly bumping up the caches to match what the final expected sizes
is probably a good idea but I don't see it influencing import speed all
that much.

Right.


One idea I've had is to add the users in batches of 1000. What you'd do
is create 120 non-POSIX user groups, ipausers1..ipausers120, and add
them as members of ipausers.

Then for each batch change the default ipausers group with:

  $ ipa config-mod --defaultgroup=<name>

This should keep the user-add command fairly peppy and keep the ipausers
group somewhat in check via the nesting.

I imagine that the UI would blow up if you tried to view the ipausers
group as it tried to dereference 120k users.

You'll probably also want to disable the compat module for the import.

I assume you've already done some amount of testing with a smaller batch
of users to ensure they migrate ok, passwords work, etc?

rob


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to