On 05/05/2016 03:44 PM, Petr Vobornik wrote:
On 05/04/2016 02:20 PM, thierry bordaz wrote:

     I have been doing some tests/measures using
     The tool creates a set of typical users/hosts/groups... to import with a

     I wrote down some finding in
     I still have to do some cleanup around the performance but the basic of a
     possible improvement is to do provisioning in several steps (disabling
     plugins, provisioning, enabling plugin, running fixup tasks).

     Before going further in the design I wanted to share those ideas and know 
     it raise any concern.


Hi Thierry,

Thanks for the analysis. Very nice.

Knowing this will help us suggesting workarounds also for old releases.

Couple questions:

Have you tested retrCL disabled with memberOf enabled. It seems that it
would eliminate 550K adds and 0.8M searches. What would be the time

Basically provisioning/migrate does not scale well because of memberOf.

Each time you add a member to a group you need to reevalute its membership (15M search) and finally update the member (that is a MOD for the member + ADD in retroCL). If a same entry is added to a group and later added to an other group, the first evaluation/updates are useless.

Disabling memberof drops SRCH from 15M -> ~1M. You can get an additional gain ~1M -> ~0.1M if you disable retroCL.

I tested retroCL disabled / memberOf enabled but on a different dataset. It gaves low improvement in term of duration.
I think MODs are here more expensive that ADD.
I will test your suggestion to confirm the numbers.

Do you know what is the time when memberof is enabled but slapi-nis and
retroCL are disabled?

No. I will test it.

 From the text it was not clear to me, if you find or investigate
possible improvements in memberof plugin which would improve the
performance without stopping and starting DS.
memberof does not scale well because of #SRCH and #MOD.
Ludwig suggested a group caching that is looking close to other in-memory implementations of memberof. Such cache would reduce the #SRCH possibly from 15M->1M, but the implementation is not immediate. Also note that filling the cache has a cost that can delay startup (like we saw in slapi-nis) and a memory footprint.

Regarding the MODs we can make memberof attribute a virtual attribute and stop updating the entries. This is also not immediate to implement: rewrite memberof plugin.

From 7.3 schedule, I think we can get an important improvement if DS start-stop is acceptable.
Improving memberof is more a longer term option.


Manage your subscription for the Freeipa-devel mailing list:
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Reply via email to