thierry bordaz wrote:


Hello,

Thanks for all the feedbacks. I updated the design accordingly and with
additional tests results
(http://www.freeipa.org/page/V4/Performance_Improvements#Proposed_improvements)
Several improvements can be done, in particular in DS plugins (memberof,
retroCL), but for "easy" benefit provisioning will be done with memberof
disabled followed by fixup.

It remains some aspects that are not clear to me:

  * For best performance, DS tuning and provisioning/fixup would
    preferably be done under 'directory manager'
    That means prompting DM password and writing it into temporary file.
    Is that a concern ?
  * Fixup requires that we know the filters matching the provisioned
    entries. For example :
      o (objectClass=inetorgperson)
      o (objectClass=ipausergroup)
      o (objectClass=ipahost)
      o (objectClass=ipahostgroup)
      o (objectClass=ipasudorule)
      o (objectClass=ipahbacrule)

        The set of objectclass could be hardcode or provided in the
        provisioning CLI option
        What to do if an entry in in the provision file does not match
        any of those filter ? Should it stop without starting the
        provisioning ?
  * The CLI doing the provisioning could be something like 'ipa
    provision <options>' or should it be a separated command e.g.
    ipa-bulk-load ?

It depends. There is a migration command now, ipa migrate-ds, that adds records and is impacted by this. There is also the possibility of looping calls to ipa [user|group|etc]-add.

Would it be reasonable to require bulk import to be done on an IPA master so we can leverage the ldapi socket?

rob


thanks
thierry

On 05/13/2016 10:18 AM, Ludwig Krispenz wrote:

On 05/13/2016 09:42 AM, Petr Spacek wrote:
On 13.5.2016 09:26, Martin Kosek wrote:
On 05/12/2016 04:16 PM, Ludwig Krispenz wrote:
On 05/12/2016 03:45 PM, Ludwig Krispenz wrote:
On 05/12/2016 02:16 PM, Petr Vobornik wrote:
On 05/10/2016 05:50 PM, thierry bordaz wrote:
On 05/05/2016 03:44 PM, Petr Vobornik wrote:
On 05/04/2016 02:20 PM, thierry bordaz wrote:
Hello,

       I have been doing some tests/measures using
https://github.com/freeipa/freeipa-tools/blob/master/create-test-data.py.


       The tool creates a set of typical users/hosts/groups... to
import with a
       ldapadd.

       I wrote down some finding in
http://www.freeipa.org/page/V4/Performance_Improvements#Provisioning_throughput_and_DS_plugins.



       I still have to do some cleanup around the performance
but the
basic of a
       possible improvement is to do provisioning in several
steps
(disabling
       plugins, provisioning, enabling plugin, running fixup
tasks).

       Before going further in the design I wanted to share
those ideas
and know if
       it raise any concern.

       thanks
       thierry

Hi Thierry,

Thanks for the analysis. Very nice.

Knowing this will help us suggesting workarounds also for old
releases.

Couple questions:

Have you tested retrCL disabled with memberOf enabled. It seems
that it
would eliminate 550K adds and 0.8M searches. What would be the
time
improvement?

Do you know what is the time when memberof is enabled but
slapi-nis and
retroCL are disabled?
The culprit of the performance issue is very likely related to SRCH
(internal) triggered by memberof.

If retroCL is disabled and memberof enabled, #SRCH is 13.8M.
If retroCL is disabled, slapi-nis disabled and memberof enabled
#SRCH is
14.8
When all of them are enabled the #SRCH is 15M.

You are right if retroCL is disabled the #ADD drops but it has no
significant effect on the duration.
ok, thanks for the analysis

Regarding the duration of the provisioning, values are not
really stable
as performance of VM fluctuates. But as soon as memberof is
enabled the
provisioning lasts > 4hours where the same provisioning lasts
6mins as
soon as memberof is disabled.

I need to confirm the average time for internal searches but
assuming
1ms per SRCH it consumes >90% of the provisioning.


   From the text it was not clear to me, if you find or
investigate
possible improvements in memberof plugin which would improve the
performance without stopping and starting DS.
As was discussed at mtg, have you tried if the DS restart is really
necessary?
memberof plugin can be enabled and disabled while the server is
running, BUT
to achieve this the "enable-dynamic-plugins" feature has to be
turned on. And
then any enable/disable of a plugin would try to do it dynamically
an dnot
wait for the restart.
And I think not all plugins are able to handle this, TomasB was
once working
on it for IPA plugins, but it was not completed as far as I know
but enabling dynamic plugins can be done without restart, so what
can be done is.
- enable dynamic plugins
- disable memberof
- do some work
- enable memberof
- disable dynamic plugins
Please see
https://fedorahosted.org/freeipa/ticket/4203#comment:9
I do not think this will be that easy. We would first need to invest
into
updating FreeIPA plugins to work with dynamic plugins setting and
then we could
do things alike above.

It looks like that for FreeIPA 4.4, we will need to live with DS
restart unless
there is some workaround...
couldn't the scenario I outline above with enabling dynamic plugins
only temporary work, are there any attempts to enable/disable plugins
during provisioning ? If that would be the case that would also
require a restart
One more thing:

How does it affect topologies with replicas?

I might be wrong, but if memberOf is always computed locally then we
have to
disable it on *all* replicas.

If we disabled it only on one replica and not others, the chosen
replica would
be way faster than rest of the topology and I'm not sure what would
happen
later on.
good point. we exclude memberof from replication as it is regenerated
on every server, so each replica would suffer from the performance
problem

Thierry, Ludwig, can you comment on this?






--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Reply via email to