subject:"\[Freeipa\-users\] Very slow kerberos performance after upgrade to IPA 2.2"

Re: [Freeipa-users] Very slow kerberos performance after upgrade to IPA 2.2

2012-07-31 Thread Sigbjorn Lie

On 07/31/2012 11:25 PM, Simo Sorce wrote:

On Tue, 2012-07-31 at 21:08 +0200, Sigbjorn Lie wrote:

On 07/31/2012 01:50 PM, Simo Sorce wrote:

On Tue, 2012-07-31 at 10:50 +0200, Sigbjorn Lie wrote:

On Tue, July 31, 2012 10:20, Petr Spacek wrote:

On 07/30/2012 10:37 PM, Sigbjorn Lie wrote:

Hi,

I've been having performance issues after I upgraded to RHEL 6.3 / IPA 2.2. I
still have a LDAP server having unusual high cpu usage even after it's been
removed from the SRV
records and is serving almost no clients anymore, but it would seem as my main
issues is with
the kerberos server.

All kerberos services are performing very slowly, and the IPA servers has much
higher CPU load now then what they had with IPA 2.1. Some services are timing
out, like
kerberized web servers, other kerberized services perform authentication very
slowly. I had to
switch our automounter away from kerberos authentication as it is no longer
usable.

Using SSH to log on to SSSD enabled hosts are also very slow, a login takes
anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA 2.2.

The IPA web admin interface is definitely not faster than in IPA 2.1.

For a comparison, listing out all the folders in an automount map, causing
them to be looked up from LDAP and mounted takes over 5 minutes with IPA 2.2
when using kerberos
authentication for the automounter. There are approx 130 folders in that
automount map.

After unmounting all the mounted folders, and changing to using a username and
password authentication with a TLS connection, attempting the same operating
again, and it now
finishes in about 14 seconds for both the lookup from LDAP and the mount
operation.

After unmounting all the mounted folders again, changing to username and
password authentication with a simple unencrypted bind, and then attempting the
same operation
and it now finishes both lookup and mount in just over 5 seconds!

I don't have any timing for kerberized automount pre IPA-2.2, but we we're not
talking about several minutes to mount all the folders in this automount map.
Unfortunately
mounting all the folders is what happens when the users use konqueror to browse
the automount
maps, so this is a very noticable issue.

Even loading a new gnome-terminal or konsole terminal which causes an
automount folder to be mounted takes anything between 5 - 15 seconds after the
upgrade. There
we're no notiable delay when opening a new terminal window pre IPA-2.2.

I am not using SSSD for the automounter.

I do notice that the dbmodule for the kerberos server has changed from "kldap"
to "ipadb.so" Perhaps there is some issues with the new library?

Regards,
Siggi

Hello,

I'm not a Kerberos guy, so I can give only general advice:
"Overloaded-CPU-problems" can be troubleshooted with OProfile.

Oprofile is lightweight statistic profiler (AFAIK it was designed for
production environment).

Step-by-step documentation for RHEL 6 is available from:
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.ht
ml#ch-OProfile

As you can see in section 22.5.1., it allows to break whole CPU usage between
processes, libraries and even individual symbols (if proper debuginfos are
installed).

I recommend to run OProfile on problematic system - results from opreport can
provide missing clue to us.

OProfile gives best results on bare-metal machines. On virtual machines you
has to use timer mode in place of hardware performance counters, please see the
documentation.

Short getting started guide:
http://oprofile.sourceforge.net/doc/overview.html#getting-started

Nice article with theory&& examples:
http://people.redhat.com/wcohen/Oprofile.pdf

Homepage with a lot of useful information:
http://oprofile.sourceforge.net/

Thank you.

All 3 IPA servers are close to idle now after switching from kerberos to
user/pwd bind for the
Linux automounter.

Still there is an issue with kerberos failing to issue a ticket every now and
then and it's
responding very slowly.

There seem to be low activity on this list just now. Is the kerberos people
away on vacation?

Hi Siggi,
some people are on vacation, some are busy covering others :-)

Would you be able to take a wireshark trace of an automount going on ?
I would like to see precise timing of packets on the wire to make a
first assesment of where is the bottleneck.

We did change from ldap.so to ipadb.so, but the structure of the drivers
is not much different, so I am surprised it would be much slower,
however it is possible, I would like to find out what is going on with
your help.

OK, I will get that done when I'm back in the office tomorrow. I suspect
it will be somewhat better than my first results as the load on the IPA
servers are now much lower when the linux automounters are no longer
using kerberos for authentication.

It seem like there is a race condition going on as the shit didn't hit
the fan until the week after the upgrade to IPA 2.2 when p

Re: [Freeipa-users] Very slow kerberos performance after upgrade to IPA 2.2

2012-07-31 Thread Simo Sorce

On Tue, 2012-07-31 at 21:08 +0200, Sigbjorn Lie wrote:
> On 07/31/2012 01:50 PM, Simo Sorce wrote:
> > On Tue, 2012-07-31 at 10:50 +0200, Sigbjorn Lie wrote:
> >> On Tue, July 31, 2012 10:20, Petr Spacek wrote:
> >>> On 07/30/2012 10:37 PM, Sigbjorn Lie wrote:
> >>>
>  Hi,
> 
> 
>  I've been having performance issues after I upgraded to RHEL 6.3 / IPA 
>  2.2. I
>  still have a LDAP server having unusual high cpu usage even after it's 
>  been removed from the SRV
>  records and is serving almost no clients anymore, but it would seem as 
>  my main issues is with
>  the kerberos server.
> 
>  All kerberos services are performing very slowly, and the IPA servers 
>  has much
>  higher CPU load now then what they had with IPA 2.1. Some services are 
>  timing out, like
>  kerberized web servers, other kerberized services perform authentication 
>  very slowly. I had to
>  switch our automounter away from kerberos authentication as it is no 
>  longer usable.
> 
>  Using SSH to log on to SSSD enabled hosts are also very slow, a login 
>  takes
>  anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA 
>  2.2.
> 
>  The IPA web admin interface is definitely not faster than in IPA 2.1.
> 
> 
>  For a comparison, listing out all the folders in an automount map, 
>  causing
>  them to be looked up from LDAP and mounted takes over 5 minutes with IPA 
>  2.2 when using kerberos
>  authentication for the automounter. There are approx 130 folders in that 
>  automount map.
> 
>  After unmounting all the mounted folders, and changing to using a 
>  username and
>  password authentication with a TLS connection, attempting the same 
>  operating again, and it now
>  finishes in about 14 seconds for both the lookup from LDAP and the mount 
>  operation.
> 
>  After unmounting all the mounted folders again, changing to username and
>  password authentication with a simple unencrypted bind, and then 
>  attempting the same operation
>  and it now finishes both lookup and mount in just over 5 seconds!
> 
>  I don't have any timing for kerberized automount pre IPA-2.2, but we 
>  we're not
>  talking about several minutes to mount all the folders in this automount 
>  map. Unfortunately
>  mounting all the folders is what happens when the users use konqueror to 
>  browse the automount
>  maps, so this is a very noticable issue.
> 
>  Even loading a new gnome-terminal or konsole terminal which causes an
>  automount folder to be mounted takes anything between 5 - 15 seconds 
>  after the upgrade. There
>  we're no notiable delay when opening a new terminal window pre IPA-2.2.
> 
> 
>  I am not using SSSD for the automounter.
> 
> 
>  I do notice that the dbmodule for the kerberos server has changed from 
>  "kldap"
>  to "ipadb.so" Perhaps there is some issues with the new library?
> 
> 
> 
> 
>  Regards,
>  Siggi
> 
> >>>
> >>> Hello,
> >>>
> >>>
> >>> I'm not a Kerberos guy, so I can give only general advice:
> >>> "Overloaded-CPU-problems" can be troubleshooted with OProfile.
> >>>
> >>>
> >>> Oprofile is lightweight statistic profiler (AFAIK it was designed for
> >>> production environment).
> >>>
> >>> Step-by-step documentation for RHEL 6 is available from:
> >>> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.ht
> >>> ml#ch-OProfile
> >>>
> >>> As you can see in section 22.5.1., it allows to break whole CPU usage 
> >>> between
> >>> processes, libraries and even individual symbols (if proper debuginfos 
> >>> are installed).
> >>>
> >>> I recommend to run OProfile on problematic system - results from opreport 
> >>> can
> >>> provide missing clue to us.
> >>>
> >>> OProfile gives best results on bare-metal machines. On virtual machines 
> >>> you
> >>> has to use timer mode in place of hardware performance counters, please 
> >>> see the documentation.
> >>>
> >>>
> >>> Short getting started guide:
> >>> http://oprofile.sourceforge.net/doc/overview.html#getting-started
> >>>
> >>>
> >>> Nice article with theory&&  examples:
> >>> http://people.redhat.com/wcohen/Oprofile.pdf
> >>>
> >>>
> >>> Homepage with a lot of useful information:
> >>> http://oprofile.sourceforge.net/
> >>>
> >>>
> >>>
> >> Thank you.
> >>
> >> All 3 IPA servers are close to idle now after switching from kerberos to 
> >> user/pwd bind for the
> >> Linux automounter.
> >>
> >> Still there is an issue with kerberos failing to issue a ticket every now 
> >> and then and it's
> >> responding very slowly.
> >>
> >> There seem to be low activity on this list just now. Is the kerberos 
> >> people away on vacation?
> > Hi Siggi,
> > some people are on vacation, some are busy covering others :-)
> >