On 07/31/2012 01:50 PM, Simo Sorce wrote:
On Tue, 2012-07-31 at 10:50 +0200, Sigbjorn Lie wrote:
On Tue, July 31, 2012 10:20, Petr Spacek wrote:
On 07/30/2012 10:37 PM, Sigbjorn Lie wrote:

Hi,


I've been having performance issues after I upgraded to RHEL 6.3 / IPA 2.2. I
still have a LDAP server having unusual high cpu usage even after it's been 
removed from the SRV
records and is serving almost no clients anymore, but it would seem as my main 
issues is with
the kerberos server.

All kerberos services are performing very slowly, and the IPA servers has much
higher CPU load now then what they had with IPA 2.1. Some services are timing 
out, like
kerberized web servers, other kerberized services perform authentication very 
slowly. I had to
switch our automounter away from kerberos authentication as it is no longer 
usable.

Using SSH to log on to SSSD enabled hosts are also very slow, a login takes
anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA 2.2.

The IPA web admin interface is definitely not faster than in IPA 2.1.


For a comparison, listing out all the folders in an automount map, causing
them to be looked up from LDAP and mounted takes over 5 minutes with IPA 2.2 
when using kerberos
authentication for the automounter. There are approx 130 folders in that 
automount map.

After unmounting all the mounted folders, and changing to using a username and
password authentication with a TLS connection, attempting the same operating 
again, and it now
finishes in about 14 seconds for both the lookup from LDAP and the mount 
operation.

After unmounting all the mounted folders again, changing to username and
password authentication with a simple unencrypted bind, and then attempting the 
same operation
and it now finishes both lookup and mount in just over 5 seconds!

I don't have any timing for kerberized automount pre IPA-2.2, but we we're not
talking about several minutes to mount all the folders in this automount map. 
Unfortunately
mounting all the folders is what happens when the users use konqueror to browse 
the automount
maps, so this is a very noticable issue.

Even loading a new gnome-terminal or konsole terminal which causes an
automount folder to be mounted takes anything between 5 - 15 seconds after the 
upgrade. There
we're no notiable delay when opening a new terminal window pre IPA-2.2.


I am not using SSSD for the automounter.


I do notice that the dbmodule for the kerberos server has changed from "kldap"
to "ipadb.so" Perhaps there is some issues with the new library?




Regards,
Siggi


Hello,


I'm not a Kerberos guy, so I can give only general advice:
"Overloaded-CPU-problems" can be troubleshooted with OProfile.


Oprofile is lightweight statistic profiler (AFAIK it was designed for
production environment).

Step-by-step documentation for RHEL 6 is available from:
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.ht
ml#ch-OProfile

As you can see in section 22.5.1., it allows to break whole CPU usage between
processes, libraries and even individual symbols (if proper debuginfos are 
installed).

I recommend to run OProfile on problematic system - results from opreport can
provide missing clue to us.

OProfile gives best results on bare-metal machines. On virtual machines you
has to use timer mode in place of hardware performance counters, please see the 
documentation.


Short getting started guide:
http://oprofile.sourceforge.net/doc/overview.html#getting-started


Nice article with theory&&  examples:
http://people.redhat.com/wcohen/Oprofile.pdf


Homepage with a lot of useful information:
http://oprofile.sourceforge.net/



Thank you.

All 3 IPA servers are close to idle now after switching from kerberos to 
user/pwd bind for the
Linux automounter.

Still there is an issue with kerberos failing to issue a ticket every now and 
then and it's
responding very slowly.

There seem to be low activity on this list just now. Is the kerberos people 
away on vacation?
Hi Siggi,
some people are on vacation, some are busy covering others :-)

Would you be able to take a wireshark trace of an automount going on ?
I would like to see precise timing of packets on the wire to make a
first assesment of where is the bottleneck.

We did change from ldap.so to ipadb.so, but the structure of the drivers
is not much different, so I am surprised it would be much slower,
however it is possible, I would like to find out what is going on with
your help.


OK, I will get that done when I'm back in the office tomorrow. I suspect it will be somewhat better than my first results as the load on the IPA servers are now much lower when the linux automounters are no longer using kerberos for authentication.

It seem like there is a race condition going on as the shit didn't hit the fan until the week after the upgrade to IPA 2.2 when people returned to work. The slowness issues then gradually became worse and worse.

I will send you the captures in a private email. Do you need anything besides TCP 389, 636 and TCP/UDP 88 ?



Rgds,
Siggi

_______________________________________________
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users

Reply via email to