Re: [Freeipa-users] IPA very very slow

Rich Megginson Fri, 12 Jun 2015 15:37:32 -0700

On 06/12/2015 03:25 PM, William Graboyes wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512


Hi Ken,

I ran this command back to back, I am snipping some of the results.

First time I ran the command:

time ldapsearch -x -h 127.0.0.1 "(uid=admin)"
# extended LDIF
#
# LDAPv3
# base <dc=foo,dc=org> (default) with scope subtree
# filter: (uid=admin)
# requesting: ALL
#

- --snip--

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2

real    0m0.056s
user    0m0.003s
sys     0m0.004s


Run on the same server not 5 seconds after the previous command:

time ldapsearch -x -h 127.0.0.1 "(uid=admin)"
# extended LDIF
#
# LDAPv3
# base <dc=foo,dc=org> (default) with scope subtree
# filter: (uid=admin)
# requesting: ALL
#

- -- snip --

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2

real    0m31.756s
user    0m0.003s
sys     0m0.005s

Ok. First, seehttp://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes


You'll also have to do
# debuginfo-install ipa-server slapi-nis
to get all of the ipa packages.

Next, see http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs

Reproduce the problem, and during the 30 seconds the directory server isprocessing the search request, run the gdb command several times to getstack traces during the search request.



I am starting to see this error in the dirserv logs:

[12/Jun/2015:14:06:51 -0700] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[12/Jun/2015:14:11:51 -0700] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[12/Jun/2015:14:16:51 -0700] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)
[12/Jun/2015:14:21:51 -0700] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server) errno 107
(Transport endpoint is not connected)

I doubt this is related to the performance. This looks like the serveris attempting to contact a replica which is down, and has backed off forthe full 5 minute max backoff.


Thanks,
Bill Graboyes


On 6/12/15 1:36 PM, Rich Megginson wrote:

On 06/12/2015 02:10 PM, Martin Kosek wrote:

On 06/12/2015 09:15 PM, William Graboyes wrote:

Hi Martin,

Here are the outputs of the various commands, cleaned of course:

time ldapsearch SASL/EXTERNAL authentication started
ldap_sasl_interactive_bind_s: Unknown authentication method (-6)
additional info: SASL(-4): no mechanism available:

real    0m32.464s user    0m0.385s sys    0m0.052s

This is quite long time. We should check respective dirsrv
errors and access logs snippets.

Also, the command above did not exit successfully, I would
recommend doing at least

# ldapsearch -x -h `hostname` "(uid=admin)"

To eliminate DNS from the equation, use
# time ldapsearch -x -h 127.0.0.1 "(uid=admin)"

time host ipa-server-2.foo.org <-- server with issues
ipa-server-2.foo.org has address 10.0.0.2

real    0m0.070s user    0m0.010s sys    0m0.006s

time host ipa-server-1.foo.org <-- replicant with no issues
ipa-server-1.foo.org has address 10.0.0.3

real    0m0.073s user    0m0.012s sys    0m0.006s

time kinit kinit: Cannot contact any KDC for realm 'FOO.ORG' while
getting initial credentials

real    0m27.049s user    0m0.013s sys    0m0.004s

^^^ has been something I have been seeing intermittently



On 6/12/15 12:11 AM, Martin Kosek wrote:

Hi List,

This is a problem that has surfaced after a reboot of
this system in particular. It is being really, really
slow.  In terms of hardware usage issues, there are none.
It is taking 3-5 minutes to list users in the gui.
Running commands like ipa-replica-manage list is taking
between 30seconds and 3 minutes.  Memory usage is low,
cpu usage is low, iops are low.  I really have no idea
where to start here, there is noting really damning in
the logs.  I have tried restarting IPA (ipactl restart)
stopping and starting IPA (ipactl stop wait... ipactl
start), and rebooting the entire server.

The oddest thing is that there have been some krb errors
saying that they cannot contact the krb server.. logging
into the gui saying your session has timed out..

It is just general strangeness.

ipa-server-4.1.0-18.el7.centos.3.x86_64
sssd-ipa-1.12.2-58.el7_1.6.x86_64
krb5-server-1.12.2-14.el7.x86_64

Any help would be greatly appreciated.

Thanks, Bill

I would recommend starting with simple things, seeing the
performance and then following with more complex stuff:

- Try bare "ldapsearch" against the FreeIPA LDAP server,
see the response rate. If it is also slow, we have the root
cause. Before ringing on DS people doors, see if for
example DNS is not slow and there are no DNS timeouts in
play - "host ipa.server.test" will tell you that

- If DS is OK, try Kerberos - kinit, kvno commands

- If Kerberos is also OK and "ipa-replica-manage list" is
still slow, maybe we should just "strace" it to see what it
waits on.

HTH, Martin


--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA very very slow

Reply via email to