Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
Roughly 128 clients.. But when the services start going gaa-gaa it also causes time-outs with the naming service running on the same server. I may be wrong, but wouldn't that basically kill any chance of a client connecting to the server anyway? I'm just lucky that the clients aren't really there for general usage, so the only times that users actually try to log in is when it is support staff that needs to check or restart a pretty static, stable service. I just want to add that the server is not hung when doing an ipactl status and the KDC service is not started as below, so are we barking up the right tree looking at the DS service? (PS, the KDC service was not manually stopped by me. *sudo ipactl status* *Directory Service: RUNNING* *KDC Service: STOPPED* *KPASSWD Service: RUNNING* *DNS Service: RUNNING* *MEMCACHE Service: RUNNING* *HTTP Service: RUNNING* *CA Service: RUNNING* *ADTRUST Service: RUNNING* *EXTID Service: RUNNING* On Thu, Nov 13, 2014 at 12:27 PM, Ludwig Krispenz lkris...@redhat.com wrote: Hmm, the symbols are there now, but all threads are idle, DS is just waiting on work to do. Which client do you expect to connect to DS, maybe you need to debug this client. On 11/13/2014 11:02 AM, Walter van Lille wrote: Thanks Rich, I have installed the packages and run gdb again. Hopefully the attached file is more useful. On Wed, Nov 12, 2014 at 4:16 PM, Rich Megginson rmegg...@redhat.com wrote: On 11/12/2014 05:42 AM, Walter van Lille wrote: Thanks again for the assistance guys. I have saved two files and included it here. Hope it tells you more than it does me. These stack traces contain no useful symbols. Please read http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes to find out how to install the correct debuginfo packages on your system. For IPA you will also need debuginfo packages for ipa and slapi-nis If debuginfo-install is not working, see https://www.centos.org/forums/viewtopic.php?t=1710 If you simply cannot figure out how to install the debuginfo packages, please email me with the following information: $ rpm -q ipa-server slapi-nis 389-ds-base openldap db4 nss nspr glibc and I will make them available to you NOTE: With debuginfo packages, the version of the debuginfo package _must exactly match_ the version of the associated package e.g. for 389-ds-base.x86_64 1.2.11.15-34.el6_5 you must have 389-ds-base-debuginfo.x86_64 1.2.11.15-34.el6_5 If the versions do not match exactly, you will not get the symbols. Regards, Wallter On Tue, Nov 11, 2014 at 8:03 PM, Rich Megginson rmegg...@redhat.com wrote: On 11/11/2014 10:37 AM, Martin Basti wrote: On 11/11/14 15:58, Rich Megginson wrote: On 11/11/2014 06:20 AM, Ludwig Krispenz wrote: On 11/11/2014 02:14 PM, Martin Basti wrote: Ludiwg (CCed) this seems like old (fixed?) DS bug. hmm, it says limit is 2097152, so it already has the new setting, but the error message says the packet is 800MB *Right. That usually means the server was expecting an encrypted SASL buffer from the client, but instead the client thinks SASL encryption negotiation failed and just sent a plain LDAP buffer. What version of 389-ds-base are you using? rpm -q 389-ds-base https://fedorahosted.org/389/ticket/47416 https://fedorahosted.org/389/ticket/47416 So, DO NOT increase your sasl io buffer size - it will not fix the problem, and it will leave you open to DoS attacks. * He is using CentOS release 6.5 (Final) 389-ds-base.x86_64 1.2.11.15-34.el6_5 Ignore the SASL encrypted packet length exceeds maximum allowed limit error. All it means is the client has some error and is canceling the connection. The bugzilla associated with *47416 https://fedorahosted.org/389/ticket/47416 is targeted for RHEL 7.1. The problem is that we were never able to figure out what error the client was getting that caused the client to stop establishing the *SASL encryption, and the original customer who reported the problem dropped the case - everything just started working, no further errors. Note that 47416 doesn't really fix the problem or address the root cause - the root cause is some error on the client side that causes it to stop doing encrypted SASL I/O and send an LDAP unbind request. Let's get back to the actual problem - what is the actual problem? The ldap server is hanging or unresponsive? If so, then see http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs. Let's get some dirsrv stack traces during the period of time when it appears to be unresponsive. On 11/11/14 13:13, Walter van Lille wrote: I've just cleaned out a ton of slapd_poll timed out messages from the output and changed the names to protect the innocent, :-) Here is the output as requested: *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds maximum allowed limit (length=805565, limit=2097152). Change the nsslapd-maxsasliosize
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
Hmm, the symbols are there now, but all threads are idle, DS is just waiting on work to do. Which client do you expect to connect to DS, maybe you need to debug this client. On 11/13/2014 11:02 AM, Walter van Lille wrote: Thanks Rich, I have installed the packages and run gdb again. Hopefully the attached file is more useful. On Wed, Nov 12, 2014 at 4:16 PM, Rich Megginson rmegg...@redhat.com mailto:rmegg...@redhat.com wrote: On 11/12/2014 05:42 AM, Walter van Lille wrote: Thanks again for the assistance guys. I have saved two files and included it here. Hope it tells you more than it does me. These stack traces contain no useful symbols. Please read http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes to find out how to install the correct debuginfo packages on your system. For IPA you will also need debuginfo packages for ipa and slapi-nis If debuginfo-install is not working, see https://www.centos.org/forums/viewtopic.php?t=1710 If you simply cannot figure out how to install the debuginfo packages, please email me with the following information: $ rpm -q ipa-server slapi-nis 389-ds-base openldap db4 nss nspr glibc and I will make them available to you NOTE: With debuginfo packages, the version of the debuginfo package _must exactly match_ the version of the associated package e.g. for 389-ds-base.x86_64 1.2.11.15-34.el6_5 you must have 389-ds-base-debuginfo.x86_64 1.2.11.15-34.el6_5 If the versions do not match exactly, you will not get the symbols. Regards, Wallter On Tue, Nov 11, 2014 at 8:03 PM, Rich Megginson rmegg...@redhat.com mailto:rmegg...@redhat.com wrote: On 11/11/2014 10:37 AM, Martin Basti wrote: On 11/11/14 15:58, Rich Megginson wrote: On 11/11/2014 06:20 AM, Ludwig Krispenz wrote: On 11/11/2014 02:14 PM, Martin Basti wrote: Ludiwg (CCed) this seems like old (fixed?) DS bug. hmm, it says limit is 2097152, so it already has the new setting, but the error message says the packet is 800MB* * *Right. That usually means the server was expecting an encrypted SASL buffer from the client, but instead the client thinks SASL encryption negotiation failed and just sent a plain LDAP buffer. What version of 389-ds-base are you using? rpm -q 389-ds-base https://fedorahosted.org/389/ticket/47416 So, DO NOT increase your sasl io buffer size - it will not fix the problem, and it will leave you open to DoS attacks. * He is using CentOS release 6.5 (Final) 389-ds-base.x86_64 1.2.11.15-34.el6_5 Ignore the SASL encrypted packet length exceeds maximum allowed limit error. All it means is the client has some error and is canceling the connection. The bugzilla associated with *47416 https://fedorahosted.org/389/ticket/47416 is targeted for RHEL 7.1. The problem is that we were never able to figure out what error the client was getting that caused the client to stop establishing the *SASL encryption, and the original customer who reported the problem dropped the case - everything just started working, no further errors. Note that 47416 doesn't really fix the problem or address the root cause - the root cause is some error on the client side that causes it to stop doing encrypted SASL I/O and send an LDAP unbind request. Let's get back to the actual problem - what is the actual problem? The ldap server is hanging or unresponsive? If so, then see http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs. Let's get some dirsrv stack traces during the period of time when it appears to be unresponsive. * * ** On 11/11/14 13:13, Walter van Lille wrote: I've just cleaned out a ton of slapd_poll timed out messages from the output and changed the names to protect the innocent, :-) Here is the output as requested: *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds maximum allowed limit (length=805565, limit=2097152). Change the nsslapd-maxsasliosize attribute in cn=config to increase limit.* * * *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out* *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL security on connection in CLOSING state* *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers from connection* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling operation threads* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for 30 threads to terminate*
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
On 11/13/2014 03:02 AM, Walter van Lille wrote: Thanks Rich, I have installed the packages and run gdb again. Hopefully the attached file is more useful. The symbols are there. However, the server is almost completely idle - no hangs, no deadlocks, no waiting on I/O. You must catch dirsrv while it is hung. You might want to run that gdb script every few seconds during the time it appears to be hung. On Wed, Nov 12, 2014 at 4:16 PM, Rich Megginson rmegg...@redhat.com mailto:rmegg...@redhat.com wrote: On 11/12/2014 05:42 AM, Walter van Lille wrote: Thanks again for the assistance guys. I have saved two files and included it here. Hope it tells you more than it does me. These stack traces contain no useful symbols. Please read http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes to find out how to install the correct debuginfo packages on your system. For IPA you will also need debuginfo packages for ipa and slapi-nis If debuginfo-install is not working, see https://www.centos.org/forums/viewtopic.php?t=1710 If you simply cannot figure out how to install the debuginfo packages, please email me with the following information: $ rpm -q ipa-server slapi-nis 389-ds-base openldap db4 nss nspr glibc and I will make them available to you NOTE: With debuginfo packages, the version of the debuginfo package _must exactly match_ the version of the associated package e.g. for 389-ds-base.x86_64 1.2.11.15-34.el6_5 you must have 389-ds-base-debuginfo.x86_64 1.2.11.15-34.el6_5 If the versions do not match exactly, you will not get the symbols. Regards, Wallter On Tue, Nov 11, 2014 at 8:03 PM, Rich Megginson rmegg...@redhat.com mailto:rmegg...@redhat.com wrote: On 11/11/2014 10:37 AM, Martin Basti wrote: On 11/11/14 15:58, Rich Megginson wrote: On 11/11/2014 06:20 AM, Ludwig Krispenz wrote: On 11/11/2014 02:14 PM, Martin Basti wrote: Ludiwg (CCed) this seems like old (fixed?) DS bug. hmm, it says limit is 2097152, so it already has the new setting, but the error message says the packet is 800MB* * *Right. That usually means the server was expecting an encrypted SASL buffer from the client, but instead the client thinks SASL encryption negotiation failed and just sent a plain LDAP buffer. What version of 389-ds-base are you using? rpm -q 389-ds-base https://fedorahosted.org/389/ticket/47416 So, DO NOT increase your sasl io buffer size - it will not fix the problem, and it will leave you open to DoS attacks. * He is using CentOS release 6.5 (Final) 389-ds-base.x86_64 1.2.11.15-34.el6_5 Ignore the SASL encrypted packet length exceeds maximum allowed limit error. All it means is the client has some error and is canceling the connection. The bugzilla associated with *47416 https://fedorahosted.org/389/ticket/47416 is targeted for RHEL 7.1. The problem is that we were never able to figure out what error the client was getting that caused the client to stop establishing the *SASL encryption, and the original customer who reported the problem dropped the case - everything just started working, no further errors. Note that 47416 doesn't really fix the problem or address the root cause - the root cause is some error on the client side that causes it to stop doing encrypted SASL I/O and send an LDAP unbind request. Let's get back to the actual problem - what is the actual problem? The ldap server is hanging or unresponsive? If so, then see http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs. Let's get some dirsrv stack traces during the period of time when it appears to be unresponsive. * * ** On 11/11/14 13:13, Walter van Lille wrote: I've just cleaned out a ton of slapd_poll timed out messages from the output and changed the names to protect the innocent, :-) Here is the output as requested: *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds maximum allowed limit (length=805565, limit=2097152). Change the nsslapd-maxsasliosize attribute in cn=config to increase limit.* * * *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out* *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL security on connection in CLOSING state* *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers from connection* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling operation threads* *[11/Nov/2014:11:48:09 +0200] - slapd
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
IMHO It's DS bug, can you share DS error log? pspacek CCed to examine named logs. Martin^2 On 11/11/14 12:13, Walter van Lille wrote: Hi Martin, thanks for the reply. My version: bind-dyndb-ldap-2.3-5.el6.x86_64 The server doesn't have journalctl installed but I have the outputs from the messages and named.run files that I included here: Messages: *Nov 11 12:30:13 freeipa named[1481]: error (network unreachable) resolving 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* Named.run: *client 10.123.123.123#42639: transfer of 'example.example/IN': AXFR-style IXFR started* *client 10.123.123.123#42639: transfer of ''example.example/IN': AXFR-style IXFR ended* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR started* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR ended* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* I just replaced the IPs and the actual names with something more generic. Regards, Walter On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti mba...@redhat.com mailto:mba...@redhat.com wrote: On 06/11/14 14:58, Walter van Lille wrote: Hi, I need some assistance please. I've taken over an IPA server to manage a few months ago, and it was working fine until recently when it started acting up seemingly off its own accord. When I do an ipactl status it basically gives an output as shown below: *Directory Service: RUNNING * * * *Loong pause... (To the tune of 7 minutes sometimes)* * * *KDC Service: RUNNING* *KPASSWD Service: RUNNING* *DNS Service: RUNNING* *MEMCACHE Service: RUNNING* *HTTP Service: RUNNING* *CA Service: RUNNING* *ADTRUST Service: RUNNING* *EXTID Service: RUNNING* Running top showed that ns-slapd was munching almost all my resources, but I got that fixed by upping the cache. Unfortunately this did not correct the issue and it still reacts in the same fashion, although the resources have been freed up now. I've noticed that when I run dig on either the local server or a remote machine that the query basically just times out as shown here: *dig freeipa.myexample.sample* * * *; DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 freeipa.myexample.sample* *;; global options: +cmd* *;; connection timed out; no servers could be reached* When the KDC service fails to start, then name lookups seem OK, but authentication fails. otherwise it's dead in the water. This also happens: *sudo ipactl status* *Directory Service: RUNNING* *Unknown error when retrieving list of services from LDAP:* * * My software setup is as follows: *CentOS release 6.5 (Final) * *389-ds-base.x86_64 1.2.11.15-34.el6_5 * *bind.x86_64 32:9.8.2-0.23.rc1.el6_5.1 * *bind-dyndb-ldap.x86_64* *bind-libs.x86_64 32:9.8.2-0.23.rc1.el6_5.1* *bind-utils.x86_64 32:9.8.2-0.23.rc1.el6_5.1* *rpcbind.x86_64 0.2.0-11.el6 @anaconda-CentOS-201311291202.x86_64/6.5* *samba4-winbind.x86_64* *krb5-server.x86_64 1.10.3-15.el6_5.1 * * * *Linux 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux * It's not a permanent situation as it sometimes runs 100% for a while, but 80% of the time it is unusable. If anybody can assist me, please be so kind. Regards, Walter Hello please which version of bind-dyndb-ldap do you use? I had similar issue with bind-dyndb-ldap, but it was development version, I'm not sure if this is your case. When named was failing, dirserv was really slow. Can you send journalctl -b -u named log when dig doesn't work?? -- Martin Basti -- Martin Basti -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
I've just cleaned out a ton of slapd_poll timed out messages from the output and changed the names to protect the innocent, :-) Here is the output as requested: *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds maximum allowed limit (length=805634565, limit=2097152). Change the nsslapd-maxsasliosize attribute in cn=config to increase limit.* *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out* *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL security on connection in CLOSING state* *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers from connection* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling operation threads* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for 30 threads to terminate* *[11/Nov/2014:13:14:12 +0200] - slapd shutting down - closing down internal subsystems and plugins* *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database threads to stop* *[11/Nov/2014:13:14:13 +0200] - All database threads now stopped* *[11/Nov/2014:13:14:13 +0200] - slapd stopped.* *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15 http://1.2.11.15 B2014.219.179 starting up* *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=sample,dc=example* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - slapd started. Listening on All Interfaces port 389 for LDAP requests* *[11/Nov/2014:13:26:36 +0200] - Listening on All Interfaces port 636 for LDAPS requests* *[11/Nov/2014:13:26:36 +0200] - Listening on /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests* *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out* On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti mba...@redhat.com wrote: IMHO It's DS bug, can you share DS error log? pspacek CCed to examine named logs. Martin^2 On 11/11/14 12:13, Walter van Lille wrote: Hi Martin, thanks for the reply. My version: bind-dyndb-ldap-2.3-5.el6.x86_64 The server doesn't have journalctl installed but I have the outputs from the messages and named.run files that I included here: Messages: *Nov 11 12:30:13 freeipa named[1481]: error (network unreachable) resolving 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* Named.run: *client 10.123.123.123#42639: transfer of 'example.example/IN': AXFR-style IXFR started* *client 10.123.123.123#42639: transfer of ''example.example/IN': AXFR-style IXFR ended* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR started* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR ended* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* I just replaced the IPs and the actual names with something more generic. Regards, Walter On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti mba...@redhat.com wrote: On 06/11/14 14:58, Walter van Lille wrote: Hi, I need some assistance please. I've taken over an IPA server to manage a few months ago, and it was working fine until recently when it started acting up seemingly off its own accord. When I do an ipactl status it basically gives an output as shown below: *Directory Service: RUNNING * *Loong pause... (To the tune of 7 minutes sometimes)* *KDC Service: RUNNING* *KPASSWD Service: RUNNING* *DNS Service: RUNNING* *MEMCACHE Service: RUNNING* *HTTP Service: RUNNING* *CA Service: RUNNING* *ADTRUST Service: RUNNING* *EXTID Service: RUNNING* Running top showed that ns-slapd was munching almost all my resources, but I got that fixed by upping the cache. Unfortunately this did not correct the issue and it still reacts in the same fashion, although the resources have been freed up now. I've noticed that when I run dig on either the local server or a remote machine that the query basically just times out as shown here: *dig freeipa.myexample.sample* *; DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 freeipa.myexample.sample* *;; global options: +cmd* *;; connection timed out; no servers could be reached* When
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
On 11.11.2014 13:13, Walter van Lille wrote: SASL encrypted packet length exceeds maximum allowed limit Martin, do you remember where is the appropriate knob? -- Petr^2 Spacek -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
Ludiwg (CCed) this seems like old (fixed?) DS bug. On 11/11/14 13:13, Walter van Lille wrote: I've just cleaned out a ton of slapd_poll timed out messages from the output and changed the names to protect the innocent, :-) Here is the output as requested: *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds maximum allowed limit (length=805634565, limit=2097152). Change the nsslapd-maxsasliosize attribute in cn=config to increase limit.* * * *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out* *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL security on connection in CLOSING state* *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers from connection* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling operation threads* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for 30 threads to terminate* *[11/Nov/2014:13:14:12 +0200] - slapd shutting down - closing down internal subsystems and plugins* *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database threads to stop* *[11/Nov/2014:13:14:13 +0200] - All database threads now stopped* *[11/Nov/2014:13:14:13 +0200] - slapd stopped.* *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15 http://1.2.11.15 B2014.219.179 starting up* *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=sample,dc=example* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - slapd started. Listening on All Interfaces port 389 for LDAP requests* *[11/Nov/2014:13:26:36 +0200] - Listening on All Interfaces port 636 for LDAPS requests* *[11/Nov/2014:13:26:36 +0200] - Listening on /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests* *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out* * * * * * * On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti mba...@redhat.com mailto:mba...@redhat.com wrote: IMHO It's DS bug, can you share DS error log? pspacek CCed to examine named logs. Martin^2 On 11/11/14 12:13, Walter van Lille wrote: Hi Martin, thanks for the reply. My version: bind-dyndb-ldap-2.3-5.el6.x86_64 The server doesn't have journalctl installed but I have the outputs from the messages and named.run files that I included here: Messages: *Nov 11 12:30:13 freeipa named[1481]: error (network unreachable) resolving 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* Named.run: *client 10.123.123.123#42639: transfer of 'example.example/IN': AXFR-style IXFR started* *client 10.123.123.123#42639: transfer of ''example.example/IN': AXFR-style IXFR ended* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR started* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR ended* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* I just replaced the IPs and the actual names with something more generic. Regards, Walter On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti mba...@redhat.com mailto:mba...@redhat.com wrote: On 06/11/14 14:58, Walter van Lille wrote: Hi, I need some assistance please. I've taken over an IPA server to manage a few months ago, and it was working fine until recently when it started acting up seemingly off its own accord. When I do an ipactl status it basically gives an output as shown below: *Directory Service: RUNNING * * * *Loong pause... (To the tune of 7 minutes sometimes)* * * *KDC Service: RUNNING* *KPASSWD Service: RUNNING* *DNS Service: RUNNING* *MEMCACHE Service: RUNNING* *HTTP Service: RUNNING* *CA Service: RUNNING* *ADTRUST Service: RUNNING* *EXTID Service: RUNNING* Running top showed that ns-slapd was munching almost all my resources, but I got that fixed by upping the cache.
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
On 11/11/2014 01:29 PM, Petr Spacek wrote: On 11.11.2014 13:13, Walter van Lille wrote: SASL encrypted packet length exceeds maximum allowed limit Martin, do you remember where is the appropriate knob? Do you mean nsslapd-sasl-max-buffer-size setting in cn=config? This is a related ticket https://fedorahosted.org/389/ticket/47457 This is the public, RHEL-7.1 variant of it: https://bugzilla.redhat.com/show_bug.cgi?id=1044193 Martin -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
On 11/11/2014 02:14 PM, Martin Basti wrote: Ludiwg (CCed) this seems like old (fixed?) DS bug. hmm, it says limit is 2097152, so it already has the new setting, but the error message says the packet is 800MB* * On 11/11/14 13:13, Walter van Lille wrote: I've just cleaned out a ton of slapd_poll timed out messages from the output and changed the names to protect the innocent, :-) Here is the output as requested: *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds maximum allowed limit (length=805565, limit=2097152). Change the nsslapd-maxsasliosize attribute in cn=config to increase limit.* * * *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out* *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL security on connection in CLOSING state* *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers from connection* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling operation threads* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for 30 threads to terminate* *[11/Nov/2014:13:14:12 +0200] - slapd shutting down - closing down internal subsystems and plugins* *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database threads to stop* *[11/Nov/2014:13:14:13 +0200] - All database threads now stopped* *[11/Nov/2014:13:14:13 +0200] - slapd stopped.* *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15 http://1.2.11.15 B2014.219.179 starting up* *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=sample,dc=example* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - slapd started. Listening on All Interfaces port 389 for LDAP requests* *[11/Nov/2014:13:26:36 +0200] - Listening on All Interfaces port 636 for LDAPS requests* *[11/Nov/2014:13:26:36 +0200] - Listening on /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests* *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out* * * * * * * On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti mba...@redhat.com mailto:mba...@redhat.com wrote: IMHO It's DS bug, can you share DS error log? pspacek CCed to examine named logs. Martin^2 On 11/11/14 12:13, Walter van Lille wrote: Hi Martin, thanks for the reply. My version: bind-dyndb-ldap-2.3-5.el6.x86_64 The server doesn't have journalctl installed but I have the outputs from the messages and named.run files that I included here: Messages: *Nov 11 12:30:13 freeipa named[1481]: error (network unreachable) resolving 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* Named.run: *client 10.123.123.123#42639: transfer of 'example.example/IN': AXFR-style IXFR started* *client 10.123.123.123#42639: transfer of ''example.example/IN': AXFR-style IXFR ended* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR started* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR ended* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* I just replaced the IPs and the actual names with something more generic. Regards, Walter On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti mba...@redhat.com mailto:mba...@redhat.com wrote: On 06/11/14 14:58, Walter van Lille wrote: Hi, I need some assistance please. I've taken over an IPA server to manage a few months ago, and it was working fine until recently when it started acting up seemingly off its own accord. When I do an ipactl status it basically gives an output as shown below: *Directory Service: RUNNING * * * *Loong pause... (To the tune of 7 minutes sometimes)* * * *KDC Service: RUNNING* *KPASSWD Service: RUNNING* *DNS Service: RUNNING* *MEMCACHE Service: RUNNING* *HTTP Service: RUNNING* *CA Service: RUNNING* *ADTRUST Service: RUNNING*
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
On 11/11/2014 06:20 AM, Ludwig Krispenz wrote: On 11/11/2014 02:14 PM, Martin Basti wrote: Ludiwg (CCed) this seems like old (fixed?) DS bug. hmm, it says limit is 2097152, so it already has the new setting, but the error message says the packet is 800MB* * *Right. That usually means the server was expecting an encrypted SASL buffer from the client, but instead the client thinks SASL encryption negotiation failed and just sent a plain LDAP buffer. What version of 389-ds-base are you using? rpm -q 389-ds-base https://fedorahosted.org/389/ticket/47416 So, DO NOT increase your sasl io buffer size - it will not fix the problem, and it will leave you open to DoS attacks. * ** On 11/11/14 13:13, Walter van Lille wrote: I've just cleaned out a ton of slapd_poll timed out messages from the output and changed the names to protect the innocent, :-) Here is the output as requested: *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds maximum allowed limit (length=805565, limit=2097152). Change the nsslapd-maxsasliosize attribute in cn=config to increase limit.* * * *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out* *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL security on connection in CLOSING state* *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers from connection* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling operation threads* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for 30 threads to terminate* *[11/Nov/2014:13:14:12 +0200] - slapd shutting down - closing down internal subsystems and plugins* *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database threads to stop* *[11/Nov/2014:13:14:13 +0200] - All database threads now stopped* *[11/Nov/2014:13:14:13 +0200] - slapd stopped.* *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15 http://1.2.11.15 B2014.219.179 starting up* *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=sample,dc=example* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - slapd started. Listening on All Interfaces port 389 for LDAP requests* *[11/Nov/2014:13:26:36 +0200] - Listening on All Interfaces port 636 for LDAPS requests* *[11/Nov/2014:13:26:36 +0200] - Listening on /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests* *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out* * * * * * * On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti mba...@redhat.com mailto:mba...@redhat.com wrote: IMHO It's DS bug, can you share DS error log? pspacek CCed to examine named logs. Martin^2 On 11/11/14 12:13, Walter van Lille wrote: Hi Martin, thanks for the reply. My version: bind-dyndb-ldap-2.3-5.el6.x86_64 The server doesn't have journalctl installed but I have the outputs from the messages and named.run files that I included here: Messages: *Nov 11 12:30:13 freeipa named[1481]: error (network unreachable) resolving 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* Named.run: *client 10.123.123.123#42639: transfer of 'example.example/IN': AXFR-style IXFR started* *client 10.123.123.123#42639: transfer of ''example.example/IN': AXFR-style IXFR ended* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR started* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR ended* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* I just replaced the IPs and the actual names with something more generic. Regards, Walter On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti mba...@redhat.com mailto:mba...@redhat.com wrote: On 06/11/14 14:58, Walter van Lille wrote: Hi, I need some assistance please. I've taken over an IPA server to manage a few months ago, and it was working fine until recently when it started acting up seemingly off its own accord. When I do an ipactl status it
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
On 11/11/14 15:58, Rich Megginson wrote: On 11/11/2014 06:20 AM, Ludwig Krispenz wrote: On 11/11/2014 02:14 PM, Martin Basti wrote: Ludiwg (CCed) this seems like old (fixed?) DS bug. hmm, it says limit is 2097152, so it already has the new setting, but the error message says the packet is 800MB* * *Right. That usually means the server was expecting an encrypted SASL buffer from the client, but instead the client thinks SASL encryption negotiation failed and just sent a plain LDAP buffer. What version of 389-ds-base are you using? rpm -q 389-ds-base https://fedorahosted.org/389/ticket/47416 So, DO NOT increase your sasl io buffer size - it will not fix the problem, and it will leave you open to DoS attacks. * He is using CentOS release 6.5 (Final) 389-ds-base.x86_64 1.2.11.15-34.el6_5 * * ** On 11/11/14 13:13, Walter van Lille wrote: I've just cleaned out a ton of slapd_poll timed out messages from the output and changed the names to protect the innocent, :-) Here is the output as requested: *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds maximum allowed limit (length=805565, limit=2097152). Change the nsslapd-maxsasliosize attribute in cn=config to increase limit.* * * *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out* *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL security on connection in CLOSING state* *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers from connection* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling operation threads* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for 30 threads to terminate* *[11/Nov/2014:13:14:12 +0200] - slapd shutting down - closing down internal subsystems and plugins* *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database threads to stop* *[11/Nov/2014:13:14:13 +0200] - All database threads now stopped* *[11/Nov/2014:13:14:13 +0200] - slapd stopped.* *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15 http://1.2.11.15 B2014.219.179 starting up* *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=sample,dc=example* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - slapd started. Listening on All Interfaces port 389 for LDAP requests* *[11/Nov/2014:13:26:36 +0200] - Listening on All Interfaces port 636 for LDAPS requests* *[11/Nov/2014:13:26:36 +0200] - Listening on /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests* *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out* * * * * * * On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti mba...@redhat.com mailto:mba...@redhat.com wrote: IMHO It's DS bug, can you share DS error log? pspacek CCed to examine named logs. Martin^2 On 11/11/14 12:13, Walter van Lille wrote: Hi Martin, thanks for the reply. My version: bind-dyndb-ldap-2.3-5.el6.x86_64 The server doesn't have journalctl installed but I have the outputs from the messages and named.run files that I included here: Messages: *Nov 11 12:30:13 freeipa named[1481]: error (network unreachable) resolving 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* Named.run: *client 10.123.123.123#42639: transfer of 'example.example/IN': AXFR-style IXFR started* *client 10.123.123.123#42639: transfer of ''example.example/IN': AXFR-style IXFR ended* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR started* *client 10.123.123.123#46912: transfer of '10.123.123.123.in-addr.arpa/IN': AXFR-style IXFR ended* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* *LDAP query timed out. Try to adjust timeout parameter* I just replaced the IPs and the actual names with something more generic. Regards, Walter On Thu, Nov 6, 2014 at 5:00 PM, Martin Basti mba...@redhat.com mailto:mba...@redhat.com wrote: On 06/11/14 14:58, Walter van Lille wrote: Hi, I need some assistance please. I've taken over an IPA server to manage a few months ago, and it was
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
On 11/11/2014 10:37 AM, Martin Basti wrote: On 11/11/14 15:58, Rich Megginson wrote: On 11/11/2014 06:20 AM, Ludwig Krispenz wrote: On 11/11/2014 02:14 PM, Martin Basti wrote: Ludiwg (CCed) this seems like old (fixed?) DS bug. hmm, it says limit is 2097152, so it already has the new setting, but the error message says the packet is 800MB* * *Right. That usually means the server was expecting an encrypted SASL buffer from the client, but instead the client thinks SASL encryption negotiation failed and just sent a plain LDAP buffer. What version of 389-ds-base are you using? rpm -q 389-ds-base https://fedorahosted.org/389/ticket/47416 So, DO NOT increase your sasl io buffer size - it will not fix the problem, and it will leave you open to DoS attacks. * He is using CentOS release 6.5 (Final) 389-ds-base.x86_64 1.2.11.15-34.el6_5 Ignore the SASL encrypted packet length exceeds maximum allowed limit error. All it means is the client has some error and is canceling the connection. The bugzilla associated with *47416 is targeted for RHEL 7.1. The problem is that we were never able to figure out what error the client was getting that caused the client to stop establishing the *SASL encryption, and the original customer who reported the problem dropped the case - everything just started working, no further errors. Note that 47416 doesn't really fix the problem or address the root cause - the root cause is some error on the client side that causes it to stop doing encrypted SASL I/O and send an LDAP unbind request. Let's get back to the actual problem - what is the actual problem? The ldap server is hanging or unresponsive? If so, then see http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-hangs. Let's get some dirsrv stack traces during the period of time when it appears to be unresponsive. * * ** On 11/11/14 13:13, Walter van Lille wrote: I've just cleaned out a ton of slapd_poll timed out messages from the output and changed the names to protect the innocent, :-) Here is the output as requested: *[05/Nov/2014:11:44:05 +0200] - SASL encrypted packet length exceeds maximum allowed limit (length=805565, limit=2097152). Change the nsslapd-maxsasliosize attribute in cn=config to increase limit.* * * *[10/Nov/2014:14:45:19 +0200] - slapd_poll(115) timed out* *[10/Nov/2014:14:45:19 +0200] sasl_io_enable - Cannot enable SASL security on connection in CLOSING state* *[10/Nov/2014:14:45:19 +0200] - Error: could not add/remove IO layers from connection* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - signaling operation threads* *[11/Nov/2014:11:48:09 +0200] - slapd shutting down - waiting for 30 threads to terminate* *[11/Nov/2014:13:14:12 +0200] - slapd shutting down - closing down internal subsystems and plugins* *[11/Nov/2014:13:14:12 +0200] - Waiting for 4 database threads to stop* *[11/Nov/2014:13:14:13 +0200] - All database threads now stopped* *[11/Nov/2014:13:14:13 +0200] - slapd stopped.* *[11/Nov/2014:13:26:35 +0200] - 389-Directory/1.2.11.15 http://1.2.11.15 B2014.219.179 starting up* *[11/Nov/2014:13:26:35 +0200] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=sample,dc=example* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=sample,dc=example--no CoS Templates found, which should be added before the CoS Definition.* *[11/Nov/2014:13:26:36 +0200] - slapd started. Listening on All Interfaces port 389 for LDAP requests* *[11/Nov/2014:13:26:36 +0200] - Listening on All Interfaces port 636 for LDAPS requests* *[11/Nov/2014:13:26:36 +0200] - Listening on /var/run/slapd-SAMPLE-EXAMPLE.socket for LDAPI requests* *[11/Nov/2014:13:57:08 +0200] - slapd_poll(78) timed out* * * * * * * On Tue, Nov 11, 2014 at 1:19 PM, Martin Basti mba...@redhat.com mailto:mba...@redhat.com wrote: IMHO It's DS bug, can you share DS error log? pspacek CCed to examine named logs. Martin^2 On 11/11/14 12:13, Walter van Lille wrote: Hi Martin, thanks for the reply. My version: bind-dyndb-ldap-2.3-5.el6.x86_64 The server doesn't have journalctl installed but I have the outputs from the messages and named.run files that I included here: Messages: *Nov 11 12:30:13 freeipa named[1481]: error (network unreachable) resolving 'example.example.com.10.123.123.123/A/IN': 2001:500:2f::f#53* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:23 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query timed out. Try to adjust timeout parameter* *Nov 11 12:30:33 freeipa named[1481]: LDAP query
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
On 6.11.2014 16:41, Dmitri Pal wrote: On 11/06/2014 10:00 AM, Martin Basti wrote: On 06/11/14 14:58, Walter van Lille wrote: Hi, I need some assistance please. I've taken over an IPA server to manage a few months ago, and it was working fine until recently when it started acting up seemingly off its own accord. When I do an ipactl status it basically gives an output as shown below: *Directory Service: RUNNING * * * *Loong pause... (To the tune of 7 minutes sometimes)* * * *KDC Service: RUNNING* *KPASSWD Service: RUNNING* *DNS Service: RUNNING* *MEMCACHE Service: RUNNING* *HTTP Service: RUNNING* *CA Service: RUNNING* *ADTRUST Service: RUNNING* *EXTID Service: RUNNING* Running top showed that ns-slapd was munching almost all my resources, but I got that fixed by upping the cache. Unfortunately this did not correct the issue and it still reacts in the same fashion, although the resources have been freed up now. I've noticed that when I run dig on either the local server or a remote machine that the query basically just times out as shown here: *dig freeipa.myexample.sample* * * *; DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 freeipa.myexample.sample* *;; global options: +cmd* *;; connection timed out; no servers could be reached* When the KDC service fails to start, then name lookups seem OK, but authentication fails. otherwise it's dead in the water. This also happens: *sudo ipactl status* *Directory Service: RUNNING* *Unknown error when retrieving list of services from LDAP:* * * My software setup is as follows: *CentOS release 6.5 (Final) * *389-ds-base.x86_64 1.2.11.15-34.el6_5 * *bind.x86_64 32:9.8.2-0.23.rc1.el6_5.1 * *bind-dyndb-ldap.x86_64* *bind-libs.x86_64 32:9.8.2-0.23.rc1.el6_5.1* *bind-utils.x86_6432:9.8.2-0.23.rc1.el6_5.1* *rpcbind.x86_64 0.2.0-11.el6 @anaconda-CentOS-201311291202.x86_64/6.5* *samba4-winbind.x86_64* *krb5-server.x86_64 1.10.3-15.el6_5.1 * * * *Linux 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux * It's not a permanent situation as it sometimes runs 100% for a while, but 80% of the time it is unusable. If anybody can assist me, please be so kind. Regards, Walter Hello please which version of bind-dyndb-ldap do you use? I had similar issue with bind-dyndb-ldap, but it was development version, I'm not sure if this is your case. When named was failing, dirserv was really slow. Can you send journalctl -b -u named log when dig doesn't work?? -- Martin Basti You also want to look at the directory server logs especially at startup and see what is it doing. Also check the diskspace. May be you do not have much room on the volume and it might cause DS to slow down. One thing to keep in mind: FreeIPA DNS service is backed by LDAP so can't possibly work if your LDAP server is down or is unresponsive. If you encounter a problem with DNS again please try to follow steps 1-5 described on page https://fedorahosted.org/bind-dyndb-ldap/wiki/BIND9/NamedCannotStart . I'm mainly interested in results obtained in step 5 (other steps are just prerequisite). It will tell us if DNS (bind-dyndb-ldap) is broken or if LDAP server does not respond and DNS is failing because of the cascade/domino effect. Good luck! -- Petr^2 Spacek -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
On 06/11/14 14:58, Walter van Lille wrote: Hi, I need some assistance please. I've taken over an IPA server to manage a few months ago, and it was working fine until recently when it started acting up seemingly off its own accord. When I do an ipactl status it basically gives an output as shown below: *Directory Service: RUNNING * * * *Loong pause... (To the tune of 7 minutes sometimes)* * * *KDC Service: RUNNING* *KPASSWD Service: RUNNING* *DNS Service: RUNNING* *MEMCACHE Service: RUNNING* *HTTP Service: RUNNING* *CA Service: RUNNING* *ADTRUST Service: RUNNING* *EXTID Service: RUNNING* Running top showed that ns-slapd was munching almost all my resources, but I got that fixed by upping the cache. Unfortunately this did not correct the issue and it still reacts in the same fashion, although the resources have been freed up now. I've noticed that when I run dig on either the local server or a remote machine that the query basically just times out as shown here: *dig freeipa.myexample.sample* * * *; DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 freeipa.myexample.sample* *;; global options: +cmd* *;; connection timed out; no servers could be reached* When the KDC service fails to start, then name lookups seem OK, but authentication fails. otherwise it's dead in the water. This also happens: *sudo ipactl status* *Directory Service: RUNNING* *Unknown error when retrieving list of services from LDAP:* * * My software setup is as follows: *CentOS release 6.5 (Final) * *389-ds-base.x86_64 1.2.11.15-34.el6_5 * *bind.x86_64 32:9.8.2-0.23.rc1.el6_5.1 * *bind-dyndb-ldap.x86_64* *bind-libs.x86_64 32:9.8.2-0.23.rc1.el6_5.1* *bind-utils.x86_6432:9.8.2-0.23.rc1.el6_5.1* *rpcbind.x86_64 0.2.0-11.el6 @anaconda-CentOS-201311291202.x86_64/6.5* *samba4-winbind.x86_64* *krb5-server.x86_64 1.10.3-15.el6_5.1 * * * *Linux 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux * It's not a permanent situation as it sometimes runs 100% for a while, but 80% of the time it is unusable. If anybody can assist me, please be so kind. Regards, Walter Hello please which version of bind-dyndb-ldap do you use? I had similar issue with bind-dyndb-ldap, but it was development version, I'm not sure if this is your case. When named was failing, dirserv was really slow. Can you send journalctl -b -u named log when dig doesn't work?? -- Martin Basti -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] FreeIPA unresponsive - Causes DOS situations
On 11/06/2014 10:00 AM, Martin Basti wrote: On 06/11/14 14:58, Walter van Lille wrote: Hi, I need some assistance please. I've taken over an IPA server to manage a few months ago, and it was working fine until recently when it started acting up seemingly off its own accord. When I do an ipactl status it basically gives an output as shown below: *Directory Service: RUNNING * * * *Loong pause... (To the tune of 7 minutes sometimes)* * * *KDC Service: RUNNING* *KPASSWD Service: RUNNING* *DNS Service: RUNNING* *MEMCACHE Service: RUNNING* *HTTP Service: RUNNING* *CA Service: RUNNING* *ADTRUST Service: RUNNING* *EXTID Service: RUNNING* Running top showed that ns-slapd was munching almost all my resources, but I got that fixed by upping the cache. Unfortunately this did not correct the issue and it still reacts in the same fashion, although the resources have been freed up now. I've noticed that when I run dig on either the local server or a remote machine that the query basically just times out as shown here: *dig freeipa.myexample.sample* * * *; DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 freeipa.myexample.sample* *;; global options: +cmd* *;; connection timed out; no servers could be reached* When the KDC service fails to start, then name lookups seem OK, but authentication fails. otherwise it's dead in the water. This also happens: *sudo ipactl status* *Directory Service: RUNNING* *Unknown error when retrieving list of services from LDAP:* * * My software setup is as follows: *CentOS release 6.5 (Final) * *389-ds-base.x86_64 1.2.11.15-34.el6_5 * *bind.x86_64 32:9.8.2-0.23.rc1.el6_5.1 * *bind-dyndb-ldap.x86_64* *bind-libs.x86_64 32:9.8.2-0.23.rc1.el6_5.1* *bind-utils.x86_6432:9.8.2-0.23.rc1.el6_5.1* *rpcbind.x86_64 0.2.0-11.el6 @anaconda-CentOS-201311291202.x86_64/6.5* *samba4-winbind.x86_64* *krb5-server.x86_64 1.10.3-15.el6_5.1 * * * *Linux 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux * It's not a permanent situation as it sometimes runs 100% for a while, but 80% of the time it is unusable. If anybody can assist me, please be so kind. Regards, Walter Hello please which version of bind-dyndb-ldap do you use? I had similar issue with bind-dyndb-ldap, but it was development version, I'm not sure if this is your case. When named was failing, dirserv was really slow. Can you send journalctl -b -u named log when dig doesn't work?? -- Martin Basti You also want to look at the directory server logs especially at startup and see what is it doing. Also check the diskspace. May be you do not have much room on the volume and it might cause DS to slow down. -- Thank you, Dmitri Pal Sr. Engineering Manager IdM portfolio Red Hat, Inc. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
