RE: [Freeipa-users] FreeIPA "crashes" after many mystery connections
There are 26 IPA clients, 28 users, and 4 FreeIPA servers (of which only 2 are used by clients for authentication at present). Andy -Original Message- From: Simo Sorce [mailto:sso...@redhat.com] Sent: 26 October 2009 12:30 To: Andy Singleton Cc: freeipa-users@redhat.com Subject: RE: [Freeipa-users] FreeIPA "crashes" after many mystery connections On Mon, 2009-10-26 at 08:46 +, Andy Singleton wrote: > As far as I can see, whatever was trying to connect kept trying, and > filling up new slots as they became available until I rebooted. How many clients do you have ? Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
RE: [Freeipa-users] FreeIPA "crashes" after many mystery connections
On Mon, 2009-10-26 at 14:13 +, Andy Singleton wrote: > There are 26 IPA clients, 28 users, and 4 FreeIPA servers (of which only > 2 are used by clients for authentication at present). They are not many so even the default of ~1000 available FDs shouldn't be a problem. I guess I can't help you further unless we can find what caused so many connections. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
RE: [Freeipa-users] FreeIPA "crashes" after many mystery connections
On Mon, 2009-10-26 at 08:46 +, Andy Singleton wrote: > As far as I can see, whatever was trying to connect kept trying, and > filling up new slots as they became available until I rebooted. How many clients do you have ? Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
RE: [Freeipa-users] FreeIPA "crashes" after many mystery connections
The DS log entries look like this: [22/Oct/2009:12:29:51 +0200] - Not listening for new connections - too many fds open [22/Oct/2009:12:30:12 +0200] - Listening for new connections again [22/Oct/2009:12:30:12 +0200] - Not listening for new connections - too many fds open [22/Oct/2009:13:19:50 +0200] - Listening for new connections again ...repeated x 170... [22/Oct/2009:13:20:08 +0200] - Not listening for new connections - too many fds open [22/Oct/2009:13:20:08 +0200] - Listening for new connections again [22/Oct/2009:13:20:13 +0200] - slapd shutting down - signaling operation threads [22/Oct/2009:13:20:13 +0200] - slapd shutting down - closing down internal subsystems and plugins [22/Oct/2009:13:20:16 +0200] - Waiting for 4 database threads to stop [22/Oct/2009:13:20:16 +0200] - All database threads now stopped [22/Oct/2009:13:20:16 +0200] - slapd stopped. As far as I can see, whatever was trying to connect kept trying, and filling up new slots as they became available until I rebooted. Thanks Andy -Original Message- From: Simo Sorce [mailto:sso...@redhat.com] Sent: 23 October 2009 13:51 To: Andy Singleton Cc: freeipa-users@redhat.com Subject: RE: [Freeipa-users] FreeIPA "crashes" after many mystery connections On Fri, 2009-10-23 at 09:59 +0100, Andy Singleton wrote: > There isn't much in the krb5kdc.logs. > Server A has a few entries about a minute before the incident. Then > nothing until we had to reboot the box. Very strange ... Do yo ustill have the DS error log ? Anything in there ? Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
RE: [Freeipa-users] FreeIPA "crashes" after many mystery connections
There isn't much in the krb5kdc.logs.
Server A has a few entries about a minute before the incident. Then nothing
until we had to reboot the box.
Oct 22 12:27:53 a.office.tipp24.de krb5kdc[2114](info): TGS_REQ (1 etypes {18})
192.168.0.11: IS
SUE: authtime 1255946532, etypes {rep=18 tkt=18 ses=18}, us...@live.tipp24.net
for krbtgt/LIVE.TIPP2
4@live.tipp24.net
Oct 22 12:28:08 a.office.tipp24.de krb5kdc[2114](info): AS_REQ (7 etypes {18 17
16 23 1 3 2}) 192.168.0.12: CLIENT_NOT_FOUND: r...@live.tipp24.net for
krbtgt/live.tipp24@live.tipp24.net, Clien
t not found in Kerberos database
Oct 22 12:28:13 a.office.tipp24.de krb5kdc[2114](info): AS_REQ (7 etypes {18 17
16 23 1 3 2}) 192.168.0.12: NEEDED_PREAUTH: us...@live.tipp24.net for
krbtgt/live.tipp24@live.tipp24.net, Additi
onal pre-authentication required
Oct 22 12:28:13 a.office.tipp24.de krb5kdc[2114](info): AS_REQ (7 etypes {18 17
16 23 1 3 2}) 192.168.0.12: ISSUE: authtime 1256207293, etypes {rep=18 tkt=18
ses=18}, us...@live.tipp24.net for krb
tgt/live.tipp24@live.tipp24.net
Oct 22 13:21:40 a.office.tipp24.de krb5kdc[2080](info): setting up network...
Server B has even less: No entries for an hour before it gets the same problem.
Oct 22 11:32:34 b.office.tipp24.de krb5kdc[11838](info): AS_REQ (7 etypes {18
17 16 23 1 3 2})
192.168.0.10: NEEDED_PREAUTH: us...@live.tipp24.net for
krbtgt/live.tipp24@live.tipp24.net,
Additional pre-authentication required
Oct 22 11:32:34 b.office.tipp24.de krb5kdc[11838](info): AS_REQ (7 etypes {18
17 16 23 1 3 2})
192.168.0.10: ISSUE: authtime 1256203954, etypes {rep=18 tkt=18 ses=18},
us...@live.tipp24.net f
or krbtgt/live.tipp24@live.tipp24.net
All hostnames and users have been changed to protect the innocent.
Andy
-Original Message-
From: Simo Sorce [mailto:sso...@redhat.com]
Sent: 22 October 2009 18:02
To: Andy Singleton
Cc: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] FreeIPA "crashes" after many mystery connections
On Thu, 2009-10-22 at 16:22 +0100, Andy Singleton wrote:
> Hello,
>
>
>
> I am trying to solve a mystery. We have 2 replicated FreeIPA servers.
>
> Today they both stopped receiving requests because the Directory
> Server had begun to refuse connections.
>
> The relevant message is “Not listening for new connections - too many
> fds open”
>
>
>
> That’s all well and good: I can increase the file descriptor
> allowance.
>
> However, the reason the fds limit was reached was a massive number of
> connections from the servers themselves.
>
> Can someone provide me with an idea for what this might be?
>
>
>
> We received 1024 connections in under 1 second: Here is an example
> dirsrv access log entry:
>
>
>
> [22/Oct/2009:12:29:53 +0200] conn=679021 fd=464 slot=464 connection
> from 127.0.0.1 to 127.0.0.1
>
> [22/Oct/2009:12:29:53 +0200] conn=679021 op=0 BIND
> dn="uid=kdc,cn=sysaccounts,cn=etc,dc=live,dc=tipp
>
> 24,dc=net" method=128 version=3
>
> [22/Oct/2009:12:29:53 +0200] conn=679021 op=0 RESULT err=0 tag=97
> nentries=0 etime=0 dn="uid=kdc,cn=
>
> sysaccounts,cn=etc,dc=live,dc=tipp24,dc=net"
>
>
>
>
>
> Some final notes:
>
> Both servers stopped one after the other. First server A, then 1
> second afterwards, server B.
>
>
>
> I’m pretty stuck as to what might have caused this.
Can you check the krb5kdc logs ?
dn="uid=kdc,cn=sysaccounts,cn=etc,dc=live,dc=tipp24,dc=net" is the
account used by the kdc (in v1). So it looks like the KDC went crazy
trying to connect to the ldap server.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
RE: [Freeipa-users] FreeIPA "crashes" after many mystery connections
On Fri, 2009-10-23 at 09:59 +0100, Andy Singleton wrote: > There isn't much in the krb5kdc.logs. > Server A has a few entries about a minute before the incident. Then > nothing until we had to reboot the box. Very strange ... Do yo ustill have the DS error log ? Anything in there ? Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] FreeIPA "crashes" after many mystery connections
On Thu, 2009-10-22 at 16:22 +0100, Andy Singleton wrote: > Hello, > > > > I am trying to solve a mystery. We have 2 replicated FreeIPA servers. > > Today they both stopped receiving requests because the Directory > Server had begun to refuse connections. > > The relevant message is “Not listening for new connections - too many > fds open” > > > > That’s all well and good: I can increase the file descriptor > allowance. > > However, the reason the fds limit was reached was a massive number of > connections from the servers themselves. > > Can someone provide me with an idea for what this might be? > > > > We received 1024 connections in under 1 second: Here is an example > dirsrv access log entry: > > > > [22/Oct/2009:12:29:53 +0200] conn=679021 fd=464 slot=464 connection > from 127.0.0.1 to 127.0.0.1 > > [22/Oct/2009:12:29:53 +0200] conn=679021 op=0 BIND > dn="uid=kdc,cn=sysaccounts,cn=etc,dc=live,dc=tipp > > 24,dc=net" method=128 version=3 > > [22/Oct/2009:12:29:53 +0200] conn=679021 op=0 RESULT err=0 tag=97 > nentries=0 etime=0 dn="uid=kdc,cn= > > sysaccounts,cn=etc,dc=live,dc=tipp24,dc=net" > > > > > > Some final notes: > > Both servers stopped one after the other. First server A, then 1 > second afterwards, server B. > > > > I’m pretty stuck as to what might have caused this. Can you check the krb5kdc logs ? dn="uid=kdc,cn=sysaccounts,cn=etc,dc=live,dc=tipp24,dc=net" is the account used by the kdc (in v1). So it looks like the KDC went crazy trying to connect to the ldap server. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
