Re: [Freeipa-users] krb5kdc process at 100%
On Fri, 2011-09-09 at 19:28 -0400, Dmitri Pal wrote: > On 09/09/2011 03:14 PM, Smith, Martin R. [smma0...@stcloudstate.edu] > wrote: > > I have linked a zip the whole directory from abrt. After typing > > "abrt-cli -l" it outputted: > > - > > Directory: /var/spool/abrt/ccpp-2011-09-09-13:41:51-972 > > count: 1 > > executable: /usr/sbin/krb5kdc > > package:krb5-server-1.9.1-5.fc15 > > time: Fri 09 Sep 2011 01:41:51 PM CDT > > uid:0 > > - > > > > Link to crash.zip > > > > This appears to be my current ldap "openldap-2.4.24-3.fc15.x86_64". > > > > Can you please file a BZ? https://bugzilla.redhat.com > I assume it is on Fedora 15 right? FWIW I think I reproduced this yesterday evening. I will take a deeper look at it next week if it reproduces again. It seem to happen only when multiple worker processes are in use and one of them segfaults. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] krb5kdc process at 100%
On 09/09/2011 07:28 PM, Dmitri Pal wrote: > On 09/09/2011 03:14 PM, Smith, Martin R. [smma0...@stcloudstate.edu] > wrote: >> I have linked a zip the whole directory from abrt. After typing >> "abrt-cli -l" it outputted: >> - >> Directory: /var/spool/abrt/ccpp-2011-09-09-13:41:51-972 >> count: 1 >> executable: /usr/sbin/krb5kdc >> package:krb5-server-1.9.1-5.fc15 >> time: Fri 09 Sep 2011 01:41:51 PM CDT >> uid:0 >> - >> >> Link to _crash.zip_ >> <http://studentweb.stcloudstate.edu/smma0901/crash.zip> >> >> This appears to be my current ldap "openldap-2.4.24-3.fc15.x86_64". >> > > Can you please file a BZ? https://bugzilla.redhat.com > I assume it is on Fedora 15 right? End of day... Did not notice that the package name has fc15. I opened it myself: https://bugzilla.redhat.com/show_bug.cgi?id=737224 Feel free to add. > >> >> -Martin >> >> >> -Original Message- >> From: Simo Sorce _[mailto:s...@redhat.com]_ >> <mailto:[mailto:s...@redhat.com]> >> Sent: Friday, September 09, 2011 12:38 PM >> To: Smith, Martin R. [smma0...@stcloudstate.edu] >> Cc: _freeipa-users@redhat.com_ <mailto:freeipa-users@redhat.com> >> Subject: Re: [Freeipa-users] krb5kdc process at 100% >> >> If it crashes it is a bug in the KDC. >> Can you please get us the core dump when it crashes ? >> >> If you have abtrd installed it should be somewhere in /var/cache/abrt >> (check /var/log/messages) to see where. >> >> Alternatively you can run service krb5kdc stop then as root in a >> shell run ulimit -c unlimited and manually start /usr/sbin/krb5kdc >> wait for the crash then take the core file generated. >> >> Please also tell what is the exact version of the krb5-server package >> and the related ldap driver package. >> >> Simo. >> >> On Fri, 2011-09-09 at 16:27 +, Smith, Martin R. >> [smma0...@stcloudstate.edu] wrote: >> > I removed the -w 4 from the config file. Here is what happens now. >> > >> > When a user with expired password logs in the krb5kdc process now >> crashes, instead of running at 100%. >> > If I attach gdb to the process before it crashes and attempt to >> login the process doesn't crash. Here are the results of "bt" >> > - >> > #0 0x7fe84e0ea1d3 in __select_nocancel () >> > at ../sysdeps/unix/syscall-template.S:82 >> > #1 0x7fe84f2a8047 in krb5int_cm_call_select (in=, >> > out=0x7fe8501d8780, sret=0x7fff421862b4) at sendto_kdc.c:564 >> > #2 0x7fe84ffd05ee in listen_and_process (handle=0x0, >> > prog=0x7fff42187f52 "krb5kdc", reset=0x7fe84ffc6e10 >> ) >> > at net-server.c:1835 >> > #3 0x00007fe84ffbcf68 in main (argc=3, argv=) at >> > main.c:1069 >> > >> > >> > I have also attached the /var/log/krb5kdc >> > >> > -Martin >> > >> > -Original Message- >> > From: Simo Sorce _[mailto:s...@redhat.com]_ >> <mailto:[mailto:s...@redhat.com]> >> > Sent: Friday, September 09, 2011 8:56 AM >> > To: Smith, Martin R. [smma0...@stcloudstate.edu] >> > Cc: _freeipa-users@redhat.com_ <mailto:freeipa-users@redhat.com> >> > Subject: Re: [Freeipa-users] krb5kdc process at 100% >> > >> > On Fri, 2011-09-09 at 05:09 +, Smith, Martin R. >> > [smma0...@stcloudstate.edu] wrote: >> > > When I attach gdb to the process, I have tried the main process and >> > > the four child processes, it provides no output. >> > > Here are the steps I'm taking: >> > > 1. On freeipa-server run htop and find the pid (or ps aux) >> > > 1. Shows one parent PID and four child processes >> > > 1. 934 root 20 0 46784 2656 388 S 0.0 0.1 >> > > 0:00.00 `- /usr/sbin/krb5kdc >> > > -P /var/run/krb5kdc.pid -w 4 >> > > 2. 1939 root 20 0 78664 4460 2056 S 0.0 >> > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc >> > > -P /var/run/krb5kdc.pid -w 4 >> > > 3. 1938 root 20 0 78664 4460 2056 S 0.0 >> > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc >> > > -P /var/run/krb5kdc.pid -w 4 >> &
Re: [Freeipa-users] krb5kdc process at 100%
On 09/09/2011 03:14 PM, Smith, Martin R. [smma0...@stcloudstate.edu] wrote: > I have linked a zip the whole directory from abrt. After typing > "abrt-cli -l" it outputted: > - > Directory: /var/spool/abrt/ccpp-2011-09-09-13:41:51-972 > count: 1 > executable: /usr/sbin/krb5kdc > package:krb5-server-1.9.1-5.fc15 > time: Fri 09 Sep 2011 01:41:51 PM CDT > uid:0 > - > > Link to _crash.zip_ > <http://studentweb.stcloudstate.edu/smma0901/crash.zip> > > This appears to be my current ldap "openldap-2.4.24-3.fc15.x86_64". > Can you please file a BZ? https://bugzilla.redhat.com I assume it is on Fedora 15 right? > > -Martin > > > -Original Message- > From: Simo Sorce _[mailto:s...@redhat.com]_ > <mailto:[mailto:s...@redhat.com]> > Sent: Friday, September 09, 2011 12:38 PM > To: Smith, Martin R. [smma0...@stcloudstate.edu] > Cc: _freeipa-users@redhat.com_ <mailto:freeipa-users@redhat.com> > Subject: Re: [Freeipa-users] krb5kdc process at 100% > > If it crashes it is a bug in the KDC. > Can you please get us the core dump when it crashes ? > > If you have abtrd installed it should be somewhere in /var/cache/abrt > (check /var/log/messages) to see where. > > Alternatively you can run service krb5kdc stop then as root in a shell > run ulimit -c unlimited and manually start /usr/sbin/krb5kdc wait for > the crash then take the core file generated. > > Please also tell what is the exact version of the krb5-server package > and the related ldap driver package. > > Simo. > > On Fri, 2011-09-09 at 16:27 +, Smith, Martin R. > [smma0...@stcloudstate.edu] wrote: > > I removed the -w 4 from the config file. Here is what happens now. > > > > When a user with expired password logs in the krb5kdc process now > crashes, instead of running at 100%. > > If I attach gdb to the process before it crashes and attempt to login > the process doesn't crash. Here are the results of "bt" > > - > > #0 0x7fe84e0ea1d3 in __select_nocancel () > > at ../sysdeps/unix/syscall-template.S:82 > > #1 0x7fe84f2a8047 in krb5int_cm_call_select (in=, > > out=0x7fe8501d8780, sret=0x7fff421862b4) at sendto_kdc.c:564 > > #2 0x7fe84ffd05ee in listen_and_process (handle=0x0, > > prog=0x7fff42187f52 "krb5kdc", reset=0x7fe84ffc6e10 > ) > > at net-server.c:1835 > > #3 0x7fe84ffbcf68 in main (argc=3, argv=) at > > main.c:1069 > > > > > > I have also attached the /var/log/krb5kdc > > > > -Martin > > > > -Original Message- > > From: Simo Sorce _[mailto:s...@redhat.com]_ > <mailto:[mailto:s...@redhat.com]> > > Sent: Friday, September 09, 2011 8:56 AM > > To: Smith, Martin R. [smma0...@stcloudstate.edu] > > Cc: _freeipa-users@redhat.com_ <mailto:freeipa-users@redhat.com> > > Subject: Re: [Freeipa-users] krb5kdc process at 100% > > > > On Fri, 2011-09-09 at 05:09 +, Smith, Martin R. > > [smma0...@stcloudstate.edu] wrote: > > > When I attach gdb to the process, I have tried the main process and > > > the four child processes, it provides no output. > > > Here are the steps I'm taking: > > > 1. On freeipa-server run htop and find the pid (or ps aux) > > > 1. Shows one parent PID and four child processes > > > 1. 934 root 20 0 46784 2656 388 S 0.0 0.1 > > > 0:00.00 `- /usr/sbin/krb5kdc > > > -P /var/run/krb5kdc.pid -w 4 > > > 2. 1939 root 20 0 78664 4460 2056 S 0.0 > > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > > -P /var/run/krb5kdc.pid -w 4 > > > 3. 1938 root 20 0 78664 4460 2056 S 0.0 > > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > > -P /var/run/krb5kdc.pid -w 4 > > > 4. 1936 root 20 0 78664 4460 2056 S 0.0 > > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > > -P /var/run/krb5kdc.pid -w 4 > > > 5. 1935 root 20 0 78664 4212 1808 S 0.0 > > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > > -P /var/run/krb5kdc.pid -w 4 > > > 2. run sudo gdb > > > 1. attach 934 > > > 2. press &quo
Re: [Freeipa-users] krb5kdc process at 100%
I have linked a zip the whole directory from abrt. After typing "abrt-cli -l" it outputted: - Directory: /var/spool/abrt/ccpp-2011-09-09-13:41:51-972 count: 1 executable: /usr/sbin/krb5kdc package:krb5-server-1.9.1-5.fc15 time: Fri 09 Sep 2011 01:41:51 PM CDT uid:0 - Link to crash.zip<http://studentweb.stcloudstate.edu/smma0901/crash.zip> This appears to be my current ldap "openldap-2.4.24-3.fc15.x86_64". -Martin -Original Message- From: Simo Sorce [mailto:s...@redhat.com]<mailto:[mailto:s...@redhat.com]> Sent: Friday, September 09, 2011 12:38 PM To: Smith, Martin R. [smma0...@stcloudstate.edu] Cc: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com> Subject: Re: [Freeipa-users] krb5kdc process at 100% If it crashes it is a bug in the KDC. Can you please get us the core dump when it crashes ? If you have abtrd installed it should be somewhere in /var/cache/abrt (check /var/log/messages) to see where. Alternatively you can run service krb5kdc stop then as root in a shell run ulimit -c unlimited and manually start /usr/sbin/krb5kdc wait for the crash then take the core file generated. Please also tell what is the exact version of the krb5-server package and the related ldap driver package. Simo. On Fri, 2011-09-09 at 16:27 +, Smith, Martin R. [smma0...@stcloudstate.edu] wrote: > I removed the -w 4 from the config file. Here is what happens now. > > When a user with expired password logs in the krb5kdc process now crashes, > instead of running at 100%. > If I attach gdb to the process before it crashes and attempt to login the > process doesn't crash. Here are the results of "bt" > - > #0 0x7fe84e0ea1d3 in __select_nocancel () > at ../sysdeps/unix/syscall-template.S:82 > #1 0x7fe84f2a8047 in krb5int_cm_call_select (in=, > out=0x7fe8501d8780, sret=0x7fff421862b4) at sendto_kdc.c:564 > #2 0x7fe84ffd05ee in listen_and_process (handle=0x0, > prog=0x7fff42187f52 "krb5kdc", reset=0x7fe84ffc6e10 ) > at net-server.c:1835 > #3 0x7fe84ffbcf68 in main (argc=3, argv=) at > main.c:1069 > > > I have also attached the /var/log/krb5kdc > > -Martin > > -Original Message- > From: Simo Sorce [mailto:s...@redhat.com]<mailto:[mailto:s...@redhat.com]> > Sent: Friday, September 09, 2011 8:56 AM > To: Smith, Martin R. [smma0...@stcloudstate.edu] > Cc: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com> > Subject: Re: [Freeipa-users] krb5kdc process at 100% > > On Fri, 2011-09-09 at 05:09 +, Smith, Martin R. > [smma0...@stcloudstate.edu] wrote: > > When I attach gdb to the process, I have tried the main process and > > the four child processes, it provides no output. > > Here are the steps I'm taking: > > 1. On freeipa-server run htop and find the pid (or ps aux) > > 1. Shows one parent PID and four child processes > > 1. 934 root 20 0 46784 2656 388 S 0.0 0.1 > > 0:00.00 `- /usr/sbin/krb5kdc > > -P /var/run/krb5kdc.pid -w 4 > > 2. 1939 root 20 0 78664 4460 2056 S 0.0 > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > -P /var/run/krb5kdc.pid -w 4 > > 3. 1938 root 20 0 78664 4460 2056 S 0.0 > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > -P /var/run/krb5kdc.pid -w 4 > > 4. 1936 root 20 0 78664 4460 2056 S 0.0 > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > -P /var/run/krb5kdc.pid -w 4 > > 5. 1935 root 20 0 78664 4212 1808 S 0.0 > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > -P /var/run/krb5kdc.pid -w 4 > > 2. run sudo gdb > > 1. attach 934 > > 2. press "c" > > 3. Wait for output… > > 2. Attempt to login with user that has an expired password. > > 3. Now the krb5kdc process 934 starts running at 100% and the > > user is unable to login. > > 4. Only way to get the process back to normal is to type "service > > ipa restart" > > > > > I've never debugged a program before so if I'm missing a step please > > let me know. > > Ok, let's simplify the problem first. > > apperently you have a quadcore cpu so by default we configured krb5kdc to > spawn 4 worker processes. Let'
Re: [Freeipa-users] krb5kdc process at 100%
If it crashes it is a bug in the KDC. Can you please get us the core dump when it crashes ? If you have abtrd installed it should be somewhere in /var/cache/abrt (check /var/log/messages) to see where. Alternatively you can run service krb5kdc stop then as root in a shell run ulimit -c unlimited and manually start /usr/sbin/krb5kdc wait for the crash then take the core file generated. Please also tell what is the exact version of the krb5-server package and the related ldap driver package. Simo. On Fri, 2011-09-09 at 16:27 +, Smith, Martin R. [smma0...@stcloudstate.edu] wrote: > I removed the -w 4 from the config file. Here is what happens now. > > When a user with expired password logs in the krb5kdc process now crashes, > instead of running at 100%. > If I attach gdb to the process before it crashes and attempt to login the > process doesn't crash. Here are the results of "bt" > - > #0 0x7fe84e0ea1d3 in __select_nocancel () > at ../sysdeps/unix/syscall-template.S:82 > #1 0x7fe84f2a8047 in krb5int_cm_call_select (in=, > out=0x7fe8501d8780, sret=0x7fff421862b4) at sendto_kdc.c:564 > #2 0x7fe84ffd05ee in listen_and_process (handle=0x0, > prog=0x7fff42187f52 "krb5kdc", reset=0x7fe84ffc6e10 ) > at net-server.c:1835 > #3 0x7fe84ffbcf68 in main (argc=3, argv=) at main.c:1069 > > > I have also attached the /var/log/krb5kdc > > -Martin > > -Original Message- > From: Simo Sorce [mailto:s...@redhat.com] > Sent: Friday, September 09, 2011 8:56 AM > To: Smith, Martin R. [smma0...@stcloudstate.edu] > Cc: freeipa-users@redhat.com > Subject: Re: [Freeipa-users] krb5kdc process at 100% > > On Fri, 2011-09-09 at 05:09 +, Smith, Martin R. > [smma0...@stcloudstate.edu] wrote: > > When I attach gdb to the process, I have tried the main process and > > the four child processes, it provides no output. > > Here are the steps I'm taking: > > 1. On freeipa-server run htop and find the pid (or ps aux) > > 1. Shows one parent PID and four child processes > > 1. 934 root 20 0 46784 2656 388 S 0.0 0.1 > > 0:00.00 `- /usr/sbin/krb5kdc > > -P /var/run/krb5kdc.pid -w 4 > > 2. 1939 root 20 0 78664 4460 2056 S 0.0 > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > -P /var/run/krb5kdc.pid -w 4 > > 3. 1938 root 20 0 78664 4460 2056 S 0.0 > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > -P /var/run/krb5kdc.pid -w 4 > > 4. 1936 root 20 0 78664 4460 2056 S 0.0 > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > -P /var/run/krb5kdc.pid -w 4 > > 5. 1935 root 20 0 78664 4212 1808 S 0.0 > > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > > -P /var/run/krb5kdc.pid -w 4 > > 2. run sudo gdb > > 1. attach 934 > > 2. press "c" > > 3. Wait for output… > > 2. Attempt to login with user that has an expired password. > > 3. Now the krb5kdc process 934 starts running at 100% and the > > user is unable to login. > > 4. Only way to get the process back to normal is to type "service > > ipa restart" > > > > > I've never debugged a program before so if I'm missing a step please > > let me know. > > Ok, let's simplify the problem first. > > apperently you have a quadcore cpu so by default we configured krb5kdc to > spawn 4 worker processes. Let's bring it down to not spawning any worker > process so we can simplify debugging. > > Go to /etc/sysconfig/krb5kdc and remove the "-w 4" argument from it. > > Then simply do a service krb5kdc restart (no need to restart the whole ipa > service for this). > > > If krb5kdc locks up again, gdb the process like you have done before but do > not press c, type 'bt' instead and copy the log then you can exit gdb. > > Simo. > > > ___ > Freeipa-users mailing list > Freeipa-users@redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] krb5kdc process at 100%
I removed the -w 4 from the config file. Here is what happens now.
When a user with expired password logs in the krb5kdc process now crashes,
instead of running at 100%.
If I attach gdb to the process before it crashes and attempt to login the
process doesn't crash. Here are the results of "bt"
-
#0 0x7fe84e0ea1d3 in __select_nocancel ()
at ../sysdeps/unix/syscall-template.S:82
#1 0x7fe84f2a8047 in krb5int_cm_call_select (in=,
out=0x7fe8501d8780, sret=0x7fff421862b4) at sendto_kdc.c:564
#2 0x7fe84ffd05ee in listen_and_process (handle=0x0,
prog=0x7fff42187f52 "krb5kdc", reset=0x7fe84ffc6e10 )
at net-server.c:1835
#3 0x7fe84ffbcf68 in main (argc=3, argv=) at main.c:1069
I have also attached the /var/log/krb5kdc
-Martin
-Original Message-
From: Simo Sorce [mailto:s...@redhat.com]
Sent: Friday, September 09, 2011 8:56 AM
To: Smith, Martin R. [smma0...@stcloudstate.edu]
Cc: freeipa-users@redhat.com
Subject: Re: [Freeipa-users] krb5kdc process at 100%
On Fri, 2011-09-09 at 05:09 +, Smith, Martin R.
[smma0...@stcloudstate.edu] wrote:
> When I attach gdb to the process, I have tried the main process and
> the four child processes, it provides no output.
> Here are the steps I'm taking:
> 1. On freeipa-server run htop and find the pid (or ps aux)
> 1. Shows one parent PID and four child processes
> 1. 934 root 20 0 46784 2656 388 S 0.0 0.1
> 0:00.00 `- /usr/sbin/krb5kdc
> -P /var/run/krb5kdc.pid -w 4
> 2. 1939 root 20 0 78664 4460 2056 S 0.0
> 0.1 0:00.26 | `- /usr/sbin/krb5kdc
> -P /var/run/krb5kdc.pid -w 4
> 3. 1938 root 20 0 78664 4460 2056 S 0.0
> 0.1 0:00.26 | `- /usr/sbin/krb5kdc
> -P /var/run/krb5kdc.pid -w 4
> 4. 1936 root 20 0 78664 4460 2056 S 0.0
> 0.1 0:00.26 | `- /usr/sbin/krb5kdc
> -P /var/run/krb5kdc.pid -w 4
> 5. 1935 root 20 0 78664 4212 1808 S 0.0
> 0.1 0:00.26 | `- /usr/sbin/krb5kdc
> -P /var/run/krb5kdc.pid -w 4
> 2. run sudo gdb
> 1. attach 934
> 2. press "c"
> 3. Wait for output…
> 2. Attempt to login with user that has an expired password.
> 3. Now the krb5kdc process 934 starts running at 100% and the
> user is unable to login.
> 4. Only way to get the process back to normal is to type "service
> ipa restart"
>
> I've never debugged a program before so if I'm missing a step please
> let me know.
Ok, let's simplify the problem first.
apperently you have a quadcore cpu so by default we configured krb5kdc to spawn
4 worker processes. Let's bring it down to not spawning any worker process so
we can simplify debugging.
Go to /etc/sysconfig/krb5kdc and remove the "-w 4" argument from it.
Then simply do a service krb5kdc restart (no need to restart the whole ipa
service for this).
If krb5kdc locks up again, gdb the process like you have done before but do not
press c, type 'bt' instead and copy the log then you can exit gdb.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
Sep 09 11:08:46 server1.FAKE.COM krb5kdc[10618](info): listening on fd 12: tcp
0.0.0.0.88
Sep 09 11:08:46 server1.FAKE.COM krb5kdc[10618](info): listening on fd 11: tcp
::.88
Sep 09 11:08:46 server1.FAKE.COM krb5kdc[10618](info): set up 4 sockets
Sep 09 11:08:46 server1.FAKE.COM krb5kdc[10619](info): commencing operation
Sep 09 11:08:57 server1.FAKE.COM krb5kdc[10619](info): AS_REQ (4 etypes {18 17
16 23}) 199.17.59.191: NEEDED_PREAUTH: host/client1.fake@fake.com for
krbtgt/fake@fake.com, Additional pre-authentication required
Sep 09 11:08:57 server1.FAKE.COM krb5kdc[10619](info): AS_REQ (4 etypes {18 17
16 23}) 199.17.59.191: ISSUE: authtime 1315584537, etypes {rep=18 tkt=18
ses=18}, host/client1.fake@fake.com for krbtgt/fake@fake.com
Sep 09 11:08:57 server1.FAKE.COM krb5kdc[10619](info): TGS_REQ (4 etypes {18 17
16 23}) 199.17.59.191: ISSUE: authtime 1315584537, etypes {rep=18 tkt=18
ses=18}, host/client1.fake@fake.com for ldap/server1.fake@fake.com
Sep 09 11:08:58 server1.FAKE.COM krb5kdc[10619](info): AS_REQ (4 etypes {18 17
16 23}) 199.17.59.191: CLIENT KEY EXPIRED: as...@fake.com for
krbtgt/fake@fake.com, Password has expired
Sep 09 11:08:58 server1.FAKE.COM krb5kdc[10619](info): AS_REQ (4 etypes {18 17
16 23}) 199.17.59.191: NEEDED_PREAUTH: as...@fake.com for
kadmin/chang...@fake.com, Addit
Re: [Freeipa-users] krb5kdc process at 100%
On Fri, 2011-09-09 at 05:09 +, Smith, Martin R. [smma0...@stcloudstate.edu] wrote: > When I attach gdb to the process, I have tried the main process and > the four child processes, it provides no output. > Here are the steps I'm taking: > 1. On freeipa-server run htop and find the pid (or ps aux) > 1. Shows one parent PID and four child processes > 1. 934 root 20 0 46784 2656 388 S 0.0 0.1 > 0:00.00 `- /usr/sbin/krb5kdc > -P /var/run/krb5kdc.pid -w 4 > 2. 1939 root 20 0 78664 4460 2056 S 0.0 > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > -P /var/run/krb5kdc.pid -w 4 > 3. 1938 root 20 0 78664 4460 2056 S 0.0 > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > -P /var/run/krb5kdc.pid -w 4 > 4. 1936 root 20 0 78664 4460 2056 S 0.0 > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > -P /var/run/krb5kdc.pid -w 4 > 5. 1935 root 20 0 78664 4212 1808 S 0.0 > 0.1 0:00.26 | `- /usr/sbin/krb5kdc > -P /var/run/krb5kdc.pid -w 4 > 2. run sudo gdb > 1. attach 934 > 2. press "c" > 3. Wait for output… > 2. Attempt to login with user that has an expired password. > 3. Now the krb5kdc process 934 starts running at 100% and the > user is unable to login. > 4. Only way to get the process back to normal is to type "service > ipa restart" > > I've never debugged a program before so if I'm missing a step please > let me know. Ok, let's simplify the problem first. apperently you have a quadcore cpu so by default we configured krb5kdc to spawn 4 worker processes. Let's bring it down to not spawning any worker process so we can simplify debugging. Go to /etc/sysconfig/krb5kdc and remove the "-w 4" argument from it. Then simply do a service krb5kdc restart (no need to restart the whole ipa service for this). If krb5kdc locks up again, gdb the process like you have done before but do not press c, type 'bt' instead and copy the log then you can exit gdb. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] krb5kdc process at 100%
When I attach gdb to the process, I have tried the main process and the four
child processes, it provides no output.
Here are the steps I'm taking:
1. On freeipa-server run htop and find the pid (or ps aux)
* Shows one parent PID and four child processes
* 934 root 20 0 46784 2656 388 S 0.0 0.1 0:00.00 `-
/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid -w 4
*1939 root 20 0 78664 4460 2056 S 0.0 0.1 0:00.26 | `-
/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid -w 4
*1938 root 20 0 78664 4460 2056 S 0.0 0.1 0:00.26 | `-
/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid -w 4
*1936 root 20 0 78664 4460 2056 S 0.0 0.1 0:00.26 | `-
/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid -w 4
*1935 root 20 0 78664 4212 1808 S 0.0 0.1 0:00.26 | `-
/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid -w 4
* run sudo gdb
* attach 934
* press "c"
* Wait for output…
2. Attempt to login with user that has an expired password.
3. Now the krb5kdc process 934 starts running at 100% and the user is unable
to login.
4. Only way to get the process back to normal is to type "service ipa
restart"
I've never debugged a program before so if I'm missing a step please let me
know.
-Martin
On Sep 8, 2011, at 1:24 PM, Simo Sorce wrote:
Also any chance you can attach gdb to the krb5kdc process and take a
backtrace ?
Hopefully we will find out where it is hanging.
Simo.
On Thu, 2011-09-08 at 14:04 -0400, Simo Sorce wrote:
Is the ns-slapd instance for the ipa domain running when this happens ?
Simo.
On Thu, 2011-09-08 at 17:56 +, Smith, Martin R.
[smma0...@stcloudstate.edu<mailto:smma0...@stcloudstate.edu>] wrote:
Update: It appears to lockup immediately after a user with an expired
password attempts to login. This happens when a user attempts to login
at the freeipa-server itself or one of the clients.
From: freeipa-users-boun...@redhat.com<mailto:freeipa-users-boun...@redhat.com>
[mailto:freeipa-users-boun...@redhat.com] On Behalf Of Smith, Martin
R. [smma0...@stcloudstate.edu<mailto:smma0...@stcloudstate.edu>]
Sent: Thursday, September 08, 2011 12:49 PM
To: freeipa-users@redhat.com<mailto:freeipa-users@redhat.com>
Subject: [Freeipa-users] krb5kdc process at 100%
Hello all,
I’m running a fairly new install of Freeipa-server and we are running
into a problem that is preventing users from logging in. We have two
SSH servers that authenticate to our freeipa-server and after 15 min
to 4 hrs of runtime the process Krb5kdc will consume 100% of the
processor and the freeipa-server will no longer respond to ldap
requests from the other machines.
Here are some specs:
The freeipa-server is running as a virtual machine on a Xen 5.6 box
Fedora 15 with all current updates
The /home directory is a NFS mount to a different server, also running
freeipa-client
I updated the freeipa-server package to the “testing” repo today, the
problem still exists. The only additional components I’ve installed
are fail2ban, and rsyslog.
Some of the error messages include:
(krb5kdc.log)
Sep 08 12:10:23 client1.fake.com<http://client1.fake.com> krb5kdc[1867](info):
AS_REQ (7 etypes
{18 17 16 23 1 3 2}) 199.17.59.5: NEEDED_PREAUTH:
host/client1.fake@fake.com<mailto:host/client1.fake@fake.com> for
krbtgt/fake@fake.com<mailto:krbtgt/fake@fake.com>,
Additional pre-authentication required
(pki-ca-system-log)
Attached. This log is from the freeipa-server, it appears to be
complaining that it can’t connect to itself.
I can provide more logs to a personal email if needed.
Thanks for your help in resolving this issue.
-Martin Smith
___
Freeipa-users mailing list
Freeipa-users@redhat.com<mailto:Freeipa-users@redhat.com>
https://www.redhat.com/mailman/listinfo/freeipa-users
--
Simo Sorce * Red Hat, Inc * New York
___
Freeipa-users mailing list
Freeipa-users@redhat.com<mailto:Freeipa-users@redhat.com>
https://www.redhat.com/mailman/listinfo/freeipa-users
--
Simo Sorce * Red Hat, Inc * New York
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] krb5kdc process at 100%
Also any chance you can attach gdb to the krb5kdc process and take a
backtrace ?
Hopefully we will find out where it is hanging.
Simo.
On Thu, 2011-09-08 at 14:04 -0400, Simo Sorce wrote:
> Is the ns-slapd instance for the ipa domain running when this happens ?
>
> Simo.
>
> On Thu, 2011-09-08 at 17:56 +, Smith, Martin R.
> [smma0...@stcloudstate.edu] wrote:
> > Update: It appears to lockup immediately after a user with an expired
> > password attempts to login. This happens when a user attempts to login
> > at the freeipa-server itself or one of the clients.
> >
> >
> >
> >
> >
> > From: freeipa-users-boun...@redhat.com
> > [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Smith, Martin
> > R. [smma0...@stcloudstate.edu]
> > Sent: Thursday, September 08, 2011 12:49 PM
> > To: freeipa-users@redhat.com
> > Subject: [Freeipa-users] krb5kdc process at 100%
> >
> >
> >
> >
> > Hello all,
> >
> > I’m running a fairly new install of Freeipa-server and we are running
> > into a problem that is preventing users from logging in. We have two
> > SSH servers that authenticate to our freeipa-server and after 15 min
> > to 4 hrs of runtime the process Krb5kdc will consume 100% of the
> > processor and the freeipa-server will no longer respond to ldap
> > requests from the other machines.
> >
> >
> >
> > Here are some specs:
> >
> > The freeipa-server is running as a virtual machine on a Xen 5.6 box
> >
> > Fedora 15 with all current updates
> >
> > The /home directory is a NFS mount to a different server, also running
> > freeipa-client
> >
> >
> >
> > I updated the freeipa-server package to the “testing” repo today, the
> > problem still exists. The only additional components I’ve installed
> > are fail2ban, and rsyslog.
> >
> >
> >
> > Some of the error messages include:
> >
> > (krb5kdc.log)
> >
> > Sep 08 12:10:23 client1.fake.com krb5kdc[1867](info): AS_REQ (7 etypes
> > {18 17 16 23 1 3 2}) 199.17.59.5: NEEDED_PREAUTH:
> > host/client1.fake@fake.com for krbtgt/fake@fake.com,
> > Additional pre-authentication required
> >
> >
> >
> > (pki-ca-system-log)
> >
> > Attached. This log is from the freeipa-server, it appears to be
> > complaining that it can’t connect to itself.
> >
> >
> >
> > I can provide more logs to a personal email if needed.
> >
> >
> >
> > Thanks for your help in resolving this issue.
> >
> > -Martin Smith
> >
> >
> >
> >
> > ___
> > Freeipa-users mailing list
> > Freeipa-users@redhat.com
> > https://www.redhat.com/mailman/listinfo/freeipa-users
>
> --
> Simo Sorce * Red Hat, Inc * New York
>
> ___
> Freeipa-users mailing list
> Freeipa-users@redhat.com
> https://www.redhat.com/mailman/listinfo/freeipa-users
--
Simo Sorce * Red Hat, Inc * New York
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] krb5kdc process at 100%
Is the ns-slapd instance for the ipa domain running when this happens ?
Simo.
On Thu, 2011-09-08 at 17:56 +, Smith, Martin R.
[smma0...@stcloudstate.edu] wrote:
> Update: It appears to lockup immediately after a user with an expired
> password attempts to login. This happens when a user attempts to login
> at the freeipa-server itself or one of the clients.
>
>
>
>
>
> From: freeipa-users-boun...@redhat.com
> [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Smith, Martin
> R. [smma0...@stcloudstate.edu]
> Sent: Thursday, September 08, 2011 12:49 PM
> To: freeipa-users@redhat.com
> Subject: [Freeipa-users] krb5kdc process at 100%
>
>
>
>
> Hello all,
>
> I’m running a fairly new install of Freeipa-server and we are running
> into a problem that is preventing users from logging in. We have two
> SSH servers that authenticate to our freeipa-server and after 15 min
> to 4 hrs of runtime the process Krb5kdc will consume 100% of the
> processor and the freeipa-server will no longer respond to ldap
> requests from the other machines.
>
>
>
> Here are some specs:
>
> The freeipa-server is running as a virtual machine on a Xen 5.6 box
>
> Fedora 15 with all current updates
>
> The /home directory is a NFS mount to a different server, also running
> freeipa-client
>
>
>
> I updated the freeipa-server package to the “testing” repo today, the
> problem still exists. The only additional components I’ve installed
> are fail2ban, and rsyslog.
>
>
>
> Some of the error messages include:
>
> (krb5kdc.log)
>
> Sep 08 12:10:23 client1.fake.com krb5kdc[1867](info): AS_REQ (7 etypes
> {18 17 16 23 1 3 2}) 199.17.59.5: NEEDED_PREAUTH:
> host/client1.fake@fake.com for krbtgt/fake@fake.com,
> Additional pre-authentication required
>
>
>
> (pki-ca-system-log)
>
> Attached. This log is from the freeipa-server, it appears to be
> complaining that it can’t connect to itself.
>
>
>
> I can provide more logs to a personal email if needed.
>
>
>
> Thanks for your help in resolving this issue.
>
> -Martin Smith
>
>
>
>
> ___
> Freeipa-users mailing list
> Freeipa-users@redhat.com
> https://www.redhat.com/mailman/listinfo/freeipa-users
--
Simo Sorce * Red Hat, Inc * New York
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] krb5kdc process at 100%
Update: It appears to lockup immediately after a user with an expired password
attempts to login. This happens when a user attempts to login at the
freeipa-server itself or one of the clients.
From: freeipa-users-boun...@redhat.com
[mailto:freeipa-users-boun...@redhat.com] On Behalf Of Smith, Martin R.
[smma0...@stcloudstate.edu]
Sent: Thursday, September 08, 2011 12:49 PM
To: freeipa-users@redhat.com
Subject: [Freeipa-users] krb5kdc process at 100%
Hello all,
I'm running a fairly new install of Freeipa-server and we are running into a
problem that is preventing users from logging in. We have two SSH servers that
authenticate to our freeipa-server and after 15 min to 4 hrs of runtime the
process Krb5kdc will consume 100% of the processor and the freeipa-server will
no longer respond to ldap requests from the other machines.
Here are some specs:
The freeipa-server is running as a virtual machine on a Xen 5.6 box
Fedora 15 with all current updates
The /home directory is a NFS mount to a different server, also running
freeipa-client
I updated the freeipa-server package to the "testing" repo today, the problem
still exists. The only additional components I've installed are fail2ban, and
rsyslog.
Some of the error messages include:
(krb5kdc.log)
Sep 08 12:10:23 client1.fake.com krb5kdc[1867](info): AS_REQ (7 etypes {18 17
16 23 1 3 2}) 199.17.59.5: NEEDED_PREAUTH:
host/client1.fake@fake.com<mailto:host/client1.fake@fake.com> for
krbtgt/fake@fake.com<mailto:krbtgt/fake@fake.com>, Additional
pre-authentication required
(pki-ca-system-log)
Attached. This log is from the freeipa-server, it appears to be complaining
that it can't connect to itself.
I can provide more logs to a personal email if needed.
Thanks for your help in resolving this issue.
-Martin Smith
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
[Freeipa-users] krb5kdc process at 100%
Hello all,
I'm running a fairly new install of Freeipa-server and we are running into a
problem that is preventing users from logging in. We have two SSH servers that
authenticate to our freeipa-server and after 15 min to 4 hrs of runtime the
process Krb5kdc will consume 100% of the processor and the freeipa-server will
no longer respond to ldap requests from the other machines.
Here are some specs:
The freeipa-server is running as a virtual machine on a Xen 5.6 box
Fedora 15 with all current updates
The /home directory is a NFS mount to a different server, also running
freeipa-client
I updated the freeipa-server package to the "testing" repo today, the problem
still exists. The only additional components I've installed are fail2ban, and
rsyslog.
Some of the error messages include:
(krb5kdc.log)
Sep 08 12:10:23 client1.fake.com krb5kdc[1867](info): AS_REQ (7 etypes {18 17
16 23 1 3 2}) 199.17.59.5: NEEDED_PREAUTH: host/client1.fake@fake.com for
krbtgt/fake@fake.com, Additional pre-authentication required
(pki-ca-system-log)
Attached. This log is from the freeipa-server, it appears to be complaining
that it can't connect to itself.
I can provide more logs to a personal email if needed.
Thanks for your help in resolving this issue.
-Martin Smith
4692.Thread-13 - [14/Aug/2011:17:04:05 CDT] [3] [3] CRLIssuingPoint MasterCRL -
Cannot store the CRL cache in the internaldb. Error Failed to connect LDAP
server Could not connect to LDAP server host server1.fake.com port 7389 Error
netscape.ldap.LDAPException: failed to connect to server
ldap://server1.fake.com:7389 (91)
4692.Thread-13 - [14/Aug/2011:17:04:05 CDT] [8] [3] In Ldap (bound) connection
pool to host server1.fake.com port 7389, Cannot connect to LDAP server. Error:
netscape.ldap.LDAPException: failed to connect to server
ldap://server1.fake.com:7389 (91)
4692.Thread-13 - [14/Aug/2011:17:04:05 CDT] [5] [3] Failed to get a connection
to the LDAP server. Error Could not connect to LDAP server host
server1.fake.com port 7389 Error netscape.ldap.LDAPException: failed to connect
to server ldap://server1.fake.com:7389 (91)
4692.Thread-13 - [14/Aug/2011:17:04:05 CDT] [3] [3] CRLIssuingPoint MasterCRL -
Cannot store the CRL cache in the internaldb. Error Failed to connect LDAP
server Could not connect to LDAP server host server1.fake.com port 7389 Error
netscape.ldap.LDAPException: failed to connect to server
ldap://server1.fake.com:7389 (91)
1105.Thread-14 - [15/Aug/2011:16:07:47 CDT] [8] [3] Publishing: Could not
publish certificate serial number 0xc. Error Failed to publish using rule: No
rules enabled
1105.Thread-15 - [15/Aug/2011:16:23:02 CDT] [8] [3] Publishing: Could not
publish certificate serial number 0xd. Error Failed to publish using rule: No
rules enabled
1105.Thread-16 - [15/Aug/2011:16:26:23 CDT] [8] [3] Publishing: Could not
publish certificate serial number 0xe. Error Failed to publish using rule: No
rules enabled
1105.Thread-17 - [16/Aug/2011:18:57:17 CDT] [8] [3] Publishing: Could not
publish certificate serial number 0xf. Error Failed to publish using rule: No
rules enabled
1105.Thread-18 - [16/Aug/2011:19:03:18 CDT] [8] [3] Publishing: Could not
publish certificate serial number 0x10. Error Failed to publish using rule: No
rules enabled
1105.Thread-19 - [16/Aug/2011:20:08:28 CDT] [8] [3] Publishing: Could not
publish certificate serial number 0x11. Error Failed to publish using rule: No
rules enabled
1096.Thread-15 - [18/Aug/2011:14:32:48 CDT] [8] [3] Publishing: Could not
publish certificate serial number 0x12. Error Failed to publish using rule: No
rules enabled
30655.Thread-14 - [23/Aug/2011:10:37:58 CDT] [8] [3] Publishing: Could not
publish certificate serial number 0x13. Error Failed to publish using rule: No
rules enabled
3129.Thread-12 - [29/Aug/2011:12:06:33 CDT] [8] [3] In Ldap (bound) connection
pool to host server1.fake.com port 7389, Cannot connect to LDAP server. Error:
netscape.ldap.LDAPException: failed to connect to server
ldap://server1.fake.com:7389 (91)
3129.Thread-12 - [29/Aug/2011:12:06:33 CDT] [5] [3] Failed to get a connection
to the LDAP server. Error Could not connect to LDAP server host
server1.fake.com port 7389 Error netscape.ldap.LDAPException: failed to connect
to server ldap://server1.fake.com:7389 (91)
3129.Thread-12 - [29/Aug/2011:12:06:33 CDT] [3] [3] CRLIssuingPoint MasterCRL -
Cannot store the CRL cache in the internaldb. Error Failed to connect LDAP
server Could not connect to LDAP server host server1.fake.com port 7389 Error
netscape.ldap.LDAPException: failed to connect to server
ldap://server1.fake.com:7389 (91)
3129.Thread-12 - [29/Aug/2011:12:06:33 CDT] [8] [3] In Ldap (bound) connection
pool to host server1.fake.com port 7389, Cannot connect to LDAP server. Error:
netscape.ldap.LDAPException: failed to connect to server
ldap://server1.fake.com:7389 (91)
3129.Thread-12 - [29/Aug/2011:12:06:33 CDT] [5] [3] Failed to get a connection
t
