Re: High CPU usage after SQL failure when using buffered-sql
Stephen Fulton wrote: Is there any way to mitigate these CPU issues in version 2.1.4? No. You will need to either upgrade, or *manually* pull the patches from git into a local copy of 2.1.4. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU usage after SQL failure when using buffered-sql
Thanks Alan, I'll do that. -- Stephen Alan DeKok wrote: Stephen Fulton wrote: Is there any way to mitigate these CPU issues in version 2.1.4? No. You will need to either upgrade, or *manually* pull the patches from git into a local copy of 2.1.4. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU usage after SQL failure when using buffered-sql
Alan, I forgot to ask, is the fix part of stable or development? Thanks, -- Stephen Stephen Fulton wrote: Thanks Alan, I'll do that. -- Stephen Alan DeKok wrote: Stephen Fulton wrote: Is there any way to mitigate these CPU issues in version 2.1.4? No. You will need to either upgrade, or *manually* pull the patches from git into a local copy of 2.1.4. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU usage after SQL failure when using buffered-sql
Stephen Fulton wrote: I forgot to ask, is the fix part of stable or development? Both. The next release will be off of the stable tree. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU usage after SQL failure when using buffered-sql
Thanks Alan. To follow-up, 2.1.5 was tested using my parameters, and the condition did not reappear. Cheers, -- Stephen Alan DeKok wrote: Stephen Fulton wrote: I forgot to ask, is the fix part of stable or development? Both. The next release will be off of the stable tree. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: high cpu
On Fri, 2005-02-04 at 20:44 -0600, Michael Griego wrote: Try running with LD_ASSUME_KERNEL=2.4.19. This will force runtime linking against the standard libc libs instead of the thread-local storage (tls) libs. So, on the command line, run LD_ASSUME_KERNEL=2.4.19 radiusd -X and see if that segfaults. Yup. Still segfaults. --Mike Alan DeKok wrote: Daniel J McDonald [EMAIL PROTECTED] wrote: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1076829024 (LWP 17140)] 0x40079e54 in pthread_mutex_lock () from /lib/tls/libpthread.so.0 (gdb) bt #0 0x40079e54 in pthread_mutex_lock () from /lib/tls/libpthread.so.0 #1 0x406215dd in ldap_pvt_thread_mutex_lock () from /usr/lib/libldap_r.so.2 #2 0x40628443 in ldap_pvt_sasl_mutex_lock () from /usr/lib/libldap_r.so.2 #3 0x4042028e in sasl_dispose () from /usr/lib/libsasl2.so.2 Sounds like a library conflict to me. I'm not sure what to suggest... -- Daniel J McDonald, CCIE # 2495, CNX Austin Energy [EMAIL PROTECTED] - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: high cpu
Won't help much, but today I had an issue with a seg fault. Commented out a bit of code where the error was supposedly happening, seg fault went away... put the code back in...seg fault didn't return??? Did a make clean; make and everything seemed to be fine again. I guess in the end I just had my radius server in a mess as I'd been doing several configure's and compiles while trying to get snmp support to work on Solaris... Starting from a cleaner environment can sometimes be a good start to elimiate weird errors. :o) Daniel J McDonald wrote: On Fri, 2005-02-04 at 20:44 -0600, Michael Griego wrote: Try running with LD_ASSUME_KERNEL=2.4.19. This will force runtime linking against the standard libc libs instead of the thread-local storage (tls) libs. So, on the command line, run LD_ASSUME_KERNEL=2.4.19 radiusd -X and see if that segfaults. Yup. Still segfaults. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: high cpu
On Tue, 2005-02-08 at 00:08 +1100, Michael Mitchell wrote: Won't help much, but today I had an issue with a seg fault. Commented out a bit of code where the error was supposedly happening, seg fault went away... put the code back in...seg fault didn't return??? Did a make clean; make and everything seemed to be fine again. I guess in the end I just had my radius server in a mess as I'd been doing several configure's and compiles while trying to get snmp support to work on Solaris... Starting from a cleaner environment can sometimes be a good start to elimiate weird errors. :o) I've dug around and can't find any out-of-date libraries. I rebuilt this machine about a month ago - wiped it and installed everything from RPMs. I built an rpm for freeradius 1.0.1 using the 1.0.0 specfile from Mandrake Cooker and a few minor tweaks. My build machine doesn't appear to have any extra libraries either, and all of the versions match. But Mandrake has some fairly old openLDAP code in main - 2.1.30. There is a 2.2.23 package in cooker, so I'll try upgrading and relinking against that. The bug seems to be in the LDAP code anyway... Daniel J McDonald wrote: On Fri, 2005-02-04 at 20:44 -0600, Michael Griego wrote: Try running with LD_ASSUME_KERNEL=2.4.19. This will force runtime linking against the standard libc libs instead of the thread-local storage (tls) libs. So, on the command line, run LD_ASSUME_KERNEL=2.4.19 radiusd -X and see if that segfaults. Yup. Still segfaults. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: high cpu
On Fri, 2005-02-04 at 14:42 -0600, Daniel J McDonald wrote: I have an instance of freeradius 1.0.0 that is consuming 60-100% of a cpu (I have a two-processor box, so I can watch it do this). I am using ldap for the backend database. Following up to myself, I just compiled 1.0.1 and had the same issues - 97% cpu and does not send the authentication response, radiusd -X generates a segmentation fault. I changed radiusd.conf to allow corefiles, but it did not produce one that I could find. I tried to use gdb to get a stack dump, but radiusd -X wouldn't respond to radius queries while running gdb. Environment is Mandrake 10.1 - Linux ldap2.austin-energy.net 2.6.8.1-24mdksmp #1 SMP Thu Jan 13 23:11:43 MST 2005 i686 Intel(R) Pentium(R) III CPU - S 1400MHz unknown GNU/Linux -- Daniel J McDonald, CCIE # 2495, CNX Austin Energy [EMAIL PROTECTED] - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: high cpu
Daniel J McDonald [EMAIL PROTECTED] wrote: Following up to myself, I just compiled 1.0.1 and had the same issues - 97% cpu and does not send the authentication response, radiusd -X generates a segmentation fault. That's very weird. I've never seen that myself... I changed radiusd.conf to allow corefiles, but it did not produce one that I could find. I tried to use gdb to get a stack dump, but radiusd -X wouldn't respond to radius queries while running gdb. Try: $ gdb radiusd (gdb) set args -X . (gdb) run and you should see any error there. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: high cpu
On Fri, 2005-02-04 at 18:15 -0500, Alan DeKok wrote: Daniel J McDonald [EMAIL PROTECTED] wrote: Following up to myself, I just compiled 1.0.1 and had the same issues - 97% cpu and does not send the authentication response, radiusd -X generates a segmentation fault. That's very weird. I've never seen that myself... I changed radiusd.conf to allow corefiles, but it did not produce one that I could find. I tried to use gdb to get a stack dump, but radiusd -X wouldn't respond to radius queries while running gdb. Try: $ gdb radiusd (gdb) set args -X . (gdb) run ok... rlm_ldap: waiting for bind result ... rlm_ldap: Bind was successful rlm_ldap: user mcdonalddj authenticated succesfully Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1076829024 (LWP 17140)] 0x40079e54 in pthread_mutex_lock () from /lib/tls/libpthread.so.0 (gdb) bt #0 0x40079e54 in pthread_mutex_lock () from /lib/tls/libpthread.so.0 #1 0x406215dd in ldap_pvt_thread_mutex_lock () from /usr/lib/libldap_r.so.2 #2 0x40628443 in ldap_pvt_sasl_mutex_lock () from /usr/lib/libldap_r.so.2 #3 0x4042028e in sasl_dispose () from /usr/lib/libsasl2.so.2 #4 0x406277bb in ldap_int_sasl_close () from /usr/lib/libldap_r.so.2 #5 0x40633c8c in ldap_free_connection () from /usr/lib/libldap_r.so.2 #6 0x4062a7d9 in ldap_ld_free () from /usr/lib/libldap_r.so.2 #7 0x4062aa74 in ldap_unbind_ext () from /usr/lib/libldap_r.so.2 #8 0x4062abd9 in ldap_unbind_s () from /usr/lib/libldap_r.so.2 #9 0x40604acd in ?? () from /usr/lib/freeradius/rlm_ldap.so #10 0x0818f6a8 in ?? () #11 0x0818e80c in ?? () #12 0xbfffce7c in ?? () #13 0x0611487b in ?? () #14 0xbfffcee8 in ?? () #15 0x40007930 in _dl_lookup_symbol_x (undef_name=0x0, undef_map=0x1, ref=0x0, symbol_scope=0x81868c8, version=0x818e308, type_class=-1073752104, flags=134547776, skip_map=0x1) at dl-lookup.c:246 #16 0x08056958 in modcall () #17 0x08056aa9 in modcall () #18 0x08055e38 in find_module_instance () #19 0x08052b4a in rad_check_password () #20 0x08052fdf in rad_authenticate () ---Type return to continue, or q return to quit--- #21 0x0804c515 in rad_respond () #22 0x0804dd86 in main () -- Daniel J McDonald, CCIE # 2495, CNX Austin Energy [EMAIL PROTECTED] - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: high cpu
Daniel J McDonald [EMAIL PROTECTED] wrote: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1076829024 (LWP 17140)] 0x40079e54 in pthread_mutex_lock () from /lib/tls/libpthread.so.0 (gdb) bt #0 0x40079e54 in pthread_mutex_lock () from /lib/tls/libpthread.so.0 #1 0x406215dd in ldap_pvt_thread_mutex_lock () from /usr/lib/libldap_r.so.2 #2 0x40628443 in ldap_pvt_sasl_mutex_lock () from /usr/lib/libldap_r.so.2 #3 0x4042028e in sasl_dispose () from /usr/lib/libsasl2.so.2 Sounds like a library conflict to me. I'm not sure what to suggest... Alan DEKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: high cpu
Try running with LD_ASSUME_KERNEL=2.4.19. This will force runtime linking against the standard libc libs instead of the thread-local storage (tls) libs. So, on the command line, run LD_ASSUME_KERNEL=2.4.19 radiusd -X and see if that segfaults. --Mike Alan DeKok wrote: Daniel J McDonald [EMAIL PROTECTED] wrote: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1076829024 (LWP 17140)] 0x40079e54 in pthread_mutex_lock () from /lib/tls/libpthread.so.0 (gdb) bt #0 0x40079e54 in pthread_mutex_lock () from /lib/tls/libpthread.so.0 #1 0x406215dd in ldap_pvt_thread_mutex_lock () from /usr/lib/libldap_r.so.2 #2 0x40628443 in ldap_pvt_sasl_mutex_lock () from /usr/lib/libldap_r.so.2 #3 0x4042028e in sasl_dispose () from /usr/lib/libsasl2.so.2 Sounds like a library conflict to me. I'm not sure what to suggest... Alan DEKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU load, on accounting only server.
On Fri, 28 Jan 2005 22:42:08 -0800, Justin LaVelle [EMAIL PROTECTED] wrote: Thank you for your suggestion, though I didn't recompile the rpm, while I was searching for a source rpm to recompile I found that yum had installed, freeradius-1.0.1-1.i386.rpm, and Fedora had freeradius-1.0.1-3.i386.rpm available on... Nice... http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386 /Fedora/RPMS/ I installed this rpm and now things seem to be working properly, its using about 2-7% cpu now. Awesome! I wonder if the previous rpm was compiled on a different platform, or perhaps there was a bug in the rpm? What does the changelog say? (I'm just curous now..) -- Justin -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jason Frisvold Sent: Friday, January 28, 2005 8:04 PM To: freeradius-users@lists.freeradius.org Subject: Re: High CPU load, on accounting only server. On Fri, 28 Jan 2005 19:25:56 -0800, Justin LaVelle [EMAIL PROTECTED] wrote: High CPU load, on accounting only server. It using all available CPU, and not keeping up with what's being sent to it. It's a P3 900Mhz. Fedora Core 3, freeradius 1.0.1 installed from yum install. I'm accepting radius accounting data from several Redback SMS1800s (7 in total) I'm just logging to the plain text detail file. Peaked at about 20MB per hour of detail logs. That's not a lot at all... If there was a bottleneck, I would expect it to be i/o based, not cpu based... Have you tried re-compiling the rpm? I didn't think logging this would be very cpu intensive. Is this just too high a load that needs a more powerful system, or is there some processing of the incoming data that may be going on that I can disable? I presume you've tried restarting radius, as well as a complete reboot? -- Justin -- Jason 'XenoPhage' Frisvold [EMAIL PROTECTED] - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html -- Jason 'XenoPhage' Frisvold [EMAIL PROTECTED] - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU load, on accounting only server.
On Fri, 28 Jan 2005 19:25:56 -0800, Justin LaVelle [EMAIL PROTECTED] wrote: High CPU load, on accounting only server. It using all available CPU, and not keeping up with what's being sent to it. It's a P3 900Mhz. Fedora Core 3, freeradius 1.0.1 installed from yum install. I'm accepting radius accounting data from several Redback SMS1800s (7 in total) I'm just logging to the plain text detail file. Peaked at about 20MB per hour of detail logs. That's not a lot at all... If there was a bottleneck, I would expect it to be i/o based, not cpu based... Have you tried re-compiling the rpm? I didn't think logging this would be very cpu intensive. Is this just too high a load that needs a more powerful system, or is there some processing of the incoming data that may be going on that I can disable? I presume you've tried restarting radius, as well as a complete reboot? -- Justin -- Jason 'XenoPhage' Frisvold [EMAIL PROTECTED] - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU usage after FreeBSD Upgrade
Tuc [EMAIL PROTECTED] wrote: Still, is there something if I do run the debug mode again that we need to look for about these threads that seem to get used up, or unresponsive children? Look for pauses. If a thread is dead, that means it's blocking for more than 5 seconds. If you run in debug mode, you should see the server *stop* for 5 seconds. That will be indicative. Seems like this won't be necessary now. The perl scripts were pointing to the system perl which got overwritten during the upgrade. Once we pointed it to a correctly locally installed perl, its been fine ever since. Odd, but true. Thanks! Tuc/TTSG Internet Services, Inc. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU usage after FreeBSD Upgrade
Tuc [EMAIL PROTECTED] wrote: We've started to see things like : Mon Jun 7 11:00:13 2004 : Info: The maximum number of threads (32) are active, cannot spawn new thread to handle request Mon Jun 7 11:00:14 2004 : Error: Dropping packet from client L3-LasVegas:58096 - ID: 220 due to dead request 42218 Something is blocking the server. Run it in debugging mode to see what's taking so long. This is of course a production server. What impact will this have on performance/disk/etc? Is there something we can do a scan of the logs for (Some identifier, like *ERR*-) that we can trap on? Thanks, Tuc/TTSG Internet Services, Inc. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU usage after FreeBSD Upgrade
Tuc [EMAIL PROTECTED] wrote: When it starts to chew CPU, I see alot of : poll(0x81c7c00,0x3,0x0) = 0 (0x0) gettimeofday(0xbfbfeabc,0x0) = 0 (0x0) ... Does this seem odd? Yes. It looks like the main loop which reads requests is getting poked, even when there's nothing to do. As a result, it keeps waking up, and looking for things to do. Without access to a FreeBSD system, there isn't much more I can say. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU usage after FreeBSD Upgrade
Tuc [EMAIL PROTECTED] wrote: BEGIN failed--compilation aborted at /usr/local/radius/etc/raddb/scripts/login.p l line 15. Could this be related to the Perl issue your seeing in GNA? I'm not sure what you mean by that. Sorry, faded out there for a second. This was supposed to be my customer. I wouldn't be surprised if that problem was the root cause of the server slowdowns. The server really isn't slowing down, just chewing CPU. But we've fixed this, restarted, and will see if it fixes it. Still, is there something if I do run the debug mode again that we need to look for about these threads that seem to get used up, or unresponsive children? Thanks, Tuc/TTSG Internet Services, Inc. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: High CPU usage after FreeBSD Upgrade
Tuc [EMAIL PROTECTED] wrote: Still, is there something if I do run the debug mode again that we need to look for about these threads that seem to get used up, or unresponsive children? Look for pauses. If a thread is dead, that means it's blocking for more than 5 seconds. If you run in debug mode, you should see the server *stop* for 5 seconds. That will be indicative. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
