Re: Slow DB / outstanding requests
On 2011/09/15 06:35 PM, Alan Buxey wrote: Just a quick question, how many DB connections do you have in your config? 5 Server was a VM with too little RAM. And therefore a simple SQL query could take seconds instead of milliseconds. (Suspect swap meltdown if such a thing exists) Had to wait for the middle of the night to reboot (so more ram could be allocated). Server is now perfectly fine. It was just the first time that I didn't see FR recover after the DB eventually finished with slow queries. Cheers, -- Johan Meiring Cape PC Services CC Tel: (021) 883-8271 Fax: (021) 886-7782 Before acting on this email or opening any attachments you should read Cape PC Service's email disclaimer at: http://www.pcservices.co.za/disclaimer.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Slow DB / outstanding requests
Johan Meiring wrote: It was just the first time that I didn't see FR recover after the DB eventually finished with slow queries. I made a minor change for 2.1.12 which will clear out the pending queue a lot more quickly. old: for each new request clear out one dead request from queue If the dead requests originally arrived at a higher rate than the new requests, it can take a long time to clean them out. new: for each new request clear out all dead requests from queue Much better. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Slow DB / outstanding requests
On Fri, Sep 16, 2011 at 1:22 PM, Alan DeKok al...@deployingradius.com wrote: old: for each new request clear out one dead request from queue If the dead requests originally arrived at a higher rate than the new requests, it can take a long time to clean them out. new: for each new request clear out all dead requests from queue Thanks, one more reason to update to 2.1.12 when it comes out :D -- Fajar - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Slow DB / outstanding requests
Alan please do not forget to add this fix to changelog so it will be easier for a new FR users... Best regards On 9/16/2011 8:45 AM, Fajar A. Nugraha wrote: On Fri, Sep 16, 2011 at 1:22 PM, Alan DeKokal...@deployingradius.com wrote: old: for each new request clear out one dead request from queue If the dead requests originally arrived at a higher rate than the new requests, it can take a long time to clean them out. new: for each new request clear out all dead requests from queue Thanks, one more reason to update to 2.1.12 when it comes out :D - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Slow DB / outstanding requests
Marinko Tarlać wrote: Alan please do not forget to add this fix to changelog so it will be easier for a new FR users... to do...? It doesn't require anyone to do anything. There are no configuration changes. It's just code internal to the server. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Slow DB / outstanding requests
Of course we don't need to do anything but if this small change inside code is announced in changelog, more people will upgrade to 2.1.12 and they will stop bothering us on this list with the same questions, over and over again :) On 9/16/2011 9:59 AM, Alan DeKok wrote: Marinko Tarlać wrote: Alan please do not forget to add this fix to changelog so it will be easier for a new FR users... to do...? It doesn't require anyone to do anything. There are no configuration changes. It's just code internal to the server. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Slow DB / outstanding requests
Marinko Tarlać wrote: Of course we don't need to do anything but if this small change inside code is announced in changelog, more people will upgrade to 2.1.12 and they will stop bothering us on this list with the same questions, over and over again :) Good point. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Slow DB / outstanding requests
Hi, Seeing as the list is quiet, something that might be of interest. I am running 2.1.12 on a production server. The server is (was) severly underpowered and the database was straining. I was expenting the normal - rejecting duplicate request. (Cant remember exact error). Then the server received about 30 access requests in about .1 second It then went into a state where the following appeared over and over in the log: Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 647 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 648 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 649 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 650 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 651 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 652 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 690 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 685 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 691 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 692 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 693 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 694 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 677 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 686 in component core module queue. Mon Sep 12 11:33:48 2011 : Info: WARNING: Child is hung for request 625 in component core module queue. Mon Sep 12 11:33:48 2011 : Info: WARNING: Child is hung for request 626 in component core module queue. Mon Sep 12 11:33:48 2011 : Info: WARNING: Child is hung for request 627 in component core module queue. Mon Sep 12 11:33:48 2011 : Info: WARNING: Child is hung for request 628 in component core module queue. It never recovered. Had to be restarted. Even with no database queries outstanding. Dont know if 2.1.12 has more difficulty with slow databases? Unfortunately dont have more info, so this can probably be ignored. Cheers, -- Johan Meiring Cape PC Services CC Tel: (021) 883-8271 Fax: (021) 886-7782 Before acting on this email or opening any attachments you should read Cape PC Service's email disclaimer at: http://www.pcservices.co.za/disclaimer.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
RE: Slow DB / outstanding requests
Be careful what you wish for :) -Original Message- From: freeradius-users-bounces+david.peterson=acc-corp@lists.freeradius.org [mailto:freeradius-users-bounces+david.peterson=acc-corp.net@lists.freeradiu s.org] On Behalf Of Johan Meiring Sent: Thursday, September 15, 2011 11:32 AM To: FreeRadius users mailing list Subject: Slow DB / outstanding requests Hi, Seeing as the list is quiet, something that might be of interest. I am running 2.1.12 on a production server. The server is (was) severly underpowered and the database was straining. I was expenting the normal - rejecting duplicate request. (Cant remember exact error). Then the server received about 30 access requests in about .1 second It then went into a state where the following appeared over and over in the log: Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 647 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 648 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 649 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 650 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 651 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 652 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 690 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 685 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 691 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 692 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 693 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 694 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 677 in component core module queue. Mon Sep 12 11:33:47 2011 : Info: WARNING: Child is hung for request 686 in component core module queue. Mon Sep 12 11:33:48 2011 : Info: WARNING: Child is hung for request 625 in component core module queue. Mon Sep 12 11:33:48 2011 : Info: WARNING: Child is hung for request 626 in component core module queue. Mon Sep 12 11:33:48 2011 : Info: WARNING: Child is hung for request 627 in component core module queue. Mon Sep 12 11:33:48 2011 : Info: WARNING: Child is hung for request 628 in component core module queue. It never recovered. Had to be restarted. Even with no database queries outstanding. Dont know if 2.1.12 has more difficulty with slow databases? Unfortunately dont have more info, so this can probably be ignored. Cheers, -- Johan Meiring Cape PC Services CC Tel: (021) 883-8271 Fax: (021) 886-7782 Before acting on this email or opening any attachments you should read Cape PC Service's email disclaimer at: http://www.pcservices.co.za/disclaimer.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Slow DB / outstanding requests
Johan Meiring wrote: I am running 2.1.12 on a production server. The server is (was) severly underpowered and the database was straining. Don't do that I was expenting the normal - rejecting duplicate request. (Cant remember exact error). Pretty much, yes. Then the server received about 30 access requests in about .1 second It then went into a state where the following appeared over and over in the log: Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 647 in component core module queue. Mon Sep 12 11:33:46 2011 : Info: WARNING: Child is hung for request 648 in component core module queue. Hmmm... it's *really* slow. The requests are placed in a queue for the child threads to pick up. But... if the system is locked, they'll sit in the queue for a long time. It never recovered. Had to be restarted. Even with no database queries outstanding. Dont know if 2.1.12 has more difficulty with slow databases? It shouldn't. Unfortunately dont have more info, so this can probably be ignored. There are ways of addressing it... Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Slow DB / outstanding requests
Just a quick question, how many DB connections do you have in your config? alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
