Re: What does FR 2.2.2 fix?
Hi, If everyone's in favor, I'll release 2.2.2 on Monday. hold request now its monday AM and the load has gone back to higher levels the server is freaking out and freezing witht he last message in the log being Mon Oct 7 07:50:28 2013 : Error: [event.c:2318] Internal sanity check failed (thats it...no other output - the server needs a restart, it doesnt process anything else once it hits this error) alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
a.l.m.bu...@lboro.ac.uk wrote: now its monday AM and the load has gone back to higher levels the server is freaking out and freezing witht he last message in the log being Mon Oct 7 07:50:28 2013 : Error: [event.c:2318] Internal sanity check failed At least that's clearer. It would be nice to be able to debug the exact state for that, but the fix should be simple. I'll push something to git later today. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
Hi, clarification/agreement from Stefan or others? tried the newest GIT this morning and the proxy issues were gone. I haven't seen your Internal sanity check failed just yet (and am not looking forward to it :-/ ). Stefan alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html -- Stefan WINTER Ingenieur de Recherche Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et de la Recherche 6, rue Richard Coudenhove-Kalergi L-1359 Luxembourg Tel: +352 424409 1 Fax: +352 422473 PGP key updated to 4096 Bit RSA - I will encrypt all mails if the recipient's key is known to me http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xC0DE6A358A39DC66 0x8A39DC66.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
Hi, More debug output would help. The last patch came from output sent by Stefan. The patch seems to help. But there's an underlying issue which is harder to debug. It looks like a Linux specific IPv6 problem. I don't see any issue with v4. interesting..the culprit may have been found. put HEAD onto server this afternoon... the logs had plenty of core messages but look Sun Oct 6 15:13:55 2013 : Error: WARNING: Unresponsive child for request 1821224, in component core module thread Sun Oct 6 15:13:56 2013 : Error: WARNING: Unresponsive child for request 1821229, in component core module thread Sun Oct 6 15:13:56 2013 : Info: WARNING: Child is hung for request 1821224 in component core module thread. Sun Oct 6 15:13:57 2013 : Info: WARNING: Child is hung for request 1821229 in component core module thread. Sun Oct 6 15:13:58 2013 : Info: WARNING: Child is hung for request 1821224 in component core module thread. Sun Oct 6 15:13:58 2013 : Info: WARNING: Child is hung for request 1821229 in component core module thread. Sun Oct 6 15:14:00 2013 : Info: WARNING: Child is hung for request 1821224 in component core module thread. Sun Oct 6 15:14:00 2013 : Info: WARNING: Child is hung for request 1821229 in component core module thread. Sun Oct 6 15:14:03 2013 : Info: WARNING: Child is hung for request 1820598 in component core module thread. Sun Oct 6 15:14:04 2013 : Info: WARNING: Child is hung for request 1821224 in component core module thread. Sun Oct 6 15:14:04 2013 : Info: WARNING: Child is hung for request 1821229 in component core module thread. Sun Oct 6 15:14:09 2013 : Info: WARNING: Child is hung for request 1821224 in component core module thread. Sun Oct 6 15:14:09 2013 : Info: WARNING: Child is hung for request 1821229 in component core module thread. Sun Oct 6 15:14:18 2013 : Info: Ready to process requests. no 'bad logs' since that restart logged. clarification/agreement from Stefan or others? alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
a.l.m.bu...@lboro.ac.uk wrote: interesting..the culprit may have been found. put HEAD onto server this afternoon... the logs had plenty of core messages but look ... no 'bad logs' since that restart logged. Good. It's the problem I thought it was, but the earlier fixes weren't complete The odd thing is that code hadn't changed from 2.2.0. So it looks like there were two bugs. One which hid the second one. When I fixed the first one, the second one caused this issue. clarification/agreement from Stefan or others? If everyone's in favor, I'll release 2.2.2 on Monday. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
On 4 Oct 2013, at 10:19, Alex Sharaz alex.sha...@york.ac.uk wrote: Hi, Yesterday caught an email about the release of FR 2.2.2 on Monday to fix a proxy problem. As I've just migrated 2 of my servers from 2.2.0 to 2.2.1 the sudden release of 2.2.2 sounds important. What does 2.2.2 fix? Issue with workers not marking requests are being done correctly. Workers appear to get hung, leading to issues. I would upgrade to latest 2.x.x HEAD to avoid disruption if the proxying functionality is heavily used. There were also quite a few issues with the policy language. -Arran Arran Cudbard-Bell a.cudba...@freeradius.org FreeRADIUS Development Team - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
On 4 Oct 2013, at 10:37, Arran Cudbard-Bell a.cudba...@freeradius.org wrote: On 4 Oct 2013, at 10:19, Alex Sharaz alex.sha...@york.ac.uk wrote: Hi, Yesterday caught an email about the release of FR 2.2.2 on Monday to fix a proxy problem. As I've just migrated 2 of my servers from 2.2.0 to 2.2.1 the sudden release of 2.2.2 sounds important. What does 2.2.2 fix? Issue with workers not marking requests are being done correctly. Workers appear to get hung, leading to issues. I would upgrade to latest 2.x.x HEAD to avoid disruption if the proxying functionality is heavily used. Eek! that's what I'm seeing on our outward facing eduroam servers that do nothing but proxy stuff. Time to fix it methinks There were also quite a few issues with the policy language. -Arran Arran Cudbard-Bell a.cudba...@freeradius.org FreeRADIUS Development Team - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
Hi, a couple of logic issues that meant case/switch and if() worked different to 2.x - thats been fixed. ..and an issue if your server does a lot of proxying work - in which worker threads arent dealt with properly - your log file will be full of core and module messages if you are being hit. this *MIGHT* be fixed in HEAD. we are testing at the moment (looking good). if you arent doing the former and not hit by the latter you dont need to worry. alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
Hmm like these then? Fri Oct 4 11:24:12 2013 : Info: WARNING: Child is hung for request 17630 in com ponent core module thread. Fri Oct 4 11:24:13 2013 : Info: WARNING: Child is hung for request 17635 in com ponent core module thread. Fri Oct 4 11:24:14 2013 : Info: WARNING: Child is hung for request 17634 in com ponent core module thread. Fri Oct 4 11:24:17 2013 : Info: WARNING: Child is hung for request 17636 in com ponent core module thread. Fri Oct 4 11:24:44 2013 : Info: WARNING: Child is hung for request 17633 in com ponent core module thread. Fri Oct 4 11:24:52 2013 : Info: WARNING: Child is hung for request 17635 in com ponent core module thread. Fri Oct 4 11:24:53 2013 : Info: WARNING: Child is hung for request 17634 in com ponent core module thread. Fri Oct 4 11:24:55 2013 : Info: WARNING: Child is hung for request 17636 in com ponent core module thread. Reverted back to 2.2.0 as I never saw these errors with it Rgs A On 4 Oct 2013, at 11:53, a.l.m.bu...@lboro.ac.uk wrote: Hi, a couple of logic issues that meant case/switch and if() worked different to 2.x - thats been fixed. ..and an issue if your server does a lot of proxying work - in which worker threads arent dealt with properly - your log file will be full of core and module messages if you are being hit. this *MIGHT* be fixed in HEAD. we are testing at the moment (looking good). if you arent doing the former and not hit by the latter you dont need to worry. alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
RE: What does FR 2.2.2 fix?
Yep, those are the ones. :-) Stefan Hmm like these then? Fri Oct 4 11:24:12 2013 : Info: WARNING: Child is hung for request 17630 in com ponent core module thread. Fri Oct 4 11:24:13 2013 : Info: WARNING: Child is hung for request 17635 in com ponent core module thread. Fri Oct 4 11:24:14 2013 : Info: WARNING: Child is hung for request 17634 in com ponent core module thread. Fri Oct 4 11:24:17 2013 : Info: WARNING: Child is hung for request 17636 in com ponent core module thread. Fri Oct 4 11:24:44 2013 : Info: WARNING: Child is hung for request 17633 in com ponent core module thread. Fri Oct 4 11:24:52 2013 : Info: WARNING: Child is hung for request 17635 in com ponent core module thread. Fri Oct 4 11:24:53 2013 : Info: WARNING: Child is hung for request 17634 in com ponent core module thread. Fri Oct 4 11:24:55 2013 : Info: WARNING: Child is hung for request 17636 in com ponent core module thread. Reverted back to 2.2.0 as I never saw these errors with it Rgs A -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
On 4 Oct 2013, at 12:00, Alex Sharaz alex.sha...@york.ac.uk wrote: Hmm like these then? Fri Oct 4 11:24:12 2013 : Info: WARNING: Child is hung for request 17630 in com ponent core module thread. Fri Oct 4 11:24:13 2013 : Info: WARNING: Child is hung for request 17635 in com ponent core module thread. Fri Oct 4 11:24:14 2013 : Info: WARNING: Child is hung for request 17634 in com ponent core module thread. Fri Oct 4 11:24:17 2013 : Info: WARNING: Child is hung for request 17636 in com ponent core module thread. Fri Oct 4 11:24:44 2013 : Info: WARNING: Child is hung for request 17633 in com ponent core module thread. Fri Oct 4 11:24:52 2013 : Info: WARNING: Child is hung for request 17635 in com ponent core module thread. Fri Oct 4 11:24:53 2013 : Info: WARNING: Child is hung for request 17634 in com ponent core module thread. Fri Oct 4 11:24:55 2013 : Info: WARNING: Child is hung for request 17636 in com ponent core module thread. Those would be the ones. Reverted back to 2.2.0 as I never saw these errors with it If I asked particularly nicely, and promised you a beer at the next networkshop we were both in attendance at, would you be willing to try git head? I'll roll a v2.2.2_rc0 if it sweetens the deal any? It'd just be really good to know that that particular issue was fixed before rolling out 2.2.2 and then finding it was something else and having to roll 2.2.3 a few weeks later. -Arran Arran Cudbard-Bell a.cudba...@freeradius.org FreeRADIUS Development Team - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
On 04/10/13 13:46, Arran Cudbard-Bell wrote: If I asked particularly nicely, and promised you a beer at the next networkshop we were both in attendance at, would you be willing to try git head? I'll roll a v2.2.2_rc0 if it sweetens the deal any? It'd just be really good to know that that particular issue was fixed before rolling out 2.2.2 and then finding it was something else and having to roll 2.2.3 a few weeks later. I for one will be happy to run git head and see what it does. I'm building as we speak. Cheers, Jonathan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
Hi, If I asked particularly nicely, and promised you a beer at the next networkshop we were both in attendance at, would you be willing to try git head? I'll take the beer - am running HEAD since last night on one server :-) (as I said to Alan, i'll report at end of day) alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
Woah! that's getting g to be lots of beer. I'll run it on one of my outward facing servers. Point me at something I can build and run A On 4 Oct 2013, at 14:33, a.l.m.bu...@lboro.ac.uk wrote: Hi, If I asked particularly nicely, and promised you a beer at the next networkshop we were both in attendance at, would you be willing to try git head? I'll take the beer - am running HEAD since last night on one server :-) (as I said to Alan, i'll report at end of day) alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
Hi early report :( 2.2.2 HEAD still showing: Fri Oct 4 13:20:43 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:20:45 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:20:47 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:20:51 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:20:52 2013 : Info: WARNING: Child is hung for request 3766906 in component core module thread. Fri Oct 4 13:20:56 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:21:03 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:21:15 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:21:17 2013 : Info: WARNING: Child is hung for request 3766906 in component core module thread. Fri Oct 4 13:21:32 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:21:57 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:27:32 2013 : Info: WARNING: Child is hung for request 3797280 in component core module thread. Fri Oct 4 13:27:40 2013 : Info: WARNING: Child is hung for request 3797280 in component core module thread. Fri Oct 4 13:27:51 2013 : Info: WARNING: Child is hung for request 3797280 in component core module thread. alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
On 10/04/2013 06:53 AM, a.l.m.bu...@lboro.ac.uk wrote: a couple of logic issues that meant case/switch and if() worked different to 2.x - thats been fixed. I need a clarification. Do you mean worked differently ONLY IN 2.2.1? But 2.2.2 is 100% logic consistent with all 2.x, except 2.2.1? -- John - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Some things started acting differently in 2.2.1 compared to previous releases of 2.x 2.2.2 should revert that so things behave the same - so far that seems to be true but we are still seeing stalled module in core messages that we did not see with 2.2.0 alan - -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -BEGIN PGP SIGNATURE- Version: APG v1.0.8 iHkEAREIADkFAlJO8E8yHEFsYW4gQnV4ZXkgKEFsYW4gQnV4ZXkpIDxhLmwubS5i dXhleUBsYm9yby5hYy51az4ACgkQobRdvRSkLC4v6wCcDkljo+wc582+s9TDOJEr Zz7YKAoAnjM3sq4jiTOJdOn7sKwLN83aycJh =/vny -END PGP SIGNATURE- - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
On 4 Oct 2013, at 17:43, Alan Buxey a.l.m.bu...@lboro.ac.uk wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Some things started acting differently in 2.2.1 compared to previous releases of 2.x 2.2.2 should revert that so things behave the same - so far that seems to be true but we are still seeing stalled module in core messages that we did not see with 2.2.0 Any chance you could connect to one of the running processes and generate a core? -Arran Arran Cudbard-Bell a.cudba...@freeradius.org FreeRADIUS Development Team - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: What does FR 2.2.2 fix?
More debug output would help. The last patch came from output sent by Stefan. The patch seems to help. But there's an underlying issue which is harder to debug. It looks like a Linux specific IPv6 problem. I don't see any issue with v4. Alan DeKok. On 2013-10-04, at 9:41 AM, a.l.m.bu...@lboro.ac.uk wrote: Hi early report :( 2.2.2 HEAD still showing: Fri Oct 4 13:20:43 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:20:45 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:20:47 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:20:51 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:20:52 2013 : Info: WARNING: Child is hung for request 3766906 in component core module thread. Fri Oct 4 13:20:56 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:21:03 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:21:15 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:21:17 2013 : Info: WARNING: Child is hung for request 3766906 in component core module thread. Fri Oct 4 13:21:32 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:21:57 2013 : Info: WARNING: Child is hung for request 3767589 in component core module thread. Fri Oct 4 13:27:32 2013 : Info: WARNING: Child is hung for request 3797280 in component core module thread. Fri Oct 4 13:27:40 2013 : Info: WARNING: Child is hung for request 3797280 in component core module thread. Fri Oct 4 13:27:51 2013 : Info: WARNING: Child is hung for request 3797280 in component core module thread. alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
