Thanks for the info. OK, it sounds like your sprout node is definitely not overloaded, so the issue I linked to below is not likely to be relevant. I’m now more interested to see the logs, as that clearly doesn’t sound right. Given that your system should not be overloaded, can I ask you to turn on debug logging for Sprout, and then send the logs? Instructions for turning on debug logging can be found here http://clearwater.readthedocs.io/en/latest/Troubleshooting_and_Recovery.html#sprout under “To turn on debug logging for Sprout…”
Thanks, Seb. From: Clearwater [mailto:[email protected]] On Behalf Of Steven Adams Sent: 20 January 2017 10:32 To: [email protected] Subject: Re: [Project Clearwater] Sprout regularly crashes Hi Seb, While I'm collecting logs, etc, I'll just mention that I've tried Umbreon and that was more unstable than the previous version. Spout crashed on every REGISTER that was sent to it. I've gone back to Quilava and that is more stable but does still crash on occasion. While I'm here, some answers to the easy questions: - How much load are you putting through this system? Practically zero. I'm working on an end-to-end test with only two SIP clients which (try to) register and I'm manually making a single call between the two. - How big is your deployment? (Both the number of sprout nodes and the size of each sprout node) It's a CW all-in-one installation, so whatever default metrics you define for this. Regards, Steve On 20 January 2017 at 10:15, Sebastian Rex <[email protected]<mailto:[email protected]>> wrote: Steve, Further to the below, we’ve also fixed a couple of bugs in this area recently, most notably https://github.com/Metaswitch/sprout/issues/1570 which caused problems on Sprout overload. This particular issue was fixed in release-111 (the Rapidash release) so if you haven’t already it would also be a good idea to try upgrading to at least that release. Seb. From: Clearwater [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Sebastian Rex Sent: 20 January 2017 09:44 To: [email protected]<mailto:[email protected]> Subject: Re: [Project Clearwater] Sprout regularly crashes Hi Steve, It looks like the Sprout health monitoring script (possibly erroneously) thinks that Sprout is in a bad state and is killing it. Could you send us the monit logs from a time covering a crash? That should allow us to check whether that’s the case. They’re at /var/log/monit.log Also, do you have debug logging turned on? If so, then the sprout logs (/var/log/sprout/sprout_x.log) would also be useful. Could you also give us some more context here? i.e. - How much load are you putting through this system? - How big is your deployment? (Both the number of sprout nodes and the size of each sprout node) (If you both have debug logging turned on and are putting load through the system, we would suggest turning debug logging off as it considerably worsens performance and makes this more likely to occur.) Thanks, Seb. From: Clearwater [mailto:[email protected]] On Behalf Of Steven Adams Sent: 06 January 2017 21:04 To: [email protected]<mailto:[email protected]> Subject: [Project Clearwater] Sprout regularly crashes Started using Tauros recently and are noticing that sprout crashes a lot with this error: Signal 6 caught Basic stack dump: /usr/share/clearwater/bin/sprout(_ZN6Logger9backtraceEPKc+0x6d)[0x51467d] /usr/share/clearwater/bin/sprout(_ZN3Log9backtraceEPKcz+0x10d)[0x5d488d] /usr/share/clearwater/bin/sprout(_Z14signal_handleri+0x2c)[0x63ce4c] /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7fe1660a7cb0] /lib/x86_64-linux-gnu/libpthread.so.0(sem_wait+0x2e)[0x7fe1673af66e] /usr/share/clearwater/bin/sprout(main+0xb6fa)[0x5138ca] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fe166092f45] /usr/share/clearwater/bin/sprout[0x51451c] Advanced stack dump (requires gdb): sh: 1: /usr/bin/gdb: not found gdb failed with return code 32512 Typically seems to happen after sprout has received an incoming 200 OK message. Sprout does restart itself so it's not the end of the world but it doesn't bode well for stability. _______________________________________________ Clearwater mailing list [email protected]<mailto:[email protected]> http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
