Hi, after a reboot everything works fine again. The problem is more complex than it seemed on the first look. Let me explain it:
The hole telephony was down. The system ran for several month without any problem. It was reachable by ssh and http. sipxproc on console showed all services running. SipXconfig showed all services undefined. About one hour before the monitoring alerted the CPU load there were some strange lines in the proxy log: "2011-09-07T18:26:49.312936Z":3516699:KERNEL:ERR:master.voip.asdf.local:SipClientTcp-26:FFFFFFFF:SipXProxy:"OsMsgQShared::doSendCore message send failed for queue 'SipTcpServer-3' - no room, ret = 9" Later also : "2011-09-08T00:20:28.966534Z":3528586:SIP:CRIT:master.voip.asdf.local:SipClientUdp-8:B7A8FB90:SipXProxy:"SipUserAgent::queueMessageToInterestedObservers send failed with status 12 (numMsgs = 2000, maxMsgs = 2000)" "2011-09-08T00:20:28.966559Z":3528587:SIP:CRIT:master.voip.asdf.local:SipClientUdp-8:B7A8FB90:SipXProxy:"SipUserAgent::queueMessageToInterestedObservers send failed to queue named 'SipRouter-11'" "2011-09-08T00:20:28.966577Z":3528588:SIP:CRIT:master.voip.asdf.local:SipClientUdp-8:B7A8FB90:SipXProxy:"SipUserAgent::queueMessageToInterestedObservers observerQueue 0x9860ec4, observerData (nil), SIP method '', wantsRequests 1, wantsResponses 0, wantsIncoming 1, wantsOutGoing 0, eventName '', SipSession (nil)" When I tried to restart the presence service In the sipxpresence.log "2011-09-08T07:00:09.394617Z":4:KERNEL:ERR:master.voip.asdf.local:SipServerBroker-4:FFFFFFFF:sipxpresence:"OsServerSocket: accept(3) error: 22=Invalid argument" "2011-09-08T07:00:29.390130Z":5:ACD:CRIT:master.voip.asdf.local:pid-3269:B7F446E0:sipxpresence:"sigHandler: caught signal: 6" It seems that the network stack worked partially (it accepted packets on existing socket. That’s why ssh and http worked.). But after receiving the packets it could not process it internally. Loopback interface for internal communication crashed? An local application that normally subscribes presence from the rls got no answer on its subscribe request. Kernel crashed? To me this seems like a very strange operating system issue. What do you think? Sincerely *Von:* [email protected] [mailto: [email protected]] *Im Auftrag von *Tony Graziano *Gese**ndet:* Donnerstag, 8. September 2011 10:41 *An:* Discussion list for users of sipXecs software *Betreff:* Re: [sipx-users] SipXrls stopped working I think this has been discussed before. RLS has a performance issue in larger installations especially with XMPP roles enabled. In your case updating to 4.4 would help to eliminate a lot of the RLS issues, but in the meantime have you attempted a reboot? On Thu, Sep 8, 2011 at 4:27 AM, Michael Picher <[email protected]> wrote: Jan, I'd move to 4.4.0 to pick up large improvements to RLS. Thanks, Mike On Thu, Sep 8, 2011 at 2:56 AM, Jan Fricke <[email protected]> wrote: Hi, since yesterday we have a problem with the resourcelistserver of a sipx 4.2.1 installation. Our system monitoring threw an alert CPU 100% Load > 70. /var/log/messages.1 contains the following information several times: Sep 7 20:23:47 master kernel: INFO: task sipxrls:3896 blocked for more than 120 seconds. Sep 7 20:23:47 master kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 7 20:23:47 master kernel: sipxrls D 000C5F27 2604 3896 3160 3949 (NOTLB) Sep 7 20:23:47 master kernel: f68cde58 00000082 634ecdc0 000c5f27 000c5f27 f62eb000 00000092 0000000a Sep 7 20:23:47 master kernel: f6227550 634ecdc0 000c5f27 00000000 00000001 f622765c c2013944 f74c1e40 Sep 7 20:23:47 master kernel: 00000002 c2013944 00000000 00000001 00000070 00000080 018cc900 634ecdc0 Sep 7 20:23:47 master kernel: Call Trace: Sep 7 20:23:47 master kernel: [<c061c8ab>] wait_for_completion+0x6b/0x8f Sep 7 20:23:47 master kernel: [<c041f7a7>] default_wake_function+0x0/0xc Sep 7 20:23:47 master kernel: [<c0426be4>] exit_mm+0x69/0xda Sep 7 20:23:47 master kernel: [<c04280cd>] do_exit+0x20c/0x794 Sep 7 20:23:47 master kernel: [<c04286cb>] sys_exit_group+0x0/0xd Sep 7 20:23:47 master kernel: [<c042fddf>] get_signal_to_deliver+0x3a2/0x3c9 Sep 7 20:23:48 master kernel: [<c0404583>] do_notify_resume+0x77/0x67d Sep 7 20:23:48 master kernel: [<c041f7a7>] default_wake_function+0x0/0xc Sep 7 20:23:48 master kernel: [<c044be1e>] audit_syscall_entry+0x15a/0x18c Sep 7 20:23:48 master kernel: [<c044c1b8>] audit_syscall_exit+0x2da/0x301 Sep 7 20:23:48 master kernel: [<c0404fa6>] work_notifysig+0x13/0x19 Sep 7 20:23:48 master kernel: ======================= Sep 7 20:23:48 master kernel: INFO: task sipxrls:3949 blocked for more than 120 seconds. Sep 7 20:23:48 master kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 7 20:23:48 master kernel: sipxrls D 000C5F27 2868 3949 3160 3951 3896 (NOTLB) Sep 7 20:23:48 master kernel: f6295e58 00000082 635e1000 000c5f27 000c5f27 f62eb000 00000092 00000007 Sep 7 20:23:48 master kernel: f6294aa0 635e1000 000c5f27 00000000 00000001 f6294bac c2013944 f74c1e40 Sep 7 20:23:48 master kernel: 00000002 c2013944 00000000 00000000 00000000 c0405a89 f74c1e78 635e1000 Sep 7 20:23:48 master kernel: Call Trace: Sep 7 20:23:48 master kernel: [<c0405a89>] error_code+0x39/0x40 Sep 7 20:23:48 master kernel: [<c061c8ab>] wait_for_completion+0x6b/0x8f Sep 7 20:23:48 master kernel: [<c041f7a7>] default_wake_function+0x0/0xc Sep 7 20:23:48 master kernel: [<c0426be4>] exit_mm+0x69/0xda Sep 7 20:23:48 master kernel: [<c04280cd>] do_exit+0x20c/0x794 Sep 7 20:23:48 master kernel: [<c04286cb>] sys_exit_group+0x0/0xd Sep 7 20:23:48 master kernel: [<c042fddf>] get_signal_to_deliver+0x3a2/0x3c9 Sep 7 20:23:48 master kernel: [<c0404583>] do_notify_resume+0x77/0x67d Sep 7 20:23:48 master kernel: [<c042f837>] dequeue_signal+0x34/0xa8 Sep 7 20:23:48 master kernel: [<c042e0d2>] recalc_sigpending+0xe/0x20 Sep 7 20:23:48 master kernel: [<c04305a1>] sys_rt_sigtimedwait+0x253/0x2c2 Sep 7 20:23:48 master kernel: [<c044be1e>] audit_syscall_entry+0x15a/0x18c Sep 7 20:23:48 master kernel: [<c044c1b8>] audit_syscall_exit+0x2da/0x301 Sep 7 20:23:48 master kernel: [<c0404fa6>] work_notifysig+0x13/0x19 Sep 7 20:23:48 master kernel: ======================= After a while the heavy load was gone but since then the rls does not work. I tried to restart with sipxproc –restart ResourceListServer but it does not answer any subscribe requests an /var/log/sipxpbx/sipxrls.log is empty. Any ideas? _____________________________ Jan Fricke B.Sc. Cs. IANT- APPLIED NGN-TECHNOLOGIES Schlüsselfertige VoIP-Lösungen und mehr... Member of GROUPLINK IANT GmbH Salzdahlumer Straße 46/48 D-38302 Wolfenbüttel Fon: +49/(0)5331/ 900989-0 Fax: +49/(0)5331/ 900989-499 Mail: [email protected] Internet: www.iant.de Bankverbindung: IANT GmbH, Konto-Nr. 12 95 98 50 00, Volksbank BraWo (BLZ 269 910 66); IBAN-Nr.: DE02 2699 1066 1295 9850 00, BIC: GENODEF1WOB; Steuer-Nr.: 51/DP2013; Ust.-IdNr: DE264352710; HRB 201710, Amtsgericht Braunschweig; Geschäftsführer: Dipl.-Ing. Jan Schumacher, Prof. Dr.-Ing. Diederich Wermser Diese E-Mail kann vertrauliche und/oder rechtlich geschützte Informationen enthalten. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify the sender immediately and delete this e-mail. _______________________________________________ sipx-users mailing list [email protected] List Archive: http://list.sipfoundry.org/archive/sipx-users/ -- Michael Picher eZuce Director of Technical Services O.978-296-1005 X2015 M.207-956-0262 @mpicher <http://twitter.com/mpicher> www.ezuce.com _______________________________________________ sipx-users mailing list [email protected] List Archive: http://list.sipfoundry.org/archive/sipx-users/ -- ====================== Tony Graziano, Manager Telephone: 434.984.8430 sip: [email protected] Fax: 434.465.6833 Email: [email protected] LAN/Telephony/Security and Control Systems Helpdesk: Telephone: 434.984.8426 sip: [email protected] Helpdesk Contract Customers: http://support.myitdepartment.net Blog: http://blog.myitdepartment.net Linked-In Profile: http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4 Ask about our Internet faxservices!
_______________________________________________ sipx-users mailing list [email protected] List Archive: http://list.sipfoundry.org/archive/sipx-users/
