Hi Bogdan, Coming back to you after some holidays, and unfortunately, new Opensips crashes... Some weeks ago, we upgraded our servers to OpenSIPS 1.11, but we faced a new crash, always with the "out of pkg memory" error, last week.
Here is the exact version : > root@asbc2:/home/kemathy# opensips -V > version: opensips 1.11.2-notls (x86_64/linux) > flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, > SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT > ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, > MAX_URI_SIZE 1024, BUF_SIZE 65535 > poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > git revision: d41eb41 > main.c compiled on 19:23:35 Jul 23 2014 with gcc 4.7 Here are an extract of logs we get at the crash-time (I masked phone numbers and IPs for security purposes) : Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: > ERROR:core:parse_contacts: failed to parse contact parameters > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: > ERROR:core:contact_parser: failed to parse contacts > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: > ERROR:core:parse_contact: failed to parse contact > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: > ERROR:core:pv_get_contact_body: failed to parse contact hdr > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- > New REGISTER received from 188.94.XXX.XXX with Contact : -- IP > 188.94.XXX.XXX -- PORT 5060 > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g@pbx -- > REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: > ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 526 > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: > ERROR:sl:sl_send_reply_helper: response building failed > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: > ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: > ERROR:auth:challenge: failed to send the response > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- > New REGISTER received from 188.94.XXX.XXX with Contact : > <sip:[email protected]:5060;transport=udp;line=6364d3f0> -- IP > 188.94.XXX.XXX -- PORT 5060 > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r@pbx -- > REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: iy4l7y1r@pbx -- > New REGISTER received from 188.94.XXX.XXX with Contact : > <sip:[email protected]:5060;transport=udp;line=6364d3f0> -- IP > 188.94.XXX.XXX -- PORT 5060 > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: > ERROR:core:db_allocate_rows: no memory left > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: > ERROR:db_mysql:db_mysql_convert_rows: no private memory left > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: > ERROR:db_mysql:db_mysql_convert_result: error while converting rows > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: > ERROR:db_mysql:db_mysql_store_result: error while converting result > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: > ERROR:core:db_do_query: error while storing result for query [select > contact,expires,q,callid,cseq,flags,cflags,user_agent,received,path,socket,methods,last_modified,sip_instance,attr > from location where username='0326xxxxxx' order by q] > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: > ERROR:usrloc:db_load_urecord: db_query failed > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: > CRITICAL:db_mysql:wrapper_single_mysql_stmt_execute: driver error (1062): > Duplicate entry > '0326xxxxxx--sip:[email protected]:5060;transport=udp;line' > for key 'account_contact_idx' > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: > ERROR:usrloc:db_insert_ucontact: inserting contact in db failed > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: > ERROR:usrloc:insert_ucontact: failed to insert in database > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: > ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 373 > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: > ERROR:sl:sl_send_reply_helper: response building failed > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: > ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module > Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]: > ERROR:options:opt_reply: failed to send 200 via send_reply Unfortunately, we don't have the memory dump :-s ... So, I don't know if there's something we can try to avoid those errors to happen again; maybe we made something wrong in our configuration file ? We could provide you anything you want to help us find where's the problem. Thanks a lot for your help, Kevin *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP 2014-07-08 8:43 GMT+02:00 Kevin Mathy <[email protected]>: > Hi Bogdan, > > Sounds really good :-) ! I'm going to check if there's no configuration > change between 1.9's and 1.11's modules version, and then will upgrade our > version ! > > I hope this will correct our problem ;-) > > Thanks for your help, > > Kevin > > > > > *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP > > > > 2014-07-07 18:40 GMT+02:00 Bogdan-Andrei Iancu <[email protected]>: > > Hi Kevin, >> >> OK, found the problem. As expected, this was fixed in 1.11, but not >> ported to 1.9 as this version is not maintained any more. Here is the >> link to the fix for your crash: >> >> >> https://github.com/OpenSIPS/opensips/commit/eef59c1d7b47ecc35438c01e7dcbdfeea903115a#diff-30710a8823b167e0aae5a14bb4878b75 >> >> >> Regards, >> >> Bogdan-Andrei Iancu >> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com >> >> On 07.07.2014 18:15, Kevin Mathy wrote: >> >> Hi Bogdan, >> >> Opensips has crashed once again; I hope this time the memlog will be >> right to be analysed ! >> Here is the link to download the file : .................. >> >> If you need further informations, feel free to ask me ! >> >> Thanks for your help, >> >> Kevin >> >> >> >> >> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP >> >> >> >> 2014-07-04 11:54 GMT+02:00 Bogdan-Andrei Iancu <[email protected]>: >> >>> Hi Kevin, >>> >>> Indeed, it looks compiled ok now (with the memory debugger). Keep in >>> mind to set (in this order): >>> mem_log=4 >>> mem_dump=1 >>> to avoid runtime logs, but to get the mem dumps. >>> >>> Regards, >>> >>> Bogdan-Andrei Iancu >>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com >>> >>> On 04.07.2014 12:41, Kevin Mathy wrote: >>> >>> Hi Bogdan, >>> >>> We've recompiled OpenSIPS with requested modules (I think) : >>> >>>> root@asbc2:/home/kemathy# opensips -V >>>> version: opensips 1.9.2-notls (x86_64/linux) >>>> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, >>>> SHM_MMAP, PKG_MALLOC, *DBG_QM_MALLOC*, FAST_LOCK-ADAPTIVE_WAIT >>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, >>>> MAX_URI_SIZE 1024, BUF_SIZE 65535 >>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >>>> svnrevision: unknown >>>> @(#) $Id$ >>>> main.c compiled on 22:12:10 Jul 3 2014 with gcc 4.7 >>> >>> >>> Now we'll keep an eye on our server to check if everything is OK, and >>> if the memory error still occur; as we upgraded from 1.9.1 to 1.9.2... >>> >>> I'll get back to you with some logs if needed ;-) >>> >>> Kevin >>> >>> >>> >>> >>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP >>> >>> >>> >>> 2014-07-02 9:37 GMT+02:00 Kevin Mathy <[email protected]>: >>> >>>> Hi Bogdan, >>>> >>>> Hummm, right, opensips doesn't seem to have been compiled with the >>>> requested modules for memory debugging... >>>> >>>> root@asbc2:/home/kemathy# opensips -V >>>>> version: opensips 1.9.2-notls (x86_64/linux) >>>>> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, >>>>> SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT >>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, >>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535 >>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >>>>> svnrevision: unknown >>>>> @(#) $Id$ >>>>> main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7 >>>> >>>> >>>> So I think I'll have to re-compile opensips with QM_DBG_MALLOC, and >>>> try again to export the memdump log... >>>> >>>> I'll get back to you when done. >>>> >>>> Thanks a lot for your help ! >>>> >>>> Kevin >>>> >>>> >>>> >>>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP >>>> >>>> >>>> >>>> 2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu <[email protected]>: >>>> >>>> Hi Kevin, >>>>> >>>>> Unfortunately the logs are not correct - are you sure you properly >>>>> compiled the mem debug ? like adding the QM_DBG_MALLOC and removing the >>>>> FM_MALLOC flags ? As the logs show the standard memory manager (without >>>>> debugging). >>>>> Check it with "opensips -V" to see the list of compiled flags. >>>>> >>>>> I tried to get some ideas by only looking at the available memory and >>>>> how many fragments were allocated in each process - indeed, there are some >>>>> processes using maybe like 2 or 3 times more PKG, but not sure if a leak. >>>>> Getting the proper logs (which will be huge) will tell us more. >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Bogdan-Andrei Iancu >>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com >>>>> >>>>> On 01.07.2014 18:11, Kevin Mathy wrote: >>>>> >>>>> Hi Bogdan, >>>>> >>>>> I have now a memdump log, as we restarted opensips this afternoon >>>>> for a configuration maintenance... But the file is too big, even if I try >>>>> to put it to pastebin.com ... So, here is the file; I don't want to >>>>> give the link on the mailing-list :-) >>>>> >>>>> >>>>> [removed] >>>>> >>>>> I hope this will help understanding our problem's cause :-) ... >>>>> >>>>> Thanks for your help, >>>>> >>>>> Kevin >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP >>>>> >>>>> >>>>> >>>>> 2014-06-30 16:30 GMT+02:00 Kevin Mathy <[email protected]>: >>>>> >>>>>> Hi Bogdan, >>>>>> >>>>>> Ooops, I thought my two first mails have been cancelled :-) >>>>>> I prefer waiting till there's no traffic, so I'll send a SIGUSR1 >>>>>> comme this evening, and reply to this topic with the log. >>>>>> >>>>>> I'll try working with MI statistics to make some memory usage >>>>>> graphs better than with Cacti... >>>>>> >>>>>> I'll come back to you with logs; thanks for all ! >>>>>> >>>>>> Kevin >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP >>>>>> >>>>>> >>>>>> >>>>>> 2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu <[email protected]>: >>>>>> >>>>>> >>>>>> Hi Kevin, >>>>>>> >>>>>>> There is no need to send your email three times ;). One time is >>>>>>> enough. >>>>>>> >>>>>>> Waiting and taking the dump when there is not traffic is good (but >>>>>>> not a must) - the idea is to be sure that all temporary memory (used for >>>>>>> processing the current traffic) was freed - so what you still have in >>>>>>> memory is configuration data or possible leaks. >>>>>>> If you do not have the luxury of waiting, you can do it whenever you >>>>>>> can. >>>>>>> >>>>>>> Once again, do not look at the memory usage reported by OS - it is >>>>>>> irrelevant as OpenSIPS is doing its own internal memory management. >>>>>>> >>>>>>> Check the memory usage via MI, see the mem related statistics: >>>>>>> >>>>>>> http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11 >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Bogdan-Andrei Iancu >>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com >>>>>>> >>>>>>> On 30.06.2014 12:01, Kevin Mathy wrote: >>>>>>> >>>>>>> Hi Bogdan, >>>>>>> >>>>>>> If I want to send a SIGUSR1, may I have to wait 20 minutes after >>>>>>> the last call ? 20 minutes without any call ? I don't understand well >>>>>>> this >>>>>>> sentence : >>>>>>> >>>>>>>> It is highly recommended to do this after waiting about 20 >>>>>>>> minutes to be sure that as much as possile memory is freed - all >>>>>>>> temporary >>>>>>>> memory used during processing is freed by lack of load on the proxy >>>>>>> >>>>>>> >>>>>>> Also, last week-end, the traffic reduced a lot, and between last >>>>>>> friday, when the free system's memory was around 170M, and this morning, >>>>>>> the free memory seems to have increased : this morning, it was around >>>>>>> 700M, >>>>>>> before the traffic comes back. >>>>>>> >>>>>>> So, opensips seems to well free the memory, isn't it ? >>>>>>> >>>>>>> Thanks for your help, >>>>>>> >>>>>>> Kevin >>>>>>> >>>>>>> >>>>>>> >>>>>>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu <[email protected]> >>>>>>> : >>>>>>> >>>>>>>> Hi Kevin, >>>>>>>> >>>>>>>> There is no need to wait for a crash. From time to time, you can >>>>>>>> send a SIGUSR1 to a worker process (or a process you suspect as >>>>>>>> running out >>>>>>>> of pkg mem) -> the process will do a pkg dump to the log. >>>>>>>> >>>>>>>> Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9 >>>>>>>> is no longer maintained and there were some fixes in the memory manager >>>>>>>> since then. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Bogdan-Andrei Iancu >>>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com >>>>>>>> >>>>>>>> On 27.06.2014 10 <27.06.2014%2010>:36, Kevin Mathy wrote: >>>>>>>> >>>>>>>> Hi Bogdan, >>>>>>>> >>>>>>>> I've set given options, and now I'm waiting for a new crash of >>>>>>>> the service... Where the memdump will be located ? In another logfile >>>>>>>> than >>>>>>>> opensips.log, or in the same ? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur >>>>>>>> VoIP >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu <[email protected] >>>>>>>> >: >>>>>>>> >>>>>>>>> Kevin, >>>>>>>>> >>>>>>>>> Restarting should not make you loose ongoing calls (even if you >>>>>>>>> use the dialog module), do do not worry on that. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Bogdan-Andrei Iancu >>>>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>> >>> >>> >> >> >> > --
_______________________________________________ Users mailing list [email protected] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
