Re: [OpenSIPS-Devel] Memory leak in DR module, I think
To apply the patch, simply place yourself in the 2.1 source dir, and run this command: git apply <(base64 -d
Re: [OpenSIPS-Devel] Memory leak in DR module, I think
Glad I could be of help. Please could you email me the diff on that file, routing.c, so I can fix it on my proxy servers. Thanks. John Quick -Original Message- From: Devel [mailto:devel-boun...@lists.opensips.org] On Behalf Of Liviu Chircu Sent: 16 December 2016 14:42 To: devel@lists.opensips.org Subject: Re: [OpenSIPS-Devel] Memory leak in DR module, I think That's right on the spot, John! It seems this issue had been caught a long time ago, but, unfortunately, the fix did not make its way into 2.1 until now, as I just backported it. Thank you for all the help! Best regards, Liviu Chircu OpenSIPS Developer http://www.opensips-solutions.com On 16.12.2016 16:11, John Quick wrote: > For attention of Liviu Chircu, > > Thanks for your help with diagnosing memory leaks. I think I have now > found the culprit. The smoking gun! > > I kept increasing the syslog rate limiting value until it reached > 4 burst. This was still not enough, but I can see the one that is > frequent and it fits with the symptoms too. > Here is a small sample from the end of the dump: > > 2016-12-16 13:59:30 13335. N address=0x7f4be90020d8 > frag=0x7f4be90020a8 size=64 used=1 > 2016-12-16 13:59:30 alloc'd from drouting.c: > route2_carrier(3046) > 2016-12-16 13:59:30 start check=f0f0f0f0f0f0f0f0, end check= > c0c0c0c0c0c0c0c0, abcdefedabcdefed > 2016-12-16 13:59:30 13336. N address=0x7f4be9002178 > frag=0x7f4be9002148 size=48 used=1 > 2016-12-16 13:59:30 alloc'd from drouting.c: > route2_carrier(3046) > 2016-12-16 13:59:30 start check=f0f0f0f0f0f0f0f0, end check= > c0c0c0c0c0c0c0c0, abcdefedabcdefed > 2016-12-16 13:59:30 13337. N address=0x7f4be9002208 > frag=0x7f4be90021d8 size=40 used=1 > 2016-12-16 13:59:30 alloc'd from drouting.c: > route2_carrier(3046) > 2016-12-16 13:59:30 start check=f0f0f0f0f0f0f0f0, end check= > c0c0c0c0c0c0c0c0, abcdefedabcdefed > > I use the DR module to select a route for calls through this proxy. > The memory leak is fastest when there are lots of calls during the > day, then it is slow at night. > OpenSIPS Version 2.1.4 > > John Quick > Smartvox Limited > Web: www.smartvox.co.uk > > > > ___ > Devel mailing list > Devel@lists.opensips.org > http://lists.opensips.org/cgi-bin/mailman/listinfo/devel ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Re: [OpenSIPS-Devel] Memory leak in DR module, I think
That's right on the spot, John! It seems this issue had been caught a long time ago, but, unfortunately, the fix did not make its way into 2.1 until now, as I just backported it. Thank you for all the help! Best regards, Liviu Chircu OpenSIPS Developer http://www.opensips-solutions.com On 16.12.2016 16:11, John Quick wrote: For attention of Liviu Chircu, Thanks for your help with diagnosing memory leaks. I think I have now found the culprit. The smoking gun! I kept increasing the syslog rate limiting value until it reached 4 burst. This was still not enough, but I can see the one that is frequent and it fits with the symptoms too. Here is a small sample from the end of the dump: 2016-12-16 13:59:30 13335. N address=0x7f4be90020d8 frag=0x7f4be90020a8 size=64 used=1 2016-12-16 13:59:30 alloc'd from drouting.c: route2_carrier(3046) 2016-12-16 13:59:30 start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed 2016-12-16 13:59:30 13336. N address=0x7f4be9002178 frag=0x7f4be9002148 size=48 used=1 2016-12-16 13:59:30 alloc'd from drouting.c: route2_carrier(3046) 2016-12-16 13:59:30 start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed 2016-12-16 13:59:30 13337. N address=0x7f4be9002208 frag=0x7f4be90021d8 size=40 used=1 2016-12-16 13:59:30 alloc'd from drouting.c: route2_carrier(3046) 2016-12-16 13:59:30 start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed I use the DR module to select a route for calls through this proxy. The memory leak is fastest when there are lots of calls during the day, then it is slow at night. OpenSIPS Version 2.1.4 John Quick Smartvox Limited Web: www.smartvox.co.uk ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
[OpenSIPS-Devel] [OpenSIPS/opensips] 1cd594: Fix PKG mem leak
Branch: refs/heads/2.1 Home: https://github.com/OpenSIPS/opensips Commit: 1cd594ccfe87f023d79f6a0145172d3096d4b586 https://github.com/OpenSIPS/opensips/commit/1cd594ccfe87f023d79f6a0145172d3096d4b586 Author: Vlad PaiuDate: 2016-12-16 (Fri, 16 Dec 2016) Changed paths: M modules/drouting/drouting.c Log Message: --- Fix PKG mem leak (cherry picked from commit f276691d05c56748c628fc9aef9e2700aff8a536) ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
[OpenSIPS-Devel] [OpenSIPS/opensips] 20ebdb: mem/module_info: adapt code to shared memory initi...
Branch: refs/heads/master Home: https://github.com/OpenSIPS/opensips Commit: 20ebdb0597e958cfa1b998d3d5a98f6f26e13a98 https://github.com/OpenSIPS/opensips/commit/20ebdb0597e958cfa1b998d3d5a98f6f26e13a98 Author: Cerghit IonelDate: 2016-06-16 (Thu, 16 Jun 2016) Changed paths: M cfg.y M mem/module_info.c M mem/module_info.h M mem/shm_mem.c Log Message: --- mem/module_info: adapt code to shared memory initialisation change the statistics vector is exteded as more memory groups are declared if SHM_SHOW_DEFAULT_GROUP is defined allocations done util grups are defined are added to the default group Commit: db310d8b8c3efd62ccd0b7f7de4d142a366dd55b https://github.com/OpenSIPS/opensips/commit/db310d8b8c3efd62ccd0b7f7de4d142a366dd55b Author: Cerghit Ionel Date: 2016-06-17 (Fri, 17 Jun 2016) Changed paths: M Makefile.conf.template M mem/module_info.c Log Message: --- module_info.c: change obsolete flag Commit: e9c3b44bfb326441b60c4bc003228c75ce7bf27f https://github.com/OpenSIPS/opensips/commit/e9c3b44bfb326441b60c4bc003228c75ce7bf27f Author: Cerghit Ionel Date: 2016-12-09 (Fri, 09 Dec 2016) Changed paths: M mem/module_info.c M mem/module_info.h M mem/shm_mem.c Log Message: --- module_info.c: change group reallocating policy Commit: 130d3447426d3ae3ce27d099633042c9876018e3 https://github.com/OpenSIPS/opensips/commit/130d3447426d3ae3ce27d099633042c9876018e3 Author: Razvan Crainea Date: 2016-12-16 (Fri, 16 Dec 2016) Changed paths: M Makefile.conf.template M cfg.y M mem/module_info.c M mem/module_info.h M mem/shm_mem.c Log Message: --- Merge branch 'module-memory-rework' of https://github.com/ionel-cerghit/opensips into ionel-cerghit-module-memory-rework Commit: fe5d9763845aa05972de130a44ce78d56d352e26 https://github.com/OpenSIPS/opensips/commit/fe5d9763845aa05972de130a44ce78d56d352e26 Author: Razvan Crainea Date: 2016-12-16 (Fri, 16 Dec 2016) Changed paths: M Makefile.conf.template M mem/module_info.c Log Message: --- memgroups: decrease INFO to DEBUG Commit: a471509dd5e9fe63e2af59c2427577f0a29cf4a1 https://github.com/OpenSIPS/opensips/commit/a471509dd5e9fe63e2af59c2427577f0a29cf4a1 Author: Razvan Crainea Date: 2016-12-16 (Fri, 16 Dec 2016) Changed paths: M Makefile.conf.template M cfg.y M mem/module_info.c M mem/module_info.h M mem/shm_mem.c Log Message: --- Merge branch 'ionel-cerghit-module-memory-rework' Compare: https://github.com/OpenSIPS/opensips/compare/c2c496481c50...a471509dd5e9___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
[OpenSIPS-Devel] Memory leak in DR module, I think
For attention of Liviu Chircu, Thanks for your help with diagnosing memory leaks. I think I have now found the culprit. The smoking gun! I kept increasing the syslog rate limiting value until it reached 4 burst. This was still not enough, but I can see the one that is frequent and it fits with the symptoms too. Here is a small sample from the end of the dump: 2016-12-16 13:59:30 13335. N address=0x7f4be90020d8 frag=0x7f4be90020a8 size=64 used=1 2016-12-16 13:59:30 alloc'd from drouting.c: route2_carrier(3046) 2016-12-16 13:59:30 start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed 2016-12-16 13:59:30 13336. N address=0x7f4be9002178 frag=0x7f4be9002148 size=48 used=1 2016-12-16 13:59:30 alloc'd from drouting.c: route2_carrier(3046) 2016-12-16 13:59:30 start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed 2016-12-16 13:59:30 13337. N address=0x7f4be9002208 frag=0x7f4be90021d8 size=40 used=1 2016-12-16 13:59:30 alloc'd from drouting.c: route2_carrier(3046) 2016-12-16 13:59:30 start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed I use the DR module to select a route for calls through this proxy. The memory leak is fastest when there are lots of calls during the day, then it is slow at night. OpenSIPS Version 2.1.4 John Quick Smartvox Limited Web: www.smartvox.co.uk ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
[OpenSIPS-Devel] [OpenSIPS/opensips]
Branch: refs/heads/mid-registrar Home: https://github.com/OpenSIPS/opensips ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
[OpenSIPS-Devel] [OpenSIPS/opensips] 8ec91e: usrloc API: Add a loading function
Branch: refs/heads/master Home: https://github.com/OpenSIPS/opensips Commit: 8ec91e53521119d0de952367a51fdfe4a8037960 https://github.com/OpenSIPS/opensips/commit/8ec91e53521119d0de952367a51fdfe4a8037960 Author: Liviu ChircuDate: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: M modules/usrloc/usrloc.h Log Message: --- usrloc API: Add a loading function Commit: 21fff6ba5486aebdc3d1c8247786c0c375cdba14 https://github.com/OpenSIPS/opensips/commit/21fff6ba5486aebdc3d1c8247786c0c375cdba14 Author: Liviu Chircu Date: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: A modules/mid_registrar/Makefile A modules/mid_registrar/mid_registrar.c A modules/mid_registrar/uac_timer.c A modules/mid_registrar/uac_timer.h Log Message: --- mid_registrar: initial version (PoC phase) * basic modparam outline * basic script exports * un-sorted dump of code * tm/usrloc/signaling hooks * + registered callbacks * basic timer routine outline Commit: 5117e368ff1f013ac7cace4116baad1b6465c13c https://github.com/OpenSIPS/opensips/commit/5117e368ff1f013ac7cace4116baad1b6465c13c Author: Liviu Chircu Date: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: M modules/mid_registrar/mid_registrar.c M modules/mid_registrar/uac_timer.c M modules/mid_registrar/uac_timer.h Log Message: --- mid_registrar: Fix a series of bugs * mode = 0 working * mode = 1 almost working, needs some tweaks Commit: ca8b61fe616910a40e213d9afb47bc6b9fc62923 https://github.com/OpenSIPS/opensips/commit/ca8b61fe616910a40e213d9afb47bc6b9fc62923 Author: Liviu Chircu Date: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: M modules/usrloc/ucontact.c M modules/usrloc/ucontact.h Log Message: --- usrloc: Add expires value to contact struct Commit: da1a96441f13c5556249890521b41efcc5122a7b https://github.com/OpenSIPS/opensips/commit/da1a96441f13c5556249890521b41efcc5122a7b Author: Liviu Chircu Date: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: M modules/usrloc/ucontact.c M modules/usrloc/ucontact.h Log Message: --- usrloc: Add "expires_out" value to contact struct Alpha phase. Has no restart persistency. Commit: aa94f8ba89f3e631ac40e0adf88ce2244134afeb https://github.com/OpenSIPS/opensips/commit/aa94f8ba89f3e631ac40e0adf88ce2244134afeb Author: Liviu Chircu Date: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: M modules/mid_registrar/mid_registrar.c M modules/mid_registrar/uac_timer.c Log Message: --- mid_registrar: Working version of contact throttling REGISTER requests are now properly absorbed by the mid-registrar, which will internally generate a 200 OK reply when the mid_registrar_save() script function is called. Commit: 5f5d14654948c0254b15d7dc05ce32b462b5e231 https://github.com/OpenSIPS/opensips/commit/5f5d14654948c0254b15d7dc05ce32b462b5e231 Author: Liviu Chircu Date: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: M modules/mid_registrar/mid_registrar.c Log Message: --- mid_registrar: Fix 200 OK replies for absorbed registrations Properly include a 'Contact' header field, listing all bindings Commit: 6cb240cca71a48ad049c19ffac0a21a3b7d0ab93 https://github.com/OpenSIPS/opensips/commit/6cb240cca71a48ad049c19ffac0a21a3b7d0ab93 Author: Liviu Chircu Date: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: M modules/dispatcher/dispatch.c M modules/drouting/drouting.c M modules/tm/dlg.c M modules/tm/dlg.h Log Message: --- tm API: Add Call-ID parameter when building requests Commit: 95a39a0bcf4c053ea09f3f54c39d40a5c7b0103b https://github.com/OpenSIPS/opensips/commit/95a39a0bcf4c053ea09f3f54c39d40a5c7b0103b Author: Liviu Chircu Date: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: M Makefile.sources A lib/path.c A lib/path.h M modules/path/path.c Log Message: --- path: Code refactoring Make "Path" append code reusable by moving it into lib/path.c Commit: 82b47c0cbb49d603543f49078e1700b43eab6b3c https://github.com/OpenSIPS/opensips/commit/82b47c0cbb49d603543f49078e1700b43eab6b3c Author: Liviu Chircu Date: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: M modules/mid_registrar/uac_timer.c Log Message: --- mid_registrar: Forward Register requests if not in queue De-Registrations may lead to a state where contacts are present on the mid-registrar, but no timer is ticking for the outgoing registration. Should we run into this issue, simply forward the registration and re-initialize the timer. Commit:
[OpenSIPS-Devel] [OpenSIPS/opensips] 70d303: mid_registrar: Avoid redundant del lumps
Branch: refs/heads/mid-registrar Home: https://github.com/OpenSIPS/opensips Commit: 70d303edb92cad80cd558dd67ff64c90b0f7249e https://github.com/OpenSIPS/opensips/commit/70d303edb92cad80cd558dd67ff64c90b0f7249e Author: Liviu ChircuDate: 2016-12-05 (Mon, 05 Dec 2016) Changed paths: M modules/mid_registrar/save.c Log Message: --- mid_registrar: Avoid redundant del lumps Commit: e35ab76a4b3ae79f4ace7cc2ad9639a22dcec4ab https://github.com/OpenSIPS/opensips/commit/e35ab76a4b3ae79f4ace7cc2ad9639a22dcec4ab Author: Liviu Chircu Date: 2016-12-16 (Fri, 16 Dec 2016) Changed paths: M modules/mid_registrar/lookup.c M modules/mid_registrar/mid_registrar.c M modules/mid_registrar/mid_registrar.h M modules/mid_registrar/save.c Log Message: --- mid_registrar: rename "routing_mode" to "insertion_mode" Commit: 9ace8d31115dca623d5f9a60c92e292148565c86 https://github.com/OpenSIPS/opensips/commit/9ace8d31115dca623d5f9a60c92e292148565c86 Author: Liviu Chircu Date: 2016-12-16 (Fri, 16 Dec 2016) Changed paths: M modules/mid_registrar/README M modules/mid_registrar/doc/mid_registrar_admin.xml Log Message: --- mid_registrar: Update doc Compare: https://github.com/OpenSIPS/opensips/compare/8d5c0f7e9321...9ace8d31115d___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Re: [OpenSIPS-Devel] Finding a memory leak
Hi John, Memory leak troubleshooting is more difficult on OpenSIPS versions up to, and including, 2.1. Starting from version 2.2, we have improved the output summary in such a way that leaks are immediately visible, with no additional work on behalf of the user! Now, it's still possible to spot the leak with your current version, we only need to work a bit more. Useful steps: * if needed, make sure any syslog rate-limiting is disabled. These dumps can be quite large! * use "opensipsctl fifo ps" to find the PID of each OpenSIPS process you may want to send the "kill -SIGUSR1" command * "pv_add_extra(4480)" points to an allocation at line 4480, of the "pv_add_extra()" function * if there were a leak at "pv_add_extra(4480)", then periodical dumps of a single process will contain more and more lines indicating "pv_add_extra(4480)". You may need to do some output text manipulation magic in order to sum up all this information, for each dump. Best regards, Liviu Chircu OpenSIPS Developer http://www.opensips-solutions.com On 15.12.2016 18:20, John Quick wrote: Please can someone provide advice on interpreting OpenSIPS memory dumps and identifying the cause of a memory leak. A Proxy application I wrote, running v2.1.4 with DBG_QM_MALLOC enabled, is leaking package memory so badly that I have to restart the service once a week. When I use this command: opensipsctl fifo get_statistics pkmem: | grep max ...it lists 57 lines of output. Most of these show a similar value from day to day, but there are three instances near the middle of the list that increase from one day to the next. For example: pkmem:0-max_used_size:: 1359072 pkmem:1-max_used_size:: 1427568 pkmem:2-max_used_size:: 1463256 ... pkmem:23-max_used_size:: 3013848 pkmem:24-max_used_size:: 2541240 pkmem:25-max_used_size:: 3799672 ... pkmem:55-max_used_size:: 1420112 pkmem:56-max_used_size:: 1422104 pkmem:57-max_used_size:: 1396384 When I trigger a memory report (dump) using the command kill -SIGUSR1 971 ...the resulting output in my opensips.log file looks almost the same from one day top the next. Do I need to use a different Process ID so I can get a memory dump for the pkmem:25 child thread? If so, how do I know which PID to use? The command "ps ax | grep opensips" returns a list of 59 process ID's. Even if I get the right PID, what am I looking for in the memory dump? Typical memory dump output looks like this: 2016-12-15 15:54:44 41. N address=0x7f4be8db9048 frag=0x7f4be8db9018 size=48 used=1 2016-12-15 15:54:44 alloc'd from sr_module.c: register_module(151) 2016-12-15 15:54:44 start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed 2016-12-15 15:54:44 42. N address=0x7f4be8db90d8 frag=0x7f4be8db90a8 size=128 used=1 2016-12-15 15:54:44 alloc'd from cfg.lex: addstr(925) 2016-12-15 15:54:44 start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed ... 2016-12-15 15:54:44 63. N address=0x7f4be8db9f70 frag=0x7f4be8db9f40 size=80 used=1 2016-12-15 15:54:44 alloc'd from pvar.c: pv_add_extra(4480) 2016-12-15 15:54:44 start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed 2016-12-15 15:54:44 64. N address=0x7f4be8dba020 frag=0x7f4be8db9ff0 size=80 used=1 2016-12-15 15:54:44 alloc'd from pvar.c: pv_add_extra(4480) I have not found any cases where the value for "frag size" exceeds 240. The line that starts "alloc'd from" never seems to have a value in parentheses exceeding 4480. I cannot see anything that looks like a smoking gun. The list never goes beyond item 64. John Quick Smartvox Limited Tel: +44-1727-221221 ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel ___ Devel mailing list Devel@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/devel