Re: [Kamailio-Users] Understanding memory-leaks by inspecting PKG/SHM memory status at shutdown
2010/4/8 Iñaki Baz Castillo i...@aliax.net: 2010/4/8 Daniel-Constantin Mierla mico...@gmail.com: it does not look as a dump with memory debugging on. SOrry, I just applied MEMDBG=1 in one of the servers and got the output in the other. When memdbg is on, you should get something like: 0(17665) 1. N address=0xb5ab2440 frag=0xb5ab2428 size=4 used=1 0(17665) alloc'd from timer.c: init_timer(52) Notice the alloc'd ...' line which specifies the place where the memory was allocated. A leak is signaled by many occurrences of allocation from same place (skipping the part of allocation done for config parsing and module initialization which happen only one, at startup). Hi again. I already have a kamailio 1.5.4 compiled with mem debugging (as kamailio -V shows DBG_QM_MALLOC flag). In config file I have: debug=3 memlog=3 # Same behaviour with 1 or 2 as it equal or less than 'debug'. Unfortunatelly the ammount of logs it generates makes it unusable for production environment (~ 10 calls per second). Just restarting kamailio when memlog is enabled takes really long time (unfortuantelly I must restart it when adding new entries to 'address' table due to the issue when performing fifo address_reload which completely freezes kamailio sometimes). Do I miss something? is it possible to log allocated and freeded memory without generating so many logs? If not, I could use memlog=1 without memory debugging compiled and I would check periodically the ammounf of PKG memory used. This is, I get this output: kamailio[11770]: Memory status (pkg): kamailio[11770]: fm_status (0x701a40): kamailio[11770]: heap size= 16777216 kamailio[11770]: used= 190936, used+overhead=250696, free=16526520 kamailio[11770]: max used (+overhead)= 258464 I can check it periodically and inspect if the used memory is increasing. If so there must be a memleak. Am I right? Thanks. -- Iñaki Baz Castillo i...@aliax.net ___ Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Re: [Kamailio-Users] Understanding memory-leaks by inspecting PKG/SHM memory status at shutdown
On 4/8/10 1:21 PM, Iñaki Baz Castillo wrote: 2010/4/8 Iñaki Baz Castilloi...@aliax.net: 2010/4/8 Daniel-Constantin Mierlamico...@gmail.com: it does not look as a dump with memory debugging on. SOrry, I just applied MEMDBG=1 in one of the servers and got the output in the other. When memdbg is on, you should get something like: 0(17665) 1. N address=0xb5ab2440 frag=0xb5ab2428 size=4 used=1 0(17665) alloc'd from timer.c: init_timer(52) Notice the alloc'd ...' line which specifies the place where the memory was allocated. A leak is signaled by many occurrences of allocation from same place (skipping the part of allocation done for config parsing and module initialization which happen only one, at startup). Hi again. I already have a kamailio 1.5.4 compiled with mem debugging (as kamailio -V shows DBG_QM_MALLOC flag). In config file I have: debug=3 memlog=3 # Same behaviour with 1 or 2 as it equal or less than 'debug'. Unfortunatelly the ammount of logs it generates makes it unusable for production environment (~ 10 calls per second). Just restarting kamailio when memlog is enabled takes really long time (unfortuantelly I must restart it when adding new entries to 'address' table due to the issue when performing fifo address_reload which completely freezes kamailio sometimes). Do I miss something? is it possible to log allocated and freeded memory without generating so many logs? the goal is to see the places where the memory was allocated. That will give the proper hints about the leak. What you can do is to print pkg status only when you send SIGUSR1 -- I attached a patch for that. In this way, a restart does not print pkg and shm status, so it is fast. At runtime, when you send SIGUSR1 to a pid, the others can work just fine, so processing should not be affected that much. Use kamctl ps to spot the pid of an udp worker. Cheers, Daniel If not, I could use memlog=1 without memory debugging compiled and I would check periodically the ammounf of PKG memory used. This is, I get this output: kamailio[11770]: Memory status (pkg): kamailio[11770]: fm_status (0x701a40): kamailio[11770]: heap size= 16777216 kamailio[11770]: used= 190936, used+overhead=250696, free=16526520 kamailio[11770]: max used (+overhead)= 258464 I can check it periodically and inspect if the used memory is increasing. If so there must be a memleak. Am I right? Thanks. -- Daniel-Constantin Mierla * http://www.asipto.com/ * http://twitter.com/miconda * http://www.linkedin.com/in/danielconstantinmierla Index: main.c === --- main.c (revision 5999) +++ main.c (working copy) @@ -457,7 +457,7 @@ while(wait(0) 0); /* Wait for all the children to terminate */ signal(SIGALRM, sig_alarm_abort); - cleanup(1); /* cleanup show status*/ + cleanup(0); /* cleanup show status*/ alarm(0); signal(SIGALRM, SIG_IGN); dprint(Thank you for flying NAME \n); @@ -469,8 +469,8 @@ LM_GEN1(memlog, Memory status (pkg):\n); pkg_status(); #endif - LM_GEN1(memlog, Memory status (shm):\n); - shm_status(); + //LM_GEN1(memlog, Memory status (shm):\n); + //shm_status(); break; case SIGCHLD: ___ Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Re: [Kamailio-Users] Understanding memory-leaks by inspecting PKG/SHM memory status at shutdown
2010/4/8 Daniel-Constantin Mierla mico...@gmail.com: Do I miss something? is it possible to log allocated and freeded memory without generating so many logs? the goal is to see the places where the memory was allocated. That will give the proper hints about the leak. What you can do is to print pkg status only when you send SIGUSR1 -- I attached a patch for that. Great, I'll use it in the standby server and will test with SIPp in order to determine the possible memory leak. In this way, a restart does not print pkg and shm status, so it is fast. At runtime, when you send SIGUSR1 to a pid, the others can work just fine, so processing should not be affected that much. Use kamctl ps to spot the pid of an udp worker. For this purpose I've created a bash script which does the job for me :) I attach it. In the production server (no mem debugging compiled) I've created a cron job which invokes the attached script every 5 minutes. Then with other scritp I filter the PKG status from kamailio log file (rsyslog logs it to a separate file) and get this output: Apr 8 13:55:01 kamailio[11770]: used= 190936, used+overhead=250696, free=16526520 Apr 8 13:55:01 kamailio[11770]: max used (+overhead)= 258464 Apr 8 13:55:01 kamailio[11758]: used= 191176, used+overhead=250648, free=16526568 Apr 8 13:55:01 kamailio[11758]: max used (+overhead)= 257864 Apr 8 14:05:01 kamailio[11770]: used= 190936, used+overhead=250696, free=16526520 Apr 8 14:05:01 kamailio[11770]: max used (+overhead)= 258464 Apr 8 14:05:01 kamailio[11758]: used= 191176, used+overhead=250648, free=16526568 Apr 8 14:05:01 kamailio[11758]: max used (+overhead)= 257864 I cannot determine the memory leak cause with this, but I can determine that some memory exists if the 'used' value is increased, right? Thanks a lot. -- Iñaki Baz Castillo i...@aliax.net kamailio_log_pkg_mem.sh Description: Bourne shell script ___ Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
