Hi Anca,
On 20.04.2011, at 13:24, Anca Vamanu wrote:
> Hi John,
>
> Thanks for the logs.
> Do you get the memory statistics from opensips statistics or from top?
> In the logs I don't see a shared memory leak..
Hm... Maybe the memory leak in the system libraries?
Memory usage is calculated as follows:
# ps aux | head -n 1
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
# ps aux | grep -E "/usr/sbin/opensips -[P]"
opensips 8389 0.0 0.0 451444 9880 ? S Apr19 0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8395 0.2 0.0 451448 5216 ? S Apr19 2:06
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8396 0.0 0.0 451448 5184 ? S Apr19 0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8398 0.0 0.0 451452 5160 ? S Apr19 0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8399 0.0 0.1 451684 12732 ? S Apr19 0:10
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8400 0.0 0.1 451704 12708 ? S Apr19 0:09
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8402 0.0 0.1 451708 12644 ? S Apr19 0:09
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8403 0.0 0.1 451708 12640 ? S Apr19 0:10
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8405 0.0 0.0 451448 5088 ? S Apr19 0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8406 0.0 0.0 451448 5088 ? S Apr19 0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8409 0.0 0.0 451448 5088 ? S Apr19 0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8410 0.0 0.0 451448 5088 ? S Apr19 0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8411 0.0 0.0 450644 3148 ? S Apr19 0:07
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8412 0.0 0.1 451448 11572 ? S Apr19 0:11
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8413 0.0 0.0 451444 5304 ? S Apr19 0:05
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8414 0.0 0.0 451452 9660 ? S Apr19 0:01
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8418 0.0 0.0 451452 9336 ? S Apr19 0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8426 0.0 0.0 451452 9684 ? S Apr19 0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8428 0.0 0.0 451452 5920 ? S Apr19 0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
opensips 8429 0.0 0.0 451416 6544 ? S Apr19 0:01
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g
opensips
# ps aux | grep -E "/usr/sbin/opensips -[P]" | awk '{a += $6} END {print a}'
157684
i.e. sum of RSS value, where RSS (man ps)
RSS - resident set size, the non-swapped physical memory that a task has used
(in kiloBytes). (alias rssize, rsz).
How can I help?
> On 04/20/2011 10:18 AM, John Khvatov wrote:
>> Hello Anca,
>>
>> On 19.04.2011, at 13:21, Anca Vamanu wrote:
>>
>>> Hi John,
>>>
>>> That looks very bad...
>>> Please help me investigate this.
>>>
>>> What you can do is to compile with memory debugging support - in
>>> Makefile.defs search -DDBG_QM_MALLOC line, uncomment it, move it above
>>> -DF_MALLOC line and comment this line with -DF_MALLOC. Then recompile and
>>> reinstall all. Also set mem_dump=1 in your config.
>> I've built OpenSIPS from latest SVN head (branch 1.6) as you said.
>>
>>> After a couple of hours, when you see that the memory is high - you can
>>> either shutdown opensips or ask it to dump the memory by calling 'kill
>>> -SIGUSR1 lowest_pid' , where lowest_pid is the smallest pid when doing 'ps
>>> aux | grep opensips'. It will dump lots of info in the log file(this why is
>>> better to do this quite soon - maybe after 1-2 hours in the rhythm the
>>> memory increases for you). In the logs you will see all the chunks of
>>> memory and where they were allocated. Look from the end if there is a
>>> function repeated many times. You can also send the part of the log with
>>> the dump to me ( not on the list, because it will be quite large).
>> Done.
>>
>> SIGUSR1:
>> http://dev.sgu.ru/pub/memdump.log.gz
>>
>> SIGTERM:
>> http://dev.sgu.ru/pub/memdump.log.shutdown.gz
>>
>>
>>> On 04/19/2011 10:30 AM, John Khvatov wrote:
>>>> Hello Anca,
>>>>
>>>> Sorry, the memory leak is still here:
>>>>
>>>> http://dev.sgu.ru/pub/opensips_memory_usage.2.png
>>>>
>>>> BTW, dialog module was not used in my script. I've removed loadmodule
>>>> before update.
>>>>
>>>>
>>>> On 12.04.2011, at 0:00, John Khvatov wrote:
>>>>
>>>>> Hello Anca,
>>>>>
>>>>> Thank you for the advice. I built OpenSIPS from rev7863. BTW, rev7824
>>>>> was used in production.
>>>>>
>>>>> I'll let you know the result within a few days.
>>>>>
>>>>>
>>>>> On 11.04.2011, at 17:48, Anca Vamanu wrote:
>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> Can you please take the pua and pua_usrloc and dialog modules from svn?
>>>>>> There were some problems discovered in there and fixed.
>>>>>> The svn version of these modules ( 1.6 branch) is stable, we are using
>>>>>> it in production.
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Anca Vamanu
>>>>>> OpenSIPS Developer
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 04/11/2011 12:45 AM, John Khvatov wrote:
>>>>>>> On 10.04.2011, at 8:35, Ronald Cepres wrote:
>>>>>>>
>>>>>>>> Have you tried increasing the amount of allocated memory for OpenSIPS?
>>>>>>> I have same problem with OpenSIPS since 1.6.0 release. OpenSIPS has
>>>>>>> huge memory leak somewhere. Increasing the amount of allocated memory
>>>>>>> helps only delay crash.
>>>>>>>
>>>>>>> Memory usage graph (1.6.4, KkB is MB):
>>>>>>> http://dev.sgu.ru/pub/opensips_memory_usage.png
>>>>>>>
>>>>>>> I use the following modules:
>>>>>>>
>>>>>>> $ grep loadmodule opensips.cfg
>>>>>>> loadmodule "signaling.so"
>>>>>>> loadmodule "sl.so"
>>>>>>> loadmodule "tm.so"
>>>>>>> loadmodule "maxfwd.so"
>>>>>>> loadmodule "rr.so"
>>>>>>> loadmodule "textops.so"
>>>>>>> loadmodule "localcache.so"
>>>>>>> loadmodule "mi_fifo.so"
>>>>>>> loadmodule "mi_datagram.so"
>>>>>>> loadmodule "db_flatstore.so"
>>>>>>> loadmodule "siptrace.so"
>>>>>>> loadmodule "db_text.so"
>>>>>>> loadmodule "db_postgres.so"
>>>>>>> loadmodule "db_http.so"
>>>>>>> loadmodule "usrloc.so"
>>>>>>> loadmodule "registrar.so"
>>>>>>> loadmodule "auth.so"
>>>>>>> loadmodule "auth_db.so"
>>>>>>> loadmodule "uri.so"
>>>>>>> loadmodule "domain.so"
>>>>>>> loadmodule "dialog.so"
>>>>>>> loadmodule "mediaproxy.so"
>>>>>>> loadmodule "nat_traversal.so"
>>>>>>> loadmodule "permissions.so"
>>>>>>> loadmodule "alias_db.so"
>>>>>>> loadmodule "avpops.so"
>>>>>>> loadmodule "mi_xmlrpc.so"
>>>>>>> loadmodule "dialplan.so"
>>>>>>> loadmodule "pike.so"
>>>>>>> loadmodule "pua.so"
>>>>>>> loadmodule "pua_usrloc.so"
>>>>>>>
>>>>>>> --
>>>>>>> WBR, John Khvatov
>>>>> --
>>>>> WBR, John Khvatov
>
> _______________________________________________
> Users mailing list
> [email protected]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
--
WBR, John Khvatov
_______________________________________________
Users mailing list
[email protected]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
