Hello,
can you try the attached patch? It's the same patch, just for two
versions, one is for 3.3.x and the other for devel version
It initializes the SIP message variable that is passed to perl after
creating the temporary environment, so it is actually destroyed by the
perl embedded interpreter.
Cheers,
Daniel
On 7/25/13 1:29 AM, David Cunningham wrote:
Hi Daniel,
The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server
release 5.4. If I remember right programs running under Valgrind can
have issues, so I'm not sure if the customer will want to do that.
Ideally we'd do it on a test system, but I'm not sure if we have any
RHEL available.
I'll see what we can do. Thanks again.
On 25 July 2013 04:55, Daniel-Constantin Mierla <mico...@gmail.com
<mailto:mico...@gmail.com>> wrote:
Hello,
I would say that perl_exec() is the one with the highest chances
to be the reason for the leak. Next is line would be db_mysql
module, if liked with some custom mysql client library, although
even in this case will be unlikely.
Back to perl, the module itself does not call any malloc, so it
might be the embedding Perl API that is not used properly in the
module.
Can you use some testbed, set children=1 and run kamailio under
valgrind, then do some calls and see if it detects the source of
the leak?
I'm not using the perl module, I will try to check it whenever I
get a chance in the next days. What version of perl do you have
installed?
Cheers,
Daniel
On 7/24/13 10:31 AM, David Cunningham wrote:
Hello,
We don't do any kamctl commands at all. We do have various
modules loaded, as follows. The primary functions we use Kamailio
for are phone registrations through usrloc, and routing calls to
Asterisk through logic contained in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla
<mico...@gmail.com <mailto:mico...@gmail.com>> wrote:
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed
Kamailio.
then all the instructions I gave are useless, they are for
debugging kamailio's internal memory manager, which handles
pkg and shm mallocs.
The chances to be from kamailio itself are very low now. Do
you do lot of mi commands (e.g., kamctl ...)? The mi api uses
system malloc, but the rest of code should use internal
memory manager which does not go beyond the limits set with
-m and -M, thus not causing an OS memory exhaustion.
Can you list what modules are you loading? At some point it
was a leak in libssl, in case you use tls a lot. But could be
another external library...
Cheers,
Daniel
2. You're right, DEBUG_MEMORY is a local configuration
setting. If defined it sets memdbg to -2, and memlog to -2.
The debug setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx 8.5kb. Is
that a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla
<mico...@gmail.com <mailto:mico...@gmail.com>> wrote:
Hello,
first, to clarify, is the system memory or kamailio's
pkg/shm memory running out? If the operating system runs
out of memory, then should be a leak in a library,
because kamailio modules uses only from a pre-allocated
chunk, not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
Hello,
We're running a Kamailio 3.3.4 system, and Kamailio
is slowly using more and more memory. Over a couple
of weeks it will run out of system memory.
We tried to enable memory debugging doing the
following, but it resulted in Kamailio not
responding to any SIP packets. Would anyone have
advice please on how to debug the situation?
1. In Makefile.defs set MEMDBG to 1 and recompile
Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
do you set something special in config when DEBUG_MEMORY
is 1? It is not by default there, so I assume you added
some rules based on this pre-processor directive.
For memory troubleshooting, set memlog to a value lower
than debug parameter in config file and try with
mem_summary=12 for a more compact output. See more about
these parameters in the wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a while in normal conditions, then
restart it to get the memory usage summaries. There
should be indication if there is some leak, by seeing
memory chunks allocated many times from a function used
at runtime. You can send the memory summary for a
process here, we can look at it.
While this was running and Kamailio didn't respond
to packets, it logged lots of lines like this:
Do you have syslog to be configured in asynchronous
mode? See the notes from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less than debug value, that means printing
few log messages for each memory operation. You can make
memdbg higher and rely on memlog for memory summaries,
otherwise will be lot of log messages related to memory.
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128)
called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128)
returns address 0x40048918 frag. 0x40048900
(size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128)
called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128)
returns address 0x400489c8 frag. 0x400489b0
(size=128) on 1 -th hit
addstr() is a function used only for parsing
configuration file, as long as you can still see them,
the configuration file parsing was not finish. addstr()
is not a source of leaks because it is not used at runtime.
If you have large config file, then you can get close to
the limits of the private memory, which is set to 4MB.
You can increase its value using -M parameter (e.g.,
start kamailio with -M 8 to set it to use 8MB of memory).
Over the time, the private memory can get used due to
fragmentation, you can set the mem_join parameter in
config file to avoid it (works when compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then you can
use sercmd with pkg.stats command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory stats are printed by 'kamctl fifo
get_statistics shmem:'
When you see significant increase of the memory usage,
then you can restart to get the summaries.
You should run these commands after start, just to see
the initial usage of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing list
sr-users@lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
<tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
diff --git a/modules/app_perl/perlfunc.c b/modules/app_perl/perlfunc.c
index 96f7596..90a80d4 100644
--- a/modules/app_perl/perlfunc.c
+++ b/modules/app_perl/perlfunc.c
@@ -127,16 +127,15 @@ int perl_exec2(struct sip_msg* _msg, char* fnc, char*
mystr) {
return -1;
}
- m = sv_newmortal();
- sv_setref_pv(m, "Kamailio::Message", (void *)_msg);
- SvREADONLY_on(SvRV(m));
-
-
ENTER; /* everything created after here */
SAVETMPS; /* ...is a temporary variable. */
- PUSHMARK(SP); /* remember the stack pointer */
+ PUSHMARK(SP); /* remember the stack pointer */
XPUSHs(m); /* Our reference to the stack... */
+ m = sv_newmortal();
+ sv_setref_pv(m, "Kamailio::Message", (void *)_msg);
+ SvREADONLY_on(SvRV(m));
+
if (mystr)
XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
/* Our string to the stack... */
diff --git a/modules_k/perl/perlfunc.c b/modules_k/perl/perlfunc.c
index f65b9c0..9684a0d 100644
--- a/modules_k/perl/perlfunc.c
+++ b/modules_k/perl/perlfunc.c
@@ -127,16 +127,16 @@ int perl_exec2(struct sip_msg* _msg, char* fnc, char*
mystr) {
return -1;
}
- m = sv_newmortal();
- sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
- SvREADONLY_on(SvRV(m));
-
ENTER; /* everything created after here */
SAVETMPS; /* ...is a temporary variable. */
PUSHMARK(SP); /* remember the stack pointer */
XPUSHs(m); /* Our reference to the stack... */
+ m = sv_newmortal();
+ sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
+ SvREADONLY_on(SvRV(m));
+
if (mystr)
XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
/* Our string to the stack... */
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
