Hi Daniel,
I have run into a leak in 1.2 and I assume it is the same one that
Ovidiu ran into. I see in your response that it was "backported to
1.2", but I'm not sure how to get the fix. When I look at the SVN
repository at:
http://www.openser.org/pub/openser/latest-1.2.x/, the date is earlier
than the date of your email exchange so I don't think the fix has been
added there. Can you please let me know how I can get it?
thanks,
Tim
On 3/23/07, Daniel-Constantin Mierla <[EMAIL PROTECTED]> wrote:
> Hello Ovidiu,
>
> On 03/23/07 17:04, Ovidiu Sas wrote:
> > Hi Daniel,
> >
> > Can we backport this one to 1.2?
> already done, two minutes after the commit in trunk.
>
> Cheers,
> Daniel
>
> >
> >
> > Regards,
> > Ovidiu Sas
> >
> > On 3/22/07, Daniel-Constantin Mierla <[EMAIL PROTECTED]> wrote:
> >> Hello,
> >>
> >> the supposed fragmentation turned out to be a mem leak in pkg.
Please
> >> take the latest SVN version and try again to see if you got same
> >> results.
> >>
> >> Thanks,
> >> Daniel
> >>
> >> On 03/19/07 18:52, Christian Schlatter wrote:
> >> > ...
> >> >>> The memory statistics indeed show a high number of memory
fragments:
> >> >>>
> >> >>> before 'out of memory':
> >> >>>
> >> >>> shmem:total_size = 536870912
> >> >>> shmem:used_size = 59607040
> >> >>> shmem:real_used_size = 60106488
> >> >>> shmem:max_used_size = 68261536
> >> >>> shmem:free_size = 476764424
> >> >>> shmem:fragments = 9897
> >> >>>
> >> >>> after 'out of memory' (about 8000 calls per process):
> >> >>>
> >> >>> shmem:total_size = 536870912
> >> >>> shmem:used_size = 4171160
> >> >>> shmem:real_used_size = 4670744
> >> >>> shmem:max_used_size = 68261536
> >> >>> shmem:free_size = 532200168
> >> >>> shmem:fragments = 57902
> >> >>>
> >> >>>>
> >> >>>> You can try to compile openser with -DQM_JOIN_FREE (add it
in DEFS
> >> >>>> variable of Makefile.defs) and test again. Free fragments
should be
> >> >>>> merged and fragmentation should not occur -- processing
will be
> >> >>>> slower. We will try for next release to provide a better
solution
> >> >>>> for that.
> >> >>>
> >> >>> Compiling openser with -DQM_JOIN_FREE did not help. I'm not
sure how
> >> >>> big of a problem this fragmentation issue is.
> >> >> What is the number of fragments with QM_JOIN_FREE after
flooding?
> >> >
> >> > The numbers included above are with QM_JOIN_FREE enabled.
> >> >
> >> >>> Do you think it would make sense to restart our production
openser
> >> >>> instances from time to time just to make sure they're not
running
> >> >>> into this memory fragmentation limits?
> >> >> The issue will occur only when the call rate reaches the
limits of
> >> >> the proxy's memory. Otherwise the chunks are reused.
Transactions and
> >> >> avps are rounded up to be sure there will be minimized the
number of
> >> >> different sizes for memory chunks. It wasn't reported too often,
> >> >> maybe that's why no big attention was paid to it. This memory
system
> >> >> is in place since the beginning of ser. Alternative is to use
sysv
> >> >> shared memory, but is much slower, along with libc private
memory
> >> >> manager.
> >> >
> >> > I've done some more testing and the same out-of-memory stuff
happens
> >> > when I run sipp with 10 calls per second only. I tested with
> >> > 'children=1' and I only could get through about 8200 calls (again
> >> > those 8000 calls / process). And this is with QM_JOIN_FREE
enabled.
> >> >
> >> > Memory statistics:
> >> >
> >> > before:
> >> > shmem:total_size = 536870912
> >> > shmem:used_size = 2311976
> >> > shmem:real_used_size = 2335720
> >> > shmem:max_used_size = 2465816
> >> > shmem:free_size = 534535192
> >> > shmem:fragments = 183
> >> >
> >> > after:
> >> > shmem:total_size = 536870912
> >> > shmem:used_size = 1853472
> >> > shmem:real_used_size = 1877224
> >> > shmem:max_used_size = 2465816
> >> > shmem:free_size = 534993688
> >> > shmem:fragments = 547
> >> >
> >> > So I'm not sure if this is really a fragmentation issue. 10
cps surely
> >> > doesn't reach the proxy's memory.
> >> >
> >> > Thoughts?
> >> >
> >> > Christian
> >> >
> >> >
> >> >
> >> >> Cheers,
> >> >> Daniel
> >> >>
> >> >>>
> >> >>> thanks,
> >> >>> Christian
> >> >>>
> >> >>>>
> >> >>>> Cheers,
> >> >>>> Daniel
> >> >>>>
> >> >>>> On 03/18/07 01:21, Christian Schlatter wrote:
> >> >>>>> Christian Schlatter wrote:
> >> >>>>> ...
> >> >>>>>>
> >> >>>>>> I always had 768MB shared memory configured though, so I
still
> >> >>>>>> can't explain the memory allocation errors I got. Some
more test
> >> >>>>>> runs revealed that I only get these errors when using a more
> >> >>>>>> production oriented config that loads more modules than
the one
> >> >>>>>> posted in my earlier email. I now try to figure out what
exactly
> >> >>>>>> causes these memory allocation errors that happen
reproducibly
> >> >>>>>> after about 220s at 400 cps.
> >> >>>>>
> >> >>>>> I think I found the cause for the memory allocation
errors. As
> >> >>>>> soon as I include an AVP write operation in the routing
script, I
> >> >>>>> get 'out of memory' messages after a certain number of calls
> >> >>>>> generated with sipp.
> >> >>>>>
> >> >>>>> The routing script to reproduce this behavior looks like
(full
> >> >>>>> config available at
> >> >>>>> http://www.unc.edu/~cschlatt/openser/openser.cfg):
> >> >>>>>
> >> >>>>> route{
> >> >>>>> $avp(s:ct) = $ct; # commenting this line solves
> >> >>>>> # the memory problem
> >> >>>>>
> >> >>>>> if (!method=="REGISTER") record_route();
> >> >>>>> if (loose_route()) route(1);
> >> >>>>>
> >> >>>>> if (uri==myself) rewritehost("xx.xx.xx.xx");
> >> >>>>> route(1);
> >> >>>>> }
> >> >>>>>
> >> >>>>> route[1] {
> >> >>>>> if (!t_relay()) sl_reply_error();
> >> >>>>> exit;
> >> >>>>> }
> >> >>>>>
> >> >>>>> An example log file showing the 'out of memory' messages is
> >> >>>>> available at
http://www.unc.edu/~cschlatt/openser/openser.log .
> >> >>>>>
> >> >>>>> Some observations:
> >> >>>>>
> >> >>>>> - The 'out of memory' messages always appear after about
8000 test
> >> >>>>> calls per worker process. One call consists of two SIP
> >> >>>>> transactions and six end-to-end SIP messages. An openser
with 8
> >> >>>>> children handles about 64'000 calls, whereas 4 children only
> >> >>>>> handle about 32'000 calls. The sipp call rate doesn't
matter, only
> >> >>>>> number of calls.
> >> >>>>>
> >> >>>>> - The 8000 calls per worker process are independent from the
> >> >>>>> amount of shared memory available. Running openser with -m
128 or
> >> >>>>> -m 768 does not make a difference.
> >> >>>>>
> >> >>>>> - The more AVP writes are done in the script, the less
calls go
> >> >>>>> through. It looks like each AVP write is leaking memory
(unnoticed
> >> >>>>> by the memory statistics).
> >> >>>>>
> >> >>>>> - The fifo memory statistics do not reflect the 'out of
memory'
> >> >>>>> syslog messages. Even if openser does not route a single SIP
> >> >>>>> message because of memory issues, the statistics still
show a lot
> >> >>>>> of 'free' memory.
> >> >>>>>
> >> >>>>>
> >> >>>>> All tests were done with openser SVN 1.2 branch on Ubuntu
dapper
> >> >>>>> x86. I think the same is true for 1.1 version but I
haven't tested
> >> >>>>> that yet.
> >> >>>>>
> >> >>>>>
> >> >>>>> Christian
> >> >>>>>
> >> >>>
> >> >>>
> >> >
> >> >
> >> > _______________________________________________
> >> > Users mailing list
> >> > [email protected]
> >> > http://openser.org/cgi-bin/mailman/listinfo/users
> >> >
> >>
> >> _______________________________________________
> >> Users mailing list
> >> [email protected]
> >> http://openser.org/cgi-bin/mailman/listinfo/users
> >>
> >
>
> _______________________________________________
> Users mailing list
> [email protected]
> http://openser.org/cgi-bin/mailman/listinfo/users
>
