I think you can. Check the documentation of rtpengine on github. And if you can, please use the latest commit.
On Tue, 3 Dec 2019, 18:02 Callum Guy, <[email protected]> wrote: > Hi All, > > I'm working through this memory issue and have some additional data. The > server crashed this afternoon due to memory exhaustion on a UDP listener > process, as far as I know at this time. Note that the processes all have > 4GB assigned so this is a gradual and constant growth issue. > > At this point I have my suspicions that it could be related to the > RTPEngine module - the first memory allocation error messages are all > related to what I believe the be RTCP session reports coming in from the > RTP engine servers. These came through about every second for an hour > before OpenSIPs finally gave up the ghost and restarted. > > Is this normal behaviour? We have only recently moved to RTPEngine from > RTPProxy so I am new to this software. Does anyone know if it is possible > to prevent RTPEngine from sending this data to OpenSIPs - its not something > that we require and I'd like to check if this is at all related to the > memory growth. Any other ideas would also be appreciated! > > Many thanks, > > Callum > > 2019-12-03T11:48:14.653225+00:00 TH-P-SIPREG-1 opensips[2521]: > ERROR:core:fm_malloc: not enough free pkg memory (214628248 bytes left, > need 536), please increase the "-M" command line parameter! > 2019-12-03T11:48:14.653851+00:00 TH-P-SIPREG-1 opensips[2521]: > INFO:core:fm_malloc: attempting defragmentation... > 2019-12-03T11:48:15.651495+00:00 TH-P-SIPREG-1 opensips[2521]: > INFO:core:fm_malloc: unable to alloc a big enough fragment! > 2019-12-03T11:48:15.652218+00:00 TH-P-SIPREG-1 opensips[2521]: > ERROR:rtpengine:rtpe_function_call: failed to decode bencoded reply from > proxy: d7:createdi1575373650e10:created_usi505483e11:last > signali1575373670e4:SSRCd10:3831331386d11:average > MOSd3:MOSi44e15:round-trip timei6754e6:jitteri0e11:packet > lossi0e7:samplesi9ee10:lowest MOSd3:MOSi44e15:round-trip > timei6175e6:jitteri0e11:packet lossi0e11:reported ati1575373656ee11:highest > MOSd3:MOSi44e15:round-trip timei6175e6:jitteri0e11:packet > lossi0e11:reported ati1575373656ee15:MOS > progressiond8:intervali3e7:entriesld3:MOSi44e15:round-trip > timei6175e6:jitteri0e11:packet lossi0e11:reported > ati1575373656eed3:MOSi44e15:round-trip timei6307e6:jitteri0e11:packet > lossi0e11:reported ati1575373660eed3:MOSi44e15:round-trip > timei6884e6:jitteri0e11:packet lossi0e11:reported > ati1575373664eed3:MOSi44e15:round-trip timei7016e6:jitteri0e11:packet > lossi0e11:reported ati1575373668eed3:MOSi44e15:round-trip > timei6368e6:jitteri0e11:packet lossi0e11:reported > ati1575373674eed3:MOSi44e15:round-trip timei6929e6:jitteri0e11:packet > lossi0e11:reported ati1575373679eed3:MOSi44e15:round-trip > timei6884e6:jitteri0e11:packet lossi0e11:reported > ati1575373684eed3:MOSi44e15:round-trip timei7241e6:jitteri0e11:packet > lossi0e11:reported ati1575373689eed3:MOSi44e15:round-trip > timei6988e6:jitteri0e11:packet lossi0e11:reported > ati1575373694eeeee10:1028099660d11:average MOSd3:MOSi43e15:round-trip > timei29936e6:jitteri0e11:packet lossi0e7:samplesi7ee10:lowest > MOSd3:MOSi43e15:round-trip timei29515e6:jitteri0e11:packet > lossi0e11:reported ati1575373661ee11:highest MOSd3:MOSi43e15:round-trip > timei29515e6:jitteri0e11:packet lossi0e11:reported ati1575373661ee15:MOS > progressiond8:intervali3e7:entriesld3:MOSi43e15:round-trip > timei29515e6:jitteri0e11:packet lossi0e11:reported > ati1575373661eed3:MOSi43e15:round-trip timei29817e6:jitteri0e11:packet > lossi0e11:reported ati1575373666eed3:MOSi43e15:round-trip > timei30695e6:jitteri0e11:packet lossi0e11:reported > ati1575373671eed3:MOSi43e15:round-trip timei29541e6:jitteri0e11:packet > lossi0e11:reported ati1575373676eed3:MOSi43e15:round-trip > timei30073e6:jitteri0e11:packet lossi0e11:reported > ati1575373681eed3:MOSi43e15:round-trip timei29526e6:jitteri0e11:packet > lossi0e11:reported ati1575373686eed3:MOSi43e15:round-trip > timei30391e6:jitteri0e11:packet lossi0e11:reported > ati1575373691eeeeee4:tagsd10:3044367218d3:tag10:30443672187:createdi1575373650e16:in > dialogue > with13:N8Nr5mDpcNt1a6:mediasld5:indexi1e4:type5:audio8:protocol7:RTP/AVP7:streamsld10:local > porti32494e8:endpointd6:family4:IPv47:address13:51.51.51.514:porti47360ee19:advertised > endpointd6:family4:IPv47:address9:10.0.0.154:porti12672ee11:last > packeti1575373694e5:flagsl3:RTP6:filled9:confirmed10:kernelizede4:SSRCi1028099660e5:statsd7:packetsi2045e5:bytesi351740e6:errorsi0eeed10:local > porti32495e8:endpointd6:family4:IPv47:address13:51.51.51.514:porti47361ee19:advertised > endpointd6:family4:IPv47:address9:10.0.0.154:porti12673ee11:last > packeti1575373691e5:flagsl4:RTCP6:filled9:confirmed10:kernelized17:no > kernel > supporte4:SSRCi1028099660e5:statsd7:packetsi8e5:bytesi1240e6:errorsi0eeee5:flagsl11:initialized4:send4:recveeee13:N8Nr5mDpcNt1ad3:tag13:N8Nr5mDpcNt1a7:createdi1575373650e16:in > dialogue > with10:30443672186:mediasld5:indexi1e4:type5:audio8:protocol7:RTP/AVP7:streamsld10:local > porti34226e8:endpointd6:family4:IPv47:address15:192.168.163.2224:porti11484ee19:advertised > endpointd6:family4:IPv47:address15:192.168.163.2224:porti11484ee11:last > packeti1575373694e5:flagsl3:RTP6:filled9:confirmed10:kernelizede4:SSRCi3831331386e5:statsd7:packetsi2018e5:bytesi347096e6:errorsi0eeed10:local > porti34227e8:endpointd6:family4:IPv47:address15:192.168.163.2224:porti11485ee19:advertised > endpointd6:family4:IPv47:address15:192.168.163.2224:porti11485ee11:last > packeti1575373694e5:flagsl4:RTCP6:filled9:confirmed10:kernelized17:no > kernel > supporte4:SSRCi3831331386e5:statsd7:packetsi9e5:bytesi1008e6:errorsi0eeee5:flagsl11:initialized4:send4:recv15:ICE > controllingeeeee6:totalsd3:RTPd7:packetsi4063e5:bytesi698836e6:errorsi0ee4:RTCPd7:packetsi17e5:bytesi2248e6:errorsi0eee6:result2:oke > > > On Sat, 30 Nov 2019 at 22:51, Callum Guy <[email protected]> wrote: > >> Hi Ben, >> >> Thank you for your reply and insight here, very helpful to know you're >> running a drastically different setting for the package memory. >> >> I presumed if it preallocated that I would have seen some issues during >> testing, hence I've ended up with figures that were intended to provide 75% >> of the system memory to the application. >> >> Memory usage had been creeping up all day at the time of writing however >> migrations to this platform had been on hold since the initial capture of >> memory usage although call traffic would have been relatively even during >> daytime hours where the increase continued. On that basis I'm still >> concerned that there is an issue with my config causing this growth however >> I've now increased available memory and restarted so I should have ample >> time to investigate this week, I'll report back any findings for the >> community benefit. I will give some serious thought to lowering the package >> allocation value once I've got to grips with the situation. >> >> Usefully this implementation shares a lot of common components to another >> variant which acts as a pure proxy and does not deal with registrations >> where I'm not seeing this issue so that will narrow down the search area >> somewhat. >> >> Thanks again for your time, >> >> Callum >> >> On Sat, 30 Nov 2019, 15:46 Ben Newlin, <[email protected]> wrote: >> >>> Callum, >>> >>> >>> >>> It’s my understanding that OpenSIPS does not release memory back to the >>> OS, but it also pre-allocates all memory at startup into its private pool >>> and then allocates from that internally. Normally shared memory should be >>> significantly higher than package memory. For reference, on our system we >>> run with “-m 1024 -M 64” and that is sufficient for us to process very high >>> traffic volume. We don’t do registration though, so that may affect the >>> sizes you need. >>> >>> >>> >>> You are setting your package memory size to 4G, so that will allocate 4G >>> memory for every package (process) that loads and then 2G for shared >>> memory. That will use up all the memory on your machine extremely quickly >>> for sure. The statistics you provided seem like the memory increase is >>> consistent with higher traffic levels on the second reading. You can see in >>> your case that all of your “pkmem” processes have an extremely high amount >>> of free memory (~3GB!). But that memory is still allocated from the OS, so >>> you are instructing OpenSIPS to allocate much more than your system memory >>> right at startup. >>> >>> >>> >>> Your shared memory also has just under 2GB free, so you have a lot of >>> headroom there too. Since OpenSIPS pre-allocates, the amount of memory >>> being used by the system overall should be fairly steady; if it is >>> continuously increasing that implies a leak somewhere. IIRC there are a few >>> processes/modules/commands in OpenSIPS or libraries it uses that do >>> allocate memory directly from the system and not from OpenSIPS’ pool. You >>> may need to investigate some of those to find out where your memory is >>> going, or look at other processes/daemons you have running that could be >>> using that memory. >>> >>> >>> >>> Ben Newlin >>> >>> >>> >>> *From: *Users <[email protected]> on behalf of Callum >>> Guy <[email protected]> >>> *Reply-To: *OpenSIPS users mailling list <[email protected]> >>> *Date: *Friday, November 29, 2019 at 10:57 AM >>> *To: *OpenSIPS users mailling list <[email protected]> >>> *Subject: *[OpenSIPS-Users] Memory Leak - runtime flags? >>> >>> >>> >>> Hi All, >>> >>> >>> >>> I have recently deployed a new registrar and have been seeing a gradual >>> increase in the memory footprint - enough that I'm having to expand the RAM >>> (its virtualised) to ensure it doesn't run out. >>> >>> >>> >>> You can see a diff of the statistics collected last night at 11pm and >>> today at 3pm here: >>> https://gist.github.com/spacetourist/2103503674e134bd598c7f1e3a82674c/revisions >>> >>> >>> >>> Processes 5-9 are my UDP SIP receiver threads (autoscaled down from an >>> initial footprint of 20 threads). >>> >>> >>> >>> Using 3.0.1 on CentOS 7 8GB RAM (soon to be 32GB!). Currently OpenSIPs >>> is using all the RAM (minus OS usage) and 2GB of swap. Trying to use dialog >>> and dr clustering if that is significant. Alos have NAT pings configured >>> for all registrations (4000 at time of writing). >>> >>> >>> >>> I am using runtime configuration flags of "*-m 2048 -M 4096*" and am >>> concerned that these were (way) too high, I think I've misinterpreted their >>> meaning during initial setup. Is this a ridiculous setting for my >>> environment? Is it just as simple as OpenSIPs being greedy with the memory >>> such that it doesn't bother to free anything while each process free space >>> remaining? Should my -M value * max number of processes fit into my RAM? I >>> guess with an 8GB system that would mean dropping this to "-M 256"? >>> >>> >>> >>> I've done some research into the issue however I haven't found anything >>> else that would be an obvious target so wondered if the community might >>> have some ideas of where I can begin investigations. >>> >>> >>> >>> Many thanks, >>> >>> >>> >>> Callum >>> >>> >>> >>> [image: Image removed by sender.] >>> >>> >>> >>> *0333 332 0000 | www.x-on.co.uk <http://www.x-on.co.uk> | * *[image: >>> Image removed by sender.] <https://www.linkedin.com/company/x-on> [image: >>> Image removed by sender.] <https://www.facebook.com/XonTel> [image: Image >>> removed by sender.] <https://twitter.com/xonuk>* >>> >>> X-on is a trading name of Storacall Technology Ltd a limited company >>> registered in England and Wales. >>> Registered Office : Avaland House, 110 London Road, Apsley, Hemel >>> Hempstead, Herts, HP3 9SD. Company Registration No. 2578478. >>> The information in this e-mail is confidential and for use by the >>> addressee(s) only. If you are not the intended recipient, please notify >>> X-on immediately on +44(0)333 332 0000 and delete the >>> message from your computer. If you are not a named addressee you must >>> not use, disclose, disseminate, distribute, copy, print or reply to this >>> email. Views or opinions expressed by an individual >>> within this email may not necessarily reflect the views of X-on or its >>> associated companies. Although X-on routinely screens for viruses, >>> addressees should scan this email and any attachments >>> for viruses. X-on makes no representation or warranty as to the absence >>> of viruses in this email or any attachments. >>> _______________________________________________ >>> Users mailing list >>> [email protected] >>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users >>> >> > > *0333 332 0000 | www.x-on.co.uk <http://www.x-on.co.uk> | ** > <https://www.linkedin.com/company/x-on> <https://www.facebook.com/XonTel> > <https://twitter.com/xonuk> * > > X-on is a trading name of Storacall Technology Ltd a limited company > registered in England and Wales. > Registered Office : Avaland House, 110 London Road, Apsley, Hemel > Hempstead, Herts, HP3 9SD. Company Registration No. 2578478. > The information in this e-mail is confidential and for use by the > addressee(s) only. If you are not the intended recipient, please notify > X-on immediately on +44(0)333 332 0000 and delete the > message from your computer. If you are not a named addressee you must not > use, disclose, disseminate, distribute, copy, print or reply to this email. > Views > or opinions expressed by an individual > within this email may not necessarily reflect the views of X-on or its > associated companies. Although X-on routinely screens for viruses, > addressees should scan this email and any attachments > for viruses. X-on makes no representation or warranty as to the absence of > viruses in this email or any attachments. > > _______________________________________________ > Users mailing list > [email protected] > http://lists.opensips.org/cgi-bin/mailman/listinfo/users >
_______________________________________________ Users mailing list [email protected] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
