Greetings OpenSIPS gurus; I've run into an issue that I'm not making any real headway with on my own. I have an OpenSIPS 2.4.2 running on CentOS7 that does not appear to be releasing the bulk of its shared memory when calls die off. Eventually OpenSIPS runs itself out of shared memory and goes beserk when it can't reclaim memory for new transactions, or I've also seen the kernel oomkiller take it down.
The setup is, as mentioned, a 2.4.2 which has a chunk of perl living on its back that handles some API directed call decisions. Because of the inevitable delay in the API lookups I am running a large number of threads (100) to ensure that I don't start dropping packets under high levels of load (50 CPS+ - until I bumped children counts up I was seeing packets being ignored which got the End User very excitable). I appreciate having lots of children amplifies memory usage but my belief is during regular load on our side (~6000 calls at ~30 - 40CPS) I should have a full day's runtime easily w/o running into problems. The issue is that because the memory is not released overnight it starts the next day with half its shared memory gone, and then things go to pot the next day when it runs through all its remaining shared memory. I have it configured for a very large (I think) amount of memory to attempt to compensate short-term - 2048MB shared memory, 128MB package memory. This is not a well thought-out number, just me throwing RAM at the problem. (For reference this is on a VM running with 4 cores and 4096MB of RAM) I have graphed the memory stats out of the FIFO, after enabling memstats and DBG_MALLOC/etc. The graphs can be found here: https://imgur.com/a/oapKYJW You'll see that the first graph, the main shmem:, shows the usage gets tied up and remains after calls die off (these examples run for 20 minutes after the calls stopped - but I can leave it overnight and the shmem: usage won't drop). This being said, the other graphed modules show that it releases memory correctly after calls quit like they should. None of the other modules graphed (eg, _all_ of the modules I was using in OpenSIPS) showed no significant usage and I've omitted them. Interesting that the 'perl' module shows no stats ever which makes me wonder if the stats on this module works. I am not storing anything inside perl, besides lots of local session variables, and all channel variables are pushed back into OpenSIPS AVPs, but I assume there must be some memory overhead, although it may not significantly change. When I shut OpenSIPS down I get child dumps like the following: (I've included only one for brevity, but they're all much of a muchness) https://pastebin.com/KG2qxxBa If I'm reading this right (and there's a strong possibility I'm misunderstanding what I'm looking at) the memory is tied up in io_wait and the parser. I'm not sure what this signifies, however. So I guess what I'm asking is: Am I understanding what I'm looking at correctly and that OpenSIPS is not releasing its shared memory which causes it to run out of memory quicker the following day? (Which lines up with the symptoms of shorter runtimes before failure) Based on the dumps where should I look for the source of this issue? Finally, and unrelated to the above, I attempted to mail the official OpenSIPS peeps to talk about paid support but none of the 'contact us' submits appear to work - when I click on the submit button it changes shade but does not appear to act. Certainly I never received any response. If there was a direct eMail address I could send to that would be preferable. As always, my thanks for your time. - Jock PS - List of modules I'm using in this config:loadmodule "signaling.so" loadmodule "sl.so" loadmodule "tm.so" loadmodule "rr.so" loadmodule "maxfwd.so" loadmodule "textops.so" loadmodule "uri.so" loadmodule "mi_fifo.so" loadmodule "siptrace.so" loadmodule "sipmsgops.so" loadmodule "dialog.so" loadmodule "uac_auth.so" loadmodule "uac.so" loadmodule "perl.so" loadmodule "acc.so" loadmodule "proto_hep.so" loadmodule "proto_udp.so" loadmodule "compression.so" _______________________________________________ Users mailing list [email protected] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
