Re: [SOLVED] Re: net/asterisk13: memory leak under 12-CURRENT?
On 09/29/2017 10:59, O. Hartmann wrote: > On Thu, 28 Sep 2017 09:38:58 +0200 > Guido Falsi wrote: > >> On 09/28/2017 08:11, O. Hartmann wrote: >>> On Wed, 27 Sep 2017 09:05:42 +0200 >>> Guido Falsi wrote: >>> On 09/26/2017 15:41, O. Hartmann wrote: > On Tue, 26 Sep 2017 15:06:23 +0200 > Guido Falsi wrote: > Since I run net/asterisk with automatic module loading (I'm new to > asterisk), this is very likely and might cause the problem somehow. > You can exclude single modules from autoloading via modules.conf. >> Not sure, restarting the daemon should free any leaked memory the daemon >> has. If a killed process leaves memory locked at the system level there >> should be some other cause. > > Even with no runnidng asterisk, memory level drops after the last shutdown > of asterisk and keeps that low. Even for weeks! My router never shows that > high memory consumption, even under load. But while asterisk is running does the memory usage increase unbounded till filling all available memory or does it stabilize at some point? >>> >>> While Asterisk is running, it doesn't consume much memory, but stopping >>> Asterisk, I would expect that the claimed memory is freed again. It isn't, >>> not all memory is freed. Starting Asterisk then again from this reduced >>> memory level, it claims its memory, "stabilzes" at a certain point while >>> running and again, stopping Asterisk leaves the free memory now at a much >>> lower level never been leveld out - as I said. >>> >>> I played this game last night ~ 20 times until the free memory dropped >>> beneath 3 GB after asterisk has been shut down. This morning, the level was >>> at the same low level as I left it. The router has nothing special to do, >>> the workload is not memory consuming even for weeks! And if there is load, >>> after the load went away, the memory consumption always leveld out and >>> freed memory. Asterisk is relatively memory hungry, especially with all modules enabled. It also caches and logs various information in RAM, even doing "nothing" it will cache and log that "nothing" activity. If memory does stabilize after some point it's not really a leak but it's standard memory usage. To reduce it you should disable all unused modules. >>> >>> I don't understand here. Even if Asterisk is memory hungry - it has ~ 3 GB >>> to use. But after stopping it, it should free the memory. BUT the system is >>> then after the stop with less memory! that is the point. Not the running >>> asterisk's memory consumption bothers me, but the fact, that after 20 start >>> and stops and waiting for days the memory once gone is never put back. > > First of all, my questions weren't ment as an offense, more a question of > interest by asking thing that I whitnessed. If I sounded harsh or not polite, sorry, I did not mean to do that! I was trying to get a message through, but did not feel like I was succeeding and that was frustrating me. So sorry for any lack of respect I could have shown. > >> >> VM system is not composed only of "free" and "used" ram, there ar3e >> other categories. Depending on how a program allocates and uses memory >> it's not automatically sent to the "free" pool after being reclaimed. >> >> Allocated memory can be dirty buffers which are reclaimed after time, or >> other types of buffered data which is never reclaimed until there is >> memor7y pressure. How do you know the game you were playing has a >> similar memory usage as asterisk, which, I assure you, has some complex >> memory usage patterns in it's source. >> >> Also asterisk leverages many parts of the kernel which a game does not >> leverage. I'm quite sure after quitting asterisk you have an higher >> "wired" memory than before starting it. That memory was not allocated by >> asterisk itself, but by the kernel while "serving" asterisk requests. >> The kernel keeps running and does not free that memory right away. in >> fact will not free it until forced by VM pressure AFAIK. > > After a couple of other starts and stops yesterday, it seems, 12-Current and > Asterisk 13.17.2 have a barrier at ~ 3 GB of "free" memory - always having in > mind that this is within the very specific parameters of my setup. It's quite possible that on a system with less RAM (for example an older ALIX board) these extra buffers allocated while asterisk was running get freed earlier because memory pressure is felt sooner. > I expect "in circumstantial occurences of some phenomena" nothing. But my > observations triggered this explanation - and I'm glad I did and I'm glad you > explained! I'm glad I was useful! -- Guido Falsi ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
[SOLVED] Re: net/asterisk13: memory leak under 12-CURRENT?
On Thu, 28 Sep 2017 07:57:01 +0200 "O. Hartmann" wrote: > On Wed, 27 Sep 2017 12:51:18 +0200 > Hans Petter Selasky wrote: > > > On 09/27/17 09:05, Guido Falsi wrote: > > > On 09/26/2017 15:41, O. Hartmann wrote: > > >> On Tue, 26 Sep 2017 15:06:23 +0200 > > >> Guido Falsi wrote: > > > > > >> Since I run net/asterisk with automatic module loading (I'm new to > > >> asterisk), this is very likely and might cause the problem somehow. > > >> > > > > > > You can exclude single modules from autoloading via modules.conf. > > > > > >>> Not sure, restarting the daemon should free any leaked memory the daemon > > >>> has. If a killed process leaves memory locked at the system level there > > >>> should be some other cause. > > >> > > >> Even with no runnidng asterisk, memory level drops after the last > > >> shutdown of asterisk and keeps that low. Even for weeks! My router never > > >> shows that high memory consumption, even under load. > > > > > > But while asterisk is running does the memory usage increase unbounded > > > till filling all available memory or does it stabilize at some point? > > > > > > Asterisk is relatively memory hungry, especially with all modules > > > enabled. It also caches and logs various information in RAM, even doing > > > "nothing" it will cache and log that "nothing" activity. If memory does > > > stabilize after some point it's not really a leak but it's standard > > > memory usage. To reduce it you should disable all unused modules. > > > > > >> > > >> The question would be: how to use vmstat to give hints for those familiar > > >> with memory subsystems to indicate a real bug? > > >> > > >> I tried to find some advices, but maybe my English isn't good enough to > > >> make google help. > > > > > > I'm not able to give you a correct indication, but if the memory usage > > > is not increasing indefinitely but is stabilizing I'd say it's not > > > really a leak. > > > > > > > Did you look at the output from "vmstat -m" and "vmstat -z" ? > > > > --HPS > > I did not, but now I will ;-) > > Thanks for the hint! > > Kind regards, > Oliver For the completeness. Guido already explained that things I considered "stange' could be normal. I think I missed the point, that in some cases, caching data can also persist even when the process, which I considered to be the consumer, died. The vmstat output is taken after a couple of restarts of asterisk and while asterisk was still running. I had a very simplistic, outdated memory model in mind and Guido made that clear that it isn't. I was concerned about if it comes to the use of a small SoC with only 1 or 2 GB of RAM. Sorry for the noise :-( Kind regards, Oliver [...] last pid: 19557; load averages: 0.22, 0.18, 0.14 up 1+14:17:56 11:14:44 25 processes: 1 running, 24 sleeping CPU: 0.3% user, 0.0% nice, 0.8% system, 0.0% interrupt, 98.9% idle Mem: 27M Active, 104M Inact, 392M Wired, 146M Buf, 3421M Free Swap: 8192M Total, 8192M Free PID USERNAME THR PRI NICE SIZERES STATE C TIMEWCPU COMMAND 36617 asterisk70 200 147M 74236K piperd 2 3:38 2.42% asterisk 19557 root 1 200 13180K 3508K CPU33 0:00 0.13% top 93400 root 1 200 11444K 2100K select 3 1:34 0.06% syslogd 12569 root 1 200 15136K 3576K select 1 4:35 0.04% ppp 88182 administrator1 200 18768K 7980K select 0 0:00 0.02% sshd 24901 root 1 200 10840K 1704K select 1 0:19 0.01% powerd 17169 root 1 200 18192K 18296K select 2 0:10 0.01% ntpd 10719 root 1 200 14724K 4712K bpf 1 0:03 0.00% arpwatch 9691 bind 7 520 64552K 38192K sigwai 3 0:19 0.00% named 98424 root 1 200 32100K 20948K nanslp 2 0:05 0.00% perl 39431 asterisk 1 520 13848K 5072K select 0 0:01 0.00% mpg123 73938 root 1 200 11332K 2036K nanslp 3 0:01 0.00% cron 45343 daemon 1 200 11316K 1940K select 2 0:00 0.00% rpcbind 41607 root 1 200 10976K 1848K kqread 1 0:00 0.00% autounmountd 82722 root 1 220 18396K 7920K select 1 0:00 0.00% sshd 97178 root 1 210 16216K 6780K select 2 0:00 0.00% sudo 97625 root 1 200 13604K 4356K pause 2 0:00 0.00% csh 46309 root 1 200 10880K 1784K autofs 3 0:00 0.00% automountd 90914 administrator1 200 13172K 4184K pause 0 0:00 0.00% csh 40858 asterisk 1 520 13616K 3040K pipewr 3 0:00 0.00% mpg123 97392 root 1 230 11776K 3096K wait3 0:00 0.00% su 77979 dhcpd1 200 21896K 8340K select 0 0:00 0.00% dhcpd 69003 root 1 200 180
[SOLVED] Re: net/asterisk13: memory leak under 12-CURRENT?
On Thu, 28 Sep 2017 09:38:58 +0200 Guido Falsi wrote: > On 09/28/2017 08:11, O. Hartmann wrote: > > On Wed, 27 Sep 2017 09:05:42 +0200 > > Guido Falsi wrote: > > > >> On 09/26/2017 15:41, O. Hartmann wrote: > >>> On Tue, 26 Sep 2017 15:06:23 +0200 > >>> Guido Falsi wrote: > >> > >>> Since I run net/asterisk with automatic module loading (I'm new to > >>> asterisk), this is very likely and might cause the problem somehow. > >>> > >> > >> You can exclude single modules from autoloading via modules.conf. > >> > Not sure, restarting the daemon should free any leaked memory the daemon > has. If a killed process leaves memory locked at the system level there > should be some other cause. > >>> > >>> Even with no runnidng asterisk, memory level drops after the last shutdown > >>> of asterisk and keeps that low. Even for weeks! My router never shows that > >>> high memory consumption, even under load. > >> > >> But while asterisk is running does the memory usage increase unbounded > >> till filling all available memory or does it stabilize at some point? > > > > While Asterisk is running, it doesn't consume much memory, but stopping > > Asterisk, I would expect that the claimed memory is freed again. It isn't, > > not all memory is freed. Starting Asterisk then again from this reduced > > memory level, it claims its memory, "stabilzes" at a certain point while > > running and again, stopping Asterisk leaves the free memory now at a much > > lower level never been leveld out - as I said. > > > > I played this game last night ~ 20 times until the free memory dropped > > beneath 3 GB after asterisk has been shut down. This morning, the level was > > at the same low level as I left it. The router has nothing special to do, > > the workload is not memory consuming even for weeks! And if there is load, > > after the load went away, the memory consumption always leveld out and > > freed memory. > >> > >> Asterisk is relatively memory hungry, especially with all modules > >> enabled. It also caches and logs various information in RAM, even doing > >> "nothing" it will cache and log that "nothing" activity. If memory does > >> stabilize after some point it's not really a leak but it's standard > >> memory usage. To reduce it you should disable all unused modules. > > > > I don't understand here. Even if Asterisk is memory hungry - it has ~ 3 GB > > to use. But after stopping it, it should free the memory. BUT the system is > > then after the stop with less memory! that is the point. Not the running > > asterisk's memory consumption bothers me, but the fact, that after 20 start > > and stops and waiting for days the memory once gone is never put back. First of all, my questions weren't ment as an offense, more a question of interest by asking thing that I whitnessed. > > VM system is not composed only of "free" and "used" ram, there ar3e > other categories. Depending on how a program allocates and uses memory > it's not automatically sent to the "free" pool after being reclaimed. > > Allocated memory can be dirty buffers which are reclaimed after time, or > other types of buffered data which is never reclaimed until there is > memor7y pressure. How do you know the game you were playing has a > similar memory usage as asterisk, which, I assure you, has some complex > memory usage patterns in it's source. > > Also asterisk leverages many parts of the kernel which a game does not > leverage. I'm quite sure after quitting asterisk you have an higher > "wired" memory than before starting it. That memory was not allocated by > asterisk itself, but by the kernel while "serving" asterisk requests. > The kernel keeps running and does not free that memory right away. in > fact will not free it until forced by VM pressure AFAIK. After a couple of other starts and stops yesterday, it seems, 12-Current and Asterisk 13.17.2 have a barrier at ~ 3 GB of "free" memory - always having in mind that this is within the very specific parameters of my setup. Maybe it would be better first to nail down all the modules in Asterisk not used and switch them off. > > But this is quite a complicated matter and not being a VM expert I can't > explain it in detail. What you must understand is that many things are > going on at a given time in a memory subsystem and looking at it through > a single value is not indicative. You are completely right. > > > > > At the moment, I have mpg123 suspect doing nasty things, because the > > vanishing memory is more prominent and indicated when voicemail system has > > been used and mpg123 started. Not touching VM subsystem seems to free the > > whole memory claimed by asterisk after stopping asterisk, apart from maybe > > buffers claimed by the OS released later (I did no thourough investigations > > on that). > > Please note that when I use the "VM" Acronym I mean "Virtual Memory". > > there is a new version of mpg123 also, and I'm quite su
Re: net/asterisk13: memory leak under 12-CURRENT?
On 09/28/2017 09:38, Guido Falsi wrote: > I'm not understanding what you are expecting us to do based on > circumstantial and partial data. > I'm clarifying myself here: I mean that I don't know what to do about this based on the data and the details you give. What you are seeing looks normal to me. To understand better what is happening is more data. You say you have data that shows something but in a non scientific way. But this translates to me to you having a feeling, which I don't share based on that data. So if you want help or a better explanation your only way is to give more and more detailed data. At a minimum the while memory line from top not only the "free memory:" column, which by itself gives completely incomplete information. It would be better if you could give the time evolution of that line. before launching asterisk, some samples while it is running and you are doing some tests, and after closing it. Other tools like vmstat can give more detailed information, but I'm not very good at deciphering that data, but it could be very useful anyway for people more expert about the VM system to give you an explanation or be convinced that there is actually something strange going on. In other words I'm sorry to say you need to give more scientific proof of your feeling. There's nothing personal in this. -- Guido Falsi ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On 09/28/2017 08:11, O. Hartmann wrote: > On Wed, 27 Sep 2017 09:05:42 +0200 > Guido Falsi wrote: > >> On 09/26/2017 15:41, O. Hartmann wrote: >>> On Tue, 26 Sep 2017 15:06:23 +0200 >>> Guido Falsi wrote: >> >>> Since I run net/asterisk with automatic module loading (I'm new to >>> asterisk), this is very likely and might cause the problem somehow. >>> >> >> You can exclude single modules from autoloading via modules.conf. >> Not sure, restarting the daemon should free any leaked memory the daemon has. If a killed process leaves memory locked at the system level there should be some other cause. >>> >>> Even with no runnidng asterisk, memory level drops after the last shutdown >>> of asterisk and keeps that low. Even for weeks! My router never shows that >>> high memory consumption, even under load. >> >> But while asterisk is running does the memory usage increase unbounded >> till filling all available memory or does it stabilize at some point? > > While Asterisk is running, it doesn't consume much memory, but stopping > Asterisk, I would expect that the claimed memory is freed again. It isn't, not > all memory is freed. Starting Asterisk then again from this reduced memory > level, it claims its memory, "stabilzes" at a certain point while running and > again, stopping Asterisk leaves the free memory now at a much lower level > never > been leveld out - as I said. > > I played this game last night ~ 20 times until the free memory dropped beneath > 3 GB after asterisk has been shut down. This morning, the level was at the > same > low level as I left it. The router has nothing special to do, the workload is > not memory consuming even for weeks! And if there is load, after the load went > away, the memory consumption always leveld out and freed memory. >> >> Asterisk is relatively memory hungry, especially with all modules >> enabled. It also caches and logs various information in RAM, even doing >> "nothing" it will cache and log that "nothing" activity. If memory does >> stabilize after some point it's not really a leak but it's standard >> memory usage. To reduce it you should disable all unused modules. > > I don't understand here. Even if Asterisk is memory hungry - it has ~ 3 GB to > use. But after stopping it, it should free the memory. BUT the system is then > after the stop with less memory! that is the point. Not the running asterisk's > memory consumption bothers me, but the fact, that after 20 start and stops and > waiting for days the memory once gone is never put back. VM system is not composed only of "free" and "used" ram, there ar3e other categories. Depending on how a program allocates and uses memory it's not automatically sent to the "free" pool after being reclaimed. Allocated memory can be dirty buffers which are reclaimed after time, or other types of buffered data which is never reclaimed until there is memor7y pressure. How do you know the game you were playing has a similar memory usage as asterisk, which, I assure you, has some complex memory usage patterns in it's source. Also asterisk leverages many parts of the kernel which a game does not leverage. I'm quite sure after quitting asterisk you have an higher "wired" memory than before starting it. That memory was not allocated by asterisk itself, but by the kernel while "serving" asterisk requests. The kernel keeps running and does not free that memory right away. in fact will not free it until forced by VM pressure AFAIK. But this is quite a complicated matter and not being a VM expert I can't explain it in detail. What you must understand is that many things are going on at a given time in a memory subsystem and looking at it through a single value is not indicative. > > At the moment, I have mpg123 suspect doing nasty things, because the > vanishing memory is more prominent and indicated when voicemail system has > been used and mpg123 started. Not touching VM subsystem seems to free the > whole memory claimed by asterisk after stopping asterisk, apart from maybe > buffers claimed by the OS released later (I did no thourough investigations on > that). Please note that when I use the "VM" Acronym I mean "Virtual Memory". there is a new version of mpg123 also, and I'm quite sure that mpg123 causes caching of the audio data it reads. Adsterisk Voicemail system also causes caching. Those cached are not going to be freed after program exit because other programs could take advantage of the cached data. I'm not understanding what you are expecting us to do based on circumstantial and partial data. -- Guido Falsi ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On 09/28/2017 08:01, O. Hartmann wrote: >> These numbers really don't tell us anything. The system has anyway been >> running for days, depending on configuration daemons like cron and ntp >> are running and performing tasks, things are being cached and so on, so >> that difference after three days could be perfectly normal overhead. > > I think they do, but not in a scientific way. I am unable to work with such data anyway. > > The system in question has to do always the same task and is running for > months > with never dropping below a certain memory boundary. > > Then, when asterisk started/stopped/started, memory begins to fade away and is > never getting back to "free", even with asterisk off for days, twoo weeks. > Does > this really tell me nothing? > It tells you the OS is caching some data. Which is normal. You should discover what the OS is doing with that memory. The fact that it is in some way allocated does not indicate anything. Maybe the memory used by asterisk is simply going in the inactive pool and never reclaimed by the OS because it's not required, having still lots of Free memory. There are many possibilities, and exact data is needed to investigate. I'm also not an expert about VM systems. Asterisk, even with few modules tends to use a big chunk of memory and after stopping a software which uses much RAM it's normal for the OS to not immediately reclaim that to the Free pool. In fact such reclaiming happens only if there actually is memory pressure. There are so many factors playing into this that just looking at Free RAM answers no question. What problem are you trying to solve? Are you running out of memory? >> Whatever you prefer, but trying a few days uptime with all modules >> enabled is zero cost and east to do. Also you started this report with >> THIS configuration, changing configuration would prevent from comparing >> results. Let's test one thing at a time. >> > > I will. > Now as we have a indication that there is porbably something, I'll go for > deeper investiagtions with a static config. > I disagree with the indication buyt to investigate such a problem you need to use better tools than just looking at the free ram reported in top. Please at least give us the other ram numbers (Used, inactive, laundry, wired, buffers...) -- Guido Falsi ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On Wed, 27 Sep 2017 09:05:42 +0200 Guido Falsi wrote: > On 09/26/2017 15:41, O. Hartmann wrote: > > On Tue, 26 Sep 2017 15:06:23 +0200 > > Guido Falsi wrote: > > > Since I run net/asterisk with automatic module loading (I'm new to > > asterisk), this is very likely and might cause the problem somehow. > > > > You can exclude single modules from autoloading via modules.conf. > > >> Not sure, restarting the daemon should free any leaked memory the daemon > >> has. If a killed process leaves memory locked at the system level there > >> should be some other cause. > > > > Even with no runnidng asterisk, memory level drops after the last shutdown > > of asterisk and keeps that low. Even for weeks! My router never shows that > > high memory consumption, even under load. > > But while asterisk is running does the memory usage increase unbounded > till filling all available memory or does it stabilize at some point? While Asterisk is running, it doesn't consume much memory, but stopping Asterisk, I would expect that the claimed memory is freed again. It isn't, not all memory is freed. Starting Asterisk then again from this reduced memory level, it claims its memory, "stabilzes" at a certain point while running and again, stopping Asterisk leaves the free memory now at a much lower level never been leveld out - as I said. I played this game last night ~ 20 times until the free memory dropped beneath 3 GB after asterisk has been shut down. This morning, the level was at the same low level as I left it. The router has nothing special to do, the workload is not memory consuming even for weeks! And if there is load, after the load went away, the memory consumption always leveld out and freed memory. > > Asterisk is relatively memory hungry, especially with all modules > enabled. It also caches and logs various information in RAM, even doing > "nothing" it will cache and log that "nothing" activity. If memory does > stabilize after some point it's not really a leak but it's standard > memory usage. To reduce it you should disable all unused modules. I don't understand here. Even if Asterisk is memory hungry - it has ~ 3 GB to use. But after stopping it, it should free the memory. BUT the system is then after the stop with less memory! that is the point. Not the running asterisk's memory consumption bothers me, but the fact, that after 20 start and stops and waiting for days the memory once gone is never put back. At the moment, I have mpg123 suspect doing nasty things, because the vanishing memory is more prominent and indicated when voicemail system has been used and mpg123 started. Not touching VM subsystem seems to free the whole memory claimed by asterisk after stopping asterisk, apart from maybe buffers claimed by the OS released later (I did no thourough investigations on that). > > > > > The question would be: how to use vmstat to give hints for those familiar > > with memory subsystems to indicate a real bug? > > > > I tried to find some advices, but maybe my English isn't good enough to make > > google help. > > I'm not able to give you a correct indication, but if the memory usage > is not increasing indefinitely but is stabilizing I'd say it's not > really a leak. > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On Wed, 27 Sep 2017 12:23:20 +0200 Guido Falsi wrote: > On 09/27/2017 11:27, O. Hartmann wrote: > > On Wed, 27 Sep 2017 09:05:42 +0200 > > Guido Falsi wrote: > >> But while asterisk is running does the memory usage increase unbounded > >> till filling all available memory or does it stabilize at some point? > > > > As far as I could observe, a three day test run of the > > router/firewall/asterisk box drained around 500 MB of memory: starting at > > boot time with ~3700 MB, asterisk leaves the box with ~3640 MB after bein > > started and after three days the system reached ~3150 MB. Stopping asterisk > > gave back some memory, so ~3300 MB then was for days the final result - not > > recovering anything further. I use TEMPFS, if it matters, but I > > checked /tmp and /var/, there were no remnant files so far. TMPVAR is only > > allowed to have 256 MB. > > These numbers really don't tell us anything. The system has anyway been > running for days, depending on configuration daemons like cron and ntp > are running and performing tasks, things are being cached and so on, so > that difference after three days could be perfectly normal overhead. I think they do, but not in a scientific way. The system in question has to do always the same task and is running for months with never dropping below a certain memory boundary. Then, when asterisk started/stopped/started, memory begins to fade away and is never getting back to "free", even with asterisk off for days, twoo weeks. Does this really tell me nothing? > > You need to investigate the amount of memory allocated to asterisk with > ps and top and check if that stabilizes. A few days, at most a week > would be enough. After that, if it's not stabilizing you can start > thinking on a leak, but still can't assume where the leak is happening. > > > > Can't say whether it is stabilising or not - I think the runtime is too > > short. I'll check first to disable some modules in the first place and then > > try to perform a test with several days of asterisk enabled. > > > > Whatever you prefer, but trying a few days uptime with all modules > enabled is zero cost and east to do. Also you started this report with > THIS configuration, changing configuration would prevent from comparing > results. Let's test one thing at a time. > I will. Now as we have a indication that there is porbably something, I'll go for deeper investiagtions with a static config. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On Wed, 27 Sep 2017 12:51:18 +0200 Hans Petter Selasky wrote: > On 09/27/17 09:05, Guido Falsi wrote: > > On 09/26/2017 15:41, O. Hartmann wrote: > >> On Tue, 26 Sep 2017 15:06:23 +0200 > >> Guido Falsi wrote: > > > >> Since I run net/asterisk with automatic module loading (I'm new to > >> asterisk), this is very likely and might cause the problem somehow. > >> > > > > You can exclude single modules from autoloading via modules.conf. > > > >>> Not sure, restarting the daemon should free any leaked memory the daemon > >>> has. If a killed process leaves memory locked at the system level there > >>> should be some other cause. > >> > >> Even with no runnidng asterisk, memory level drops after the last shutdown > >> of asterisk and keeps that low. Even for weeks! My router never shows that > >> high memory consumption, even under load. > > > > But while asterisk is running does the memory usage increase unbounded > > till filling all available memory or does it stabilize at some point? > > > > Asterisk is relatively memory hungry, especially with all modules > > enabled. It also caches and logs various information in RAM, even doing > > "nothing" it will cache and log that "nothing" activity. If memory does > > stabilize after some point it's not really a leak but it's standard > > memory usage. To reduce it you should disable all unused modules. > > > >> > >> The question would be: how to use vmstat to give hints for those familiar > >> with memory subsystems to indicate a real bug? > >> > >> I tried to find some advices, but maybe my English isn't good enough to > >> make google help. > > > > I'm not able to give you a correct indication, but if the memory usage > > is not increasing indefinitely but is stabilizing I'd say it's not > > really a leak. > > > > Did you look at the output from "vmstat -m" and "vmstat -z" ? > > --HPS I did not, but now I will ;-) Thanks for the hint! Kind regards, Oliver ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On Wed, 27 Sep 2017 09:05:42 +0200 Guido Falsi wrote: > On 09/26/2017 15:41, O. Hartmann wrote: > > On Tue, 26 Sep 2017 15:06:23 +0200 > > Guido Falsi wrote: > > > Since I run net/asterisk with automatic module loading (I'm new to > > asterisk), this is very likely and might cause the problem somehow. > > > > You can exclude single modules from autoloading via modules.conf. I tried, but I need to study first more documentation to explore what is prerequisite and what is optional. > > >> Not sure, restarting the daemon should free any leaked memory the daemon > >> has. If a killed process leaves memory locked at the system level there > >> should be some other cause. > > > > Even with no runnidng asterisk, memory level drops after the last shutdown > > of asterisk and keeps that low. Even for weeks! My router never shows that > > high memory consumption, even under load. > > But while asterisk is running does the memory usage increase unbounded > till filling all available memory or does it stabilize at some point? As far as I could observe, a three day test run of the router/firewall/asterisk box drained around 500 MB of memory: starting at boot time with ~3700 MB, asterisk leaves the box with ~3640 MB after bein started and after three days the system reached ~3150 MB. Stopping asterisk gave back some memory, so ~3300 MB then was for days the final result - not recovering anything further. I use TEMPFS, if it matters, but I checked /tmp and /var/, there were no remnant files so far. TMPVAR is only allowed to have 256 MB. > > Asterisk is relatively memory hungry, especially with all modules > enabled. It also caches and logs various information in RAM, even doing > "nothing" it will cache and log that "nothing" activity. If memory does > stabilize after some point it's not really a leak but it's standard > memory usage. To reduce it you should disable all unused modules. > > > > > The question would be: how to use vmstat to give hints for those familiar > > with memory subsystems to indicate a real bug? > > > > I tried to find some advices, but maybe my English isn't good enough to make > > google help. > > I'm not able to give you a correct indication, but if the memory usage > is not increasing indefinitely but is stabilizing I'd say it's not > really a leak. > Can't say whether it is stabilising or not - I think the runtime is too short. I'll check first to disable some modules in the first place and then try to perform a test with several days of asterisk enabled. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On 09/27/17 09:05, Guido Falsi wrote: On 09/26/2017 15:41, O. Hartmann wrote: On Tue, 26 Sep 2017 15:06:23 +0200 Guido Falsi wrote: Since I run net/asterisk with automatic module loading (I'm new to asterisk), this is very likely and might cause the problem somehow. You can exclude single modules from autoloading via modules.conf. Not sure, restarting the daemon should free any leaked memory the daemon has. If a killed process leaves memory locked at the system level there should be some other cause. Even with no runnidng asterisk, memory level drops after the last shutdown of asterisk and keeps that low. Even for weeks! My router never shows that high memory consumption, even under load. But while asterisk is running does the memory usage increase unbounded till filling all available memory or does it stabilize at some point? Asterisk is relatively memory hungry, especially with all modules enabled. It also caches and logs various information in RAM, even doing "nothing" it will cache and log that "nothing" activity. If memory does stabilize after some point it's not really a leak but it's standard memory usage. To reduce it you should disable all unused modules. The question would be: how to use vmstat to give hints for those familiar with memory subsystems to indicate a real bug? I tried to find some advices, but maybe my English isn't good enough to make google help. I'm not able to give you a correct indication, but if the memory usage is not increasing indefinitely but is stabilizing I'd say it's not really a leak. Did you look at the output from "vmstat -m" and "vmstat -z" ? --HPS ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On 09/27/2017 11:27, O. Hartmann wrote: > On Wed, 27 Sep 2017 09:05:42 +0200 > Guido Falsi wrote: >> But while asterisk is running does the memory usage increase unbounded >> till filling all available memory or does it stabilize at some point? > > As far as I could observe, a three day test run of the > router/firewall/asterisk > box drained around 500 MB of memory: starting at boot time with ~3700 MB, > asterisk leaves the box with ~3640 MB after bein started and after three days > the system reached ~3150 MB. Stopping asterisk gave back some memory, so ~3300 > MB then was for days the final result - not recovering anything further. I use > TEMPFS, if it matters, but I checked /tmp and /var/, there were no remnant > files so far. TMPVAR is only allowed to have 256 MB. > These numbers really don't tell us anything. The system has anyway been running for days, depending on configuration daemons like cron and ntp are running and performing tasks, things are being cached and so on, so that difference after three days could be perfectly normal overhead. You need to investigate the amount of memory allocated to asterisk with ps and top and check if that stabilizes. A few days, at most a week would be enough. After that, if it's not stabilizing you can start thinking on a leak, but still can't assume where the leak is happening. > Can't say whether it is stabilising or not - I think the runtime is too short. > I'll check first to disable some modules in the first place and then try to > perform a test with several days of asterisk enabled. > Whatever you prefer, but trying a few days uptime with all modules enabled is zero cost and east to do. Also you started this report with THIS configuration, changing configuration would prevent from comparing results. Let's test one thing at a time. -- Guido Falsi ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On 09/26/2017 15:41, O. Hartmann wrote: > On Tue, 26 Sep 2017 15:06:23 +0200 > Guido Falsi wrote: > Since I run net/asterisk with automatic module loading (I'm new to asterisk), > this is very likely and might cause the problem somehow. > You can exclude single modules from autoloading via modules.conf. >> Not sure, restarting the daemon should free any leaked memory the daemon >> has. If a killed process leaves memory locked at the system level there >> should be some other cause. > > Even with no runnidng asterisk, memory level drops after the last shutdown of > asterisk and keeps that low. Even for weeks! My router never shows that high > memory consumption, even under load. But while asterisk is running does the memory usage increase unbounded till filling all available memory or does it stabilize at some point? Asterisk is relatively memory hungry, especially with all modules enabled. It also caches and logs various information in RAM, even doing "nothing" it will cache and log that "nothing" activity. If memory does stabilize after some point it's not really a leak but it's standard memory usage. To reduce it you should disable all unused modules. > > The question would be: how to use vmstat to give hints for those familiar with > memory subsystems to indicate a real bug? > > I tried to find some advices, but maybe my English isn't good enough to make > google help. I'm not able to give you a correct indication, but if the memory usage is not increasing indefinitely but is stabilizing I'd say it's not really a leak. -- Guido Falsi ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On Tue, 26 Sep 2017 15:06:23 +0200 Guido Falsi wrote: > On 09/26/2017 14:45, O. Hartmann wrote: > > Befor starting a PR I'd liek to ask for some advice to document a supposedly > > existent memory leak in net/asterisk13 and 12-CURRENT. > > > > Background: > > > > Running recent 12-CURRENT (FreeBSD 12.0-CURRENT #58 r323999: Tue Sep 26 > > 06:18:27 CEST 2017 amd64) on a PCengine APU 2C4, equipted with 4GB of RAM > > and 4080 KB free after the most recent SeaBIOS has been started, I try to > > utilise a net/asterisk13 as PBX on that platform. Asterisk13 has been build > > via poudriere and is compiled with CLANG/LLVM, not GCC! > > > > I'm running asterisk on similar hardware and also building with clang, > and have it going for months without any problems. I have disabled many > modules in that build though. Problem could be caused by some ancillary > library pulled in by some module. Since I run net/asterisk with automatic module loading (I'm new to asterisk), this is very likely and might cause the problem somehow. > > > When FreeBSD (NanoBSD) has been started, depending on the recent revision > > of the OS/Kernel, top shows ~3680 - 3705 MBytes of free memory. Starting > > net/asterisk13 via "service asterisk onestart" indicates an overall usage > > of up to 100 MBytes of memory. After having now run the Asterisk 13.17.2 > > daemon for two days resulted in ~ 3100 MBytes of free memory, while the > > asterisk daemon was still running and doing nothing. But performing several > > restarts on a freshly started box gives the same result after ~ 10 restarts > > - and even after stopping asterisk and let the system run for a couple of > > days doesn't free the memory. I stopped after 15 restarts or so watching > > the loss of memory after each restart of asterisk, so i came to the > > conclusion that there must be a memory leak somewhere. now i try to provide > > more informations as needed for a PR. > > > > Can someone help? > > Not sure, restarting the daemon should free any leaked memory the daemon > has. If a killed process leaves memory locked at the system level there > should be some other cause. Even with no runnidng asterisk, memory level drops after the last shutdown of asterisk and keeps that low. Even for weeks! My router never shows that high memory consumption, even under load. > > Have you tried diagnosing this memory with more tools? top so far ... > > > Depending on how deep you want to investigate ps, top and vmstat are > simple to use and give you some indication. To go deeper you'll need to > use more complicated tools. Others are better suited than me for > suggesting those. DTrace could be a powerful but not easy to use instrument. Well, DTrace would be an overkill for me. I'd like to provide more informations in case any FreeBSD developer wants to look at this - in case there is a real bug in either FreeBSD CURRENT - or net/asterisk. > > Looking just at free memory (which under a UNIX OS can have various > valid definitions, depending on how you account for inactive memory, > buffers and laundry RAM) is definitely not enough. Have you seen > asterisk process allocate more and more memory? > > You should check also what other processes in the system are doing. A > computer software when running never really does "nothing"(except, > maybe, when swapped out) and some memory usage can accumulate in many > parts of the system. Well, since it is a well defined NanoBSD system which has run for weeks with the same memory consumptions ebven under load, it seems obvious that there must be a black hole for memory when it somes to loading/unloading asterisk. The memory drain can be reproduced. > The question would be: how to use vmstat to give hints for those familiar with memory subsystems to indicate a real bug? I tried to find some advices, but maybe my English isn't good enough to make google help. Kind regards, Oliver ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: net/asterisk13: memory leak under 12-CURRENT?
On 09/26/2017 14:45, O. Hartmann wrote: Befor starting a PR I'd liek to ask for some advice to document a supposedly existent memory leak in net/asterisk13 and 12-CURRENT. Background: Running recent 12-CURRENT (FreeBSD 12.0-CURRENT #58 r323999: Tue Sep 26 06:18:27 CEST 2017 amd64) on a PCengine APU 2C4, equipted with 4GB of RAM and 4080 KB free after the most recent SeaBIOS has been started, I try to utilise a net/asterisk13 as PBX on that platform. Asterisk13 has been build via poudriere and is compiled with CLANG/LLVM, not GCC! I'm running asterisk on similar hardware and also building with clang, and have it going for months without any problems. I have disabled many modules in that build though. Problem could be caused by some ancillary library pulled in by some module. When FreeBSD (NanoBSD) has been started, depending on the recent revision of the OS/Kernel, top shows ~3680 - 3705 MBytes of free memory. Starting net/asterisk13 via "service asterisk onestart" indicates an overall usage of up to 100 MBytes of memory. After having now run the Asterisk 13.17.2 daemon for two days resulted in ~ 3100 MBytes of free memory, while the asterisk daemon was still running and doing nothing. But performing several restarts on a freshly started box gives the same result after ~ 10 restarts - and even after stopping asterisk and let the system run for a couple of days doesn't free the memory. I stopped after 15 restarts or so watching the loss of memory after each restart of asterisk, so i came to the conclusion that there must be a memory leak somewhere. now i try to provide more informations as needed for a PR. Can someone help? Not sure, restarting the daemon should free any leaked memory the daemon has. If a killed process leaves memory locked at the system level there should be some other cause. Have you tried diagnosing this memory with more tools? Depending on how deep you want to investigate ps, top and vmstat are simple to use and give you some indication. To go deeper you'll need to use more complicated tools. Others are better suited than me for suggesting those. DTrace could be a powerful but not easy to use instrument. Looking just at free memory (which under a UNIX OS can have various valid definitions, depending on how you account for inactive memory, buffers and laundry RAM) is definitely not enough. Have you seen asterisk process allocate more and more memory? You should check also what other processes in the system are doing. A computer software when running never really does "nothing"(except, maybe, when swapped out) and some memory usage can accumulate in many parts of the system. -- Guido Falsi ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
net/asterisk13: memory leak under 12-CURRENT?
Befor starting a PR I'd liek to ask for some advice to document a supposedly existent memory leak in net/asterisk13 and 12-CURRENT. Background: Running recent 12-CURRENT (FreeBSD 12.0-CURRENT #58 r323999: Tue Sep 26 06:18:27 CEST 2017 amd64) on a PCengine APU 2C4, equipted with 4GB of RAM and 4080 KB free after the most recent SeaBIOS has been started, I try to utilise a net/asterisk13 as PBX on that platform. Asterisk13 has been build via poudriere and is compiled with CLANG/LLVM, not GCC! When FreeBSD (NanoBSD) has been started, depending on the recent revision of the OS/Kernel, top shows ~3680 - 3705 MBytes of free memory. Starting net/asterisk13 via "service asterisk onestart" indicates an overall usage of up to 100 MBytes of memory. After having now run the Asterisk 13.17.2 daemon for two days resulted in ~ 3100 MBytes of free memory, while the asterisk daemon was still running and doing nothing. But performing several restarts on a freshly started box gives the same result after ~ 10 restarts - and even after stopping asterisk and let the system run for a couple of days doesn't free the memory. I stopped after 15 restarts or so watching the loss of memory after each restart of asterisk, so i came to the conclusion that there must be a memory leak somewhere. now i try to provide more informations as needed for a PR. Can someone help? Thanks in advance, Oliver ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
