Hi,
ever since I have switched my server from Centos 7 to Fedora 32, asterisk
is showing memory issues and no calls are possible. I'm using the asterisk
that comes with Fedora; before that, I used a self-compiled version on
Centos. The hardware is still the same. Asterisk shows the following
message when trying to make a call:
WARNING[92530]: pbx.c:4644 increase_call_count: Available system memory
(~168MB) is below the configured low watermark (1024MB)
I was thinking that asterisk is leaking memory because the only way to get
asterisk to work again was by restarting the server. Since this is very
annoying and the Fedora bug report remains ignored, I finally started to
investigate. The relevant source is attached in asnip.c.
To see what's going on, I wrote a test program, attached as sysinfo.c. You
can simply compile it with 'cc -O2 sysinfo.c -o sysinfo'.
The output is follows:
./sysinfo
unit size: 1 byte(s)
#
freeram: 153
bufferram: 5
----
Sum: 158
#
trying to allocate 158
sleeping 5 seconds
allocate twice as much
sleeping 5 seconds
memset allocated memory to 0
sleeping 5 seconds
memory freed
So what asterisk says is about right. When I look at the info from 'cat
/proc/meminfo', I see this:
cat /proc/meminfo
MemTotal: 16361780 kB
MemFree: 151844 kB
MemAvailable: 14883060 kB
Buffers: 5468 kB
Cached: 14773136 kB
SwapCached: 224 kB
Active: 2019568 kB
Inactive: 13592124 kB
[...]
which would mean that I have 14GB buffered/cached, and 'free -h' confirms
this.
Apparently, the cache remains persistently occupied. Since I'm currently
performing backups, it's not surprising that the cache is large.
Apparently it means that asterisk fails every time I'm doing something
that fills the cache :(
After understanding that the cache remains full, I figured there might be
way to flush the buffers and the cache. [1] shows how to do this, and
after 'sync; echo 3 > /proc/sys/vm/drop_caches', 'free -h' showed
buff/cache as 1.3Gi and asterisk was working again.
However, the backups are still going on, and it doesn't take long before
the cache is back at 14GB again and asterisk is blocked.
I think asterisk needs to consider the cache as free memory as well. Isn't
the cache supposed to automatically shrink when more memory is required?
As a workaround, I can set minmemfree in asterisk.conf to a low value.
Nonetheless, I guess that should be fixed. I'll update the Fedora bug
report with this.
[1]:
https://www.tecmint.com/clear-ram-memory-cache-buffer-and-swap-space-on-linux/
#if defined(HAVE_SYSINFO)
if (option_minmemfree) {
/* Make sure that the free system memory is above the configured low watermark */
if (!sysinfo(&sys_info)) {
/* Convert the amount of available RAM from mem_units to MB. The calculation
* was done this way to avoid overflow problems */
uint64_t curfreemem = sys_info.freeram + sys_info.bufferram;
curfreemem *= sys_info.mem_unit;
curfreemem /= 1024 * 1024;
if (curfreemem < option_minmemfree) {
ast_log(LOG_WARNING, "Available system memory (~%" PRIu64 "MB) is below the configured low watermark (%ldMB)\n",
curfreemem, option_minmemfree);
failed = -1;
}
}
}
#endif
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/sysinfo.h>
#include <string.h>
#include <unistd.h>
#define DISPLAYUNIT (1024 * 1024)
void sleeping(int seconds)
{
printf("sleeping %d seconds\n", seconds);
sleep(seconds);
}
int main(int argc, char argv[])
{
struct sysinfo sys_info;
memset(&sys_info, 0, sizeof(sys_info));
if(!sysinfo(&sys_info)) {
printf(
"unit size:\t%12u byte(s)\n#\nfreeram:\t%12lu\nbufferram:\t%12lu\n----\nSum:\t\t%12lu\n",
sys_info.mem_unit,
sys_info.freeram * sys_info.mem_unit / DISPLAYUNIT,
sys_info.bufferram * sys_info.mem_unit / DISPLAYUNIT,
(sys_info.freeram + sys_info.bufferram) * sys_info.mem_unit / DISPLAYUNIT
);
unsigned long some_memory = (sys_info.freeram + sys_info.bufferram) * (unsigned long)sys_info.mem_unit;
printf("#\ntrying to allocate %lu\n", some_memory / DISPLAYUNIT);
void *allocated = malloc((size_t)some_memory);
sleeping(5);
if(allocated) {
printf("allocate twice as much\n");
free(allocated);
allocated = malloc((size_t)(some_memory + some_memory));
sleeping(5);
if(allocated) {
printf("memset allocated memory to 0\n");
memset(allocated, 0, (size_t)(some_memory + some_memory));
sleeping(5);
free(allocated);
printf("memory freed\n");
} else {
printf("larger memory allocation failed\n");
}
} else {
printf("memory allocation failed\n");
}
} else {
printf("error: sysinfo not available\n");
exit(-1);
}
exit(0);
}
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Check out the new Asterisk community forum at: https://community.asterisk.org/
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
