Re: [asterisk-users] High resident memory with 11.14.0 ?
On Wed, Nov 26, 2014 at 3:20 PM, James Lamanna jlama...@gmail.com wrote:
On Tue, Nov 25, 2014 at 10:21 AM, James Lamanna jlama...@gmail.com
wrote:
On Tue, Nov 25, 2014 at 8:14 AM, Matthew Jordan mjor...@digium.com
wrote:
On Mon, Nov 24, 2014 at 2:12 PM, James Lamanna jlama...@gmail.com
wrote:
Also, how big does the cache in frame.c grow to?
I've recompiled with MALLOC_DEBUG on that server:
asterisk -rx memory show summary
1780466242 bytes (1780181594 cache) in2352909 allocations in file
frame.c
...
Seems like a ridiculous cache.
I'm not going to respond to your new thread, since it is the same
discussion as this one.
The frame cache is a per-thread local cache of frames that prevents
having to re-allocate frames as they pass through Asterisk. Clearly,
something is abusing it.
I think you'll need to provide some more information on how you're
producing this situation. Specifically:
* Channel technologies involved, and the formats on the channels
* Dialplan that reproduces the problem
Are you using any non-core dialplan applications or channel drivers?
This PBX has about 100 registered SIP clients, along with 23 PRI
channels, 2 inbound/outbound SIP trunks and around 100 IAXModems registered
to it. It primarily handles faxing.
I am not using any non-standard channel drivers. I am using the T.38
gateway funcionality.
The jist of the dialplan is this: (example of the PRI and a SIP trunk,
inbound)
[pri-in]
exten = _X.,1,Set(__FROM_DID=${EXTEN})
exten = _X.,n,Set(FAX_IDX=700)
exten = _X.,n,Set(MAX_IDX=719)
exten = _X.,n,Goto(dial-hylafax,s,1)
[sip-trunk-in]
exten = _X.,1(normal),Set(__FROM_DID=${EXTEN})
exten = _X.,n,Set(FAX_IDX=950)
exten = _X.,n,Set(MAX_IDX=959)
exten = _X.,n,Set(FAXOPT(gateway)=yes)
exten = _X.,n,Goto(dial-hylafax,s,1)
[dial-hylafax]
exten = s,1,GotoIf($[${FROM_DID:0:1} = 1]?prune:cont)
exten = s,n(prune),Set(__FROM_DID=${FROM_DID:1})
exten = s,n(cont),GotoIf($[${FAX_IDX} = ${MAX_IDX}]?tryfax:nofax)
exten = s,n(tryfax),Set(STATE=${DEVICE_STATE(Custom:iaxmodem${FAX_IDX})})
exten = s,n,NoOp(${STATE})
exten = s,n,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=INUSE)
exten = s,n,Dial(IAX2/iaxmodem${FAX_IDX}/${FROM_DID},60,g)
exten = s,n,Goto(s-${DIALSTATUS},1)
exten = s,n(nofax),Playtones(busy)
exten = s,n,NoOp(NO MODEMS AVAILABLE)
exten = s,n,Wait(20)
exten = s,n,Hangup()
exten = s-ANSWER,1,NoOp(IAXMODEM HANGUP)
exten = s-ANSWER,n,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=NOT_INUSE)
exten = s-ANSWER,n,Hangup()
exten = _s-.,1,Set(FAX_IDX=${MATH(1+${FAX_IDX},i)})
exten = _s-.,n,Goto(s,1)
exten = h,1,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=NOT_INUSE)
The current state requires me to restart Asterisk almost every day.
I'm also seeing this on a completely different machine after upgrading
from Asterisk10 to 11.
I'm wondering if this is a problem in the SLIN converter?
I do use SLIN with iaxmodem.
Also of note,
A quick valgrind run and attempting to send a few faxes produces a bunch of
these in the valgrind output:
==30640== 217,259 bytes in 287 blocks are definitely lost in loss record
1,778 of 1,789
==30640==at 0x4C267CC: calloc (vg_replace_malloc.c:467)
==30640==by 0x4DC50E: ast_frdup (utils.h:523)
==30640==by 0x47125F: __ast_queue_frame (channel.c:1284)
==30640==by 0x1EF75589: __do_deliver (chan_iax2.c:3102)
==30640==by 0x1EF76C5A: schedule_delivery (chan_iax2.c:4374)
==30640==by 0x1EF8F497: socket_process_helper (chan_iax2.c:12010)
==30640==by 0x1EF99C37: iax2_process_thread (chan_iax2.c:12030)
==30640==by 0x56C458: dummy_start (utils.c:1192)
==30640==by 0x5E359C9: start_thread (pthread_create.c:300)
==30640==by 0x270326FF: ???
--
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] High resident memory with 11.14.0 ?
On Tue, Nov 25, 2014 at 10:21 AM, James Lamanna jlama...@gmail.com wrote:
On Tue, Nov 25, 2014 at 8:14 AM, Matthew Jordan mjor...@digium.com
wrote:
On Mon, Nov 24, 2014 at 2:12 PM, James Lamanna jlama...@gmail.com
wrote:
Also, how big does the cache in frame.c grow to?
I've recompiled with MALLOC_DEBUG on that server:
asterisk -rx memory show summary
1780466242 bytes (1780181594 cache) in2352909 allocations in file
frame.c
...
Seems like a ridiculous cache.
I'm not going to respond to your new thread, since it is the same
discussion as this one.
The frame cache is a per-thread local cache of frames that prevents
having to re-allocate frames as they pass through Asterisk. Clearly,
something is abusing it.
I think you'll need to provide some more information on how you're
producing this situation. Specifically:
* Channel technologies involved, and the formats on the channels
* Dialplan that reproduces the problem
Are you using any non-core dialplan applications or channel drivers?
This PBX has about 100 registered SIP clients, along with 23 PRI channels,
2 inbound/outbound SIP trunks and around 100 IAXModems registered to it. It
primarily handles faxing.
I am not using any non-standard channel drivers. I am using the T.38
gateway funcionality.
The jist of the dialplan is this: (example of the PRI and a SIP trunk,
inbound)
[pri-in]
exten = _X.,1,Set(__FROM_DID=${EXTEN})
exten = _X.,n,Set(FAX_IDX=700)
exten = _X.,n,Set(MAX_IDX=719)
exten = _X.,n,Goto(dial-hylafax,s,1)
[sip-trunk-in]
exten = _X.,1(normal),Set(__FROM_DID=${EXTEN})
exten = _X.,n,Set(FAX_IDX=950)
exten = _X.,n,Set(MAX_IDX=959)
exten = _X.,n,Set(FAXOPT(gateway)=yes)
exten = _X.,n,Goto(dial-hylafax,s,1)
[dial-hylafax]
exten = s,1,GotoIf($[${FROM_DID:0:1} = 1]?prune:cont)
exten = s,n(prune),Set(__FROM_DID=${FROM_DID:1})
exten = s,n(cont),GotoIf($[${FAX_IDX} = ${MAX_IDX}]?tryfax:nofax)
exten = s,n(tryfax),Set(STATE=${DEVICE_STATE(Custom:iaxmodem${FAX_IDX})})
exten = s,n,NoOp(${STATE})
exten = s,n,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=INUSE)
exten = s,n,Dial(IAX2/iaxmodem${FAX_IDX}/${FROM_DID},60,g)
exten = s,n,Goto(s-${DIALSTATUS},1)
exten = s,n(nofax),Playtones(busy)
exten = s,n,NoOp(NO MODEMS AVAILABLE)
exten = s,n,Wait(20)
exten = s,n,Hangup()
exten = s-ANSWER,1,NoOp(IAXMODEM HANGUP)
exten = s-ANSWER,n,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=NOT_INUSE)
exten = s-ANSWER,n,Hangup()
exten = _s-.,1,Set(FAX_IDX=${MATH(1+${FAX_IDX},i)})
exten = _s-.,n,Goto(s,1)
exten = h,1,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=NOT_INUSE)
The current state requires me to restart Asterisk almost every day.
I'm also seeing this on a completely different machine after upgrading
from Asterisk10 to 11.
I'm wondering if this is a problem in the SLIN converter?
I do use SLIN with iaxmodem.
-- James
--
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] High resident memory with 11.14.0 ?
On Mon, Nov 24, 2014 at 2:12 PM, James Lamanna jlama...@gmail.com wrote: Also, how big does the cache in frame.c grow to? I've recompiled with MALLOC_DEBUG on that server: asterisk -rx memory show summary 1780466242 bytes (1780181594 cache) in2352909 allocations in file frame.c ... Seems like a ridiculous cache. I'm not going to respond to your new thread, since it is the same discussion as this one. The frame cache is a per-thread local cache of frames that prevents having to re-allocate frames as they pass through Asterisk. Clearly, something is abusing it. I think you'll need to provide some more information on how you're producing this situation. Specifically: * Channel technologies involved, and the formats on the channels * Dialplan that reproduces the problem Are you using any non-core dialplan applications or channel drivers? -- Matthew Jordan Digium, Inc. | Engineering Manager 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA Check us out at: http://digium.com http://asterisk.org -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] High resident memory with 11.14.0 ?
On Tue, Nov 25, 2014 at 8:14 AM, Matthew Jordan mjor...@digium.com wrote:
On Mon, Nov 24, 2014 at 2:12 PM, James Lamanna jlama...@gmail.com wrote:
Also, how big does the cache in frame.c grow to?
I've recompiled with MALLOC_DEBUG on that server:
asterisk -rx memory show summary
1780466242 bytes (1780181594 cache) in2352909 allocations in file
frame.c
...
Seems like a ridiculous cache.
I'm not going to respond to your new thread, since it is the same
discussion as this one.
The frame cache is a per-thread local cache of frames that prevents
having to re-allocate frames as they pass through Asterisk. Clearly,
something is abusing it.
I think you'll need to provide some more information on how you're
producing this situation. Specifically:
* Channel technologies involved, and the formats on the channels
* Dialplan that reproduces the problem
Are you using any non-core dialplan applications or channel drivers?
This PBX has about 100 registered SIP clients, along with 23 PRI channels,
2 inbound/outbound SIP trunks and around 100 IAXModems registered to it. It
primarily handles faxing.
I am not using any non-standard channel drivers. I am using the T.38
gateway funcionality.
The jist of the dialplan is this: (example of the PRI and a SIP trunk,
inbound)
[pri-in]
exten = _X.,1,Set(__FROM_DID=${EXTEN})
exten = _X.,n,Set(FAX_IDX=700)
exten = _X.,n,Set(MAX_IDX=719)
exten = _X.,n,Goto(dial-hylafax,s,1)
[sip-trunk-in]
exten = _X.,1(normal),Set(__FROM_DID=${EXTEN})
exten = _X.,n,Set(FAX_IDX=950)
exten = _X.,n,Set(MAX_IDX=959)
exten = _X.,n,Set(FAXOPT(gateway)=yes)
exten = _X.,n,Goto(dial-hylafax,s,1)
[dial-hylafax]
exten = s,1,GotoIf($[${FROM_DID:0:1} = 1]?prune:cont)
exten = s,n(prune),Set(__FROM_DID=${FROM_DID:1})
exten = s,n(cont),GotoIf($[${FAX_IDX} = ${MAX_IDX}]?tryfax:nofax)
exten = s,n(tryfax),Set(STATE=${DEVICE_STATE(Custom:iaxmodem${FAX_IDX})})
exten = s,n,NoOp(${STATE})
exten = s,n,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=INUSE)
exten = s,n,Dial(IAX2/iaxmodem${FAX_IDX}/${FROM_DID},60,g)
exten = s,n,Goto(s-${DIALSTATUS},1)
exten = s,n(nofax),Playtones(busy)
exten = s,n,NoOp(NO MODEMS AVAILABLE)
exten = s,n,Wait(20)
exten = s,n,Hangup()
exten = s-ANSWER,1,NoOp(IAXMODEM HANGUP)
exten = s-ANSWER,n,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=NOT_INUSE)
exten = s-ANSWER,n,Hangup()
exten = _s-.,1,Set(FAX_IDX=${MATH(1+${FAX_IDX},i)})
exten = _s-.,n,Goto(s,1)
exten = h,1,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=NOT_INUSE)
The current state requires me to restart Asterisk almost every day.
I'm also seeing this on a completely different machine after upgrading from
Asterisk10 to 11.
-- James
--
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] High resident memory with 11.14.0 ?
cat /proc/cpuinfo lists 4 cores. So even if that's not showing hyperthreading, maximum 8. By your rule, that would be 8 cores * 0.5GB = 4GB memory. I've seen resident memory be up over 6GB. On Sat, Nov 22, 2014 at 1:29 PM, Freddi Hansen f...@danovation.dk wrote: Its up to 5.8G of resident memory with 28321 calls processed. The OOM killer is going to kill this soon at this rate (8GB RAM machine). This seems like a pretty serious problem. It looks like I'll need to restart asterisk every night Hi the number of cpu cores that you see with top times 512Mbyte is the level of ram that's needed e.g. a hp-gen8 with 2 octo core cpu's and hyperthreading enabled will be ( 2 x 8 x 2 x 0,5 gb ) = 16 gb + a bit exstra. So from start memory usage increases until it reaches 17.3 gb and then stabilizes. at that level. You can disables hypertreading and cut your ram usage to half of that. I can't see what hardware you are using but I think you need to check that the rule above fits your hardware. b.r. Freddi On Fri, Nov 21, 2014 at 10:53 AM, James Lamanna jlama...@gmail.com wrote: Hi, I have an Asterisk server that's been running now for around 2 days. I've noticed that the resident memory seems to be very high for its current call load: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 18321 asterisk 20 0 8050m 5.2g 6968 S 13 66.2 363:11.80 asterisk $ asterisk -rx core show channels 24 active channels 12 active calls 25216 calls processed This server has a bunch of IAXModems hooked up to it and is mainly used as a Fax gateway to hylafax. Is this normal? 5.2Gig of memory used after 2 days with only 12 currently active calls? I am not using any realtime peers. There are 100 registered SIP peers on this server as well. Thanks. -- James -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] High resident memory with 11.14.0 ?
Also, how big does the cache in frame.c grow to? I've recompiled with MALLOC_DEBUG on that server: asterisk -rx memory show summary 1780466242 bytes (1780181594 cache) in2352909 allocations in file frame.c ... Seems like a ridiculous cache. On Mon, Nov 24, 2014 at 9:02 AM, James Lamanna jlama...@gmail.com wrote: cat /proc/cpuinfo lists 4 cores. So even if that's not showing hyperthreading, maximum 8. By your rule, that would be 8 cores * 0.5GB = 4GB memory. I've seen resident memory be up over 6GB. On Sat, Nov 22, 2014 at 1:29 PM, Freddi Hansen f...@danovation.dk wrote: Its up to 5.8G of resident memory with 28321 calls processed. The OOM killer is going to kill this soon at this rate (8GB RAM machine). This seems like a pretty serious problem. It looks like I'll need to restart asterisk every night Hi the number of cpu cores that you see with top times 512Mbyte is the level of ram that's needed e.g. a hp-gen8 with 2 octo core cpu's and hyperthreading enabled will be ( 2 x 8 x 2 x 0,5 gb ) = 16 gb + a bit exstra. So from start memory usage increases until it reaches 17.3 gb and then stabilizes. at that level. You can disables hypertreading and cut your ram usage to half of that. I can't see what hardware you are using but I think you need to check that the rule above fits your hardware. b.r. Freddi On Fri, Nov 21, 2014 at 10:53 AM, James Lamanna jlama...@gmail.com wrote: Hi, I have an Asterisk server that's been running now for around 2 days. I've noticed that the resident memory seems to be very high for its current call load: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 18321 asterisk 20 0 8050m 5.2g 6968 S 13 66.2 363:11.80 asterisk $ asterisk -rx core show channels 24 active channels 12 active calls 25216 calls processed This server has a bunch of IAXModems hooked up to it and is mainly used as a Fax gateway to hylafax. Is this normal? 5.2Gig of memory used after 2 days with only 12 currently active calls? I am not using any realtime peers. There are 100 registered SIP peers on this server as well. Thanks. -- James -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] High resident memory with 11.14.0 ?
Its up to 5.8G of resident memory with 28321 calls processed. The OOM killer is going to kill this soon at this rate (8GB RAM machine). This seems like a pretty serious problem. It looks like I'll need to restart asterisk every night Hi the number of cpu cores that you see with top times 512Mbyte is the level of ram that's needed e.g. a hp-gen8 with 2 octo core cpu's and hyperthreading enabled will be ( 2 x 8 x 2 x 0,5 gb ) = 16 gb + a bit exstra. So from start memory usage increases until it reaches 17.3 gb and then stabilizes. at that level. You can disables hypertreading and cut your ram usage to half of that. I can't see what hardware you are using but I think you need to check that the rule above fits your hardware. b.r. Freddi On Fri, Nov 21, 2014 at 10:53 AM, James Lamanna jlama...@gmail.com mailto:jlama...@gmail.com wrote: Hi, I have an Asterisk server that's been running now for around 2 days. I've noticed that the resident memory seems to be very high for its current call load: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 18321 asterisk 20 0 8050m 5.2g 6968 S 13 66.2 363:11.80 asterisk $ asterisk -rx core show channels 24 active channels 12 active calls 25216 calls processed This server has a bunch of IAXModems hooked up to it and is mainly used as a Fax gateway to hylafax. Is this normal? 5.2Gig of memory used after 2 days with only 12 currently active calls? I am not using any realtime peers. There are 100 registered SIP peers on this server as well. Thanks. -- James -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
[asterisk-users] High resident memory with 11.14.0 ?
Hi, I have an Asterisk server that's been running now for around 2 days. I've noticed that the resident memory seems to be very high for its current call load: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 18321 asterisk 20 0 8050m 5.2g 6968 S 13 66.2 363:11.80 asterisk $ asterisk -rx core show channels 24 active channels 12 active calls 25216 calls processed This server has a bunch of IAXModems hooked up to it and is mainly used as a Fax gateway to hylafax. Is this normal? 5.2Gig of memory used after 2 days with only 12 currently active calls? I am not using any realtime peers. There are 100 registered SIP peers on this server as well. Thanks. -- James -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] High resident memory with 11.14.0 ?
Its up to 5.8G of resident memory with 28321 calls processed. The OOM killer is going to kill this soon at this rate (8GB RAM machine). This seems like a pretty serious problem. It looks like I'll need to restart asterisk every night On Fri, Nov 21, 2014 at 10:53 AM, James Lamanna jlama...@gmail.com wrote: Hi, I have an Asterisk server that's been running now for around 2 days. I've noticed that the resident memory seems to be very high for its current call load: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 18321 asterisk 20 0 8050m 5.2g 6968 S 13 66.2 363:11.80 asterisk $ asterisk -rx core show channels 24 active channels 12 active calls 25216 calls processed This server has a bunch of IAXModems hooked up to it and is mainly used as a Fax gateway to hylafax. Is this normal? 5.2Gig of memory used after 2 days with only 12 currently active calls? I am not using any realtime peers. There are 100 registered SIP peers on this server as well. Thanks. -- James -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
