[asterisk-users] Problems with the voice quality under load
B.H. Hello, all :-) We have a cluster of Asterisk (v. 11.9) servers that host IVR applications. The servers work behind SIP proxy (kamailio) for load balancing. All servers are in 2 processor configuration, 8-10 cores per CPU. When a particular server gets about 500 concurrent calls, the sound quality begins to degrade, the sound plays slowly and with clicks. As far as i understand, it's because asterisk is unable to send the voice stream in time i.e. the server is overloaded. What i don't understand is, at the time that the server appears to be overloaded and the audio quality is bad, actual server's load is no more than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for some reason the server is unable to use it's CPU capacity efficiently. May be because of some kind of thread contention inside asterisk? I have read blogs that advice to divide physical server into several VMs and they claim that this will improve the total capacity. In my own experience, this did not work very well and seems like the visualization actually made the quality worse. Do you have any advice for me (other than purchasing more servers ;-) ? Thanks! -- משיח NOW! Moshiach is coming very soon, prepare yourself! יחי אדוננו מורינו ורבינו מלך המשיח לעולם ועד! -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Problems with the voice quality under load
On Mon, 2 Mar 2015, Mordechay Kaganer wrote: When a particular server gets about 500 concurrent calls, the sound quality begins to degrade, the sound plays slowly and with clicks. As far as i understand, it's because asterisk is unable to send the voice stream in time i.e. the server is overloaded. What i don't understand is, at the time that the server appears to be overloaded and the audio quality is bad, actual server's load is no more than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for some reason the server is unable to use it's CPU capacity efficiently. May be because of some kind of thread contention inside asterisk? You may be using a feature (i.e., meetme) that is single threaded. If you view the system using 'htop' instead of 'top' (or, press '1' while running top) you may see that a single CPU is maxed out while the others are relatively idle. -- Thanks in advance, - Steve Edwards sedwa...@sedwards.com Voice: +1-760-468-3867 PST Newline Fax: +1-760-731-3000 -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Problems with the voice quality under load
Have you done the math for the network connections? BTF and external What bit rates for the sound? What codecs? How are calls coming in - SIP - analogue Disks OK(low IO per second)? Caching working OK? CPU may not be the problem if your CPU utilization is really that low. Ron On 02/03/2015 10:26 AM, Mordechay Kaganer wrote: B.H. Hello, all :-) We have a cluster of Asterisk (v. 11.9) servers that host IVR applications. The servers work behind SIP proxy (kamailio) for load balancing. All servers are in 2 processor configuration, 8-10 cores per CPU. When a particular server gets about 500 concurrent calls, the sound quality begins to degrade, the sound plays slowly and with clicks. As far as i understand, it's because asterisk is unable to send the voice stream in time i.e. the server is overloaded. What i don't understand is, at the time that the server appears to be overloaded and the audio quality is bad, actual server's load is no more than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for some reason the server is unable to use it's CPU capacity efficiently. May be because of some kind of thread contention inside asterisk? I have read blogs that advice to divide physical server into several VMs and they claim that this will improve the total capacity. In my own experience, this did not work very well and seems like the visualization actually made the quality worse. Do you have any advice for me (other than purchasing more servers ;-) ? Thanks! -- משיח NOW! Moshiach is coming very soon, prepare yourself! יחי אדוננו מורינו ורבינו מלך המשיח לעולם ועד! -- Ron Wheeler President Artifact Software Inc email: rwhee...@artifact-software.com skype: ronaldmwheeler phone: 866-970-2435, ext 102 -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Problems with the voice quality under load
On Monday 02 Mar 2015, Mordechay Kaganer wrote: When a particular server gets about 500 concurrent calls, the sound quality begins to degrade, the sound plays slowly and with clicks. As far as i understand, it's because asterisk is unable to send the voice stream in time i.e. the server is overloaded. What i don't understand is, at the time that the server appears to be overloaded and the audio quality is bad, actual server's load is no more than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for some reason the server is unable to use it's CPU capacity efficiently. May be because of some kind of thread contention inside asterisk? Do you have any advice for me (other than purchasing more servers ;-) ? Perhaps it's not CPU that is blocking things, but I/O? It should be visually and audibly obvious if this is the case -- the disk activity lights will be illuminated, and the HDDs will be making noises. If you can spare the RAM, consider setting up a tmpfs to hold your IVR prompts in memory. -- AJS Note: Originating address only accepts e-mail from list! If replying off- list, change address to asterisk1list at earthshod dot co dot uk . -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Problems with the voice quality under load
B.H. Hi, really thanks for all the relies :-) Here's my answers: On Mon, Mar 2, 2015 at 5:56 PM, Ron Wheeler rwhee...@artifact-software.com wrote: Have you done the math for the network connections? BTF and external What bit rates for the sound? What codecs? How are calls coming in - SIP - analogue Disks OK(low IO per second)? Caching working OK? There's no problem with network speed - we have 3 port bonding of 1G connections on each server and 1G external cross connect to the SIP providers (which are located in the same server farm). Even single 1G link is sufficient for more than 5000 calls with G711. All disk IO is off-loaded to the central server with NFS and as far as i can see, the load is low. In most cases, the people listen to the same files and caching is OK - each server has 32GB of RAM most of it is used for caching. We are connected to several SIP trunk providers. All the setup is SIP only. The codec is G711. On Mon, Mar 2, 2015 at 6:02 PM, A J Stiles asterisk_l...@earthshod.co.uk wrote: Perhaps it's not CPU that is blocking things, but I/O? It should be visually and audibly obvious if this is the case -- the disk activity lights will be illuminated, and the HDDs will be making noises. If you can spare the RAM, consider setting up a tmpfs to hold your IVR prompts in memory. The prompts are anyway cached in in memory by the disk cache so i don't think it's the issue in this case. As stated above, i don't see much of disk load on the NFS server as well. On Mon, Mar 2, 2015 at 6:14 PM, Steve Edwards asterisk@sedwards.com wrote: You may be using a feature (i.e., meetme) that is single threaded. If you view the system using 'htop' instead of 'top' (or, press '1' while running top) you may see that a single CPU is maxed out while the others are relatively idle. I did check the CPU load and it is distributed evenly between all the cores. Most of the load is in system, probably network handling in the kernel. We do use ConfBridge a lot, but all conferences are on a single server and the issue is on all the servers. What other features are single-threaded? Asterisk is almost the only thing that runs on the PBX servers. There are also some PHP AGI scripts (5-10 per minute per server on average) and most of the AGI functions are off-loaded to a java application on the central server with fast AGI. But there is still a lot of dialplan logic that runs on the PBX's. Maybe, huge dialplan is the key for the issue? -- משיח NOW! Moshiach is coming very soon, prepare yourself! יחי אדוננו מורינו ורבינו מלך המשיח לעולם ועד! -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
