On Tue, Jun 21, 2016 at 11:12 AM, Michael Petruzzello < [email protected]> wrote:
> >On Fri, Jun 17, 2016 at 1:37 PM, Richard Mudgett <[email protected]> > wrote: > >> > >> > >> On Fri, Jun 17, 2016 at 12:36 PM, Michael Petruzzello > >> <[email protected]> wrote: > >>> > >>> Hello, > >>> > >>> I am currently working on determining bottlenecks in Asterisk and a > Stasis > >>> App. I'm currently trying to handle 83.3 calls/second. For the most > part, > >>> Asterisk and the Stasis APP handle that well, but there is a 60+ second > >>> delay in response time. > >>> > >>> On the Asterisk side, I am seeing the following warnings. [Jun 17 > >>> 12:00:16] WARNING[23561]: taskprocessor.c:803 taskprocessor_push: The > >>> 'subm:cdr_engine-00000003' task processor queue reached 500 scheduled > tasks. > >>> [Jun 17 12:00:18] WARNING[25477][C-00000068]: taskprocessor.c:803 > >>> taskprocessor_push: The 'subm:devService-test-00000038' task processor > queue > >>> reached 500 scheduled tasks. > >>> [Jun 17 12:00:21] WARNING[26298][C-000000a3]: taskprocessor.c:803 > >>> taskprocessor_push: The 'subp:PJSIP/sippeer-00000022' task processor > queue > >>> reached 500 scheduled tasks. > >>> [Jun 17 12:00:23] WARNING[27339][C-0000010d]: taskprocessor.c:803 > >>> taskprocessor_push: The 'subm:ast_channel_topic_all-cached-00000032' > task > >>> processor queue reached 500 scheduled tasks. > >>> [Jun 17 12:01:32] WARNING[31697][C-000003b2]: taskprocessor.c:803 > >>> taskprocessor_push: The 'subm:ast_channel_topic_all-00000036' task > processor > >>> queue reached 500 scheduled tasks. > >>> [Jun 17 12:05:55] WARNING[23280]: taskprocessor.c:803 > taskprocessor_push: > >>> The 'SIP' task processor queue reached 500 scheduled tasks. > >>> > >>> I have not seen a configuration setting on Asterisk to prevent these > >>> warnings from occurring (I'm trying to avoid modifying Asterisk source > code > >>> if possible). Looking at the task processors, I see the queue to the > stasis > >>> app bottlenecks: > >>> subm:devService-test-00000038 4560990 0 > >>> 1041689. It does clear up relatively quickly. The CDR engine also > bottle > >>> necks (extremely badly), but I don't use that. Nothing else comes > close to > >>> having a large queue. > >>> > >>> The stasis app itself is extremely streamlined and is very capable of > >>> handling a large number of messages at a time. The app runs with the > JVM so > >>> I am also researching into that as well as the netty library I am > using for > >>> the websocket connections. > >>> > >>> Any insight into Asterisk's side of the equation and how it scales on > 40 > >>> vCPUs would be greatly appreciated. > >> > >> > >> There are no options to disable those taskprocessor queue size warnings. > >> They are a > >> symptom of the system being severely stressed. If the stress continues > it > >> is possible > >> that the system could consume all memory in those taskprocessor queues. > >> > >> Recent changes to the Asterisk v13 branch were made to help throttle > back > >> incoming > >> SIP requests on PJSIP when the taskprocessors become backlogged like > you are > >> seeing. > >> These changes will be in the forthcoming v13.10.0 release. If you > want, you > >> can test with > >> the current v13 branch to see how these changes affect your stress > testing. > >> > >> If you don't need CDR's then you really need to disable them as they > consume > >> a lot of > >> processing time and the CDR taskprocessor queue backlog can take > minutes to > >> clear. > >> > > > >To echo what Richard said, because Asterisk is now sharing state > >across the Stasis message bus, turning off subscribers to that bus > >will help performance. Some easy ones to disable, if you aren't using > >them, are CDRs, CEL, and AMI. Those all do a reasonable amount of > >processing, and you can get some noticeable improvement by disabling > >them. > > > >Once you get past that, you can start fiddling with some of the lower > >level options. To start, you can throttle things back further by > >disabling certain internal messages in stasis.conf. As stasis.conf > >notes, functionality within Asterisk can break (or just not happen) if > >some messages are removed. For example, disabling > >'ast_channel_snapshot_type' would break ... most things. You may > >however be able to streamline your application by looking at what ARI > >messages it cares about, what messages it doesn't, inspecting the > >code, and disabling those that you don't care about. Lots of testing > >should occur before doing this, of course. > > > >You may also be able to get some different performance characteristics > >by changing the threadpool options for the message bus in stasis.conf. > >This may make a difference, depending on the underlying machine. > > Thank you for the suggestions. > > I'm running Asterisk on 40 vCPUs with 120 GB of RAM. Changing the thread > pool options to many more threads is not increasing performance, and at a > certain point it decreases performance. > > With further testing and having implemented your suggestions, I am > realizing the subm:devService-test-00000038 task processor is a major > bottleneck. I have always read that Asterisk can handle as many calls as > the hardware allows, but I'm not seeing that. > > I can go with a different architecture of multiple Asterisk servers > working together, but I would prefer one server to reduce complexity. > > What I am doing is not really a stress test because I really need Asterisk > to handle 1000s of callers calling in within a few minutes. > The subm:devService-test-00000038 taskprocessor is servicing the stasis message bus communication with your devService-test ARI application. Since each taskprocessor is executed by one thread, that is going to be a bottleneck. One thing you can try is to register multiple copies of your ARI application and randomly spread the calls to the different copies of the application. (devService-test1, devService-test2,...) Richard
-- _____________________________________________________________________ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
