Re: [asterisk-users] 13.22.0 - HTTP session count exceeded 100 sessions - instance unusable
On Wed, Jul 1, 2020 at 11:05 AM Stefan Viljoen wrote: > Hi Joshua > > No back-off, but I am caching the last 5000 results and and first hitting > the cache to see if a recent command already provided the information I'm > seeking for a particular request. > > I'll see if I can do some simulation and see if I'm effectively DDOSing > the local HTTP interface. > > I'll have to see if I maybe have a resource leak in my code that makes the > HTTP request, which is set to "Connection: close" already (and not > "Connection: keep-alive" at HTTP request header level) hang around and keep > the HTTP connection open to Asterisk. > > Does the HTTP implementation in Asterisk heed this "Connection:" HTTP > header? > The code is there, I've never experimented with or used it. -- Joshua C. Colp Asterisk Technical Lead Sangoma Technologies Check us out at www.sangoma.com and www.asterisk.org -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- Check out the new Asterisk community forum at: https://community.asterisk.org/ New to Asterisk? Start here: https://wiki.asterisk.org/wiki/display/AST/Getting+Started asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] 13.22.0 - HTTP session count exceeded 100 sessions - instance unusable
Hi Joshua No back-off, but I am caching the last 5000 results and and first hitting the cache to see if a recent command already provided the information I'm seeking for a particular request. I'll see if I can do some simulation and see if I'm effectively DDOSing the local HTTP interface. I'll have to see if I maybe have a resource leak in my code that makes the HTTP request, which is set to "Connection: close" already (and not "Connection: keep-alive" at HTTP request header level) hang around and keep the HTTP connection open to Asterisk. Does the HTTP implementation in Asterisk heed this "Connection:" HTTP header? -- Stefan -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- Check out the new Asterisk community forum at: https://community.asterisk.org/ New to Asterisk? Start here: https://wiki.asterisk.org/wiki/display/AST/Getting+Started asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] 13.22.0 - HTTP session count exceeded 100 sessions - instance unusable
Hi Joshua No back-off, but I am caching the last 5000 result and and first hitting the cache to see if a recent command already provided the information I'm seeking for a particular request. From: Joshua C. Colp Sent: Wednesday, 01 July 2020 15:47 To: viljo...@verishare.co.za; Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] 13.22.0 - HTTP session count exceeded 100 sessions - instance unusable On Wed, Jul 1, 2020 at 10:32 AM Stefan Viljoen <mailto:viljo...@verishare.co.za> wrote: Hi Joshua HTTP is used on in our setup on http://127.0.0.1/mxml? to send commands to the server, such as http://127.0.0.1/mxml?action=login&username=myuser&secret=thesecret to log in and then http://127.0.0.1/mxml?ActionID=123&Action=BlindTransfer&Channel=Channel&Context=local&Exten=123&Priority=1 etc. to control transfers, for example. ARI is not being used. WebRTC is not being used. I have in the meantime modified httpd.conf and halved the session_inactivity from 30 000 to 20 000 and session_keep_alive from 15 000 to 10 000, and increased the sessionlimit to 1000 (instead of 100) to see if shorter sessions expiring more frequently and just plain more available slots for sessions will help? So far so good, done about 55 000 new calls that way the past 32 minutes. Or am I barking up the wrong tree? It's possible it will help - but that still doesn't explain what exactly happened. Does your code have a back off strategy if HTTP requests fail? If not and they have an immediate retry with loop, then that may have effectively DDoSed things. It would also be good to explain why 100 connections were in use - did old connections not get closed? Was your load sufficient to just meet that amount? -- Joshua C. Colp Asterisk Technical Lead Sangoma Technologies Check us out at http://www.sangoma.com and http://www.asterisk.org -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- Check out the new Asterisk community forum at: https://community.asterisk.org/ New to Asterisk? Start here: https://wiki.asterisk.org/wiki/display/AST/Getting+Started asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] 13.22.0 - HTTP session count exceeded 100 sessions - instance unusable
On Wed, Jul 1, 2020 at 10:32 AM Stefan Viljoen wrote: > Hi Joshua > > HTTP is used on in our setup on > > 127.0.0.1/mxml? > > to send commands to the server, such as > > http://127.0.0.1/mxml?action=login&username=myuser&secret=thesecret > > to log in and then > > > http://127.0.0.1/mxml?ActionID=123&Action=BlindTransfer&Channel=Channel&Context=local&Exten=123&Priority=1 > > etc. to control transfers, for example. > > ARI is not being used. > > WebRTC is not being used. > > I have in the meantime modified httpd.conf and halved the > session_inactivity from 30 000 to 20 000 and session_keep_alive from 15 000 > to 10 000, and increased the sessionlimit to 1000 (instead of 100) to see > if shorter sessions expiring more frequently and just plain more available > slots for sessions will help? > > So far so good, done about 55 000 new calls that way the past 32 minutes. > > Or am I barking up the wrong tree? > It's possible it will help - but that still doesn't explain what exactly happened. Does your code have a back off strategy if HTTP requests fail? If not and they have an immediate retry with loop, then that may have effectively DDoSed things. It would also be good to explain why 100 connections were in use - did old connections not get closed? Was your load sufficient to just meet that amount? -- Joshua C. Colp Asterisk Technical Lead Sangoma Technologies Check us out at www.sangoma.com and www.asterisk.org -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- Check out the new Asterisk community forum at: https://community.asterisk.org/ New to Asterisk? Start here: https://wiki.asterisk.org/wiki/display/AST/Getting+Started asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] 13.22.0 - HTTP session count exceeded 100 sessions - instance unusable
Hi Joshua HTTP is used on in our setup on 127.0.0.1/mxml? to send commands to the server, such as http://127.0.0.1/mxml?action=login&username=myuser&secret=thesecret to log in and then http://127.0.0.1/mxml?ActionID=123&Action=BlindTransfer&Channel=Channel&Context=local&Exten=123&Priority=1 etc. to control transfers, for example. ARI is not being used. WebRTC is not being used. I have in the meantime modified httpd.conf and halved the session_inactivity from 30 000 to 20 000 and session_keep_alive from 15 000 to 10 000, and increased the sessionlimit to 1000 (instead of 100) to see if shorter sessions expiring more frequently and just plain more available slots for sessions will help? So far so good, done about 55 000 new calls that way the past 32 minutes. Or am I barking up the wrong tree? Thanks! -- Stefan -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- Check out the new Asterisk community forum at: https://community.asterisk.org/ New to Asterisk? Start here: https://wiki.asterisk.org/wiki/display/AST/Getting+Started asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] 13.22.0 - HTTP session count exceeded 100 sessions - instance unusable
On Wed, Jul 1, 2020 at 9:49 AM Stefan Viljoen wrote: > Hi all > > I'm running an Asterisk 13.22.0 instance on Centos 7 - I7-8700 12 core HT > with 16GB of RAM. > > The server maintains a total active call count of approx 285 calls with 440 > channels at any one time. The totals never go below 200 calls concurrently > active. > > For the kernel, top reports load levels at around 5.0 to 6.0 constantly. > > I'm having issues where at random intervals, the CLI is filled with > thousands of these lines > > [Jul 1 14:32:24] WARNING[11530]: http.c:1942 httpd_helper_thread: HTTP > session count exceeded 100 sessions. > > This appears to be some form of race condition, the Asterisk instance goes > completely unresponsive, all RTP stops and all calls get hung up the moment > these appear on the CLI. All registered SIP phones then deregister off the > server withing one to two minutes. > This isn't really a race condition. The message occurs when the limit for simultaneous HTTP sessions is exceeded. You haven't explained what the HTTP server in Asterisk is being used for, so I'd start with investigating that. Is ARI being used? WebRTC? If you do a packet capture on it do you see a flood of incoming HTTP connection attempts? -- Joshua C. Colp Asterisk Technical Lead Sangoma Technologies Check us out at www.sangoma.com and www.asterisk.org -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- Check out the new Asterisk community forum at: https://community.asterisk.org/ New to Asterisk? Start here: https://wiki.asterisk.org/wiki/display/AST/Getting+Started asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
