Hi everyone, I'm trying to create a testbed to test concurrent calls in some different iPBXs. The testbed consists in a UAC and a UAS and they try to start a maximum of 500 calls through a iPBX (send INVITE and then after provision messages and OK, UAC pause for 9 minutes and both exchange rtp with each other, after the pause, UAC start ending with BYE the open calls).
The problem is I can't reach 500 concurrent calls in UAC/UAS directly connected without iPBX in the middle. Everything goes ok when starting the calls between UAC and UAS, but when the number of calls reach around 400 calls (more or less), a process called ksoftirqd (runs when the machine is under heavy soft- interrupt load) start increasing, on both UAC and UAS, in one CPU and reach 100% of that CPU. This cause the UAC and the UAS to start re- transmissions, then comes the timeouts and unexpected messages (in attachment I send an error trace from UAC). After I checked interrupts, I noticed the biggest requester is the NIC, so I instructs kernel to distribute that IRQ requests for more than one CPU trying to reduce the behavior. When I start the test after this change, everything goes as expected, I see three ksoftirqd in the cpus I defined, but when reaching about 400 calls, one random ksoftirqd of the three, reach 100% and the problems starts again. Machine loads stay low because this behavior only occurs in the defined CPUs from the 8 the machines have and they have plenty of free memory. Before I noticed the real problem I tried to change timeout and re- transmissions timers, and other specs, without any success. I already tested this behavior in machines with different specs (the first two where weaker the the last ones) with CentOS Linux release 7.2.1511 with 3.10.0-327.10.1.el7.x86_64 kernel. I tested this in sipp versions 3.5.0 and 3.5.1. Anyone already faced such behavior? Regards, Paulo -- ------------------------------------------- Paulo Ferreira VoIP@RCTS - Área de Infraestruturas Aplicacionais FCCN http://www.fccn.pt/ Av. do Brasil, n.º 101 1700-066 Lisboa - Portugal Telefone|Phone: +351 218440100; Fax: +351 218472167 Aviso de Confidencialidade/Disclaimer Esta mensagem é exclusivamente destinada ao seu destinatário, podendo conter informação CONFIDENCIAL, cuja divulgação está expressamente vedada nos termos da lei. Caso tenha rececionado indevidamente esta mensagem,solicitamos-lhe que nos comunique esse mesmo facto por esta via ou para o telefone +351 218440100 devendo apagar o seu conteúdo de imediato. This message is intended exclusively for its addressee. It may contain CONFIDENTIAL information protected by law. If this message has been received by error, please notify us via e-mail or by telephone +351 218440100 and delete it immediately.
sipp: The following events occured: 2016-03-15 18:26:31.617388 1458066391.617388: Aborting call on UDP retransmission timeout for Call-ID '448-4856@10.0.53.6'. 2016-03-15 18:26:42.118099 1458066402.118099: Aborting call on UDP retransmission timeout for Call-ID '469-4856@10.0.53.6'. 2016-03-15 18:26:45.117706 1458066405.117706: Aborting call on UDP retransmission timeout for Call-ID '475-4856@10.0.53.6'. 2016-03-15 18:26:45.618625 1458066405.618625: Aborting call on UDP retransmission timeout for Call-ID '476-4856@10.0.53.6'. 2016-03-15 18:26:47.620133 1458066407.620133: Aborting call on UDP retransmission timeout for Call-ID '480-4856@10.0.53.6'. 2016-03-15 18:26:49.619640 1458066409.619640: Aborting call on UDP retransmission timeout for Call-ID '484-4856@10.0.53.6'. 2016-03-15 18:26:53.117987 1458066413.117987: Aborting call on UDP retransmission timeout for Call-ID '491-4856@10.0.53.6'. 2016-03-15 18:31:48.631042 1458066708.631042: Aborting call on UDP retransmission timeout for Call-ID '2-4856@10.0.53.6'. 2016-03-15 18:31:50.132294 1458066710.132294: Aborting call on UDP retransmission timeout for Call-ID '5-4856@10.0.53.6'. 2016-03-15 18:31:50.627638 1458066710.627638: Aborting call on UDP retransmission timeout for Call-ID '6-4856@10.0.53.6'. 2016-03-15 18:31:51.129958 1458066711.129958: Aborting call on UDP retransmission timeout for Call-ID '7-4856@10.0.53.6'. 2016-03-15 18:31:51.628162 1458066711.628162: Aborting call on UDP retransmission timeout for Call-ID '8-4856@10.0.53.6'. 2016-03-15 18:31:52.129225 1458066712.129225: Aborting call on UDP retransmission timeout for Call-ID '9-4856@10.0.53.6'. 2016-03-15 18:31:52.629211 1458066712.629211: Aborting call on UDP retransmission timeout for Call-ID '10-4856@10.0.53.6'. 2016-03-15 18:31:53.629212 1458066713.629212: Aborting call on UDP retransmission timeout for Call-ID '12-4856@10.0.53.6'. 2016-03-15 18:31:54.628903 1458066714.628903: Aborting call on UDP retransmission timeout for Call-ID '14-4856@10.0.53.6'. 2016-03-15 18:31:55.631741 1458066715.631741: Aborting call on UDP retransmission timeout for Call-ID '16-4856@10.0.53.6'. 2016-03-15 18:31:56.131610 1458066716.131610: Aborting call on UDP retransmission timeout for Call-ID '17-4856@10.0.53.6'. 2016-03-15 18:31:56.632229 1458066716.632229: Aborting call on UDP retransmission timeout for Call-ID '18-4856@10.0.53.6'. 2016-03-15 18:31:57.131983 1458066717.131983: Aborting call on UDP retransmission timeout for Call-ID '19-4856@10.0.53.6'. 2016-03-15 18:31:58.131483 1458066718.131483: Aborting call on UDP retransmission timeout for Call-ID '21-4856@10.0.53.6'. 2016-03-15 18:31:58.627972 1458066718.627972: Aborting call on UDP retransmission timeout for Call-ID '22-4856@10.0.53.6'. 2016-03-15 18:32:00.130141 1458066720.130141: Aborting call on UDP retransmission timeout for Call-ID '25-4856@10.0.53.6'. 2016-03-15 18:32:00.629877 1458066720.629877: Aborting call on UDP retransmission timeout for Call-ID '26-4856@10.0.53.6'. 2016-03-15 18:32:01.129844 1458066721.129844: Aborting call on UDP retransmission timeout for Call-ID '27-4856@10.0.53.6'. 2016-03-15 18:32:01.632834 1458066721.632834: Aborting call on UDP retransmission timeout for Call-ID '28-4856@10.0.53.6'. 2016-03-15 18:32:02.630930 1458066722.630930: Aborting call on UDP retransmission timeout for Call-ID '30-4856@10.0.53.6'. 2016-03-15 18:32:03.131533 1458066723.131533: Aborting call on UDP retransmission timeout for Call-ID '31-4856@10.0.53.6'. 2016-03-15 18:32:03.630386 1458066723.630386: Aborting call on UDP retransmission timeout for Call-ID '32-4856@10.0.53.6'. 2016-03-15 18:32:04.129790 1458066724.129790: Aborting call on UDP retransmission timeout for Call-ID '33-4856@10.0.53.6'. 2016-03-15 18:32:04.632280 1458066724.632280: Aborting call on UDP retransmission timeout for Call-ID '34-4856@10.0.53.6'. 2016-03-15 18:32:06.130145 1458066726.130145: Aborting call on UDP retransmission timeout for Call-ID '37-4856@10.0.53.6'. 2016-03-15 18:32:06.627546 1458066726.627546: Aborting call on UDP retransmission timeout for Call-ID '38-4856@10.0.53.6'. 2016-03-15 18:32:07.131152 1458066727.131152: Aborting call on UDP retransmission timeout for Call-ID '39-4856@10.0.53.6'. 2016-03-15 18:32:08.129462 1458066728.129462: Aborting call on UDP retransmission timeout for Call-ID '41-4856@10.0.53.6'. 2016-03-15 18:32:08.628973 1458066728.628973: Aborting call on UDP retransmission timeout for Call-ID '42-4856@10.0.53.6'. 2016-03-15 18:32:09.128628 1458066729.128628: Aborting call on UDP retransmission timeout for Call-ID '43-4856@10.0.53.6'. 2016-03-15 18:32:10.629069 1458066730.629069: Aborting call on UDP retransmission timeout for Call-ID '46-4856@10.0.53.6'. 2016-03-15 18:32:11.130743 1458066731.130743: Aborting call on UDP retransmission timeout for Call-ID '47-4856@10.0.53.6'. 2016-03-15 18:32:11.630335 1458066731.630335: Aborting call on UDP retransmission timeout for Call-ID '48-4856@10.0.53.6'. 2016-03-15 18:32:12.128528 1458066732.128528: Aborting call on UDP retransmission timeout for Call-ID '49-4856@10.0.53.6'. 2016-03-15 18:32:13.127757 1458066733.127757: Aborting call on UDP retransmission timeout for Call-ID '51-4856@10.0.53.6'. 2016-03-15 18:32:13.628876 1458066733.628876: Aborting call on UDP retransmission timeout for Call-ID '52-4856@10.0.53.6'. 2016-03-15 18:32:14.127758 1458066734.127758: Aborting call on UDP retransmission timeout for Call-ID '53-4856@10.0.53.6'. 2016-03-15 18:32:15.134384 1458066735.134384: Aborting call on UDP retransmission timeout for Call-ID '55-4856@10.0.53.6'. 2016-03-15 18:32:15.632499 1458066735.632499: Aborting call on UDP retransmission timeout for Call-ID '56-4856@10.0.53.6'. 2016-03-15 18:32:16.631941 1458066736.631941: Aborting call on UDP retransmission timeout for Call-ID '58-4856@10.0.53.6'. 2016-03-15 18:32:17.128393 1458066737.128393: Aborting call on UDP retransmission timeout for Call-ID '59-4856@10.0.53.6'. 2016-03-15 18:32:17.627424 1458066737.627424: Aborting call on UDP retransmission timeout for Call-ID '60-4856@10.0.53.6'. 2016-03-15 18:32:18.130638 1458066738.130638: Aborting call on UDP retransmission timeout for Call-ID '61-4856@10.0.53.6'. 2016-03-15 18:32:18.626999 1458066738.626999: Aborting call on UDP retransmission timeout for Call-ID '62-4856@10.0.53.6'. 2016-03-15 18:32:19.129310 1458066739.129310: Aborting call on UDP retransmission timeout for Call-ID '63-4856@10.0.53.6'. 2016-03-15 18:32:22.128320 1458066742.128320: Aborting call on UDP retransmission timeout for Call-ID '69-4856@10.0.53.6'. 2016-03-15 18:32:24.632808 1458066744.632808: Aborting call on UDP retransmission timeout for Call-ID '74-4856@10.0.53.6'. 2016-03-15 18:32:25.631924 1458066745.631924: Aborting call on UDP retransmission timeout for Call-ID '76-4856@10.0.53.6'. 2016-03-15 18:32:27.632634 1458066747.632634: Aborting call on UDP retransmission timeout for Call-ID '80-4856@10.0.53.6'. 2016-03-15 18:33:04.633355 1458066784.633355: Aborting call on UDP retransmission timeout for Call-ID '154-4856@10.0.53.6'. 2016-03-15 18:33:08.634934 1458066788.634934: Aborting call on UDP retransmission timeout for Call-ID '162-4856@10.0.53.6'. 2016-03-15 18:33:10.628405 1458066790.628405: Aborting call on UDP retransmission timeout for Call-ID '166-4856@10.0.53.6'. 2016-03-15 18:33:11.131720 1458066791.131720: Aborting call on UDP retransmission timeout for Call-ID '167-4856@10.0.53.6'. 2016-03-15 18:33:12.636575 1458066792.636575: Aborting call on UDP retransmission timeout for Call-ID '170-4856@10.0.53.6'.
signature.asc
Description: This is a digitally signed message part
------------------------------------------------------------------------------ Transform Data into Opportunity. Accelerate data analysis in your applications with Intel Data Analytics Acceleration Library. Click to learn more. http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
_______________________________________________ Sipp-users mailing list Sipp-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sipp-users
