Hi All, I am trying to use the stress testing with my manual installation, but it doesn’t work.
I understand that many people run stress tests against an automated installed system and this kind of installation requires an Amazon EC2 account. Is there any way to perform an automated installation without using Amazon EC2? thanks > Il giorno 08/nov/2014, alle ore 19:01, Doctor Mescaline > <[email protected]> ha scritto: > > Hi Alex, > > Thanks for your suggestions. For the provision, it made a differences, but I > do not remember if it is because I also change some other configurations. > Anyway it’s not a problem now and I will check it later. > > Following your instructions to reduce the counts, I also got the same error. > The situation is that I can either make real call or run live test > successfully with the preset users and password ‘7kkzTyGW’ in provision,but I > cannot get stress test passed. I suspect there’s something wrong with my > configuration of sip-stress. > > Below is the logs of bono, sprout and sip-stress. > > Thanks > > bono: > > 07-11-2014 12:02:37.671 UTC Call-Disconnected: > CALL_ID=2010000000///[email protected] > <mailto:CALL_ID=2010000000///[email protected]> REASON=401 > 07-11-2014 12:02:37.689 UTC Call-Disconnected: > CALL_ID=2010000001///[email protected] > <mailto:CALL_ID=2010000001///[email protected]> REASON=401 > 07-11-2014 12:17:10.480 UTC Call-Disconnected: > CALL_ID=2010000000///[email protected] > <mailto:CALL_ID=2010000000///[email protected]> REASON=408 > 07-11-2014 12:24:12.694 UTC Call-Disconnected: > CALL_ID=2010000000///[email protected] > <mailto:CALL_ID=2010000000///[email protected]> REASON=401 > 07-11-2014 12:24:12.776 UTC Call-Disconnected: > CALL_ID=2010000001///[email protected] > <mailto:CALL_ID=2010000001///[email protected]> REASON=401 > 07-11-2014 12:32:02.353 UTC Call-Connected: > FROM=sip:[email protected] <mailto:[email protected]> > TO=sip:[email protected] <mailto:[email protected]> > CALL_ID=2010000000-1.000000///[email protected] > <mailto:CALL_ID=2010000000-1.000000///[email protected]> > 07-11-2014 12:32:02.358 UTC Call-Connected: > FROM=sip:[email protected] <mailto:[email protected]> > TO=sip:[email protected] <mailto:[email protected]> > CALL_ID=2010000000-1.000000///[email protected] > <mailto:CALL_ID=2010000000-1.000000///[email protected]> > 07-11-2014 12:35:28.383 UTC Call-Disconnected: > CALL_ID=2010000000///[email protected] > <mailto:CALL_ID=2010000000///[email protected]> REASON=401 > 07-11-2014 12:35:28.445 UTC Call-Disconnected: > CALL_ID=2010000001///[email protected] > <mailto:CALL_ID=2010000001///[email protected]> REASON=401 > 07-11-2014 12:37:27.355 UTC Call-Disconnected: > CALL_ID=NDk3MjIxZjlmNjU4MWM2NGIxYmY5NDE3NWMxMGE4MTk. REASON=403 > 07-11-2014 12:37:30.219 UTC Call-Disconnected: > CALL_ID=OWEwNTRkYzFhYWRlZjMwZDRhNjM2NWY0OTgxZDRjNDA. REASON=403 > 07-11-2014 12:37:32.211 UTC Call-Disconnected: > CALL_ID=NTNiOWE0ZmUzYTYxMjQxOTk4MWU5MTUwZTcyZDI5OTA. REASON=403 > 07-11-2014 12:37:37.552 UTC Call-Disconnected: > CALL_ID=OWJlMDc5ZTU1NDk3YWZiODhiZjkxNGIyZWFhZjljZmQ. REASON=403 > 07-11-2014 12:37:39.583 UTC Call-Disconnected: > CALL_ID=NGQwZTM2OWNmNTUyZTA5Yjc4ZjZiNDk4ZWEwMGVhYjI. REASON=403 > 07-11-2014 12:37:48.224 UTC Call-Disconnected: > CALL_ID=OTYxNDNjYmJhZTJlZTVhYjdkOGZjMTE4ZDY0MDM2MGY. REASON=403 > 07-11-2014 12:38:43.427 UTC Call-Disconnected: > CALL_ID=ZTFiNThlODM5YzAxNDRiZjdkZTA5NDkyZjdiZDI5NmQ. REASON=403 > > sprout: > > 07-11-2014 12:02:37.679 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:29895;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:02:37.695 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:29895;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:07:27.709 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:29895;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:07:27.718 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:29895;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:12:17.735 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:29895;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:12:17.745 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:29895;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:17:07.756 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:29895;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:24:12.733 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:57263;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:24:12.796 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:57263;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:29:02.832 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:57263;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:29:02.857 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:57263;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:35:28.406 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:43377;transport=UDP;ob EXPIRES=3600 > 07-11-2014 12:35:28.466 UTC Registration: > USER_URI=sip:[email protected] <mailto:[email protected]> > BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 > CONTACT_URI=sip:[email protected] > <mailto:[email protected]>:43377;transport=UDP;ob EXPIRES=3600 > > clearwater-sip-stress: > > sipp: The following events occured: > 2014-11-07 12:52:58:406 1415361178.406342: Call-Id: > [email protected] <mailto:[email protected]>, receive timeout on > message Call Load Test:34 without label to jump to (ontimeout attribute): > aborting call. > 2014-11-07 13:17:08:878 1415362628.878665: The minor watchdog timer > 500ms has been tripped (535), 120 trips remaining.. > 2014-11-07 13:17:10:483 1415362630.483070: Aborting call on > unexpected message for Call-Id '[email protected] > <mailto:[email protected]>': while expecting '200' (index 12), received > 'SIP/2.0 408 Request Timeout > Via: SIP/2.0/UDP > 192.168.3.62:29895;rport=29895;received=10.40.7.169;branch=z9hG4bK-27529-2-11-2010000000-8.000000 > Call-ID: 2010000000///[email protected] > <mailto:2010000000///[email protected]> > From: <sip:[email protected] > <sip:[email protected]>>;tag=27529SIPpTag002 > To: <sip:[email protected] > <sip:[email protected]>>;tag=z9hG4bK-27529-2-11-2010000000-8.000000 > CSeq: 9 REGISTER > Content-Length: 0 > > '. > 2014-11-07 13:27:09:230 1415363229.230236: Resetting watchdog timer > trigger counts, as it has not been triggered in over 600351ms.. > 2014-11-07 13:32:14:384 1415363534.384905: Call-Id: > [email protected] <mailto:[email protected]>, receive timeout on > message Call Load Test:34 without label to jump to (ontimeout attribute): > aborting call. > > > > >> Il giorno 06/nov/2014, alle ore 16:34, Alex Hockey >> <[email protected] <mailto:[email protected]>> ha scritto: >> >> Hi there, >> >> I am surprised that modifying the bulk provisioning script made a >> difference. The two database rows that you mentioned are functionally >> equivalent – the `known_preferred` field is no longer used, and Clearwater >> will use a quality of protection of “auth if ” `digest_qop` is not filled in. >> >> Are you able to register a SIP phone using one of the bulk provisioned >> numbers? If so, are you able to register two phones to different bulk >> provisioned numbers and make a call between them. >> >> If this works, then the problem definitely lies with the SIPp script >> somewhere. The next step is to reduce the number of virtual subscribers that >> the SIPp nodes create, and get some logs from the running system. To do this: >> - Turn on debug logging on your sprout node. By default the logging >> level is set to log level 2, which only includes errors and very high level >> events. To enable debug logs, change the log level to 5 by writing >> log_level=5 to /etc/clearwater/user_settings (creating it if it doesn't >> exist already), and restart sprout (using service sprout stop - monit >> automatically restarts it). Sprout will now be writing debug logs to >> /var/log/sprout/sprout_*. >> - Edit /usr/share/clearwater/infrastructure/scripts/sip-stress, and >> set the `count` variable at the top of the file to `2`. >> - Restart SIP stress by running `sudo service clearwater-sip-stress >> stop; sudo service clearwater-infrastructure restart; sudo service >> clearwater-sip-stress start` >> >> Could you try this out and share the output from the most recent logs from >> /var/log/sprout and /var/log/clearwater-sip-stress >> >> Thanks, >> Alex. >> >> >> From: Doctor Mescaline [mailto:[email protected] >> <mailto:[email protected]>] >> Sent: 05 November 2014 15:46 >> To: Alex Hockey >> Cc: [email protected] >> <mailto:[email protected]> >> Subject: Re: [Clearwater] Benchmarking tool >> >> Hi Alex, >> >> every node works well with ‘sudo monit status’. The Clearwater Live Tests >> pass, maybe because they perform “sigh up” with ellis to get authorization. >> >> From my understanding, what the “provision subscriber” does is to directly >> inject the authorization of the numbers into the database, hence skipping >> the sign up step on ellis. >> >> >> If I run the “provision subscriber” I get this entry >> >> private_id | digest_ha1 | digest_qop | >> digest_realm | known_preferred >> ----------------------------+----------------------------------+------------+-----------------+----------------- >> [email protected] <mailto:[email protected]> | >> bcdbbf195231a7c2285b713b373aa70a | null | demo.clearwater | >> null >> >> >> So I modified the live test suite to register with existing numbers and >> passwords in such records instead of signing up with ellis. >> But it didn’t work util I notice that the record for an authorized number >> should be like: >> >> private_id | digest_ha1 | digest_qop >> | digest_realm | known_preferred >> ----------------------------+----------------------------------+------------+-----------------+----------------- >> [email protected] <mailto:[email protected]> | >> bcdbbf195231a7c2285b713b373aa70a | auth | demo.clearwater | >> True >> >> Then I modified the provision script to form up the records in above way and >> this time the test live suite works well with basic calls, which means that >> such numbers are available for register-make a call-unregister. >> But under such configuration we still get the stress test node failed (0 >> success call in the output csv file) with messages like: >> >> 2014-11-04 07:15:38:733 1415081738.733320: Call-Id: >> [email protected] <mailto:[email protected]>, receive timeout on >> message Call Load Test:19 without label to jump to (ontimeout attribute): >> aborting call. >> >> The logs of other nodes seem normal except the bono node showing some 503. >> >> Thanks >> >> >> Il giorno 05/nov/2014, alle ore 12:59, Alex Hockey >> <[email protected] <mailto:[email protected]>> ha scritto: >> >> Hi there, >> >> Provisioning the subscribers after turning up the SIPp node should work. >> >> You are seeing two types of “disconnected” logs from Bono: >> - The 401s are expected. The first time a client registers with Clearwater >> it will not provide authentication credentials. Sprout rejects the REGISTER >> message with a 401 response, containing an authentication challenge. The >> client then sends another REGISTER with credentials. >> - The 503s are not expected. I suspect these are responses to the 2nd, >> authenticated REGISTER message. They suggest a part of your deployment is >> not running properly. >> >> The first step is to check that all the Clearwater processes are running >> successfully. Run `sudo monit status` on a node to check this, and look out >> for any entries that are reported as failed or non-existent. If you are >> using an all-in-one node, just run `sudo monit status` from that node. If >> you are running a multi-node deployment, run the command on all your nodes. >> >> Alex. >> >> From: Doctor Mescaline [mailto:[email protected] >> <mailto:[email protected]>] >> Sent: 04 November 2014 12:17 >> To: Alex Hockey >> Cc: [email protected] >> <mailto:[email protected]> >> Subject: Re: [Clearwater] Benchmarking tool >> >> Hi Alex, >> >> I already did it and calls were always disconnected with REASON=401 or >> REASON=503. >> >> I created the new SIPp node and then bulked provision subscribers, while you >> suggest to bulk provision subscribers and then create the new SIPp node. >> >> Is this order important? >> >> >> Thanks >> >> Il giorno 04/nov/2014, alle ore 12:47, Alex Hockey >> <[email protected] <mailto:[email protected]>> ha scritto: >> >> Hi there, >> >> I suspect the problem here is that you haven’t provisioned any subscribers >> on your deployment. When the SIPp nodes try to register, clearwater rejects >> their requests with a 403 Forbidden. >> >> The stress testing wiki page >> (https://github.com/Metaswitch/clearwater-docs/wiki/Clearwater-stress-testing >> >> <https://github.com/Metaswitch/clearwater-docs/wiki/Clearwater-stress-testing>) >> was missing a “provision subscribers” step. I have updated it so that is >> contains a link to the instructions for how to do this. It is a fairly >> simple process, that just involves running some scripts on your homer and >> homestead nodes. >> >> I hope this helps. We haven’t had many people running stress against a >> manually installed system before, and you’ve helped us find some gaps in our >> documentation! >> >> Thanks, >> Alex. >> >> From: The Doctor [mailto:[email protected] >> <mailto:[email protected]>] >> Sent: 31 October 2014 15:58 >> To: Alex Hockey >> Subject: Re: [Clearwater] Benchmarking tool >> >> Hi Alex, >> >> I got it. I have configured the file and installed the package, but it 'is >> not able to successfully run the scenario. When running the stress test, I >> got these message in the bono node >> >> UTC Call-Disconnected: CALL_ID=7000069028///[email protected] >> <mailto:[email protected]> REASON=403 >> >> Do you think the CALL_ID format is correct? >> >> Thanks >> >> >> >> 2014-10-31 14:03 GMT+01:00 Alex Hockey <[email protected] >> <mailto:[email protected]>>: >> Hi there, >> >> I assume that you’ve set up a Clearwater deployment using the Manual Install >> instructions >> (https://github.com/Metaswitch/clearwater-docs/wiki/Manual-Install >> <https://github.com/Metaswitch/clearwater-docs/wiki/Manual-Install>). The >> SIP stress node should be a completely new virtual machine. You should >> bootstrap this machine in the same way you bootstrapped the nodes as part of >> setting up the deployment (this bit wasn’t clear in the page I pointed you >> at). You can then install clearwater-sip-stress as a normal debian package >> by running `sudo apt-get install clearwater-sip-stress`. >> >> Alex. >> >> p.s. Please could you CC the list on responses? That way other members of >> the community benefit from the discussion :-). >> >> From: The Doctor [mailto:[email protected] >> <mailto:[email protected]>] >> Sent: 31 October 2014 11:02 >> To: Alex Hockey >> Subject: Re: [Clearwater] Benchmarking tool >> >> Hi Alex, >> >> I have read the wiki, but I don't understand where I can downaload the >> clearwater-sip-stress Debian package and in which node I have to edit the >> /etc/clearwater/config. >> >> Thanks >> >> 2014-10-31 11:07 GMT+01:00 Alex Hockey <[email protected] >> <mailto:[email protected]>>: >> Hi there, >> >> In the Project Clearwater team we use our clearwater-sip-stress package for >> stress testing and benchmarking. This is built on SIPp >> (http://sipp.sourceforge.net/ <http://sipp.sourceforge.net/>). There is more >> information about clearwater-sip-stress on the wiki >> (https://github.com/Metaswitch/clearwater-docs/wiki/Clearwater-stress-testing >> >> <https://github.com/Metaswitch/clearwater-docs/wiki/Clearwater-stress-testing>). >> >> By default clearwater-sip-stress runs a fairly simple scenario, where >> simulated endpoints REGISTER, then periodically re-REGISTER and make simple >> calls. However you can edit the SIPp scenario file to emulate different >> behaviour. >> >> I'm not sure if this is what you're after, but I hope it helps. Other users >> on the list may have better suggestions though! >> >> Alex. >> >> -----Original Message----- >> From: [email protected] >> <mailto:[email protected]> >> [mailto:[email protected] >> <mailto:[email protected]>] On Behalf Of The >> Doctor >> Sent: 31 October 2014 09:12 >> To: [email protected] >> <mailto:[email protected]> >> Subject: [Clearwater] Benchmarking tool >> >> Hi list, >> >> I am looking for a benchmarking tool in order to emulates users behavior >> according to variours workload patterns. Is there any tool or Step-by-Step >> guide to do that? >> >> Thanks >> _______________________________________________ >> Clearwater mailing list >> [email protected] >> <mailto:[email protected]> >> http://lists.projectclearwater.org/listinfo/clearwater >> <http://lists.projectclearwater.org/listinfo/clearwater> _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
