Hi Alex, Thanks for your suggestions. For the provision, it made a differences, but I do not remember if it is because I also change some other configurations. Anyway it’s not a problem now and I will check it later.
Following your instructions to reduce the counts, I also got the same error. The situation is that I can either make real call or run live test successfully with the preset users and password ‘7kkzTyGW’ in provision,but I cannot get stress test passed. I suspect there’s something wrong with my configuration of sip-stress. Below is the logs of bono, sprout and sip-stress. Thanks bono: 07-11-2014 12:02:37.671 UTC Call-Disconnected: CALL_ID=2010000000///[email protected] REASON=401 07-11-2014 12:02:37.689 UTC Call-Disconnected: CALL_ID=2010000001///[email protected] REASON=401 07-11-2014 12:17:10.480 UTC Call-Disconnected: CALL_ID=2010000000///[email protected] REASON=408 07-11-2014 12:24:12.694 UTC Call-Disconnected: CALL_ID=2010000000///[email protected] REASON=401 07-11-2014 12:24:12.776 UTC Call-Disconnected: CALL_ID=2010000001///[email protected] REASON=401 07-11-2014 12:32:02.353 UTC Call-Connected: FROM=sip:[email protected] TO=sip:[email protected] CALL_ID=2010000000-1.000000///[email protected] 07-11-2014 12:32:02.358 UTC Call-Connected: FROM=sip:[email protected] TO=sip:[email protected] CALL_ID=2010000000-1.000000///[email protected] 07-11-2014 12:35:28.383 UTC Call-Disconnected: CALL_ID=2010000000///[email protected] REASON=401 07-11-2014 12:35:28.445 UTC Call-Disconnected: CALL_ID=2010000001///[email protected] REASON=401 07-11-2014 12:37:27.355 UTC Call-Disconnected: CALL_ID=NDk3MjIxZjlmNjU4MWM2NGIxYmY5NDE3NWMxMGE4MTk. REASON=403 07-11-2014 12:37:30.219 UTC Call-Disconnected: CALL_ID=OWEwNTRkYzFhYWRlZjMwZDRhNjM2NWY0OTgxZDRjNDA. REASON=403 07-11-2014 12:37:32.211 UTC Call-Disconnected: CALL_ID=NTNiOWE0ZmUzYTYxMjQxOTk4MWU5MTUwZTcyZDI5OTA. REASON=403 07-11-2014 12:37:37.552 UTC Call-Disconnected: CALL_ID=OWJlMDc5ZTU1NDk3YWZiODhiZjkxNGIyZWFhZjljZmQ. REASON=403 07-11-2014 12:37:39.583 UTC Call-Disconnected: CALL_ID=NGQwZTM2OWNmNTUyZTA5Yjc4ZjZiNDk4ZWEwMGVhYjI. REASON=403 07-11-2014 12:37:48.224 UTC Call-Disconnected: CALL_ID=OTYxNDNjYmJhZTJlZTVhYjdkOGZjMTE4ZDY0MDM2MGY. REASON=403 07-11-2014 12:38:43.427 UTC Call-Disconnected: CALL_ID=ZTFiNThlODM5YzAxNDRiZjdkZTA5NDkyZjdiZDI5NmQ. REASON=403 sprout: 07-11-2014 12:02:37.679 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:29895;transport=UDP;ob EXPIRES=3600 07-11-2014 12:02:37.695 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:29895;transport=UDP;ob EXPIRES=3600 07-11-2014 12:07:27.709 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:29895;transport=UDP;ob EXPIRES=3600 07-11-2014 12:07:27.718 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:29895;transport=UDP;ob EXPIRES=3600 07-11-2014 12:12:17.735 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:29895;transport=UDP;ob EXPIRES=3600 07-11-2014 12:12:17.745 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:29895;transport=UDP;ob EXPIRES=3600 07-11-2014 12:17:07.756 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:29895;transport=UDP;ob EXPIRES=3600 07-11-2014 12:24:12.733 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:57263;transport=UDP;ob EXPIRES=3600 07-11-2014 12:24:12.796 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:57263;transport=UDP;ob EXPIRES=3600 07-11-2014 12:29:02.832 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:57263;transport=UDP;ob EXPIRES=3600 07-11-2014 12:29:02.857 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:57263;transport=UDP;ob EXPIRES=3600 07-11-2014 12:35:28.406 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:43377;transport=UDP;ob EXPIRES=3600 07-11-2014 12:35:28.466 UTC Registration: USER_URI=sip:[email protected] BINDING_ID=<urn:uuid:00000000-0000-0000-0000-000000000001>:1 CONTACT_URI=sip:[email protected]:43377;transport=UDP;ob EXPIRES=3600 clearwater-sip-stress: sipp: The following events occured: 2014-11-07 12:52:58:406 1415361178.406342: Call-Id: [email protected], receive timeout on message Call Load Test:34 without label to jump to (ontimeout attribute): aborting call. 2014-11-07 13:17:08:878 1415362628.878665: The minor watchdog timer 500ms has been tripped (535), 120 trips remaining.. 2014-11-07 13:17:10:483 1415362630.483070: Aborting call on unexpected message for Call-Id '[email protected]': while expecting '200' (index 12), received 'SIP/2.0 408 Request Timeout Via: SIP/2.0/UDP 192.168.3.62:29895;rport=29895;received=10.40.7.169;branch=z9hG4bK-27529-2-11-2010000000-8.000000 Call-ID: 2010000000///[email protected] From: <sip:[email protected]>;tag=27529SIPpTag002 To: <sip:[email protected]>;tag=z9hG4bK-27529-2-11-2010000000-8.000000 CSeq: 9 REGISTER Content-Length: 0 '. 2014-11-07 13:27:09:230 1415363229.230236: Resetting watchdog timer trigger counts, as it has not been triggered in over 600351ms.. 2014-11-07 13:32:14:384 1415363534.384905: Call-Id: [email protected], receive timeout on message Call Load Test:34 without label to jump to (ontimeout attribute): aborting call. > Il giorno 06/nov/2014, alle ore 16:34, Alex Hockey > <[email protected]> ha scritto: > > Hi there, > > I am surprised that modifying the bulk provisioning script made a difference. > The two database rows that you mentioned are functionally equivalent – the > `known_preferred` field is no longer used, and Clearwater will use a quality > of protection of “auth if ” `digest_qop` is not filled in. > > Are you able to register a SIP phone using one of the bulk provisioned > numbers? If so, are you able to register two phones to different bulk > provisioned numbers and make a call between them. > > If this works, then the problem definitely lies with the SIPp script > somewhere. The next step is to reduce the number of virtual subscribers that > the SIPp nodes create, and get some logs from the running system. To do this: > - Turn on debug logging on your sprout node. By default the logging > level is set to log level 2, which only includes errors and very high level > events. To enable debug logs, change the log level to 5 by writing > log_level=5 to /etc/clearwater/user_settings (creating it if it doesn't exist > already), and restart sprout (using service sprout stop - monit automatically > restarts it). Sprout will now be writing debug logs to > /var/log/sprout/sprout_*. > - Edit /usr/share/clearwater/infrastructure/scripts/sip-stress, and set > the `count` variable at the top of the file to `2`. > - Restart SIP stress by running `sudo service clearwater-sip-stress > stop; sudo service clearwater-infrastructure restart; sudo service > clearwater-sip-stress start` > > Could you try this out and share the output from the most recent logs from > /var/log/sprout and /var/log/clearwater-sip-stress > > Thanks, > Alex. > > > From: Doctor Mescaline [mailto:[email protected]] > Sent: 05 November 2014 15:46 > To: Alex Hockey > Cc: [email protected] > Subject: Re: [Clearwater] Benchmarking tool > > Hi Alex, > > every node works well with ‘sudo monit status’. The Clearwater Live Tests > pass, maybe because they perform “sigh up” with ellis to get authorization. > > From my understanding, what the “provision subscriber” does is to directly > inject the authorization of the numbers into the database, hence skipping the > sign up step on ellis. > > > If I run the “provision subscriber” I get this entry > > private_id | digest_ha1 | digest_qop | > digest_realm | known_preferred > ----------------------------+----------------------------------+------------+-----------------+----------------- > [email protected] <mailto:[email protected]> | > bcdbbf195231a7c2285b713b373aa70a | null | demo.clearwater | > null > > > So I modified the live test suite to register with existing numbers and > passwords in such records instead of signing up with ellis. > But it didn’t work util I notice that the record for an authorized number > should be like: > > private_id | digest_ha1 | digest_qop | > digest_realm | known_preferred > ----------------------------+----------------------------------+------------+-----------------+----------------- > [email protected] <mailto:[email protected]> | > bcdbbf195231a7c2285b713b373aa70a | auth | demo.clearwater | > True > > Then I modified the provision script to form up the records in above way and > this time the test live suite works well with basic calls, which means that > such numbers are available for register-make a call-unregister. > But under such configuration we still get the stress test node failed (0 > success call in the output csv file) with messages like: > > 2014-11-04 07:15:38:733 1415081738.733320: Call-Id: > [email protected] <mailto:[email protected]>, receive timeout on > message Call Load Test:19 without label to jump to (ontimeout attribute): > aborting call. > > The logs of other nodes seem normal except the bono node showing some 503. > > Thanks > > > Il giorno 05/nov/2014, alle ore 12:59, Alex Hockey > <[email protected] <mailto:[email protected]>> ha scritto: > > Hi there, > > Provisioning the subscribers after turning up the SIPp node should work. > > You are seeing two types of “disconnected” logs from Bono: > - The 401s are expected. The first time a client registers with Clearwater > it will not provide authentication credentials. Sprout rejects the REGISTER > message with a 401 response, containing an authentication challenge. The > client then sends another REGISTER with credentials. > - The 503s are not expected. I suspect these are responses to the 2nd, > authenticated REGISTER message. They suggest a part of your deployment is not > running properly. > > The first step is to check that all the Clearwater processes are running > successfully. Run `sudo monit status` on a node to check this, and look out > for any entries that are reported as failed or non-existent. If you are using > an all-in-one node, just run `sudo monit status` from that node. If you are > running a multi-node deployment, run the command on all your nodes. > > Alex. > > From: Doctor Mescaline [mailto:[email protected] > <mailto:[email protected]>] > Sent: 04 November 2014 12:17 > To: Alex Hockey > Cc: [email protected] > <mailto:[email protected]> > Subject: Re: [Clearwater] Benchmarking tool > > Hi Alex, > > I already did it and calls were always disconnected with REASON=401 or > REASON=503. > > I created the new SIPp node and then bulked provision subscribers, while you > suggest to bulk provision subscribers and then create the new SIPp node. > > Is this order important? > > > Thanks > > Il giorno 04/nov/2014, alle ore 12:47, Alex Hockey > <[email protected] <mailto:[email protected]>> ha scritto: > > Hi there, > > I suspect the problem here is that you haven’t provisioned any subscribers on > your deployment. When the SIPp nodes try to register, clearwater rejects > their requests with a 403 Forbidden. > > The stress testing wiki page > (https://github.com/Metaswitch/clearwater-docs/wiki/Clearwater-stress-testing > <https://github.com/Metaswitch/clearwater-docs/wiki/Clearwater-stress-testing>) > was missing a “provision subscribers” step. I have updated it so that is > contains a link to the instructions for how to do this. It is a fairly simple > process, that just involves running some scripts on your homer and homestead > nodes. > > I hope this helps. We haven’t had many people running stress against a > manually installed system before, and you’ve helped us find some gaps in our > documentation! > > Thanks, > Alex. > > From: The Doctor [mailto:[email protected] > <mailto:[email protected]>] > Sent: 31 October 2014 15:58 > To: Alex Hockey > Subject: Re: [Clearwater] Benchmarking tool > > Hi Alex, > > I got it. I have configured the file and installed the package, but it 'is > not able to successfully run the scenario. When running the stress test, I > got these message in the bono node > > UTC Call-Disconnected: CALL_ID=7000069028///[email protected] > <mailto:[email protected]> REASON=403 > > Do you think the CALL_ID format is correct? > > Thanks > > > > 2014-10-31 14:03 GMT+01:00 Alex Hockey <[email protected] > <mailto:[email protected]>>: > Hi there, > > I assume that you’ve set up a Clearwater deployment using the Manual Install > instructions > (https://github.com/Metaswitch/clearwater-docs/wiki/Manual-Install > <https://github.com/Metaswitch/clearwater-docs/wiki/Manual-Install>). The SIP > stress node should be a completely new virtual machine. You should bootstrap > this machine in the same way you bootstrapped the nodes as part of setting up > the deployment (this bit wasn’t clear in the page I pointed you at). You can > then install clearwater-sip-stress as a normal debian package by running > `sudo apt-get install clearwater-sip-stress`. > > Alex. > > p.s. Please could you CC the list on responses? That way other members of the > community benefit from the discussion :-). > > From: The Doctor [mailto:[email protected] > <mailto:[email protected]>] > Sent: 31 October 2014 11:02 > To: Alex Hockey > Subject: Re: [Clearwater] Benchmarking tool > > Hi Alex, > > I have read the wiki, but I don't understand where I can downaload the > clearwater-sip-stress Debian package and in which node I have to edit the > /etc/clearwater/config. > > Thanks > > 2014-10-31 11:07 GMT+01:00 Alex Hockey <[email protected] > <mailto:[email protected]>>: > Hi there, > > In the Project Clearwater team we use our clearwater-sip-stress package for > stress testing and benchmarking. This is built on SIPp > (http://sipp.sourceforge.net/ <http://sipp.sourceforge.net/>). There is more > information about clearwater-sip-stress on the wiki > (https://github.com/Metaswitch/clearwater-docs/wiki/Clearwater-stress-testing > <https://github.com/Metaswitch/clearwater-docs/wiki/Clearwater-stress-testing>). > > By default clearwater-sip-stress runs a fairly simple scenario, where > simulated endpoints REGISTER, then periodically re-REGISTER and make simple > calls. However you can edit the SIPp scenario file to emulate different > behaviour. > > I'm not sure if this is what you're after, but I hope it helps. Other users > on the list may have better suggestions though! > > Alex. > > -----Original Message----- > From: [email protected] > <mailto:[email protected]> > [mailto:[email protected] > <mailto:[email protected]>] On Behalf Of The > Doctor > Sent: 31 October 2014 09:12 > To: [email protected] > <mailto:[email protected]> > Subject: [Clearwater] Benchmarking tool > > Hi list, > > I am looking for a benchmarking tool in order to emulates users behavior > according to variours workload patterns. Is there any tool or Step-by-Step > guide to do that? > > Thanks > _______________________________________________ > Clearwater mailing list > [email protected] > <mailto:[email protected]> > http://lists.projectclearwater.org/listinfo/clearwater > <http://lists.projectclearwater.org/listinfo/clearwater> _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
