Sorry I could not add error log (sip-stress_543_errors.log). This is as below:
sipp: The following events occured: 2016-08-17 14:53:20:985 1471445600.985925: Aborting call on unexpected message for Call-Id '[email protected]': while expecting '401' (index 3), received 'SIP/2.0 404 Not Found^M Via: SIP/2.0/TCP 10.0.22.193:5060 ;rport=47137;received=10.0.22.193;branch=z9hG4bK-543-954-2-2010003092-^M Record-Route: <sip:scscf.cwaio:5054;transport=TCP;lr;service=scscf;billing-role=charge-orig>^M Call-ID: 2010003092///[email protected]^M From: <sip:[email protected]>;tag=543SIPpTag00954^M To: <sip:[email protected] >;tag=z9hG4bKPjbzBKmCQaMJKB5als7O2r1RyzYp37C91d^M CSeq: 1 REGISTER^M Content-Length: 0^M ^M '. 2016-08-17 14:53:21:255 1471445601.255900: Aborting call on unexpected message for Call-Id '[email protected]': while expecting '401' (index 3), received 'SIP/2.0 404 Not Found^M Via: SIP/2.0/TCP 10.0.22.193:5060 ;rport=47138;received=10.0.22.193;branch=z9hG4bK-543-1317-2-2010002366-^M Record-Route: <sip:scscf.cwaio:5054;transport=TCP;lr;service=scscf;billing-role=charge-orig>^M Call-ID: 2010002366///[email protected]^M From: <sip:[email protected]>;tag=543SIPpTag001317^M To: <sip:[email protected] >;tag=z9hG4bKPj3jbAiJfr3TrTlzsuq4n3J8VVGdfeCFNX^M CSeq: 1 REGISTER^M Content-Length: 0^M ... Thanks Kosala On Wed, Aug 17, 2016 at 3:54 PM, Kosala <[email protected]> wrote: > Hi Matt, > > I changed the provisioning range as 2010000000-2010000999. > However, I am still getting this error " unexpected message for Call-Id > .." > > > Last Error: Aborting call on unexpected message for Call-Id '2467-27374@ > ...^M > ^[[2J------------------------------ Scenario Screen -------- [1-9]: > Change Screen --^M > Users (length) Port Total-time Total-calls Remote-host^M > 2500 (0 ms) 5060 384.70 s 2500 10.0.21.192:5060 > (TCP)^M > ^M > Call limit reached (-m 2500), 1.001 s period 1 ms scheduler resolution^M > 919 calls (limit 2500) Peak was 2500 calls, after 0 s^M > 0 Running, 921 Paused, 7 Woken up^M > 0 dead call msg (discarded) 0 out-of-call msg > (discarded) ^M > 2 open sockets ^M > ^M > Messages Retrans Timeout > Unexpected-Msg^M > Pause [0ms/10:00] 2500 0 ^M > REGISTER ----------> 1581 0 ^M > 401 <---------- 0 0 0 1581 ^M > REGISTER ----------> 0 0 ^M > 200 <---------- 0 0 0 0 ^M > REGISTER ----------> 0 0 ^M > 401 <---------- 0 0 0 0 ^M > REGISTER ----------> 0 0 ^M > 200 <---------- 0 0 0 0 ^M > Pause [ 4:50] 0 0 ^M > REGISTER ----------> B-RTD1 0 0 ^M > 200 <---------- E-RTD1 0 0 0 0 ^M > REGISTER ----------> B-RTD1 0 0 ^M > 200 <---------- E-RTD1 0 0 0 0 ^M > Pause [ 4:50] 0 0 ^M > Pause [$pre_call_delay] 0 0 ^M > INVITE ----------> B-RTD2 0 0 ^M > 100 <---------- 0 0 0 0 ^M > INVITE <---------- 0 0 0 0 ^M > 100 <---------- 0 0 0 0 ^M > INVITE <---------- 0 0 0 0 ^M > 100 ----------> 0 0 ^M > 180 ----------> 0 0 ^M > 180 <---------- 0 0 0 0 ^M > Pause [ 6000ms] 0 0 ^M > 200 ----------> 0 0 ^M > 200 <---------- 0 0 0 0 ^M > ACK ----------> 0 0 ^M > ACK <---------- 0 0 0 0 ^M > UPDATE ----------> 0 0 ^M > UPDATE <---------- 0 0 0 0 ^M > 200 ----------> 0 0 ^M > 200 <---------- E-RTD2 0 0 0 0 ^M > Pause [ 24.0s] 0 0 ^M > BYE ----------> B-RTD3 0 0 ^M > BYE <---------- 0 0 0 0 ^M > 200 ----------> 0 0 ^M > 200 <---------- E-RTD3 0 0 0 0 ^M > Pause [$post_call_delay] 0 0 ^M > ------- Waiting for active calls to end. Press [q] again to force exit. > -------^M > ^M > Last Error: Aborting call on unexpected message for Call-Id '2216-27374@ > ...^M > > Any advice is appreciated. > > Kosala > > > > On Wed, Aug 17, 2016 at 2:40 PM, Matt Williams (projectclearwater.org) < > [email protected]> wrote: > >> Kosala, >> >> >> >> OK, so this says that the subscriber you're trying to register as can't >> be found. >> >> >> >> The stress nodes will by default use DNs from 201000000 onwards. What >> numbers did you provision? >> >> >> >> The docs (https://github.com/Metaswitch/crest/blob/dev/docs/Bulk- >> Provisioning%20Numbers.md) recommend provisioning >> 2010000000-2010099999. However, I recall you trying to run "sudo >> ./BulkProvision homer 21000 29999 example.com 7kkzTyGW", which would >> only provision the numbers 21000-29999. Maybe there's a mismatch here? >> >> >> >> Matt >> >> >> >> -- >> >> >> >> Matt Williams >> Lead Architect, Project Clearwater >> >> +44 (0) 20 8366 1177 >> >> >> >> *From:* Kosala [mailto:[email protected]] >> *Sent:* 17 August 2016 14:34 >> >> *To:* Matt Williams (projectclearwater.org) <[email protected]> >> *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 >> >> >> >> I am getting followings in sip-stress_18438_errors.log: >> >> 2016-08-17 12:26:26:258 1471436786.258684: Aborting call on >> unexpected message for Call-Id '[email protected]': while expecting >> '401' (index 3), received 'SIP/2.0 404 Not Found^M >> Via: SIP/2.0/TCP 10.0.22.193:5060;rport=8826;re >> ceived=10.0.22.193;branch=z9hG4bK-18438-898-2-2010003204-^M >> Record-Route: <sip:scscf.cwaio:5054;transport=TCP;lr;service=scscf; >> billing-role=charge-orig>^M >> Call-ID: 2010003204///[email protected]^M >> From: <sip:[email protected]>;tag=18438SIPpTag00898^M >> To: <sip:[email protected]>;tag=z9hG4bKPjc1xOWCH0ooae6O9npl >> 7fSSrQ-x0vmoV9^M >> CSeq: 1 REGISTER^M >> Content-Length: 0^M >> >> Any advice is appreciated. >> >> Many Thanks, >> >> Kosala >> >> >> >> >> >> On Wed, Aug 17, 2016 at 2:20 PM, Kosala <[email protected]> wrote: >> >> Hi Matt, >> >> I can see this scenario screen from sip-stress.1.out. I hope this will be >> the stress test output results. >> >> >> ------------------------------ Scenario Screen -------- [1-9]: Change >> Screen -- >> Users (length) Port Total-time Total-calls Remote-host >> 2500 (0 ms) 5060 77.15 s 2500 10.0.21.192:5060 >> (TCP) >> >> Call limit reached (-m 2500), 1.002 s period 1 ms scheduler resolution >> 2175 calls (limit 2500) Peak was 2499 calls, after 0 s >> 0 Running, 2177 Paused, 8 Woken up >> 0 dead call msg (discarded) 0 out-of-call msg >> (discarded) >> 3 open sockets >> >> Messages Retrans Timeout >> Unexpected-Msg >> Pause [0ms/10:00] 2500 0 >> REGISTER ----------> 326 0 >> 401 <---------- 0 0 0 325 >> REGISTER ----------> 0 0 >> 200 <---------- 0 0 0 0 >> REGISTER ----------> 0 0 >> 401 <---------- 0 0 0 0 >> REGISTER ----------> 0 0 >> 200 <---------- 0 0 0 0 >> Pause [ 4:50] 0 0 >> REGISTER ----------> B-RTD1 0 0 >> 200 <---------- E-RTD1 0 0 0 0 >> REGISTER ----------> B-RTD1 0 0 >> 200 <---------- E-RTD1 0 0 0 0 >> Pause [ 4:50] 0 0 >> Pause [$pre_call_delay] 0 0 >> INVITE ----------> B-RTD2 0 0 >> 100 <---------- 0 0 0 0 >> INVITE <---------- 0 0 0 0 >> 100 <---------- 0 0 0 0 >> INVITE <---------- 0 0 0 0 >> 100 ----------> 0 0 >> 180 ----------> 0 0 >> 180 <---------- 0 0 0 0 >> Pause [ 6000ms] 0 0 >> 200 ----------> 0 0 >> 200 <---------- 0 0 0 0 >> ACK ----------> 0 0 >> ACK <---------- 0 0 0 0 >> UPDATE ----------> 0 0 >> UPDATE <---------- 0 0 0 0 >> 200 ----------> 0 0 >> 200 <---------- E-RTD2 0 0 0 0 >> Pause [ 24.0s] 0 0 >> BYE ----------> B-RTD3 0 0 >> BYE <---------- 0 0 0 0 >> 200 ----------> 0 0 >> 200 <---------- E-RTD3 0 0 0 0 >> Pause [$post_call_delay] 0 0 >> ------- Waiting for active calls to end. Press [q] again to force exit. >> ------- >> Last Error: Aborting call on unexpected message for Call-Id '1079-13729@ >> ... >> >> However there is an unexpected message error at the end. >> >> Thanks >> >> Kosala >> >> >> >> >> >> On Wed, Aug 17, 2016 at 12:40 PM, Matt Williams (projectclearwater.org) < >> [email protected]> wrote: >> >> Kosala, >> >> >> >> They should log to /var/log/clearwater-sip-stress - in particular, the >> *.1.out file is a log of the output from sipp. Do you see anything there? >> >> >> >> Matt >> >> >> >> *From:* Kosala [mailto:[email protected]] >> *Sent:* 17 August 2016 12:33 >> *To:* Matt Williams (projectclearwater.org) <[email protected]>; >> [email protected] >> >> >> *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 >> >> >> >> Hi Matt, >> >> I followed above steps and run sudo apt-get install >> clearwater-sip-stress. >> >> How do I know stress tests are running? Is is said, stress tests start >> automatically but I cannot see it. >> >> Thanks >> >> Kosala >> >> >> >> >> >> >> On Tue, Aug 16, 2016 at 2:01 PM, Matt Williams (projectclearwater.org) < >> [email protected]> wrote: >> >> Kosala, >> >> >> >> I'm glad you've made progress on this. >> >> >> >> I had a look at the crest package version, but it seems to be later than >> the most recent all-in-one image uses (1.0-160811.103635). When did you >> download the all-in-one image, and did you apply any patches to it? >> >> >> >> Regarding making calls for stress testing, this is where >> http://clearwater.readthedocs.io/en/latest/Clearwater_stress >> _testing.html#manual-i-e-non-chef-stress-runs comes in - you need to >> >> · create a separate VM >> >> · set the bono_hostname and count parameters in >> /etc/clearwater/shared_config >> >> · install the clearwater-sip-stress package. >> >> I hope that helps. >> >> >> >> BTW, please remember to copy in [email protected] - >> that means that other people can see and comment on your posts. >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >> >> >> *From:* Kosala [mailto:[email protected]] >> *Sent:* 16 August 2016 10:48 >> *To:* Matt Williams (projectclearwater.org) <[email protected]> >> >> *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 >> >> >> >> Hi Matt, >> >> Bulk provisioning for all in one image is done using >> https://github.com/Metaswitch/crest/blob/dev/docs/Bulk-Provi >> sioning%20Numbers.md. >> >> I have records in tables in homestead_cache and homer. >> >> Now How to make calls for stress testing? >> >> Is there any guideline? >> >> Many Thanks, >> >> Kosala >> >> >> >> >> >> >> >> >> >> On Tue, Aug 16, 2016 at 9:04 AM, Kosala <[email protected]> wrote: >> >> Hi Matt, >> >> TBH, The document in https://github.com/Metaswitch/ >> crest/blob/dev/docs/Bulk-Provisioning%20Numbers.md is not clear to me. >> >> Step 1. Homestead node in deployment. I do not know where is that node in >> all in one image. It will be helpful the location of that folder from the >> root. >> >> Step 2. A simple example will be helpful. >> >> Thanks >> >> Kosala >> >> >> >> >> >> On Tue, Aug 16, 2016 at 8:50 AM, Kosala <[email protected]> wrote: >> >> Hi Matt, >> >> I downloaded the all in one package from http://clearwater.readthedocs. >> io/en/stable/All_in_one_OVF_Installation.html >> >> and imported using oracle virtual box on ubuntu 14.04, >> >> Crest version >> >> [cw-aio]ubuntu@cwaio:~$ sudo dpkg-query -s crest >> Package: crest >> Status: install ok installed >> Priority: optional >> Section: network >> Installed-Size: 40098 >> Maintainer: Project Clearwater Maintainers <maintainers@projectclearwater >> .org> >> Architecture: amd64 >> Version: 1.0-160815.145007 >> >> Thanks >> >> Kosala >> >> >> >> >> >> On Mon, Aug 15, 2016 at 5:56 PM, Matt Williams (projectclearwater.org) < >> [email protected]> wrote: >> >> Kosala, >> >> >> >> That's odd - I don't see that failure on my all-in-one image (spun up >> just now from the latest image) - which version of all-in-one-image are you >> running, and which version of crest does it include ("sudo dpkg-query -s >> crest")? >> >> >> >> You may be able to work-around this either by taking the latest >> all-in-one image or by following the simpler bulk-provisioning process at >> https://github.com/Metaswitch/crest/blob/dev/docs/Bulk-Provi >> sioning%20Numbers.md (when this doc talks about using the >> sstables_provisioning process for very large sets of numbers, we mean 1M+ >> numbers - the 9000 you're provisioning are well within the capabilities of >> this process). >> >> >> >> I hope that helps - please let me know how you get on. >> >> >> >> Cheers, >> >> >> >> Matt >> >> >> >> >> >> *From:* Clearwater [mailto:[email protected]] >> *On Behalf Of *Kosala >> *Sent:* 15 August 2016 16:58 >> *To:* [email protected] >> *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 >> >> >> >> HI Matt, >> >> Thanks. >> >> >> I am trying to follow https://github.com/Metaswitch/ >> crest/blob/dev/src/metaswitch/crest/tools/sstable_provisioning/README.md >> for bulk provisioning. >> >> >> When I am trying to execute sudo ./BulkProvision homer 21000 29999 >> example.com 7kkzTyGW in all-in-one image, >> I am getting an error java.lang.UnsupportedClassVersionError: >> ClearwaterBulkProvisioner : Unsupported major.minor version 52.0 >> >> Java version info in all-in-one image: >> [cw-aio]ubuntu@cwaio:/usr/share/clearwater/crest/tools/sstable_provisioning$ >> java -version >> java version "1.7.0_101" >> OpenJDK Runtime Environment (IcedTea 2.6.6) (7u101-2.6.6-0ubuntu0.14.04.1) >> OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) >> >> Thanks >> >> Kosala >> >> >> >> >> >> >> On Mon, Aug 15, 2016 at 2:17 PM, <[email protected] >> ectclearwater.org> wrote: >> >> Send Clearwater mailing list submissions to >> [email protected] >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://lists.projectclearwater.org/mailman/listinfo/ >> clearwater_lists.projectclearwater.org >> >> or, via email, send a message with subject or body 'help' to >> [email protected] >> >> You can reach the person managing the list at >> [email protected] >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Clearwater digest..." >> >> >> Today's Topics: >> >> 1. Stress Testing (Kosala) >> 2. Re: Stress Testing (Matt Williams (projectclearwater.org)) >> 3. Re: Stress Testing (Matt Williams (projectclearwater.org)) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Mon, 15 Aug 2016 10:38:57 +0100 >> From: Kosala <[email protected]> >> To: [email protected] >> Subject: [Project Clearwater] Stress Testing >> Message-ID: >> <[email protected] >> ail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Hi Team, >> >> I installed clearwater "all-in-one" image on Oracle Virtual Box. >> 1. Can I do stress testing ? >> 2. How to do that? >> >> I am trying to follow Stress Testing on >> http://clearwater.readthedocs.io/en/stable/Clearwater_stress >> _testing.html#configuring-udp-stress >> . >> >> Using chef - > It seems chef/knife install only supports in the Cloud >> approach (The automated install is only supported for deployments running >> in Amazon?s EC2 cloud). >> >> Manual approach, It looks not supporting all in one image. >> >> Many Thanks >> Kosala >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: <http://lists.projectclearwater.org/pipermail/clearwater_ >> lists.projectclearwater.org/attachments/20160815/15fc3b07/at >> tachment-0001.html> >> >> ------------------------------ >> >> Message: 2 >> Date: Mon, 15 Aug 2016 09:54:02 +0000 >> From: "Matt Williams (projectclearwater.org)" >> <[email protected]> >> To: Kosala <[email protected]> >> Cc: "[email protected]" >> <[email protected]> >> Subject: Re: [Project Clearwater] Stress Testing >> Message-ID: >> <[email protected] >> mprd02.prod.outlook.com> >> >> Content-Type: text/plain; charset="utf-8" >> >> Kosala, >> >> While it's theoretically possible to do stress testing against an >> all-in-one image, I'd recommend against it - the all-in-one image has been >> squeezed down a lot to fit on a single VM and we'd expect very low >> performance out of it, and I could believe that overload behavior might be >> suboptimal. If you're looking to stress test for performance reasons, I'd >> strongly recommend running a full deployment as this will give you much >> more reliable numbers. If you're looking to stress test for other reasons, >> please can you share a bit more about what you're trying to achieve, and we >> can have a think about what would work best? >> >> If you do really want to run stress testing against an all-in-one image, >> it should probably work, even though http://clearwater.readthedocs. >> io/en/latest/Clearwater_stress_testing.html#manual-i-e-non- >> chef-stress-runs only talks about Chef and manual installs. You'd need >> to set the "bono_servers" option in /etc/clearwater/shared_config to point >> to your all-in-one node, and I suspect you'd need to reduce the "count" >> option to something a lot lower - I'd start around 5000 and see how that >> goes. >> >> I hope that helps. Please let me know how you get on. >> >> Thanks, >> >> Matt >> >> From: Clearwater [mailto:[email protected]] >> On Behalf Of Kosala >> Sent: 15 August 2016 10:39 >> To: [email protected] >> Subject: [Project Clearwater] Stress Testing >> >> Hi Team, >> I installed clearwater "all-in-one" image on Oracle Virtual Box. >> 1. Can I do stress testing ? >> 2. How to do that? >> I am trying to follow Stress Testing on http://clearwater.readthedocs. >> io/en/stable/Clearwater_stress_testing.html#configuring-udp-stress. >> Using chef - > It seems chef/knife install only supports in the Cloud >> approach (The automated install is only supported for deployments running >> in Amazon?s EC2 cloud). >> Manual approach, It looks not supporting all in one image. >> Many Thanks >> Kosala >> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: <http://lists.projectclearwater.org/pipermail/clearwater_ >> lists.projectclearwater.org/attachments/20160815/29d6328d/at >> tachment-0001.html> >> >> ------------------------------ >> >> Message: 3 >> Date: Mon, 15 Aug 2016 13:16:47 +0000 >> From: "Matt Williams (projectclearwater.org)" >> <[email protected]> >> To: "[email protected]" >> <[email protected]> >> Subject: Re: [Project Clearwater] Stress Testing >> Message-ID: >> <[email protected] >> mprd02.prod.outlook.com> >> >> Content-Type: text/plain; charset="utf-8" >> >> Kosala, >> >> You'll need to follow all the steps in http://clearwater.readthedocs. >> io/en/latest/Clearwater_stress_testing.html#manual-i-e-non- >> chef-stress-runs, i.e. bulk provisioning subscribers and installing >> clearwater-sip-stress, but that should be it. >> >> Just to be clear, though, you'll need to set the configuration and >> install the clearwater-sip-stress package on a separate VM, not on the >> all-in-one image itself - the all-in-one image almost certainly opens up >> ports that conflict with those that clearwater-sip-stress tries to use. >> >> This isn't a very well-trodden path, though, so you may hit further >> issues - let me know if so, and we can try to work through them. >> >> Cheers, >> >> Matt >> >> From: Kosala [mailto:[email protected]] >> Sent: 15 August 2016 13:41 >> To: Matt Williams (projectclearwater.org) <[email protected]> >> Subject: Re: [Project Clearwater] Stress Testing >> >> Hi Matt, >> Thanks. I set the bono_servers option. Should I add count =5000 to >> /etc/clearwater/shared_config ? >> >> Do I need other changes to run stress tests on all-in-one image? >> Thanks >> Kosala >> >> >> >> On Mon, Aug 15, 2016 at 10:54 AM, Matt Williams (projectclearwater.org< >> http://projectclearwater.org>) <[email protected]<mailto: >> [email protected]>> wrote: >> Kosala, >> >> While it's theoretically possible to do stress testing against an >> all-in-one image, I'd recommend against it - the all-in-one image has been >> squeezed down a lot to fit on a single VM and we'd expect very low >> performance out of it, and I could believe that overload behavior might be >> suboptimal. If you're looking to stress test for performance reasons, I'd >> strongly recommend running a full deployment as this will give you much >> more reliable numbers. If you're looking to stress test for other reasons, >> please can you share a bit more about what you're trying to achieve, and we >> can have a think about what would work best? >> >> If you do really want to run stress testing against an all-in-one image, >> it should probably work, even though http://clearwater.readthedocs. >> io/en/latest/Clearwater_stress_testing.html#manual-i-e-non- >> chef-stress-runs only talks about Chef and manual installs. You'd need >> to set the "bono_servers" option in /etc/clearwater/shared_config to point >> to your all-in-one node, and I suspect you'd need to reduce the "count" >> option to something a lot lower - I'd start around 5000 and see how that >> goes. >> >> I hope that helps. Please let me know how you get on. >> >> Thanks, >> >> Matt >> >> From: Clearwater [mailto:[email protected] >> <mailto:[email protected]>] On Behalf Of >> Kosala >> Sent: 15 August 2016 10:39 >> To: [email protected]<mailto:clearwater@ >> lists.projectclearwater.org> >> Subject: [Project Clearwater] Stress Testing >> >> Hi Team, >> I installed clearwater "all-in-one" image on Oracle Virtual Box. >> 1. Can I do stress testing ? >> 2. How to do that? >> I am trying to follow Stress Testing on http://clearwater.readthedocs. >> io/en/stable/Clearwater_stress_testing.html#configuring-udp-stress. >> Using chef - > It seems chef/knife install only supports in the Cloud >> approach (The automated install is only supported for deployments running >> in Amazon?s EC2 cloud). >> Manual approach, It looks not supporting all in one image. >> Many Thanks >> Kosala >> >> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: <http://lists.projectclearwater.org/pipermail/clearwater_ >> lists.projectclearwater.org/attachments/20160815/11781755/attachment.html >> > >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> Clearwater mailing list >> [email protected] >> http://lists.projectclearwater.org/mailman/listinfo/ >> clearwater_lists.projectclearwater.org >> >> >> ------------------------------ >> >> End of Clearwater Digest, Vol 40, Issue 24 >> ****************************************** >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
