Hi Matt, Now for the numbers I have in Cassandra, at SP node, I am getting "SIP/2.0 504 Server Timeout"
2016-08-18 11:49:03:189 1471520943.189039: Aborting call on unexpected message for Call-Id '[email protected]': while expecting '401' (index 3), received 'SIP/2.0 504 Server Timeout^M Via: SIP/2.0/TCP 10.0.22.193:5060 ;rport=32939;received=10.0.22.193;branch=z9hG4bK-27637-1073-2-2010000854-^M Record-Route: <sip:scscf.cwaio:5054;transport=TCP;lr;service=scscf;billing-role=charge-orig>^M Call-ID: 2010000854///[email protected]^M From: <sip:[email protected]>;tag=27637SIPpTag001073^M To: <sip:[email protected] >;tag=z9hG4bKPjuN16eRA0bDott9z6ZA-fwIYe3VA-KMkQ^M CSeq: 1 REGISTER^M Content-Length: 0^M ^M '. 2016-08-18 11:49:04:185 1471520944.185945: Aborting call on unexpected message for Call-Id '[email protected]': while expecting '401' (index 3), received 'SIP/2.0 504 Server Timeout^M Via: SIP/2.0/TCP 10.0.22.193:5060 ;rport=32940;received=10.0.22.193;branch=z9hG4bK-27637-1154-2-2010000692-^M Record-Route: <sip:scscf.cwaio:5054;transport=TCP;lr;service=scscf;billing-role=charge-orig>^M Call-ID: 2010000692///[email protected]^M From: <sip:[email protected]>;tag=27637SIPpTag001154^M To: <sip:[email protected] >;tag=z9hG4bKPjk-5lYhadbnX7rqNirT8tjwQdDjLAQo0j^M CSeq: 1 REGISTER^M Content-Length: 0^M How do I know, stress tests are running properly? Can I see statistics such as memory usage, cpu time, etc ? Thanks Kosala On Thu, Aug 18, 2016 at 12:38 PM, Matt Williams (projectclearwater.org) < [email protected]> wrote: > Kosala, > > > > Thanks! I'm very confused about your software version, though, as the > latest all-in-one image has > > · crest 1.0-160811.103635 (i.e. older than yours) > > · homestead 1.0-160814.205535 (i.e. slightly newer than yours) > > · sprout 1.0-160814.205701 (i.e. much newer than yours) > > · the logging.bash script fix already applied. > > It looks as though your might have taken the previous all-in-one image, > and then upgraded some of the packages. Please can you confirm: have you > upgraded any packages? > > > > On the issue you're seeing, I think you're saying that > > · you're running clearwater-sip-stress (which defaults to using > numbers in the range above 2010000000) > > · you've deleted all these numbers from Homestead's Cassandra > > · you see errors in clearwater-sip-stress's log file saying that > registration are failing with "404 Not Found". > > I think this is expected - you need to provision numbers in Homestead's > Cassandra that match the numbers that clearwater-sip-stress is expecting to > register as. Please let me know if I've misunderstood. > > > > Thanks, > > > > Matt > > > > *From:* Kosala [mailto:[email protected]] > *Sent:* 18 August 2016 11:46 > > *To:* Matt Williams (projectclearwater.org) <[email protected]> > *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 > > > > > > Hi Matt, > > I am running the latest version all-in-one which i downloaded from > http://vm-images.cw-ngv.com/cw-aio.ova. > > [cw-aio]ubuntu@cwaio:~$ sudo dpkg-query -s crest > [sudo] password for ubuntu: > Package: crest > Status: install ok installed > Priority: optional > Section: network > Installed-Size: 40102 > Maintainer: Project Clearwater Maintainers <maintainers@ > projectclearwater.org> > Architecture: amd64 > Version: 1.0-160818.091648 > > homestead > Version: 1.0-160812.164051 > > sprout > Version: 1.0-160726.101022 > > And one strange thing is: > > I deleted all the records in tables from homestead_cache and > homestead_provisioning. Then when I check the SIP node for logs, > sip-stress_14320_errors.log still has records with sip: ids which are not > in Cassandra. For example, sip:[email protected] > > 2016-08-18 10:26:08:288 1471515968.288189: Aborting call on > unexpected message for Call-Id '[email protected]': while expecting > '401' (index 3), received 'SIP/2.0 404 Not Found^M > Via: SIP/2.0/TCP 10.0.22.193:5060;rport=35233;received=10.0.22.193;branch= > z9hG4bK-14320-4270-2-2010021460-^M > Record-Route: <sip:scscf.cwaio:5054;transport=TCP;lr;service= > scscf;billing-role=charge-orig>^M > Call-ID: 2010021460///[email protected]^M > From: <sip:[email protected]>;tag=14320SIPpTag004270^M > To: <sip:[email protected]>;tag=z9hG4bKPjVe2he9j28lou. > 0Dl9NmKGS7ypUMLjMJv^M > CSeq: 1 REGISTER^M > Content-Length: 0^M > > Thanks > > Kosala > > > > On Thu, Aug 18, 2016 at 10:47 AM, Matt Williams (projectclearwater.org) < > [email protected]> wrote: > > Kosala, > > > > The fix is not to *copy* this to /usr/share/clearwater/utils, but to > *move* it. However, since the error is benign, I'd suggest just ignoring > it while we get this working. > > > > As I asked below, please can you confirm what version you're running? I'm > concerned that it's going to be very hard to troubleshoot problems if you > have an all-in-one image from a few weeks ago, with some packages updated > to more recent versions and then some further fixes manually applied. > > > > Thanks, > > > > Matt > > > > > > *From:* Kosala [mailto:[email protected]] > *Sent:* 18 August 2016 10:34 > > > *To:* Matt Williams (projectclearwater.org) <[email protected]> > *Cc:* [email protected] > *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 > > > > Hi Matt, > > Yeah, Manual calls are working fine. > > As it is said in the big fix, https://github.com/Metaswitch/ > clearwater-infrastructure/issues/355 , I copied logging.bash to > /usr/share/clearwater/utils > > but I am still getting > > /etc/init.d/clearwater-infrastructure: 76: > /etc/init.d/clearwater-infrastructure: > /usr/share/clearwater/infrastructure/scripts/logging.bash: Permission > denied > Configuring monit for only localhost access > > Kosala > > > > > > On Thu, Aug 18, 2016 at 10:01 AM, Matt Williams (projectclearwater.org) < > [email protected]> wrote: > > Kosala, > > > > Yes, it's https://github.com/Metaswitch/clearwater-infrastructure/ > issues/355 and it should be benign. > > > > However, this was fixed about 2 weeks ago. Please can you confirm when > you downloaded the all-in-one image, and whether you've upgraded it all? > (As I observed earlier, your crest release was from only a few days ago.) > > > > Have you got manual calls working? > > > > Thanks, > > > > Matt > > > > > > *From:* Kosala [mailto:[email protected]] > *Sent:* 18 August 2016 09:56 > *To:* Matt Williams (projectclearwater.org) <[email protected]> > *Cc:* [email protected] > > > *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 > > > > Hi Matt, > > In SIP node, when I am trying to execute "sudo service > clearwater-sip-stress restart", I am getting an error > > > /etc/init.d/clearwater-infrastructure: 76: > /etc/init.d/clearwater-infrastructure: > /usr/share/clearwater/infrastructure/scripts/logging.bash: Permission > denied > Configuring monit for only localhost access > > Any clue about this error? > > Thanks > > Kosala > > > > > > On Wed, Aug 17, 2016 at 2:29 PM, Matt Williams (projectclearwater.org) < > [email protected]> wrote: > > Kosala, > > > > Yes, that's the correct screen, and you're correct that it shows that > stress is being attempted but failing. > > > > Do regular (non-stress) calls work through this deployment? > > > > You can look at what message the stress node is receiving in the log files > in the /var/log/clearwater-sip-stress/ directory - I think there are > "errors" logs that contain the unexpected messages. > > > > You can obviously also look at what's happening on the all-in-one node > itself, using the /var/log/bono/*, /var/log/sprout/* and > /var/log/homestead/* files, as described at http://clearwater.readthedocs. > io/en/latest/Troubleshooting_and_Recovery.html. > > > > I hope that helps. > > > > Cheers, > > > > Matt > > > > *From:* Kosala [mailto:[email protected]] > *Sent:* 17 August 2016 14:20 > > > *To:* Matt Williams (projectclearwater.org) <[email protected]> > *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 > > > > Hi Matt, > > I can see this scenario screen from sip-stress.1.out. I hope this will be > the stress test output results. > > > ------------------------------ Scenario Screen -------- [1-9]: Change > Screen -- > Users (length) Port Total-time Total-calls Remote-host > 2500 (0 ms) 5060 77.15 s 2500 10.0.21.192:5060 > (TCP) > > Call limit reached (-m 2500), 1.002 s period 1 ms scheduler resolution > 2175 calls (limit 2500) Peak was 2499 calls, after 0 s > 0 Running, 2177 Paused, 8 Woken up > 0 dead call msg (discarded) 0 out-of-call msg > (discarded) > 3 open sockets > > Messages Retrans Timeout > Unexpected-Msg > Pause [0ms/10:00] 2500 0 > REGISTER ----------> 326 0 > 401 <---------- 0 0 0 325 > REGISTER ----------> 0 0 > 200 <---------- 0 0 0 0 > REGISTER ----------> 0 0 > 401 <---------- 0 0 0 0 > REGISTER ----------> 0 0 > 200 <---------- 0 0 0 0 > Pause [ 4:50] 0 0 > REGISTER ----------> B-RTD1 0 0 > 200 <---------- E-RTD1 0 0 0 0 > REGISTER ----------> B-RTD1 0 0 > 200 <---------- E-RTD1 0 0 0 0 > Pause [ 4:50] 0 0 > Pause [$pre_call_delay] 0 0 > INVITE ----------> B-RTD2 0 0 > 100 <---------- 0 0 0 0 > INVITE <---------- 0 0 0 0 > 100 <---------- 0 0 0 0 > INVITE <---------- 0 0 0 0 > 100 ----------> 0 0 > 180 ----------> 0 0 > 180 <---------- 0 0 0 0 > Pause [ 6000ms] 0 0 > 200 ----------> 0 0 > 200 <---------- 0 0 0 0 > ACK ----------> 0 0 > ACK <---------- 0 0 0 0 > UPDATE ----------> 0 0 > UPDATE <---------- 0 0 0 0 > 200 ----------> 0 0 > 200 <---------- E-RTD2 0 0 0 0 > Pause [ 24.0s] 0 0 > BYE ----------> B-RTD3 0 0 > BYE <---------- 0 0 0 0 > 200 ----------> 0 0 > 200 <---------- E-RTD3 0 0 0 0 > Pause [$post_call_delay] 0 0 > ------- Waiting for active calls to end. Press [q] again to force exit. > ------- > Last Error: Aborting call on unexpected message for Call-Id '1079-13729@ > ... > > However there is an unexpected message error at the end. > > Thanks > > Kosala > > > > > > On Wed, Aug 17, 2016 at 12:40 PM, Matt Williams (projectclearwater.org) < > [email protected]> wrote: > > Kosala, > > > > They should log to /var/log/clearwater-sip-stress - in particular, the > *.1.out file is a log of the output from sipp. Do you see anything there? > > > > Matt > > > > *From:* Kosala [mailto:[email protected]] > *Sent:* 17 August 2016 12:33 > *To:* Matt Williams (projectclearwater.org) <[email protected]>; > [email protected] > > > *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 > > > > Hi Matt, > > I followed above steps and run sudo apt-get install clearwater-sip-stress. > > How do I know stress tests are running? Is is said, stress tests start > automatically but I cannot see it. > > Thanks > > Kosala > > > > > > > On Tue, Aug 16, 2016 at 2:01 PM, Matt Williams (projectclearwater.org) < > [email protected]> wrote: > > Kosala, > > > > I'm glad you've made progress on this. > > > > I had a look at the crest package version, but it seems to be later than > the most recent all-in-one image uses (1.0-160811.103635). When did you > download the all-in-one image, and did you apply any patches to it? > > > > Regarding making calls for stress testing, this is where > http://clearwater.readthedocs.io/en/latest/Clearwater_ > stress_testing.html#manual-i-e-non-chef-stress-runs comes in - you need to > > · create a separate VM > > · set the bono_hostname and count parameters in > /etc/clearwater/shared_config > > · install the clearwater-sip-stress package. > > I hope that helps. > > > > BTW, please remember to copy in [email protected] - > that means that other people can see and comment on your posts. > > > > Thanks, > > > > Matt > > > > > > *From:* Kosala [mailto:[email protected]] > *Sent:* 16 August 2016 10:48 > *To:* Matt Williams (projectclearwater.org) <[email protected]> > > *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 > > > > Hi Matt, > > Bulk provisioning for all in one image is done using > https://github.com/Metaswitch/crest/blob/dev/docs/Bulk- > Provisioning%20Numbers.md. > > I have records in tables in homestead_cache and homer. > > Now How to make calls for stress testing? > > Is there any guideline? > > Many Thanks, > > Kosala > > > > > > > > > > On Tue, Aug 16, 2016 at 9:04 AM, Kosala <[email protected]> wrote: > > Hi Matt, > > TBH, The document in https://github.com/Metaswitch/ > crest/blob/dev/docs/Bulk-Provisioning%20Numbers.md is not clear to me. > > Step 1. Homestead node in deployment. I do not know where is that node in > all in one image. It will be helpful the location of that folder from the > root. > > Step 2. A simple example will be helpful. > > Thanks > > Kosala > > > > > > On Tue, Aug 16, 2016 at 8:50 AM, Kosala <[email protected]> wrote: > > Hi Matt, > > I downloaded the all in one package from http://clearwater.readthedocs. > io/en/stable/All_in_one_OVF_Installation.html > > and imported using oracle virtual box on ubuntu 14.04, > > Crest version > > [cw-aio]ubuntu@cwaio:~$ sudo dpkg-query -s crest > Package: crest > Status: install ok installed > Priority: optional > Section: network > Installed-Size: 40098 > Maintainer: Project Clearwater Maintainers <maintainers@ > projectclearwater.org> > Architecture: amd64 > Version: 1.0-160815.145007 > > Thanks > > Kosala > > > > > > On Mon, Aug 15, 2016 at 5:56 PM, Matt Williams (projectclearwater.org) < > [email protected]> wrote: > > Kosala, > > > > That's odd - I don't see that failure on my all-in-one image (spun up just > now from the latest image) - which version of all-in-one-image are you > running, and which version of crest does it include ("sudo dpkg-query -s > crest")? > > > > You may be able to work-around this either by taking the latest all-in-one > image or by following the simpler bulk-provisioning process at > https://github.com/Metaswitch/crest/blob/dev/docs/Bulk- > Provisioning%20Numbers.md (when this doc talks about using the > sstables_provisioning process for very large sets of numbers, we mean 1M+ > numbers - the 9000 you're provisioning are well within the capabilities of > this process). > > > > I hope that helps - please let me know how you get on. > > > > Cheers, > > > > Matt > > > > > > *From:* Clearwater [mailto:[email protected]] > *On Behalf Of *Kosala > *Sent:* 15 August 2016 16:58 > *To:* [email protected] > *Subject:* Re: [Project Clearwater] Clearwater Digest, Vol 40, Issue 24 > > > > HI Matt, > > Thanks. > > > I am trying to follow https://github.com/Metaswitch/ > crest/blob/dev/src/metaswitch/crest/tools/sstable_provisioning/README.md > for bulk provisioning. > > > When I am trying to execute sudo ./BulkProvision homer 21000 29999 > example.com 7kkzTyGW in all-in-one image, > I am getting an error java.lang.UnsupportedClassVersionError: > ClearwaterBulkProvisioner : Unsupported major.minor version 52.0 > > Java version info in all-in-one image: > [cw-aio]ubuntu@cwaio:/usr/share/clearwater/crest/tools/sstable_provisioning$ > java -version > java version "1.7.0_101" > OpenJDK Runtime Environment (IcedTea 2.6.6) (7u101-2.6.6-0ubuntu0.14.04.1) > OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) > > Thanks > > Kosala > > > > > > > On Mon, Aug 15, 2016 at 2:17 PM, <clearwater-request@lists. > projectclearwater.org> wrote: > > Send Clearwater mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.projectclearwater.org/mailman/ > listinfo/clearwater_lists.projectclearwater.org > > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Clearwater digest..." > > > Today's Topics: > > 1. Stress Testing (Kosala) > 2. Re: Stress Testing (Matt Williams (projectclearwater.org)) > 3. Re: Stress Testing (Matt Williams (projectclearwater.org)) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 15 Aug 2016 10:38:57 +0100 > From: Kosala <[email protected]> > To: [email protected] > Subject: [Project Clearwater] Stress Testing > Message-ID: > <CAGEe17an2B_bGj8NfKFwms+nU=vJURyhT5CE+mcY1pjcm4YF+A@mail. > gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Team, > > I installed clearwater "all-in-one" image on Oracle Virtual Box. > 1. Can I do stress testing ? > 2. How to do that? > > I am trying to follow Stress Testing on > http://clearwater.readthedocs.io/en/stable/Clearwater_stress_testing.html# > configuring-udp-stress > . > > Using chef - > It seems chef/knife install only supports in the Cloud > approach (The automated install is only supported for deployments running > in Amazon?s EC2 cloud). > > Manual approach, It looks not supporting all in one image. > > Many Thanks > Kosala > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: <http://lists.projectclearwater.org/pipermail/clearwater_lists. > projectclearwater.org/attachments/20160815/15fc3b07/attachment-0001.html> > > ------------------------------ > > Message: 2 > Date: Mon, 15 Aug 2016 09:54:02 +0000 > From: "Matt Williams (projectclearwater.org)" > <[email protected]> > To: Kosala <[email protected]> > Cc: "[email protected]" > <[email protected]> > Subject: Re: [Project Clearwater] Stress Testing > Message-ID: > <CY1PR0201MB15636F1F20EB91BB180F053390120@CY1PR0201MB1563. > namprd02.prod.outlook.com> > > Content-Type: text/plain; charset="utf-8" > > Kosala, > > While it's theoretically possible to do stress testing against an > all-in-one image, I'd recommend against it - the all-in-one image has been > squeezed down a lot to fit on a single VM and we'd expect very low > performance out of it, and I could believe that overload behavior might be > suboptimal. If you're looking to stress test for performance reasons, I'd > strongly recommend running a full deployment as this will give you much > more reliable numbers. If you're looking to stress test for other reasons, > please can you share a bit more about what you're trying to achieve, and we > can have a think about what would work best? > > If you do really want to run stress testing against an all-in-one image, > it should probably work, even though http://clearwater.readthedocs. > io/en/latest/Clearwater_stress_testing.html#manual-i- > e-non-chef-stress-runs only talks about Chef and manual installs. You'd > need to set the "bono_servers" option in /etc/clearwater/shared_config to > point to your all-in-one node, and I suspect you'd need to reduce the > "count" option to something a lot lower - I'd start around 5000 and see how > that goes. > > I hope that helps. Please let me know how you get on. > > Thanks, > > Matt > > From: Clearwater [mailto:[email protected]] > On Behalf Of Kosala > Sent: 15 August 2016 10:39 > To: [email protected] > Subject: [Project Clearwater] Stress Testing > > Hi Team, > I installed clearwater "all-in-one" image on Oracle Virtual Box. > 1. Can I do stress testing ? > 2. How to do that? > I am trying to follow Stress Testing on http://clearwater.readthedocs. > io/en/stable/Clearwater_stress_testing.html#configuring-udp-stress. > Using chef - > It seems chef/knife install only supports in the Cloud > approach (The automated install is only supported for deployments running > in Amazon?s EC2 cloud). > Manual approach, It looks not supporting all in one image. > Many Thanks > Kosala > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: <http://lists.projectclearwater.org/pipermail/clearwater_lists. > projectclearwater.org/attachments/20160815/29d6328d/attachment-0001.html> > > ------------------------------ > > Message: 3 > Date: Mon, 15 Aug 2016 13:16:47 +0000 > From: "Matt Williams (projectclearwater.org)" > <[email protected]> > To: "[email protected]" > <[email protected]> > Subject: Re: [Project Clearwater] Stress Testing > Message-ID: > <CY1PR0201MB156372246DC5A77B11839F9D90120@CY1PR0201MB1563. > namprd02.prod.outlook.com> > > Content-Type: text/plain; charset="utf-8" > > Kosala, > > You'll need to follow all the steps in http://clearwater.readthedocs. > io/en/latest/Clearwater_stress_testing.html#manual-i- > e-non-chef-stress-runs, i.e. bulk provisioning subscribers and installing > clearwater-sip-stress, but that should be it. > > Just to be clear, though, you'll need to set the configuration and install > the clearwater-sip-stress package on a separate VM, not on the all-in-one > image itself - the all-in-one image almost certainly opens up ports that > conflict with those that clearwater-sip-stress tries to use. > > This isn't a very well-trodden path, though, so you may hit further issues > - let me know if so, and we can try to work through them. > > Cheers, > > Matt > > From: Kosala [mailto:[email protected]] > Sent: 15 August 2016 13:41 > To: Matt Williams (projectclearwater.org) <[email protected]> > Subject: Re: [Project Clearwater] Stress Testing > > Hi Matt, > Thanks. I set the bono_servers option. Should I add count =5000 to > /etc/clearwater/shared_config ? > > Do I need other changes to run stress tests on all-in-one image? > Thanks > Kosala > > > > On Mon, Aug 15, 2016 at 10:54 AM, Matt Williams (projectclearwater.org< > http://projectclearwater.org>) <[email protected]<mailto: > [email protected]>> wrote: > Kosala, > > While it's theoretically possible to do stress testing against an > all-in-one image, I'd recommend against it - the all-in-one image has been > squeezed down a lot to fit on a single VM and we'd expect very low > performance out of it, and I could believe that overload behavior might be > suboptimal. If you're looking to stress test for performance reasons, I'd > strongly recommend running a full deployment as this will give you much > more reliable numbers. If you're looking to stress test for other reasons, > please can you share a bit more about what you're trying to achieve, and we > can have a think about what would work best? > > If you do really want to run stress testing against an all-in-one image, > it should probably work, even though http://clearwater.readthedocs. > io/en/latest/Clearwater_stress_testing.html#manual-i- > e-non-chef-stress-runs only talks about Chef and manual installs. You'd > need to set the "bono_servers" option in /etc/clearwater/shared_config to > point to your all-in-one node, and I suspect you'd need to reduce the > "count" option to something a lot lower - I'd start around 5000 and see how > that goes. > > I hope that helps. Please let me know how you get on. > > Thanks, > > Matt > > From: Clearwater [mailto:[email protected]< > mailto:[email protected]>] On Behalf Of > Kosala > Sent: 15 August 2016 10:39 > To: [email protected]<mailto:clearwater@lists. > projectclearwater.org> > Subject: [Project Clearwater] Stress Testing > > Hi Team, > I installed clearwater "all-in-one" image on Oracle Virtual Box. > 1. Can I do stress testing ? > 2. How to do that? > I am trying to follow Stress Testing on http://clearwater.readthedocs. > io/en/stable/Clearwater_stress_testing.html#configuring-udp-stress. > Using chef - > It seems chef/knife install only supports in the Cloud > approach (The automated install is only supported for deployments running > in Amazon?s EC2 cloud). > Manual approach, It looks not supporting all in one image. > Many Thanks > Kosala > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: <http://lists.projectclearwater.org/pipermail/clearwater_lists. > projectclearwater.org/attachments/20160815/11781755/attachment.html> > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Clearwater mailing list > [email protected] > http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists. > projectclearwater.org > > > ------------------------------ > > End of Clearwater Digest, Vol 40, Issue 24 > ****************************************** > > > > > > > > > > > > > > > > > > >
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
