Hi Ellie, We tried several register attempts and each failed with "failed to get Authentication vector" error on sprout.
We observed some IP conflict in /etc/clearwater/cluster_settings. Resolving that, memcached log seems to be ok, please verify attached log. But the problem with register still persists. tcpdump at homestead shows that the sip message is recieved from sprout node but /var/log/homestead logs doesnt show up anything. To verify connectivity between homestead and sprout, we confirmed the consistency of node names in config file on all the nodes and ping works between all nodes. We are attaching detailed log of homestead (along with tcpdump capture), sprout and memcached. Thanks, Prerna Pandit Tata Consultancy Services Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Consulting ____________________________________________ -----Eleanor Merry <[email protected]> wrote: ----- To: Prerna Pandit <[email protected]> From: Eleanor Merry <[email protected]> Date: 03/13/2015 11:18PM Cc: "[email protected]" <[email protected]>, Pundalik K <[email protected]> Subject: RE: [Clearwater] Fw: Trouble installing Ellis(Manual Install) Hi Prerna, Is this just from the one register attempt? It looks like memcached isn?t running on the Sprout node. Can you check this and check the memcached logs in /var/log/memcached.log or in /var/log/syslog? If this doesn?t help then can you please send me the debug logs from the Sprout node. You can turn on debug logging by creating/editing the file /etc/clearwater/user_settings, adding log_level=5 and then restarting Sprout (run ?service sprout stop? ? it?s automatically restarted by monit). Ellie From: Prerna Pandit [mailto:[email protected]] Sent: 13 March 2015 11:27 To: Eleanor Merry Cc: [email protected]; Pundalik K Subject: RE: [Clearwater] Fw: Trouble installing Ellis(Manual Install) Hi Ellie, As per your suggestion, since we do not have any CDF in our setup, we disabled Ralf by commenting corresponding entries from /etc/clearwater/config and /etc/bind/db.example.com. Post this , we tried bringing up the ellis GUI, which was successful. Also we added auth username of the subscriber as suggested, but on registering , it seems to go into some endless loop. Observed some errors on Sprout saying - 13-03-2015 11:09:58.668 UTC Warning authentication.cpp:204: Received an authentication request for [email protected] with nonce 492c0cb606cfbf7f, but no matching AV found 13-03-2015 11:09:58.671 UTC Error memcachedstore.cpp:713: Failed to write data for av\\[email protected]\6d8b205e7a3a96af to 1 replicas 13-03-2015 11:09:58.671 UTC Error avstore.cpp:71: Failed to write Authentication Vector for private_id [email protected] 13-03-2015 11:09:58.742 UTC Error memcachedstore.cpp:519: Failed to read data for av\\[email protected]\6d8b205e7a3a96af from 1 replicas 13-03-2015 11:09:58.742 UTC Warning authentication.cpp:204: Received an authentication request for [email protected] with nonce 6d8b205e7a3a96af, but no matching AV found 13-03-2015 11:09:58.749 UTC Error memcachedstore.cpp:713: Failed to write data for av\\[email protected]\1676e38444c12b76 to 1 replicas 13-03-2015 11:09:58.749 UTC Error avstore.cpp:71: Failed to write Authentication Vector for private_id [email protected] Attached are the detailed logs of Sprout and homestead with a screenshot of Zoiper too. Regards, Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Consulting ____________________________________________ -----Eleanor Merry <[email protected]> wrote: ----- To: Prerna Pandit <[email protected]> From: Eleanor Merry <[email protected]> Date: 03/13/2015 12:31AM Cc: "[email protected]" < [email protected]>, Pundalik K <[email protected]> Subject: RE: [Clearwater] Fw: Trouble installing Ellis(Manual Install) Hi Prerna, The Sprout logs you?ve sent over seem to be corrupted ? I can?t open them. Looking at the Zoiper screenshot though, you?ll need to add in the auth username option. This should be set to the private identity of your subscriber e.g. [email protected] Also, from the Ralf logs it doesn?t look like you have a CDF in your system; if you don?t have an external CDF (see https://github.com/Metaswitch/clearwater-docs/wiki/CDF-Integration for how to integrate with one) then you don?t need to have a Ralf in your deployment. Ellie From: Prerna Pandit [mailto:[email protected]] Sent: 12 March 2015 11:25 To: Eleanor Merry Cc: [email protected]; Pundalik K Subject: RE: [Clearwater] Fw: Trouble installing Ellis(Manual Install) Thanks Ellie. Correcting the homestead DNS names at /etc/clearwater/config solves yesterday's issue, We are now successfully able to create a Internal Identity through ellis GUI. Using the SIP username and password generated by the GUI , we tried to configure a new account using Zoiper SIP client ( for Ubuntu). This activity fails. We found some errors on Sprout, Homestead and Ralf. Looks like ACR message fails between Sprout and Homestead. We are attaching the logs. Any suggestions would be of great help. Regards, Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Consulting ____________________________________________ -----Eleanor Merry <[email protected]> wrote: ----- To: Prerna Pandit <[email protected]>, " [email protected]" < [email protected]> From: Eleanor Merry <[email protected]> Date: 03/11/2015 11:58PM Cc: Pundalik K <[email protected]> Subject: RE: [Clearwater] Fw: Trouble installing Ellis(Manual Install) Hi Prerna, In the Ellis logs, Ellis is trying to contact hs.example.com (which is taken from the hs_hostname in /etc/clearwater/config). However in your DNS records you?ve set up homestead.example.com. Can you make these consistent? Ellie From: Prerna Pandit [mailto:[email protected]] Sent: 10 March 2015 12:00 To: [email protected]; Eleanor Merry Cc: Pundalik K Subject: Re: [Clearwater] Fw: Trouble installing Ellis(Manual Install) (See attached file: clearwaterLogs.zip) Hi Ellie, We did the modifications as per your suggestion to /etc/bind/db.example.com. Post that, we are now able to ping all the machines from each other. (e.g. ping homestead.example.com etc.) Then we successfully brought up the ellis GUI (http://ellis.example.com). But when we try to create a new Internal Identity , it gives error "Failed to update Server". To troubleshoot the same, we followed the steps mentioned in the link https://github.com/Metaswitch/clearwater-docs/wiki/Troubleshooting-and-Recovery . As per the ellis logs, the error seems to be with HOMESTEAD_URL not defined. Also verified the homestead and homer node configurations which looks ok to us. We are attaching the detailed logs(ellis, homer, homestead) for reference. Please guide and suggest to resolve the issue. Regards, Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Consulting ____________________________________________ -Hi Prerna, The DNS settings in https://github.com/Metaswitch/clearwater-docs/wiki/Clearwater-DNS-Usage use <process>-<number>.<domain> for the A/AAAA record for an individual node, and <process>.<domain> for the record containing all of the nodes for that process. As an example, if you had two Sprouts, you would have individual A/AAAA records for sprout-1.<domain> and sprout-2.<domain>. The cluster record for Sprout would be sprout.<domain>, and this resolves to both of the individual Sprouts. It?s this cluster record that goes into sprout_hostname in /etc/clearwater/config, and it?s used to load balance between the individual Sprouts. If you only have one node of each type you don?t need to set up cluster records. Ellie From: Prerna Pandit [mailto:[email protected]] Sent: 26 February 2015 08:10 To: Eleanor Merry Cc: clearwater; Pundalik K Subject: RE: [Clearwater] Fw: Trouble installing Ellis(Manual Install)(ScanMail has removed a file) Hi Ellie, As per your suggestion, we have modified the db.example.com file for DNS settings. PFA the same. ( Note: The same file has been copied to all the machines at location /etc/bind/db.example.com) There is a confusion here. We have given the hostnames in /etc/clearwater/config as ellis, bono, sprout and so on.. But the names in db.example.com are ellis-1, bono-1, sprout-1.. Do we need to make both these names same ? To ensure DNS is configured properly , we tried to ping bono from ellis using "ping bono" , but this didn't work. Any pointers for the same will be a great help. Regards, Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Consulting ____________________________________________ From: Eleanor Merry <[email protected]> To: Prerna Pandit <[email protected]> Cc: clearwater <[email protected]>, Pundalik K <[email protected]> Date: 20-02-2015 19:22 Subject: RE: [Clearwater] Fw: Trouble installing Ellis(Manual Install)(ScanMail has removed a file) Hi Prerna, Have you set up the Homestead and Homer nodes, and added them to DNS? You do need to do this before you can provision the subscribers through Ellis. Also, your DNS settings for sprout/bono don?t look quite right. You have multiple entries for each type of node (e.g. there?s a sprout-1 and a sprout-2), but I think you only have one of each type, and there?s also dummy IPv6 entries. You should remove the extra entries; for example ?sprout.example.com? will currently resolve to 192.168.1.3, 3.0.0.2 3::1 and 3::2, and you only want 192.168.1.3. Ellie From: Prerna Pandit [mailto:[email protected]] Sent: 20 February 2015 11:43 To: Eleanor Merry Cc: clearwater; Pundalik K Subject: Re: [Clearwater] Fw: Trouble installing Ellis(Manual Install)(ScanMail has removed a file) Re-attaching the file (db.example.com) Regards, Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Consulting ____________________________________________ [email protected] wrote: ----- To: [email protected] From: Prerna Pandit Sent by: [email protected] Date: 02/20/2015 05:04PM Cc: clearwater <[email protected]>, Pundalik K < [email protected]> Subject: Re: [Clearwater] Fw: Trouble installing Ellis(Manual Install)(ScanMail has removed a file) To add on to the previous email, We have configured the DNS using BIND. PFA the DNS configuration file (/etc/bind/db.example.com). Regards, Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Consulting ____________________________________________ [email protected] wrote: ----- To: [email protected] From: Prerna Pandit Sent by: [email protected] Date: 02/20/2015 04:59PM Cc: Pundalik K <[email protected]>, clearwater < [email protected]> Subject: Re: [Clearwater] Fw: Trouble installing Ellis(Manual Install) Hi Ellie, Thanks for your help. We are able to install and configure three nodes vis - Ellis, Bono , Sprout on one laptop with Ellis on host and Bono and Sprout on two VMs. Later we configured the DNS for Ellis node and tried launching the GUI using "http://ellis.example.com";. We signed up as a new user and signup-code being "secret" from the /etc/clearwater/config file. But it didn't successfully launched giving the error as in the attachment. Following is the syslog output: Feb 20 16:39:01 ellis CRON[29250]: (root) CMD (/usr/lib/sysstat/sadc 1 1 /var/log/sysstat/clearwater-sa`date +%d`) Feb 20 16:39:20 ellis kernel: [16390.223708] Program lshw tried to access /dev/mem between ff000->101000. Feb 20 16:40:01 ellis CRON[29619]: (root) CMD (/usr/lib/sysstat/sadc 1 1 /var/log/sysstat/clearwater-sa`date +%d`) Feb 20 16:40:20 ellis kernel: [16450.186018] Program lshw tried to access /dev/mem between ff000->101000. Feb 20 16:41:01 ellis CRON[30009]: (root) CMD (/usr/lib/sysstat/sadc 1 1 /var/log/sysstat/clearwater-sa`date +%d`) Feb 20 16:41:20 ellis kernel: [16510.285771] Program lshw tried to access /dev/mem between ff000->101000. Feb 20 16:42:01 ellis CRON[30381]: (root) CMD (/usr/lib/sysstat/sadc 1 1 /var/log/sysstat/clearwater-sa`date +%d`) Feb 20 16:42:20 ellis kernel: [16570.341833] Program lshw tried to access /dev/mem between ff000->101000. Feb 20 16:43:01 ellis CRON[30760]: (root) CMD (/usr/lib/sysstat/sadc 1 1 /var/log/sysstat/clearwater-sa`date +%d`) Feb 20 16:43:20 ellis kernel: [16630.322536] Program lshw tried to access /dev/mem between ff000->101000. Feb 20 16:43:51 ellis named[26764]: error (network unreachable) resolving 'ns1.p16.dynect.net/A/IN': 2001:500:1::803f:235#53 Feb 20 16:43:51 ellis named[26764]: error (network unreachable) resolving 'ns1.p16.dynect.net/AAAA/IN': 2001:500:1::803f:235#53 Feb 20 16:43:51 ellis named[26764]: error (network unreachable) resolving 'ns3.p16.dynect.net/A/IN': 2001:500:1::803f:235#53 Feb 20 16:43:51 ellis named[26764]: error (network unreachable) resolving './NS/IN': 2001:500:1::803f:235#53 Feb 20 16:43:51 ellis named[26764]: error (network unreachable) resolving 'ns2.p16.dynect.net/A/IN': 2001:500:1::803f:235#53 Feb 20 16:43:52 ellis named[26764]: error (network unreachable) resolving 'ns3.p16.dynect.net/AAAA/IN': 2001:500:90::100#53 Feb 20 16:43:53 ellis named[26764]: error (network unreachable) resolving 'ns1.p03.dynect.net/A/IN': 2001:500:94::100#53 Feb 20 16:43:53 ellis named[26764]: error (network unreachable) resolving 'ns1.p03.dynect.net/AAAA/IN': 2001:500:94::100#53 Feb 20 16:43:54 ellis named[26764]: error (network unreachable) resolving 'cs9.wac.edgecastcdn.net/A/IN': 2606:2800:1::6#53 Regards, Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Consulting ____________________________________________ From: Eleanor Merry <[email protected]> To: Prerna Pandit <[email protected]> Cc: clearwater <[email protected]>, Pundalik K < [email protected]> Date: 19-02-2015 00:15 Subject: RE: [Clearwater] Trouble installing Ellis(Manual Install) Hi Prerna, Only the IP addresses and the public hostname need to change between nodes. So you should update the local_ip, public_ip and public_hostname values, and the local IP address in the chronos_hostname value to be Bono?s address instead of Ellis. You do not need to change the ellis_api_key. This parameter is only used by Ellis, and will be ignored by Bono. We keep it in the main configuration file to make it easier to share the same configuration file between all your nodes. Ellie From: Prerna Pandit [mailto:[email protected]] Sent: 18 February 2015 11:32 To: Eleanor Merry Cc: clearwater; Pundalik K Subject: RE: [Clearwater] Trouble installing Ellis(Manual Install) Hi Ellie, Thanks for the response. We are now trying to install the next node-Bono. We have some doubts regarding the changes required in config file (/etc/clearwater/config). How will it be different from that of Ellis? For Ellis we have used the below config : # Deployment definitions /* Do we need to change any parameter here ? */ home_domain=example.com sprout_hostname=sprout.example.com chronos_hostname=192.168.1.1:7253 hs_hostname=hs.example.com:8888 hs_provisioning_hostname=hs.example.com:8889 ralf_hostname=ralf.example.com:10888 xdms_hostname=homer.example.com:7888 # Local IP configuration /* IP address and hostname will be of the new node Bono ? */ local_ip=192.168.1.1 public_ip=192.168.1.1 public_hostname=Ellis # Email server configuration smtp_smarthost=<smtp server> smtp_username=<username> smtp_password=<password> [email protected] # Keysÿ/* Will the keywords ellis_api_key be replaced with something like bono_api_key ? */ signup_key="secret" turn_workaround="secret" ellis_api_key="secret" ellis_cookie_key="secret"ÿ Regards, Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. ÿ ÿ ÿ ÿIT Services ÿÿ ÿ ÿ ÿ ÿ ÿ ÿ ÿ ÿ ÿ ÿ Business Solutions ÿÿ ÿ ÿ ÿ ÿ ÿ ÿ ÿ ÿ ÿ ÿ Consulting ____________________________________________ÿ From: ÿ ÿ ÿ ÿEleanor Merry <[email protected]>ÿ To: ÿ ÿ ÿ ÿPundalik K <[email protected]>? Cc: ? ? ? ?Prerna Pandit <[email protected]>, clearwater < [email protected]>? Date: ? ? ? ?17-02-2015 23:44? Subject: ? ? ? ?RE: [Clearwater] Trouble installing Ellis(Manual Install)? Hi Pundalik, ÿ? local_ip and public_ip can be the same. The local_ip should be the IP address that the node can use to communicate with the other Clearwater/IMS core components. The public_ip is used by Bono and Ellis nodes, and it should be set to an IP address that/?s accessible to external clients (e.g. SIP clients for Bono, web browsers for Ellis) /? if these are the same then that is fine. ÿ? The public_hostname should be a hostname that resolves to the public_ip /? if you don/?t have one then this should be set to the same value as the public IP. ÿ? Email server configuration is not mandatory, and can be configured later. ÿ? Ellie? ÿ? From:?Pundalik K [mailto:[email protected]] Sent:?17 February 2015 12:32 To:?Eleanor Merry Cc:?Prerna Pandit; clearwater Subject:?RE: [Clearwater] Trouble installing Ellis(Manual Install)? ÿ? Hi Ellie, I am working along with Prerna to setup clearwater IMS. Based on your guidelines, we upgraded the machine to ubuntu 12.04 64-bit and later tried installing one of the image (Ellis for now). The installation now succeeds without any errors. Thanks. As we are following the Manual way to install and test Clearwater on 6 machines, have certain queries w.r.t. /etc/clearwater/config file which needs to be configured on each machine/image. Example entries that we have configured are as given below (highlighted in bold). As this is our local setup, can local_ip?& public_ip?be same? Is Email server configuration mandatory or can it be configured later? Will the <zone> remain same across all machines? Please guide with the config so we can proceed accordingly # Deployment definitions home_domain=example.com sprout_hostname=sprout.example.com chronos_hostname=192.168.1.1:7253 hs_hostname=hs.example.com:8888 hs_provisioning_hostname=hs.example.com:8889 ralf_hostname=ralf.example.com:10888 xdms_hostname=homer.example.com:7888 # Local IP configuration local_ip=192.168.1.1 public_ip=192.168.1.1 public_hostname=Ellis # Email server configuration smtp_smarthost=<smtp server> smtp_username=<username> smtp_password=<password> [email protected] # Keys signup_key="secret" turn_workaround="secret" ellis_api_key="secret" ellis_cookie_key="secret" Regards, Pundalik Kandolcar Tata Consultancy Services Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Consulting ____________________________________________ -----Eleanor Merry <[email protected]>?wrote: -----? To: Prerna Pandit <[email protected]> From: Eleanor Merry <[email protected]> Date: 02/17/2015 12:56AM Cc: clearwater <[email protected]>, Pundalik K < [email protected]> Subject: RE: [Clearwater] Trouble installing Ellis(Manual Install)? Hi Prerna, ÿ? Clearwater was built from the ground up for virtualization /? you absolutely don/?t need to have separate physical machines. You could run all the nodes on one machine if you wished!? ÿ? That said /? your laptops are a bit underpowered if you want to put load through Clearwater (e.g. we recommend a minimum of 1.7GB of memory for each node). If you are just testing the setup this should be fine though (you may also be interested in are all-in-one node /? please see https://github.com/Metaswitch/clearwater-docs/wiki/All-in-one-Images?for more details). ÿ? Ellie? ÿ? From:?Prerna Pandit [mailto:[email protected]] Sent:?16 February 2015 07:22 To:?Eleanor Merry Cc:?clearwater; Pundalik K Subject:?RE: [Clearwater] Trouble installing Ellis(Manual Install)? ÿ? Yes, I was running it on a 32 bit machine. Thanks Ellie for pointing the same.? One more question -? We are trying to setup the entire clearwater IMS. Due to hardware limitations, we are planning to take two laptops and setup 3 nodes on each.? Two VMs and one host on each. This way we'll be able to setup the six images on two machines(ubuntu 64 bit server) .? Note: Current Laptop configuration - intel i5 processor-2.5GHz, Memory 4GB? Question is - Will the aforesaid setup work ? Or we will need 6 separate machines only ? Regards, Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. ? ? ? ?IT Services ÿ? ? ? ? ? ? ? ? ? ? ?Business Solutions ÿ? ? ? ? ? ? ? ? ? ? ?Consulting ____________________________________________? From: ? ? ? ?Eleanor Merry <[email protected]>? To: ? ? ? ?Prerna Pandit <[email protected]>? Cc: ? ? ? ?clearwater <[email protected]>? Date: ? ? ? ?13-02-2015 01:41? Subject: ? ? ? ?RE: [Clearwater] Trouble installing Ellis(Manual Install)? Hi Prerna, ÿ I/?ve tried downloading from the stable repository, and I can get the Ellis package. Can you retrieve any of the clearwater packages? Does running apt-get update report any errors?? ÿ You can hit this symptom if you/?re running the a 32bit machine rather than a 64bit (we only support 64bit). You can test this by running /?uname /?a/? on your machine. ÿ Ellie? ÿ From:?Prerna Pandit [mailto:[email protected]] Sent:?11 February 2015 08:23 To:?Eleanor Merry Cc:?clearwater Subject:?RE: [Clearwater] Trouble installing Ellis(Manual Install)? ÿ Hi Ellie,? Thanks for responding. The repository set in?/etc/apt/sources.list.d/clearwater.list-?"deb http://repo.cw-ngv.com/stable?binary/";.? And yes I am following the same link which you have given.? Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com?; ____________________________________________ Experience certainty. ? ? ? ?IT Services ÿ? ? ? ? ? ? ? ? ? ? Business Solutions ÿ? ? ? ? ? ? ? ? ? ? Consulting ____________________________________________? From: ? ? ? ?Eleanor Merry <[email protected]>? To: ? ? ? ?Prerna Pandit <[email protected]>, clearwater < [email protected]>? Date: ? ? ? ?11-02-2015 00:17? Subject: ? ? ? ?RE: [Clearwater] Trouble installing Ellis(Manual Install)? Hi Prerna, What repository have you got set in /etc/apt/sources.list.d/clearwater.list? There are a number of steps you need to do on the machine before you install the node-specific software - have you followed the steps at https://github.com/Metaswitch/clearwater-docs/wiki/Manual%20Install#bootstrapping-the-machines??These cover configuring the APT sources, and setting up the configuration file on the node. Ellie -----Original Message----- From: [email protected]?[ mailto:[email protected]] On Behalf Of Prerna Pandit Sent: 10 February 2015 10:36 To: clearwater Subject: [Clearwater] Trouble installing Ellis(Manual Install) Hello, We are trying to install clearwater IMS by following steps from " https://github.com/Metaswitch/clearwater-docs/wiki/Manual%20Install";. Following is the error observed . root@tcs:/home/tcs# sudo DEBIAN_FRONTEND=noninteractive apt-get install ellis --yes Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package ellis Kindly throw some light on how to resolve the same. Regards, Prerna Pandit Tata Consultancy Services Limited Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. ? ? ? ? ? ? ? ? IT Services ÿ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Business Solutions ÿ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Consulting ____________________________________________ =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater?; _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
