Hi Andrew, Thanks for your support. I am not in a position to modify the current dns server nor our admins would do it. I have added entry for icscf.cwsprout in /etc/hosts. I am able to ping icscf.cwsprout after adding this entry. Still i get same error in the log.
=========================================================== cwbono@cwbono:~$ ping icscf.cwsprout PING icscf.cwsprout (10.48.12.143) 56(84) bytes of data. 64 bytes from cwsprout (10.48.12.143): icmp_seq=1 ttl=64 time=0.157 ms 64 bytes from cwsprout (10.48.12.143): icmp_seq=2 ttl=64 time=0.129 ms ^C --- icscf.cwsprout ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.129/0.143/0.157/0.014 ms cwbono@cwbono:~$ cat /var/log/bono/bono_current.txt 14-07-2017 10:10:19.440 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 14-07-2017 10:10:19.440 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 14-07-2017 10:10:19.440 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 14-07-2017 10:10:19.440 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) =========================================================== If this is not working then I might have to create a dns server locally. Then i would be able to modify or add entries Is it possible for you to give me an example of how these entries are added? Thanks Hrishikesh On Thu, Jul 13, 2017 at 9:30 PM, Andrew Edmonds < [email protected]> wrote: > Hi Hrishikesh, > > > > Thank you for the updated diagnostics. > > > > We can still see the following appearing in log files: > > > > “Failed to resolve icscf.cwsprout to an IP address” > > > > The icscf.<sprout hostname> DNS record is used by the P-CSCF (in this case > Bono) to identify which I-CSCF to forward requests on to. > > > > Even when assigning nodes IP addresses through DHCP you must still > configure your DNS with all the records that Clearwater nodes require to > communicate with each other (such as icscf.<sprout hostname>). You can find > a list of all the records that Clearwater requires here > <http://clearwater.readthedocs.io/en/stable/Clearwater_DNS_Usage.html>. > > > > Please could you try updating your DNS server to contain these records and > restart your nodes, let me know if you still hit the issue. > > > > Thanks, > > > > Andrew > > > > *From:* Clearwater [mailto:[email protected]] > *On Behalf Of *Hrishikesh Karanjikar > *Sent:* Thursday, July 13, 2017 10:40 AM > *To:* [email protected] > *Subject:* Re: [Project Clearwater] restund_process Execution failed on > bono > > > > Hi, > > I have changed my deployment completely. > > No I am not using static IP addresses any more and each node is getting IP > addresses from the DHCP server. > > Each node is able to ping one another using hostname cwellis, cwsprout, > cwbono, cwvellum, cwhomer, cwdime. > > From bono node I am able to ping, ssh to cwsprout. Check the logs, > > ============================================================ > =================== > > cwbono@cwbono:~$ ping cwsprout > PING cwsprout.amcc.com (10.48.12.143) 56(84) bytes of data. > 64 bytes from cwsprout.amcc.com (10.48.12.143): icmp_seq=1 ttl=64 > time=0.126 ms > 64 bytes from cwsprout.amcc.com (10.48.12.143): icmp_seq=2 ttl=64 > time=0.203 ms > 64 bytes from cwsprout.amcc.com (10.48.12.143): icmp_seq=3 ttl=64 > time=0.210 ms > > cwbono@cwbono:~$ ssh cwsprout@cwsprout > The authenticity of host 'cwsprout (10.48.12.143)' can't be established. > ECDSA key fingerprint is 03:f8:42:81:36:ef:b1:be:7a:d8:3d:52:b5:74:f9:ba. > Are you sure you want to continue connecting (yes/no)? yes > Warning: Permanently added 'cwsprout,10.48.12.143' (ECDSA) to the list of > known hosts. > cwsprout@cwsprout's password: > Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64) > > * Documentation: https://help.ubuntu.com/ > > System information as of Thu Jul 13 14:08:07 IST 2017 > > System load: 0.12 Processes: 115 > Usage of /: 27.7% of 7.26GB Users logged in: 0 > Memory usage: 8% IP address for eth0: 10.48.12.143 > Swap usage: 0% > > Graph this data and manage this system at: > https://landscape.canonical.com/ > > New release '16.04.2 LTS' available. > Run 'do-release-upgrade' to upgrade to it. > > Last login: Thu Jul 13 14:08:07 2017 from hdk-supermicro.amcc.com > [sprout]cwsprout@cwsprout:~$ > > [sprout]cwsprout@cwsprout:~$ exit > logout > Connection to cwsprout closed. > cwbono@cwbono:~$ > cwbono@cwbono:~$ > cwbono@cwbono:~$ cat /etc/clearwater/local_config > local_ip=10.48.12.173 > public_ip=10.48.12.173 > public_hostname=cwbono > etcd_cluster="10.48.12.142,10.48.12.143,10.48.12.173,10.48. > 12.140,10.48.12.139,10.48.12.120" > cwbono@cwbono:~$ cat /etc/clearwater/shared_config > ##################################################################### > # No Shared Config has been provided > # Replace this file with the Shared Configuration for your deployment > ##################################################################### > > home_domain=example.com > sprout_hostname=cwsprout > sprout_registration_store=10.48.12.120 #vellum > hs_hostname=10.48.12.139:8888 #dime > hs_provisioning_hostname=10.48.12.139:8889 #dime > ralf_hostname= > ralf_session_store= > xdms_hostname=10.48.12.140:7888 #homer > chronos_hostname=vellum > cassandra_hostname=10.48.12.120 #vellum > > # Email server configuration > smtp_smarthost=localhost > smtp_username=username > smtp_password=password > [email protected] > > # Keys > signup_key=secret > turn_workaround=secret > ellis_api_key=secret > ellis_cookie_key=secret > > > cwbono@cwbono:~$ clearwater-etcdctl cluster-health > member 2d821d7a0a7736b4 is healthy: got healthy result from > http://10.48.12.142:4000 > member 6201151ee7f99f5c is healthy: got healthy result from > http://10.48.12.139:4000 > member 895efb70c4b1b8b4 is healthy: got healthy result from > http://10.48.12.120:4000 > member c7f5d6485fb4735b is healthy: got healthy result from > http://10.48.12.143:4000 > member f7726a4e29ec7d3d is healthy: got healthy result from > http://10.48.12.173:4000 > member ffb968d2990c63f0 is healthy: got healthy result from > http://10.48.12.140:4000 > cluster is healthy > cwbono@cwbono:~$ clearwater-etcdctl member list > 2d821d7a0a7736b4: name=10-48-12-142 peerURLs=http://10.48.12.142:2380 > clientURLs=http://10.48.12.142:4000 isLeader=false > 6201151ee7f99f5c: name=10-48-12-139 peerURLs=http://10.48.12.139:2380 > clientURLs=http://10.48.12.139:4000 isLeader=false > 895efb70c4b1b8b4: name=10-48-12-120 peerURLs=http://10.48.12.120:2380 > clientURLs=http://10.48.12.120:4000 isLeader=false > c7f5d6485fb4735b: name=10-48-12-143 peerURLs=http://10.48.12.143:2380 > clientURLs=http://10.48.12.143:4000 isLeader=false > f7726a4e29ec7d3d: name=10-48-12-173 peerURLs=http://10.48.12.173:2380 > clientURLs=http://10.48.12.173:4000 isLeader=true > ffb968d2990c63f0: name=10-48-12-140 peerURLs=http://10.48.12.140:2380 > clientURLs=http://10.48.12.140:4000 isLeader=false > > cwbono@cwbono:~$ cw-check_cluster_state > This script prints the status of the Cassandra, Chronos, and Memcached > clusters. > This node (10.48.12.173) should not be in any cluster. > > Describing the Cassandra cluster: > The cluster is stable > 10.48.12.120 is in state normal > > Describing the Chronos cluster: > The cluster is stable > 10.48.12.120 is in state normal > > Describing the Memcached cluster: > The cluster is stable > 10.48.12.120 is in state normal > > cwbono@cwbono:~$ sudo cw-check_config_sync > - /etc/clearwater/dns.json is up to date > - /etc/clearwater/shared_config is up to date > > cwbono@cwbono:~$ sudo monit summary > Monit 5.18.1 uptime: 54m > Service Name Status > Type > node-cwbono Running > System > restund_process Execution failed | Does... > Process > ntp_process Running > Process > clearwater_queue_manager_pro... Running > Process > etcd_process Running > Process > clearwater_diags_monitor_pro... Running > Process > clearwater_config_manager_pr... Running > Process > clearwater_cluster_manager_p... Running > Process > bono_process Running > Process > poll_restund Wait parent > Program > monit_uptime Status ok > Program > clearwater_queue_manager_uptime Status ok > Program > etcd_uptime Status ok > Program > poll_etcd_cluster Status ok > Program > poll_etcd Status ok > Program > poll_bono Status ok > Program > > cwbono@cwbono:~$ cat /var/log/bono/bono_current.txt > 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > > cwbono@cwbono:~$ cat /var/log/monit.log > turn: extended channels enabled > tcp: sock_bind: bind: Address already in use > [IST Jul 13 15:03:29] error : 'restund_process' process is not running > [IST Jul 13 15:03:29] info : 'restund_process' trying to restart > [IST Jul 13 15:03:29] info : 'restund_process' restart: > /etc/init.d/restund > [IST Jul 13 15:04:00] error : 'restund_process' failed to restart (exit > status 0) -- /etc/init.d/restund: httpdb: configured url > http://hs.example.com:8888/impi/%s/digest > httpd: using URI workaround > turn: server deployed behind static NAT addr=10.48.12.173:0 > turn: extended channels enabled > tcp: sock_bind: bind: Address already in use > [IST Jul 13 15:04:10] error : 'restund_process' process is not running > [IST Jul 13 15:04:10] info : 'restund_process' trying to restart > [IST Jul 13 15:04:10] info : 'restund_process' restart: > /etc/init.d/restund > [IST Jul 13 15:04:40] error : 'restund_process' failed to restart (exit > status 0) -- /etc/init.d/restund: httpdb: configured url > http://hs.example.com:8888/impi/%s/digest > httpd: using URI workaround > turn: server deployed behind static NAT addr=10.48.12.173:0 > turn: extended channels enabled > tcp: sock_bind: bind: Address already in use > [IST Jul 13 15:04:50] error : 'restund_process' process is not running > [IST Jul 13 15:04:50] info : 'restund_process' trying to restart > [IST Jul 13 15:04:50] info : 'restund_process' restart: > /etc/init.d/restund > [IST Jul 13 15:05:20] error : 'restund_process' failed to restart (exit > status 0) -- /etc/init.d/restund: httpdb: configured url > http://hs.example.com:8888/impi/%s/digest > httpd: using URI workaround > turn: server deployed behind static NAT addr=10.48.12.173:0 > turn: extended channels enabled > tcp: sock_bind: bind: Address already in use > [IST Jul 13 15:05:30] error : 'restund_process' process is not running > [IST Jul 13 15:05:30] info : 'restund_process' trying to restart > [IST Jul 13 15:05:30] info : 'restund_process' restart: > /etc/init.d/restund > [IST Jul 13 15:06:01] error : 'restund_process' failed to restart (exit > status 0) -- /etc/init.d/restund: httpdb: configured url > http://hs.example.com:8888/impi/%s/digest > httpd: using URI workaround > turn: server deployed behind static NAT addr=10.48.12.173:0 > turn: extended channels enabled > tcp: sock_bind: bind: Address already in use > [IST Jul 13 15:06:11] error : 'restund_process' process is not running > [IST Jul 13 15:06:11] info : 'restund_process' trying to restart > [IST Jul 13 15:06:11] info : 'restund_process' restart: > /etc/init.d/restund > [IST Jul 13 15:06:41] error : 'restund_process' failed to restart (exit > status 0) -- /etc/init.d/restund: httpdb: configured url > http://hs.example.com:8888/impi/%s/digest > httpd: using URI workaround > turn: server deployed behind static NAT addr=10.48.12.173:0 > turn: extended channels enabled > tcp: sock_bind: bind: Address already in use > [IST Jul 13 15:06:51] error : 'restund_process' process is not running > [IST Jul 13 15:06:51] info : 'restund_process' trying to restart > [IST Jul 13 15:06:51] info : 'restund_process' restart: > /etc/init.d/restund > [IST Jul 13 15:07:21] error : 'restund_process' failed to restart (exit > status 0) -- /etc/init.d/restund: httpdb: configured url > http://hs.example.com:8888/impi/%s/digest > httpd: using URI workaround > turn: server deployed behind static NAT addr=10.48.12.173:0 > turn: extended channels enabled > tcp: sock_bind: bind: Address already in use > [IST Jul 13 15:07:31] error : 'restund_process' process is not running > [IST Jul 13 15:07:31] info : 'restund_process' trying to restart > [IST Jul 13 15:07:31] info : 'restund_process' restart: > /etc/init.d/restund > > ============================================================ > =================== > > I don't know whats going wrong here. > > What is icscf.cwsprout? > > > > Thanks > > Hrishikesh > > > > > > On Tue, Jul 11, 2017 at 5:08 PM, Hrishikesh Karanjikar < > [email protected]> wrote: > > Hi, > > Thanks a lot for your reply. > > I am using virtualbox and host only network. > > The DHCP server runs inside virtualbox and I can only configure the IP > address range. > > However I am not using the DHCP server and assigning static IP addresses > to all nodes which are within the DHCP server IP address range. > > I added entry of cwsprout in /ets/hosts of bono node as follows, > > =============================================== > > 127.0.0.1 localhost > 127.0.1.1 cwbono > > *192.168.56.103 cwsprout 192.168.56.103 icscf.cwsprout* > > # The following lines are desirable for IPv6 capable hosts > ::1 localhost ip6-localhost ip6-loopback > ff02::1 ip6-allnodes > ff02::2 ip6-allrouters > 192.168.56.103 cwsprout > 192.168.56.103 icscf.cwsprout > ::1 localhost # added by clearwater-infrastructure 1hosts script > 192.168.56.104 cwbono #+clearwater-infrastructure > > =============================================== > > I am also able to ping cwsprout and icscf.cwsprout from bono, > > Here is the log, > > =============================================== > > [bono]cwbono@cwbono:~$ ping cwsprout > PING cwsprout (192.168.56.103) 56(84) bytes of data. > 64 bytes from cwsprout (192.168.56.103): icmp_seq=1 ttl=64 time=0.197 ms > 64 bytes from cwsprout (192.168.56.103): icmp_seq=2 ttl=64 time=0.217 ms > ^C > --- cwsprout ping statistics --- > 2 packets transmitted, 2 received, 0% packet loss, time 999ms > rtt min/avg/max/mdev = 0.197/0.207/0.217/0.010 ms > [bono]cwbono@cwbono:~$ > [bono]cwbono@cwbono:~$ > [bono]cwbono@cwbono:~$ > [bono]cwbono@cwbono:~$ ping icscf.cwsprout > PING icscf.cwsprout (192.168.56.103) 56(84) bytes of data. > 64 bytes from cwsprout (192.168.56.103): icmp_seq=1 ttl=64 time=0.112 ms > 64 bytes from cwsprout (192.168.56.103): icmp_seq=2 ttl=64 time=0.165 ms > > =============================================== > > I am using static IP addresses as I have to specify them in local_config > file of each node. > > If I use DHCP server of VirtualBox they might change. In that case I am > not sure how do I cope up with local_config. > > Can I modify local_config after all nodes are up? > > > > Thanks > > Hrishikesh > > > > > > On Tue, Jul 11, 2017 at 2:42 PM, Andrew Edmonds < > [email protected]> wrote: > > Hi Hrishikesh, > > > > Thank you for your question and the detailed logs you have provided. > > > > The issues appears to be caused by your shared config, in the manual > installation instructions > <http://clearwater.readthedocs.io/en/stable/Manual_Install.html> you’ll > see that we advise that the entries in shared config have a format like: > > > > sprout_hostname*=*sprout*.<*site_name*>.<*zone*>* > > sprout_registration_store*=*vellum*.<*site_name*>.<*zone*>* > > hs_hostname*=*hs*.<*site_name*>.<*zone*>*:8888 > > > > In your shared config you have: > > > > sprout_hostname=cwsprout > sprout_registration_store=192.168.56.107 #vellum > > I don’t think the sprout_hostname used here will resolve, we can see > evidence for this in the Bono logs, Bono attempts to resolve > icscf.<sprout_hostname> to find which location to forward SIP messages on > you, you can see it is failing to do that here: > > > > Failed to resolve icscf.cwsprout to an IP address - Not found > (PJ_ENOTFOUND) > > > > To resolve this issue you should change your shared config to use > hostnames which have been configured in your DNS server to resolve to the > appropriate location. Once you have done this run the command “sudo > cw-upload_shared_config”. Please let me know if this does not resolve the > issue. > > > > Thanks, > > > > Andrew > > > > > > *From:* Clearwater [mailto:[email protected]] > *On Behalf Of *Hrishikesh Karanjikar > *Sent:* Thursday, July 6, 2017 10:22 AM > *To:* [email protected] > *Subject:* [Project Clearwater] restund_process Execution failed on bono > > > > Hello, > > I have Manually installed all 6 nodes on VMs using virtualbox. > > I followed the procedure given @ http://clearwater.readthedocs. > io/en/stable/Manual_Install.html > > Looks like all nodes except Dime are running fine. > > I am getting error "restund_process Execution failed" in monit summary on > Bono node. > > Here is the shared and local config file, > > ##################################################################### > > [bono]cwbono@cwbono:~$ cat /etc/clearwater/shared_config > > home_domain=example.com > sprout_hostname=cwsprout > sprout_registration_store=192.168.56.107 #vellum > hs_hostname=192.168.56.106:8888 #dime > hs_provisioning_hostname=192.168.56.106:8889 #dime > ralf_hostname= > ralf_session_store= > xdms_hostname=192.168.56.105:7888 #homer > chronos_hostname=192.168.56.107 #vellum > cassandra_hostname=192.168.56.107 #vellum > > # Email server configuration > smtp_smarthost=localhost > smtp_username=username > smtp_password=password > [email protected] > > # Keys > signup_key=secret > turn_workaround=secret > ellis_api_key=secret > ellis_cookie_key=secret > [bono]cwbono@cwbono:~$ cat /etc/clearwater/local_config > local_ip=192.168.56.104 > public_ip=192.168.56.104 > public_hostname=cwbono > etcd_cluster="192.168.56.102,192.168.56.103,192.168.56.104, > 192.168.56.105,192.168.56.106,192.168.56.107" > > ##################################################################### > > The logs for bono node are as follows, > > ##################################################################### > [bono]cwbono@cwbono:~$ sudo monit summary > [sudo] password for cwbono: > Monit 5.18.1 uptime: 2d 16h 39m > Service Name Status > Type > node-cwbono Running > System > restund_process Execution failed | Does... > Process > ntp_process Running > Process > clearwater_queue_manager_pro... Running > Process > etcd_process Running > Process > clearwater_diags_monitor_pro... Running > Process > clearwater_config_manager_pr... Running > Process > clearwater_cluster_manager_p... Running > Process > bono_process Running > Process > poll_restund Wait parent > Program > monit_uptime Status ok > Program > clearwater_queue_manager_uptime Status ok > Program > etcd_uptime Status ok > Program > poll_etcd_cluster Status ok > Program > poll_etcd Status ok > Program > poll_bono Status ok > Program > > > 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:15.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:15.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:15.843 UTC Status main.cpp:1358: Quiesce signal received > 05-07-2017 11:30:15.843 UTC Status stack.cpp:125: Setting quiescing = > PJ_TRUE > 05-07-2017 11:30:15.851 UTC Status stack.cpp:156: Quiescing state changed > 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The > Quiescing Manager received input QUIESCE (0) when in state ACTIVE (0) > 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:265: Close > untrusted listening port > 05-07-2017 11:30:15.851 UTC Status stack.cpp:368: Destroyed TCP transport > for port 5060 > 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:273: Quiesce > FlowTable > 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The > Quiescing Manager received input FLOWS_GONE (1) when in state > QUIESCING_FLOWS (1) > 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:290: Closing > trusted port > 05-07-2017 11:30:15.851 UTC Status stack.cpp:368: Destroyed TCP transport > for port 5058 > 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:296: Quiescing > all connections > 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:162: Start > quiescing connections > 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:175: Quiescing 0 > transactions > 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:180: Connection > quiescing complete > 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The > Quiescing Manager received input CONNS_GONE (2) when in state > QUIESCING_CONNS (2) > 05-07-2017 11:30:15.851 UTC Status main.cpp:1380: Quiesce complete > 05-07-2017 11:30:15.853 UTC Status stack.cpp:171: PJSIP thread ended > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to > resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) > > [bono]cwbono@cwbono:~$ cat /var/log/monit.log > > httpd: using URI workaround > turn: server deployed behind static NAT addr=192.168.56.104:0 > turn: extended channels enabled > tcp: sock_bind: bind: Address already in u > [IST Jul 6 14:47:46] error : 'restund_process' process is not running > [IST Jul 6 14:47:46] info : 'restund_process' trying to restart > [IST Jul 6 14:47:46] info : 'restund_process' restart: > /etc/init.d/restund > [IST Jul 6 14:48:16] error : 'restund_process' failed to restart (exit > status 0) -- /etc/init.d/restund: httpdb: configured url > http://hs.example.com:8888/impi/%s/digest > httpd: using URI workaround > turn: server deployed behind static NAT addr=192.168.56.104:0 > turn: extended channels enabled > tcp: sock_bind: bind: Address already in u > [IST Jul 6 14:48:26] error : 'restund_process' process is not running > [IST Jul 6 14:48:26] info : 'restund_process' trying to restart > [IST Jul 6 14:48:26] info : 'restund_process' restart: > /etc/init.d/restund > [IST Jul 6 14:48:56] error : 'restund_process' failed to restart (exit > status 0) -- /etc/init.d/restund: httpdb: configured url > http://hs.example.com:8888/impi/%s/digest > httpd: using URI workaround > turn: server deployed behind static NAT addr=192.168.56.104:0 > turn: extended channels enabled > tcp: sock_bind: bind: Address already in u > [IST Jul 6 14:49:06] error : 'restund_process' process is not running > [IST Jul 6 14:49:06] info : 'restund_process' trying to restart > [IST Jul 6 14:49:06] info : 'restund_process' restart: > /etc/init.d/restund > > > [bono]cwbono@cwbono:~$ clearwater-etcdctl cluster-health > member 9c1928228d308a0f is healthy: got healthy result from > http://192.168.56.107:4000 > member b0c9c017e0d47e14 is healthy: got healthy result from > http://192.168.56.106:4000 > member d44832212a08c43f is healthy: got healthy result from > http://192.168.56.103:4000 > member ef1a9a8a2fd05283 is healthy: got healthy result from > http://192.168.56.104:4000 > member f63afbe816fb463d is healthy: got healthy result from > http://192.168.56.102:4000 > member f7132cc88f7a39fa is healthy: got healthy result from > http://192.168.56.105:4000 > cluster is healthy > [bono]cwbono@cwbono:~$ cw-check_cluster_state > This script prints out the status of the Chronos, Memcached and Cassandra > clusters. > > Describing the Vellum Chronos cluster: > The local node is *not* in this cluster > The cluster is stable > 192.168.56.107 is in state normal > > Describing the Vellum Memcached cluster: > The local node is *not* in this cluster > The cluster is stable > 192.168.56.107 is in state normal > > Describing the Vellum Cassandra cluster: > The local node is *not* in this cluster > The cluster is stable > 192.168.56.107 is in state normal > > [bono]cwbono@cwbono:~$ clearwater-etcdctl member list > 9c1928228d308a0f: name=192-168-56-107 peerURLs=http://192.168.56.107:2380 > clientURLs=http://192.168.56.107:4000 isLeader=false > b0c9c017e0d47e14: name=192-168-56-106 peerURLs=http://192.168.56.106:2380 > clientURLs=http://192.168.56.106:4000 isLeader=true > d44832212a08c43f: name=192-168-56-103 peerURLs=http://192.168.56.103:2380 > clientURLs=http://192.168.56.103:4000 isLeader=false > ef1a9a8a2fd05283: name=192-168-56-104 peerURLs=http://192.168.56.104:2380 > clientURLs=http://192.168.56.104:4000 isLeader=false > f63afbe816fb463d: name=192-168-56-102 peerURLs=http://192.168.56.102:2380 > clientURLs=http://192.168.56.102:4000 isLeader=false > f7132cc88f7a39fa: name=192-168-56-105 peerURLs=http://192.168.56.105:2380 > clientURLs=http://192.168.56.105:4000 isLeader=false > [bono]cwbono@cwbono:~$ sudo cw-check_config_sync > [sudo] password for cwbono: > - /etc/clearwater/dns.json is up to date > - /etc/clearwater/shared_config is up to date > > > ##################################################################### > > The logs for other nodes are as follows > > ##################################################################### > > [ellis]cwellis@cwellis:~$ sudo monit summary > Monit 5.18.1 uptime: 1d 21h 15m > Service Name Status > Type > node-cwellis Running > System > ntp_process Running > Process > nginx_process Running > Process > mysql_process Running > Process > ellis_process Running > Process > clearwater_queue_manager_pro... Running > Process > etcd_process Running > Process > clearwater_diags_monitor_pro... Running > Process > clearwater_config_manager_pr... Running > Process > clearwater_cluster_manager_p... Running > Process > nginx_ping Status ok > Program > nginx_uptime Status ok > Program > monit_uptime Status ok > Program > poll_ellis Status ok > Program > poll_ellis_https Status ok > Program > clearwater_queue_manager_uptime Status ok > Program > etcd_uptime Status ok > Program > poll_etcd_cluster Status ok > Program > poll_etcd Status ok > Program > > > [sprout]cwsprout@cwsprout:~$ sudo monit summary > Monit 5.18.1 uptime: 1d 20h 7m > Service Name Status > Type > node-cwsprout Running > System > sprout_process Running > Process > ntp_process Running > Process > nginx_process Running > Process > memento_process Running > Process > clearwater_queue_manager_pro... Running > Process > etcd_process Running > Process > clearwater_diags_monitor_pro... Running > Process > clearwater_config_manager_pr... Running > Process > clearwater_cluster_manager_p... Running > Process > sprout_uptime Status ok > Program > poll_sprout_sip Status ok > Program > poll_sprout_http Status ok > Program > nginx_ping Status ok > Program > nginx_uptime Status ok > Program > monit_uptime Status ok > Program > memento_uptime Status ok > Program > poll_memento Status ok > Program > poll_memento_https Status ok > Program > clearwater_queue_manager_uptime Status ok > Program > etcd_uptime Status ok > Program > poll_etcd_cluster Status ok > Program > poll_etcd Status ok Program > > > [homer]cwhomer@cwhomer:~$ sudo monit summary > Monit 5.18.1 uptime: 1d 20h 2m > Service Name Status > Type > node-cwhomer Running > System > ntp_process Running > Process > nginx_process Running > Process > homer_process Running > Process > clearwater_queue_manager_pro... Running > Process > etcd_process Running > Process > clearwater_diags_monitor_pro... Running > Process > clearwater_config_manager_pr... Running > Process > clearwater_cluster_manager_p... Running > Process > nginx_ping Status ok > Program > nginx_uptime Status ok > Program > monit_uptime Status ok > Program > poll_homer Status ok > Program > clearwater_queue_manager_uptime Status ok > Program > etcd_uptime Status ok > Program > poll_etcd_cluster Status ok > Program > poll_etcd Status ok Program > > [dime]cwdime@cwdime:~$ sudo monit summary > Monit 5.18.1 uptime: 1d 20h 2m > Service Name Status > Type > node-cwdime Running > System > snmpd_process Running > Process > ntp_process Running > Process > nginx_process Running > Process > homestead_process Running > Process > homestead-prov_process Running > Process > clearwater_queue_manager_pro... Running > Process > etcd_process Running > Process > clearwater_diags_monitor_pro... Running > Process > clearwater_config_manager_pr... Running > Process > clearwater_cluster_manager_p... Running > Process > nginx_ping Status ok > Program > nginx_uptime Status ok > Program > monit_uptime Status ok > Program > homestead_uptime Status ok > Program > poll_homestead Status ok > Program > check_cx_health Status ok > Program > poll_homestead-prov Status ok > Program > clearwater_queue_manager_uptime Status ok > Program > etcd_uptime Status ok > Program > poll_etcd_cluster Status ok > Program > poll_etcd Status ok > Program > > > [vellum]cwvellum@cwvellum:~$ sudo monit summary > Monit 5.18.1 uptime: 1d 20h 3m > Service Name Status > Type > node-cwvellum Running > System > snmpd_process Running > Process > ntp_process Running > Process > memcached_process Running > Process > clearwater_queue_manager_pro... Running > Process > etcd_process Running > Process > clearwater_diags_monitor_pro... Running > Process > clearwater_config_manager_pr... Running > Process > clearwater_cluster_manager_p... Running > Process > cassandra_process Running > Process > chronos_process Running > Process > astaire_process Running > Process > monit_uptime Status ok > Program > memcached_uptime Status ok > Program > poll_memcached Status ok > Program > clearwater_queue_manager_uptime Status ok > Program > etcd_uptime Status ok > Program > poll_etcd_cluster Status ok > Program > poll_etcd Status ok > Program > cassandra_uptime Status ok > Program > poll_cassandra Status ok > Program > poll_cqlsh Status ok > Program > chronos_uptime Status ok > Program > poll_chronos Status ok > Program > astaire_uptime Status ok > Program > > > ##################################################################### > > Please let me know if I am missing any configuration. > > Thanks > > Hrishikesh > > > > _______________________________________________ > Clearwater mailing list > [email protected] > http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists. > projectclearwater.org > > > > > > _______________________________________________ > Clearwater mailing list > [email protected] > http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists. > projectclearwater.org > >
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
