Hi, I have changed my deployment completely. No I am not using static IP addresses any more and each node is getting IP addresses from the DHCP server. Each node is able to ping one another using hostname cwellis, cwsprout, cwbono, cwvellum, cwhomer, cwdime. >From bono node I am able to ping, ssh to cwsprout. Check the logs,
=============================================================================== cwbono@cwbono:~$ ping cwsprout PING cwsprout.amcc.com (10.48.12.143) 56(84) bytes of data. 64 bytes from cwsprout.amcc.com (10.48.12.143): icmp_seq=1 ttl=64 time=0.126 ms 64 bytes from cwsprout.amcc.com (10.48.12.143): icmp_seq=2 ttl=64 time=0.203 ms 64 bytes from cwsprout.amcc.com (10.48.12.143): icmp_seq=3 ttl=64 time=0.210 ms cwbono@cwbono:~$ ssh cwsprout@cwsprout The authenticity of host 'cwsprout (10.48.12.143)' can't be established. ECDSA key fingerprint is 03:f8:42:81:36:ef:b1:be:7a:d8:3d:52:b5:74:f9:ba. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'cwsprout,10.48.12.143' (ECDSA) to the list of known hosts. cwsprout@cwsprout's password: Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64) * Documentation: https://help.ubuntu.com/ System information as of Thu Jul 13 14:08:07 IST 2017 System load: 0.12 Processes: 115 Usage of /: 27.7% of 7.26GB Users logged in: 0 Memory usage: 8% IP address for eth0: 10.48.12.143 Swap usage: 0% Graph this data and manage this system at: https://landscape.canonical.com/ New release '16.04.2 LTS' available. Run 'do-release-upgrade' to upgrade to it. Last login: Thu Jul 13 14:08:07 2017 from hdk-supermicro.amcc.com [sprout]cwsprout@cwsprout:~$ [sprout]cwsprout@cwsprout:~$ exit logout Connection to cwsprout closed. cwbono@cwbono:~$ cwbono@cwbono:~$ cwbono@cwbono:~$ cat /etc/clearwater/local_config local_ip=10.48.12.173 public_ip=10.48.12.173 public_hostname=cwbono etcd_cluster="10.48.12.142,10.48.12.143,10.48.12.173,10.48.12.140,10.48.12.139,10.48.12.120" cwbono@cwbono:~$ cat /etc/clearwater/shared_config ##################################################################### # No Shared Config has been provided # Replace this file with the Shared Configuration for your deployment ##################################################################### home_domain=example.com sprout_hostname=cwsprout sprout_registration_store=10.48.12.120 #vellum hs_hostname=10.48.12.139:8888 #dime hs_provisioning_hostname=10.48.12.139:8889 #dime ralf_hostname= ralf_session_store= xdms_hostname=10.48.12.140:7888 #homer chronos_hostname=vellum cassandra_hostname=10.48.12.120 #vellum # Email server configuration smtp_smarthost=localhost smtp_username=username smtp_password=password [email protected] # Keys signup_key=secret turn_workaround=secret ellis_api_key=secret ellis_cookie_key=secret cwbono@cwbono:~$ clearwater-etcdctl cluster-health member 2d821d7a0a7736b4 is healthy: got healthy result from http://10.48.12.142:4000 member 6201151ee7f99f5c is healthy: got healthy result from http://10.48.12.139:4000 member 895efb70c4b1b8b4 is healthy: got healthy result from http://10.48.12.120:4000 member c7f5d6485fb4735b is healthy: got healthy result from http://10.48.12.143:4000 member f7726a4e29ec7d3d is healthy: got healthy result from http://10.48.12.173:4000 member ffb968d2990c63f0 is healthy: got healthy result from http://10.48.12.140:4000 cluster is healthy cwbono@cwbono:~$ clearwater-etcdctl member list 2d821d7a0a7736b4: name=10-48-12-142 peerURLs=http://10.48.12.142:2380 clientURLs=http://10.48.12.142:4000 isLeader=false 6201151ee7f99f5c: name=10-48-12-139 peerURLs=http://10.48.12.139:2380 clientURLs=http://10.48.12.139:4000 isLeader=false 895efb70c4b1b8b4: name=10-48-12-120 peerURLs=http://10.48.12.120:2380 clientURLs=http://10.48.12.120:4000 isLeader=false c7f5d6485fb4735b: name=10-48-12-143 peerURLs=http://10.48.12.143:2380 clientURLs=http://10.48.12.143:4000 isLeader=false f7726a4e29ec7d3d: name=10-48-12-173 peerURLs=http://10.48.12.173:2380 clientURLs=http://10.48.12.173:4000 isLeader=true ffb968d2990c63f0: name=10-48-12-140 peerURLs=http://10.48.12.140:2380 clientURLs=http://10.48.12.140:4000 isLeader=false cwbono@cwbono:~$ cw-check_cluster_state This script prints the status of the Cassandra, Chronos, and Memcached clusters. This node (10.48.12.173) should not be in any cluster. Describing the Cassandra cluster: The cluster is stable 10.48.12.120 is in state normal Describing the Chronos cluster: The cluster is stable 10.48.12.120 is in state normal Describing the Memcached cluster: The cluster is stable 10.48.12.120 is in state normal cwbono@cwbono:~$ sudo cw-check_config_sync - /etc/clearwater/dns.json is up to date - /etc/clearwater/shared_config is up to date cwbono@cwbono:~$ sudo monit summary Monit 5.18.1 uptime: 54m Service Name Status Type node-cwbono Running System restund_process Execution failed | Does... Process ntp_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process bono_process Running Process poll_restund Wait parent Program monit_uptime Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program poll_bono Status ok Program cwbono@cwbono:~$ cat /var/log/bono/bono_current.txt 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 13-07-2017 09:34:34.680 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) cwbono@cwbono:~$ cat /var/log/monit.log turn: extended channels enabled tcp: sock_bind: bind: Address already in use [IST Jul 13 15:03:29] error : 'restund_process' process is not running [IST Jul 13 15:03:29] info : 'restund_process' trying to restart [IST Jul 13 15:03:29] info : 'restund_process' restart: /etc/init.d/restund [IST Jul 13 15:04:00] error : 'restund_process' failed to restart (exit status 0) -- /etc/init.d/restund: httpdb: configured url http://hs.example.com:8888/impi/%s/digest httpd: using URI workaround turn: server deployed behind static NAT addr=10.48.12.173:0 turn: extended channels enabled tcp: sock_bind: bind: Address already in use [IST Jul 13 15:04:10] error : 'restund_process' process is not running [IST Jul 13 15:04:10] info : 'restund_process' trying to restart [IST Jul 13 15:04:10] info : 'restund_process' restart: /etc/init.d/restund [IST Jul 13 15:04:40] error : 'restund_process' failed to restart (exit status 0) -- /etc/init.d/restund: httpdb: configured url http://hs.example.com:8888/impi/%s/digest httpd: using URI workaround turn: server deployed behind static NAT addr=10.48.12.173:0 turn: extended channels enabled tcp: sock_bind: bind: Address already in use [IST Jul 13 15:04:50] error : 'restund_process' process is not running [IST Jul 13 15:04:50] info : 'restund_process' trying to restart [IST Jul 13 15:04:50] info : 'restund_process' restart: /etc/init.d/restund [IST Jul 13 15:05:20] error : 'restund_process' failed to restart (exit status 0) -- /etc/init.d/restund: httpdb: configured url http://hs.example.com:8888/impi/%s/digest httpd: using URI workaround turn: server deployed behind static NAT addr=10.48.12.173:0 turn: extended channels enabled tcp: sock_bind: bind: Address already in use [IST Jul 13 15:05:30] error : 'restund_process' process is not running [IST Jul 13 15:05:30] info : 'restund_process' trying to restart [IST Jul 13 15:05:30] info : 'restund_process' restart: /etc/init.d/restund [IST Jul 13 15:06:01] error : 'restund_process' failed to restart (exit status 0) -- /etc/init.d/restund: httpdb: configured url http://hs.example.com:8888/impi/%s/digest httpd: using URI workaround turn: server deployed behind static NAT addr=10.48.12.173:0 turn: extended channels enabled tcp: sock_bind: bind: Address already in use [IST Jul 13 15:06:11] error : 'restund_process' process is not running [IST Jul 13 15:06:11] info : 'restund_process' trying to restart [IST Jul 13 15:06:11] info : 'restund_process' restart: /etc/init.d/restund [IST Jul 13 15:06:41] error : 'restund_process' failed to restart (exit status 0) -- /etc/init.d/restund: httpdb: configured url http://hs.example.com:8888/impi/%s/digest httpd: using URI workaround turn: server deployed behind static NAT addr=10.48.12.173:0 turn: extended channels enabled tcp: sock_bind: bind: Address already in use [IST Jul 13 15:06:51] error : 'restund_process' process is not running [IST Jul 13 15:06:51] info : 'restund_process' trying to restart [IST Jul 13 15:06:51] info : 'restund_process' restart: /etc/init.d/restund [IST Jul 13 15:07:21] error : 'restund_process' failed to restart (exit status 0) -- /etc/init.d/restund: httpdb: configured url http://hs.example.com:8888/impi/%s/digest httpd: using URI workaround turn: server deployed behind static NAT addr=10.48.12.173:0 turn: extended channels enabled tcp: sock_bind: bind: Address already in use [IST Jul 13 15:07:31] error : 'restund_process' process is not running [IST Jul 13 15:07:31] info : 'restund_process' trying to restart [IST Jul 13 15:07:31] info : 'restund_process' restart: /etc/init.d/restund =============================================================================== I don't know whats going wrong here. What is icscf.cwsprout? Thanks Hrishikesh On Tue, Jul 11, 2017 at 5:08 PM, Hrishikesh Karanjikar <[email protected]> wrote: > Hi, > > Thanks a lot for your reply. > > I am using virtualbox and host only network. > The DHCP server runs inside virtualbox and I can only configure the IP > address range. > However I am not using the DHCP server and assigning static IP addresses > to all nodes which are within the DHCP server IP address range. > I added entry of cwsprout in /ets/hosts of bono node as follows, > > =============================================== > > 127.0.0.1 localhost > 127.0.1.1 cwbono > > *192.168.56.103 cwsprout192.168.56.103 icscf.cwsprout* > > # The following lines are desirable for IPv6 capable hosts > ::1 localhost ip6-localhost ip6-loopback > ff02::1 ip6-allnodes > ff02::2 ip6-allrouters > 192.168.56.103 cwsprout > 192.168.56.103 icscf.cwsprout > ::1 localhost # added by clearwater-infrastructure 1hosts script > 192.168.56.104 cwbono #+clearwater-infrastructure > > =============================================== > > I am also able to ping cwsprout and icscf.cwsprout from bono, > Here is the log, > > =============================================== > > [bono]cwbono@cwbono:~$ ping cwsprout > PING cwsprout (192.168.56.103) 56(84) bytes of data. > 64 bytes from cwsprout (192.168.56.103): icmp_seq=1 ttl=64 time=0.197 ms > 64 bytes from cwsprout (192.168.56.103): icmp_seq=2 ttl=64 time=0.217 ms > ^C > --- cwsprout ping statistics --- > 2 packets transmitted, 2 received, 0% packet loss, time 999ms > rtt min/avg/max/mdev = 0.197/0.207/0.217/0.010 ms > [bono]cwbono@cwbono:~$ > [bono]cwbono@cwbono:~$ > [bono]cwbono@cwbono:~$ > [bono]cwbono@cwbono:~$ ping icscf.cwsprout > PING icscf.cwsprout (192.168.56.103) 56(84) bytes of data. > 64 bytes from cwsprout (192.168.56.103): icmp_seq=1 ttl=64 time=0.112 ms > 64 bytes from cwsprout (192.168.56.103): icmp_seq=2 ttl=64 time=0.165 ms > > =============================================== > > I am using static IP addresses as I have to specify them in local_config > file of each node. > If I use DHCP server of VirtualBox they might change. In that case I am > not sure how do I cope up with local_config. > Can I modify local_config after all nodes are up? > > Thanks > Hrishikesh > > > > On Tue, Jul 11, 2017 at 2:42 PM, Andrew Edmonds < > [email protected]> wrote: > >> Hi Hrishikesh, >> >> >> >> Thank you for your question and the detailed logs you have provided. >> >> >> >> The issues appears to be caused by your shared config, in the manual >> installation instructions >> <http://clearwater.readthedocs.io/en/stable/Manual_Install.html> you’ll >> see that we advise that the entries in shared config have a format like: >> >> >> >> sprout_hostname*=*sprout*.<*site_name*>.<*zone*>* >> >> sprout_registration_store*=*vellum*.<*site_name*>.<*zone*>* >> >> hs_hostname*=*hs*.<*site_name*>.<*zone*>*:8888 >> >> >> >> In your shared config you have: >> >> >> >> sprout_hostname=cwsprout >> sprout_registration_store=192.168.56.107 #vellum >> >> I don’t think the sprout_hostname used here will resolve, we can see >> evidence for this in the Bono logs, Bono attempts to resolve >> icscf.<sprout_hostname> to find which location to forward SIP messages on >> you, you can see it is failing to do that here: >> >> >> >> Failed to resolve icscf.cwsprout to an IP address - Not found >> (PJ_ENOTFOUND) >> >> >> >> To resolve this issue you should change your shared config to use >> hostnames which have been configured in your DNS server to resolve to the >> appropriate location. Once you have done this run the command “sudo >> cw-upload_shared_config”. Please let me know if this does not resolve the >> issue. >> >> >> >> Thanks, >> >> >> >> Andrew >> >> >> >> >> >> *From:* Clearwater [mailto:[email protected]] >> *On Behalf Of *Hrishikesh Karanjikar >> *Sent:* Thursday, July 6, 2017 10:22 AM >> *To:* [email protected] >> *Subject:* [Project Clearwater] restund_process Execution failed on bono >> >> >> >> Hello, >> >> I have Manually installed all 6 nodes on VMs using virtualbox. >> >> I followed the procedure given @ http://clearwater.readthedocs. >> io/en/stable/Manual_Install.html >> >> Looks like all nodes except Dime are running fine. >> >> I am getting error "restund_process Execution failed" in monit summary on >> Bono node. >> >> Here is the shared and local config file, >> >> ##################################################################### >> >> [bono]cwbono@cwbono:~$ cat /etc/clearwater/shared_config >> >> home_domain=example.com >> sprout_hostname=cwsprout >> sprout_registration_store=192.168.56.107 #vellum >> hs_hostname=192.168.56.106:8888 #dime >> hs_provisioning_hostname=192.168.56.106:8889 #dime >> ralf_hostname= >> ralf_session_store= >> xdms_hostname=192.168.56.105:7888 #homer >> chronos_hostname=192.168.56.107 #vellum >> cassandra_hostname=192.168.56.107 #vellum >> >> # Email server configuration >> smtp_smarthost=localhost >> smtp_username=username >> smtp_password=password >> [email protected] >> >> # Keys >> signup_key=secret >> turn_workaround=secret >> ellis_api_key=secret >> ellis_cookie_key=secret >> [bono]cwbono@cwbono:~$ cat /etc/clearwater/local_config >> local_ip=192.168.56.104 >> public_ip=192.168.56.104 >> public_hostname=cwbono >> etcd_cluster="192.168.56.102,192.168.56.103,192.168.56.104,1 >> 92.168.56.105,192.168.56.106,192.168.56.107" >> >> ##################################################################### >> >> The logs for bono node are as follows, >> >> ##################################################################### >> [bono]cwbono@cwbono:~$ sudo monit summary >> [sudo] password for cwbono: >> Monit 5.18.1 uptime: 2d 16h 39m >> Service Name Status >> Type >> node-cwbono Running >> System >> restund_process Execution failed | Does... >> Process >> ntp_process Running >> Process >> clearwater_queue_manager_pro... Running >> Process >> etcd_process Running >> Process >> clearwater_diags_monitor_pro... Running >> Process >> clearwater_config_manager_pr... Running >> Process >> clearwater_cluster_manager_p... Running >> Process >> bono_process Running >> Process >> poll_restund Wait parent >> Program >> monit_uptime Status ok >> Program >> clearwater_queue_manager_uptime Status ok >> Program >> etcd_uptime Status ok >> Program >> poll_etcd_cluster Status ok >> Program >> poll_etcd Status ok >> Program >> poll_bono Status ok >> Program >> >> >> 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:15.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:15.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:15.843 UTC Status main.cpp:1358: Quiesce signal received >> 05-07-2017 11:30:15.843 UTC Status stack.cpp:125: Setting quiescing = >> PJ_TRUE >> 05-07-2017 11:30:15.851 UTC Status stack.cpp:156: Quiescing state changed >> 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The >> Quiescing Manager received input QUIESCE (0) when in state ACTIVE (0) >> 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:265: Close >> untrusted listening port >> 05-07-2017 11:30:15.851 UTC Status stack.cpp:368: Destroyed TCP transport >> for port 5060 >> 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:273: Quiesce >> FlowTable >> 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The >> Quiescing Manager received input FLOWS_GONE (1) when in state >> QUIESCING_FLOWS (1) >> 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:290: Closing >> trusted port >> 05-07-2017 11:30:15.851 UTC Status stack.cpp:368: Destroyed TCP transport >> for port 5058 >> 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:296: Quiescing >> all connections >> 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:162: Start >> quiescing connections >> 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:175: Quiescing >> 0 transactions >> 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:180: Connection >> quiescing complete >> 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The >> Quiescing Manager received input CONNS_GONE (2) when in state >> QUIESCING_CONNS (2) >> 05-07-2017 11:30:15.851 UTC Status main.cpp:1380: Quiesce complete >> 05-07-2017 11:30:15.853 UTC Status stack.cpp:171: PJSIP thread ended >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to >> resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) >> >> [bono]cwbono@cwbono:~$ cat /var/log/monit.log >> >> httpd: using URI workaround >> turn: server deployed behind static NAT addr=192.168.56.104:0 >> turn: extended channels enabled >> tcp: sock_bind: bind: Address already in u >> [IST Jul 6 14:47:46] error : 'restund_process' process is not running >> [IST Jul 6 14:47:46] info : 'restund_process' trying to restart >> [IST Jul 6 14:47:46] info : 'restund_process' restart: >> /etc/init.d/restund >> [IST Jul 6 14:48:16] error : 'restund_process' failed to restart >> (exit status 0) -- /etc/init.d/restund: httpdb: configured url >> http://hs.example.com:8888/impi/%s/digest >> httpd: using URI workaround >> turn: server deployed behind static NAT addr=192.168.56.104:0 >> turn: extended channels enabled >> tcp: sock_bind: bind: Address already in u >> [IST Jul 6 14:48:26] error : 'restund_process' process is not running >> [IST Jul 6 14:48:26] info : 'restund_process' trying to restart >> [IST Jul 6 14:48:26] info : 'restund_process' restart: >> /etc/init.d/restund >> [IST Jul 6 14:48:56] error : 'restund_process' failed to restart >> (exit status 0) -- /etc/init.d/restund: httpdb: configured url >> http://hs.example.com:8888/impi/%s/digest >> httpd: using URI workaround >> turn: server deployed behind static NAT addr=192.168.56.104:0 >> turn: extended channels enabled >> tcp: sock_bind: bind: Address already in u >> [IST Jul 6 14:49:06] error : 'restund_process' process is not running >> [IST Jul 6 14:49:06] info : 'restund_process' trying to restart >> [IST Jul 6 14:49:06] info : 'restund_process' restart: >> /etc/init.d/restund >> >> >> [bono]cwbono@cwbono:~$ clearwater-etcdctl cluster-health >> member 9c1928228d308a0f is healthy: got healthy result from >> http://192.168.56.107:4000 >> member b0c9c017e0d47e14 is healthy: got healthy result from >> http://192.168.56.106:4000 >> member d44832212a08c43f is healthy: got healthy result from >> http://192.168.56.103:4000 >> member ef1a9a8a2fd05283 is healthy: got healthy result from >> http://192.168.56.104:4000 >> member f63afbe816fb463d is healthy: got healthy result from >> http://192.168.56.102:4000 >> member f7132cc88f7a39fa is healthy: got healthy result from >> http://192.168.56.105:4000 >> cluster is healthy >> [bono]cwbono@cwbono:~$ cw-check_cluster_state >> This script prints out the status of the Chronos, Memcached and Cassandra >> clusters. >> >> Describing the Vellum Chronos cluster: >> The local node is *not* in this cluster >> The cluster is stable >> 192.168.56.107 is in state normal >> >> Describing the Vellum Memcached cluster: >> The local node is *not* in this cluster >> The cluster is stable >> 192.168.56.107 is in state normal >> >> Describing the Vellum Cassandra cluster: >> The local node is *not* in this cluster >> The cluster is stable >> 192.168.56.107 is in state normal >> >> [bono]cwbono@cwbono:~$ clearwater-etcdctl member list >> 9c1928228d308a0f: name=192-168-56-107 peerURLs=http://192.168.56.107:2380 >> clientURLs=http://192.168.56.107:4000 isLeader=false >> b0c9c017e0d47e14: name=192-168-56-106 peerURLs=http://192.168.56.106:2380 >> clientURLs=http://192.168.56.106:4000 isLeader=true >> d44832212a08c43f: name=192-168-56-103 peerURLs=http://192.168.56.103:2380 >> clientURLs=http://192.168.56.103:4000 isLeader=false >> ef1a9a8a2fd05283: name=192-168-56-104 peerURLs=http://192.168.56.104:2380 >> clientURLs=http://192.168.56.104:4000 isLeader=false >> f63afbe816fb463d: name=192-168-56-102 peerURLs=http://192.168.56.102:2380 >> clientURLs=http://192.168.56.102:4000 isLeader=false >> f7132cc88f7a39fa: name=192-168-56-105 peerURLs=http://192.168.56.105:2380 >> clientURLs=http://192.168.56.105:4000 isLeader=false >> [bono]cwbono@cwbono:~$ sudo cw-check_config_sync >> [sudo] password for cwbono: >> - /etc/clearwater/dns.json is up to date >> - /etc/clearwater/shared_config is up to date >> >> >> ##################################################################### >> >> The logs for other nodes are as follows >> >> ##################################################################### >> >> [ellis]cwellis@cwellis:~$ sudo monit summary >> Monit 5.18.1 uptime: 1d 21h 15m >> Service Name Status >> Type >> node-cwellis Running >> System >> ntp_process Running >> Process >> nginx_process Running >> Process >> mysql_process Running >> Process >> ellis_process Running >> Process >> clearwater_queue_manager_pro... Running >> Process >> etcd_process Running >> Process >> clearwater_diags_monitor_pro... Running >> Process >> clearwater_config_manager_pr... Running >> Process >> clearwater_cluster_manager_p... Running >> Process >> nginx_ping Status ok >> Program >> nginx_uptime Status ok >> Program >> monit_uptime Status ok >> Program >> poll_ellis Status ok >> Program >> poll_ellis_https Status ok >> Program >> clearwater_queue_manager_uptime Status ok >> Program >> etcd_uptime Status ok >> Program >> poll_etcd_cluster Status ok >> Program >> poll_etcd Status ok >> Program >> >> >> [sprout]cwsprout@cwsprout:~$ sudo monit summary >> Monit 5.18.1 uptime: 1d 20h 7m >> Service Name Status >> Type >> node-cwsprout Running >> System >> sprout_process Running >> Process >> ntp_process Running >> Process >> nginx_process Running >> Process >> memento_process Running >> Process >> clearwater_queue_manager_pro... Running >> Process >> etcd_process Running >> Process >> clearwater_diags_monitor_pro... Running >> Process >> clearwater_config_manager_pr... Running >> Process >> clearwater_cluster_manager_p... Running >> Process >> sprout_uptime Status ok >> Program >> poll_sprout_sip Status ok >> Program >> poll_sprout_http Status ok >> Program >> nginx_ping Status ok >> Program >> nginx_uptime Status ok >> Program >> monit_uptime Status ok >> Program >> memento_uptime Status ok >> Program >> poll_memento Status ok >> Program >> poll_memento_https Status ok >> Program >> clearwater_queue_manager_uptime Status ok >> Program >> etcd_uptime Status ok >> Program >> poll_etcd_cluster Status ok >> Program >> poll_etcd Status ok >> Program >> >> >> [homer]cwhomer@cwhomer:~$ sudo monit summary >> Monit 5.18.1 uptime: 1d 20h 2m >> Service Name Status >> Type >> node-cwhomer Running >> System >> ntp_process Running >> Process >> nginx_process Running >> Process >> homer_process Running >> Process >> clearwater_queue_manager_pro... Running >> Process >> etcd_process Running >> Process >> clearwater_diags_monitor_pro... Running >> Process >> clearwater_config_manager_pr... Running >> Process >> clearwater_cluster_manager_p... Running >> Process >> nginx_ping Status ok >> Program >> nginx_uptime Status ok >> Program >> monit_uptime Status ok >> Program >> poll_homer Status ok >> Program >> clearwater_queue_manager_uptime Status ok >> Program >> etcd_uptime Status ok >> Program >> poll_etcd_cluster Status ok >> Program >> poll_etcd Status ok Program >> >> [dime]cwdime@cwdime:~$ sudo monit summary >> Monit 5.18.1 uptime: 1d 20h 2m >> Service Name Status >> Type >> node-cwdime Running >> System >> snmpd_process Running >> Process >> ntp_process Running >> Process >> nginx_process Running >> Process >> homestead_process Running >> Process >> homestead-prov_process Running >> Process >> clearwater_queue_manager_pro... Running >> Process >> etcd_process Running >> Process >> clearwater_diags_monitor_pro... Running >> Process >> clearwater_config_manager_pr... Running >> Process >> clearwater_cluster_manager_p... Running >> Process >> nginx_ping Status ok >> Program >> nginx_uptime Status ok >> Program >> monit_uptime Status ok >> Program >> homestead_uptime Status ok >> Program >> poll_homestead Status ok >> Program >> check_cx_health Status ok >> Program >> poll_homestead-prov Status ok >> Program >> clearwater_queue_manager_uptime Status ok >> Program >> etcd_uptime Status ok >> Program >> poll_etcd_cluster Status ok >> Program >> poll_etcd Status ok >> Program >> >> >> [vellum]cwvellum@cwvellum:~$ sudo monit summary >> Monit 5.18.1 uptime: 1d 20h 3m >> Service Name Status >> Type >> node-cwvellum Running >> System >> snmpd_process Running >> Process >> ntp_process Running >> Process >> memcached_process Running >> Process >> clearwater_queue_manager_pro... Running >> Process >> etcd_process Running >> Process >> clearwater_diags_monitor_pro... Running >> Process >> clearwater_config_manager_pr... Running >> Process >> clearwater_cluster_manager_p... Running >> Process >> cassandra_process Running >> Process >> chronos_process Running >> Process >> astaire_process Running >> Process >> monit_uptime Status ok >> Program >> memcached_uptime Status ok >> Program >> poll_memcached Status ok >> Program >> clearwater_queue_manager_uptime Status ok >> Program >> etcd_uptime Status ok >> Program >> poll_etcd_cluster Status ok >> Program >> poll_etcd Status ok >> Program >> cassandra_uptime Status ok >> Program >> poll_cassandra Status ok >> Program >> poll_cqlsh Status ok >> Program >> chronos_uptime Status ok >> Program >> poll_chronos Status ok >> Program >> astaire_uptime Status ok >> Program >> >> >> ##################################################################### >> >> Please let me know if I am missing any configuration. >> >> Thanks >> >> Hrishikesh >> >> _______________________________________________ >> Clearwater mailing list >> [email protected] >> http://lists.projectclearwater.org/mailman/listinfo/ >> clearwater_lists.projectclearwater.org >> >> >
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
