Hi Hrishikesh, Thank you for your question and the detailed logs you have provided.
The issues appears to be caused by your shared config, in the manual installation instructions<http://clearwater.readthedocs.io/en/stable/Manual_Install.html> you’ll see that we advise that the entries in shared config have a format like: sprout_hostname=sprout.<site_name>.<zone> sprout_registration_store=vellum.<site_name>.<zone> hs_hostname=hs.<site_name>.<zone>:8888 In your shared config you have: sprout_hostname=cwsprout sprout_registration_store=192.168.56.107 #vellum I don’t think the sprout_hostname used here will resolve, we can see evidence for this in the Bono logs, Bono attempts to resolve icscf.<sprout_hostname> to find which location to forward SIP messages on you, you can see it is failing to do that here: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) To resolve this issue you should change your shared config to use hostnames which have been configured in your DNS server to resolve to the appropriate location. Once you have done this run the command “sudo cw-upload_shared_config”. Please let me know if this does not resolve the issue. Thanks, Andrew From: Clearwater [mailto:[email protected]] On Behalf Of Hrishikesh Karanjikar Sent: Thursday, July 6, 2017 10:22 AM To: [email protected] Subject: [Project Clearwater] restund_process Execution failed on bono Hello, I have Manually installed all 6 nodes on VMs using virtualbox. I followed the procedure given @ http://clearwater.readthedocs.io/en/stable/Manual_Install.html Looks like all nodes except Dime are running fine. I am getting error "restund_process Execution failed" in monit summary on Bono node. Here is the shared and local config file, ##################################################################### [bono]cwbono@cwbono:~$ cat /etc/clearwater/shared_config home_domain=example.com<http://example.com> sprout_hostname=cwsprout sprout_registration_store=192.168.56.107 #vellum hs_hostname=192.168.56.106:8888<http://192.168.56.106:8888> #dime hs_provisioning_hostname=192.168.56.106:8889<http://192.168.56.106:8889> #dime ralf_hostname= ralf_session_store= xdms_hostname=192.168.56.105:7888<http://192.168.56.105:7888> #homer chronos_hostname=192.168.56.107 #vellum cassandra_hostname=192.168.56.107 #vellum # Email server configuration smtp_smarthost=localhost smtp_username=username smtp_password=password [email protected]<mailto:[email protected]> # Keys signup_key=secret turn_workaround=secret ellis_api_key=secret ellis_cookie_key=secret [bono]cwbono@cwbono:~$ cat /etc/clearwater/local_config local_ip=192.168.56.104 public_ip=192.168.56.104 public_hostname=cwbono etcd_cluster="192.168.56.102,192.168.56.103,192.168.56.104,192.168.56.105,192.168.56.106,192.168.56.107" ##################################################################### The logs for bono node are as follows, ##################################################################### [bono]cwbono@cwbono:~$ sudo monit summary [sudo] password for cwbono: Monit 5.18.1 uptime: 2d 16h 39m Service Name Status Type node-cwbono Running System restund_process Execution failed | Does... Process ntp_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process bono_process Running Process poll_restund Wait parent Program monit_uptime Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program poll_bono Status ok Program 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.843 UTC Status main.cpp:1358: Quiesce signal received 05-07-2017 11:30:15.843 UTC Status stack.cpp:125: Setting quiescing = PJ_TRUE 05-07-2017 11:30:15.851 UTC Status stack.cpp:156: Quiescing state changed 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The Quiescing Manager received input QUIESCE (0) when in state ACTIVE (0) 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:265: Close untrusted listening port 05-07-2017 11:30:15.851 UTC Status stack.cpp:368: Destroyed TCP transport for port 5060 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:273: Quiesce FlowTable 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The Quiescing Manager received input FLOWS_GONE (1) when in state QUIESCING_FLOWS (1) 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:290: Closing trusted port 05-07-2017 11:30:15.851 UTC Status stack.cpp:368: Destroyed TCP transport for port 5058 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:296: Quiescing all connections 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:162: Start quiescing connections 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:175: Quiescing 0 transactions 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:180: Connection quiescing complete 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The Quiescing Manager received input CONNS_GONE (2) when in state QUIESCING_CONNS (2) 05-07-2017 11:30:15.851 UTC Status main.cpp:1380: Quiesce complete 05-07-2017 11:30:15.853 UTC Status stack.cpp:171: PJSIP thread ended 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) [bono]cwbono@cwbono:~$ cat /var/log/monit.log httpd: using URI workaround turn: server deployed behind static NAT addr=192.168.56.104:0<http://192.168.56.104:0> turn: extended channels enabled tcp: sock_bind: bind: Address already in u [IST Jul 6 14:47:46] error : 'restund_process' process is not running [IST Jul 6 14:47:46] info : 'restund_process' trying to restart [IST Jul 6 14:47:46] info : 'restund_process' restart: /etc/init.d/restund [IST Jul 6 14:48:16] error : 'restund_process' failed to restart (exit status 0) -- /etc/init.d/restund: httpdb: configured url http://hs.example.com:8888/impi/%s/digest<http://hs.example.com:8888/impi/%25s/digest> httpd: using URI workaround turn: server deployed behind static NAT addr=192.168.56.104:0<http://192.168.56.104:0> turn: extended channels enabled tcp: sock_bind: bind: Address already in u [IST Jul 6 14:48:26] error : 'restund_process' process is not running [IST Jul 6 14:48:26] info : 'restund_process' trying to restart [IST Jul 6 14:48:26] info : 'restund_process' restart: /etc/init.d/restund [IST Jul 6 14:48:56] error : 'restund_process' failed to restart (exit status 0) -- /etc/init.d/restund: httpdb: configured url http://hs.example.com:8888/impi/%s/digest<http://hs.example.com:8888/impi/%25s/digest> httpd: using URI workaround turn: server deployed behind static NAT addr=192.168.56.104:0<http://192.168.56.104:0> turn: extended channels enabled tcp: sock_bind: bind: Address already in u [IST Jul 6 14:49:06] error : 'restund_process' process is not running [IST Jul 6 14:49:06] info : 'restund_process' trying to restart [IST Jul 6 14:49:06] info : 'restund_process' restart: /etc/init.d/restund [bono]cwbono@cwbono:~$ clearwater-etcdctl cluster-health member 9c1928228d308a0f is healthy: got healthy result from http://192.168.56.107:4000 member b0c9c017e0d47e14 is healthy: got healthy result from http://192.168.56.106:4000 member d44832212a08c43f is healthy: got healthy result from http://192.168.56.103:4000 member ef1a9a8a2fd05283 is healthy: got healthy result from http://192.168.56.104:4000 member f63afbe816fb463d is healthy: got healthy result from http://192.168.56.102:4000 member f7132cc88f7a39fa is healthy: got healthy result from http://192.168.56.105:4000 cluster is healthy [bono]cwbono@cwbono:~$ cw-check_cluster_state This script prints out the status of the Chronos, Memcached and Cassandra clusters. Describing the Vellum Chronos cluster: The local node is *not* in this cluster The cluster is stable 192.168.56.107 is in state normal Describing the Vellum Memcached cluster: The local node is *not* in this cluster The cluster is stable 192.168.56.107 is in state normal Describing the Vellum Cassandra cluster: The local node is *not* in this cluster The cluster is stable 192.168.56.107 is in state normal [bono]cwbono@cwbono:~$ clearwater-etcdctl member list 9c1928228d308a0f: name=192-168-56-107 peerURLs=http://192.168.56.107:2380 clientURLs=http://192.168.56.107:4000 isLeader=false b0c9c017e0d47e14: name=192-168-56-106 peerURLs=http://192.168.56.106:2380 clientURLs=http://192.168.56.106:4000 isLeader=true d44832212a08c43f: name=192-168-56-103 peerURLs=http://192.168.56.103:2380 clientURLs=http://192.168.56.103:4000 isLeader=false ef1a9a8a2fd05283: name=192-168-56-104 peerURLs=http://192.168.56.104:2380 clientURLs=http://192.168.56.104:4000 isLeader=false f63afbe816fb463d: name=192-168-56-102 peerURLs=http://192.168.56.102:2380 clientURLs=http://192.168.56.102:4000 isLeader=false f7132cc88f7a39fa: name=192-168-56-105 peerURLs=http://192.168.56.105:2380 clientURLs=http://192.168.56.105:4000 isLeader=false [bono]cwbono@cwbono:~$ sudo cw-check_config_sync [sudo] password for cwbono: - /etc/clearwater/dns.json is up to date - /etc/clearwater/shared_config is up to date ##################################################################### The logs for other nodes are as follows ##################################################################### [ellis]cwellis@cwellis:~$ sudo monit summary Monit 5.18.1 uptime: 1d 21h 15m Service Name Status Type node-cwellis Running System ntp_process Running Process nginx_process Running Process mysql_process Running Process ellis_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program poll_ellis Status ok Program poll_ellis_https Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program [sprout]cwsprout@cwsprout:~$ sudo monit summary Monit 5.18.1 uptime: 1d 20h 7m Service Name Status Type node-cwsprout Running System sprout_process Running Process ntp_process Running Process nginx_process Running Process memento_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process sprout_uptime Status ok Program poll_sprout_sip Status ok Program poll_sprout_http Status ok Program nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program memento_uptime Status ok Program poll_memento Status ok Program poll_memento_https Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program [homer]cwhomer@cwhomer:~$ sudo monit summary Monit 5.18.1 uptime: 1d 20h 2m Service Name Status Type node-cwhomer Running System ntp_process Running Process nginx_process Running Process homer_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program poll_homer Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program [dime]cwdime@cwdime:~$ sudo monit summary Monit 5.18.1 uptime: 1d 20h 2m Service Name Status Type node-cwdime Running System snmpd_process Running Process ntp_process Running Process nginx_process Running Process homestead_process Running Process homestead-prov_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program homestead_uptime Status ok Program poll_homestead Status ok Program check_cx_health Status ok Program poll_homestead-prov Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program [vellum]cwvellum@cwvellum:~$ sudo monit summary Monit 5.18.1 uptime: 1d 20h 3m Service Name Status Type node-cwvellum Running System snmpd_process Running Process ntp_process Running Process memcached_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process cassandra_process Running Process chronos_process Running Process astaire_process Running Process monit_uptime Status ok Program memcached_uptime Status ok Program poll_memcached Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program cassandra_uptime Status ok Program poll_cassandra Status ok Program poll_cqlsh Status ok Program chronos_uptime Status ok Program poll_chronos Status ok Program astaire_uptime Status ok Program ##################################################################### Please let me know if I am missing any configuration. Thanks Hrishikesh
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
