Hello, I have Manually installed all 6 nodes on VMs using virtualbox. I followed the procedure given @ http://clearwater.readthedocs.io/en/stable/Manual_Install.html Looks like all nodes except Dime are running fine. I am getting error "restund_process Execution failed" in monit summary on Bono node.
Here is the shared and local config file, ##################################################################### [bono]cwbono@cwbono:~$ cat /etc/clearwater/shared_config home_domain=example.com sprout_hostname=cwsprout sprout_registration_store=192.168.56.107 #vellum hs_hostname=192.168.56.106:8888 #dime hs_provisioning_hostname=192.168.56.106:8889 #dime ralf_hostname= ralf_session_store= xdms_hostname=192.168.56.105:7888 #homer chronos_hostname=192.168.56.107 #vellum cassandra_hostname=192.168.56.107 #vellum # Email server configuration smtp_smarthost=localhost smtp_username=username smtp_password=password [email protected] # Keys signup_key=secret turn_workaround=secret ellis_api_key=secret ellis_cookie_key=secret [bono]cwbono@cwbono:~$ cat /etc/clearwater/local_config local_ip=192.168.56.104 public_ip=192.168.56.104 public_hostname=cwbono etcd_cluster="192.168.56.102,192.168.56.103,192.168.56.104,192.168.56.105,192.168.56.106,192.168.56.107" ##################################################################### The logs for bono node are as follows, ##################################################################### [bono]cwbono@cwbono:~$ sudo monit summary [sudo] password for cwbono: Monit 5.18.1 uptime: 2d 16h 39m Service Name Status Type node-cwbono Running System restund_process Execution failed | Does... Process ntp_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process bono_process Running Process poll_restund Wait parent Program monit_uptime Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program poll_bono Status ok Program 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.748 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:15.843 UTC Status main.cpp:1358: Quiesce signal received 05-07-2017 11:30:15.843 UTC Status stack.cpp:125: Setting quiescing = PJ_TRUE 05-07-2017 11:30:15.851 UTC Status stack.cpp:156: Quiescing state changed 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The Quiescing Manager received input QUIESCE (0) when in state ACTIVE (0) 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:265: Close untrusted listening port 05-07-2017 11:30:15.851 UTC Status stack.cpp:368: Destroyed TCP transport for port 5060 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:273: Quiesce FlowTable 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The Quiescing Manager received input FLOWS_GONE (1) when in state QUIESCING_FLOWS (1) 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:290: Closing trusted port 05-07-2017 11:30:15.851 UTC Status stack.cpp:368: Destroyed TCP transport for port 5058 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:296: Quiescing all connections 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:162: Start quiescing connections 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:175: Quiescing 0 transactions 05-07-2017 11:30:15.851 UTC Status connection_tracker.cpp:180: Connection quiescing complete 05-07-2017 11:30:15.851 UTC Status quiescing_manager.cpp:139: The Quiescing Manager received input CONNS_GONE (2) when in state QUIESCING_CONNS (2) 05-07-2017 11:30:15.851 UTC Status main.cpp:1380: Quiesce complete 05-07-2017 11:30:15.853 UTC Status stack.cpp:171: PJSIP thread ended 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) 05-07-2017 11:30:16.749 UTC Error sip_connection_pool.cpp:189: Failed to resolve icscf.cwsprout to an IP address - Not found (PJ_ENOTFOUND) [bono]cwbono@cwbono:~$ cat /var/log/monit.log httpd: using URI workaround turn: server deployed behind static NAT addr=192.168.56.104:0 turn: extended channels enabled tcp: sock_bind: bind: Address already in u [IST Jul 6 14:47:46] error : 'restund_process' process is not running [IST Jul 6 14:47:46] info : 'restund_process' trying to restart [IST Jul 6 14:47:46] info : 'restund_process' restart: /etc/init.d/restund [IST Jul 6 14:48:16] error : 'restund_process' failed to restart (exit status 0) -- /etc/init.d/restund: httpdb: configured url http://hs.example.com:8888/impi/%s/digest httpd: using URI workaround turn: server deployed behind static NAT addr=192.168.56.104:0 turn: extended channels enabled tcp: sock_bind: bind: Address already in u [IST Jul 6 14:48:26] error : 'restund_process' process is not running [IST Jul 6 14:48:26] info : 'restund_process' trying to restart [IST Jul 6 14:48:26] info : 'restund_process' restart: /etc/init.d/restund [IST Jul 6 14:48:56] error : 'restund_process' failed to restart (exit status 0) -- /etc/init.d/restund: httpdb: configured url http://hs.example.com:8888/impi/%s/digest httpd: using URI workaround turn: server deployed behind static NAT addr=192.168.56.104:0 turn: extended channels enabled tcp: sock_bind: bind: Address already in u [IST Jul 6 14:49:06] error : 'restund_process' process is not running [IST Jul 6 14:49:06] info : 'restund_process' trying to restart [IST Jul 6 14:49:06] info : 'restund_process' restart: /etc/init.d/restund [bono]cwbono@cwbono:~$ clearwater-etcdctl cluster-health member 9c1928228d308a0f is healthy: got healthy result from http://192.168.56.107:4000 member b0c9c017e0d47e14 is healthy: got healthy result from http://192.168.56.106:4000 member d44832212a08c43f is healthy: got healthy result from http://192.168.56.103:4000 member ef1a9a8a2fd05283 is healthy: got healthy result from http://192.168.56.104:4000 member f63afbe816fb463d is healthy: got healthy result from http://192.168.56.102:4000 member f7132cc88f7a39fa is healthy: got healthy result from http://192.168.56.105:4000 cluster is healthy [bono]cwbono@cwbono:~$ cw-check_cluster_state This script prints out the status of the Chronos, Memcached and Cassandra clusters. Describing the Vellum Chronos cluster: The local node is *not* in this cluster The cluster is stable 192.168.56.107 is in state normal Describing the Vellum Memcached cluster: The local node is *not* in this cluster The cluster is stable 192.168.56.107 is in state normal Describing the Vellum Cassandra cluster: The local node is *not* in this cluster The cluster is stable 192.168.56.107 is in state normal [bono]cwbono@cwbono:~$ clearwater-etcdctl member list 9c1928228d308a0f: name=192-168-56-107 peerURLs=http://192.168.56.107:2380 clientURLs=http://192.168.56.107:4000 isLeader=false b0c9c017e0d47e14: name=192-168-56-106 peerURLs=http://192.168.56.106:2380 clientURLs=http://192.168.56.106:4000 isLeader=true d44832212a08c43f: name=192-168-56-103 peerURLs=http://192.168.56.103:2380 clientURLs=http://192.168.56.103:4000 isLeader=false ef1a9a8a2fd05283: name=192-168-56-104 peerURLs=http://192.168.56.104:2380 clientURLs=http://192.168.56.104:4000 isLeader=false f63afbe816fb463d: name=192-168-56-102 peerURLs=http://192.168.56.102:2380 clientURLs=http://192.168.56.102:4000 isLeader=false f7132cc88f7a39fa: name=192-168-56-105 peerURLs=http://192.168.56.105:2380 clientURLs=http://192.168.56.105:4000 isLeader=false [bono]cwbono@cwbono:~$ sudo cw-check_config_sync [sudo] password for cwbono: - /etc/clearwater/dns.json is up to date - /etc/clearwater/shared_config is up to date ##################################################################### The logs for other nodes are as follows ##################################################################### [ellis]cwellis@cwellis:~$ sudo monit summary Monit 5.18.1 uptime: 1d 21h 15m Service Name Status Type node-cwellis Running System ntp_process Running Process nginx_process Running Process mysql_process Running Process ellis_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program poll_ellis Status ok Program poll_ellis_https Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program [sprout]cwsprout@cwsprout:~$ sudo monit summary Monit 5.18.1 uptime: 1d 20h 7m Service Name Status Type node-cwsprout Running System sprout_process Running Process ntp_process Running Process nginx_process Running Process memento_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process sprout_uptime Status ok Program poll_sprout_sip Status ok Program poll_sprout_http Status ok Program nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program memento_uptime Status ok Program poll_memento Status ok Program poll_memento_https Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program [homer]cwhomer@cwhomer:~$ sudo monit summary Monit 5.18.1 uptime: 1d 20h 2m Service Name Status Type node-cwhomer Running System ntp_process Running Process nginx_process Running Process homer_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program poll_homer Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program [dime]cwdime@cwdime:~$ sudo monit summary Monit 5.18.1 uptime: 1d 20h 2m Service Name Status Type node-cwdime Running System snmpd_process Running Process ntp_process Running Process nginx_process Running Process homestead_process Running Process homestead-prov_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program homestead_uptime Status ok Program poll_homestead Status ok Program check_cx_health Status ok Program poll_homestead-prov Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program [vellum]cwvellum@cwvellum:~$ sudo monit summary Monit 5.18.1 uptime: 1d 20h 3m Service Name Status Type node-cwvellum Running System snmpd_process Running Process ntp_process Running Process memcached_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process cassandra_process Running Process chronos_process Running Process astaire_process Running Process monit_uptime Status ok Program memcached_uptime Status ok Program poll_memcached Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program cassandra_uptime Status ok Program poll_cassandra Status ok Program poll_cqlsh Status ok Program chronos_uptime Status ok Program poll_chronos Status ok Program astaire_uptime Status ok Program ##################################################################### Please let me know if I am missing any configuration. Thanks Hrishikesh
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
