Hi Bader, Thank you for the extra diagnostics you have provided.
Everything looks good with your local config and shared config. However we can see from the output of “sudo monit summary” below that quite a few of your nodes are in an unhealthy state. It appears that clearwater-etcd has failed to start on every node and that the homestead_process has also failed to start. Because of multiple errors I would suggest destroying the deployment and creating it again from scratch by following the instructions here<http://clearwater.readthedocs.io/en/stable/Manual_Install.html>. Install the node-specific software one node at a time, as you do this check for network connectivity between your nodes by pinging the IP address of one node in your deployment from a different node in your deployment. If this has not revealed any networking issues, once you have installed all the node-specific software you should attempt to upload the shared configure you have created on one node in /etc/clearwater/shared_config. If this doesn’t work could you please provide the some diagnostics again. Thanks, Andrew From: Bader LAMTI [mailto:[email protected]] Sent: Monday, June 19, 2017 10:51 AM To: [email protected]; Andrew Edmonds <[email protected]> Subject: Re: [Project Clearwater] [Clearwater] Manual Install Instructions Shared Configuration error Hi Andrew, Thank you for your response. [ellis]ubuntu@test-1:~$ sudo monit summary Monit 5.18.1 uptime: 15h 25m Service Name Status Type node-test-1 Running System ntp_process Running Process nginx_process Running Process mysql_process Running Process ellis_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Execution failed | Does... Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program poll_ellis Status failed Program poll_ellis_https Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Wait parent Program poll_etcd_cluster Wait parent Program poll_etcd Wait parent Program [ellis]ubuntu@test-1:~$ [bono]ubuntu@test-2:~$ sudo monit summary Monit 5.18.1 uptime: 15h 24m Service Name Status Type node-test-2 Running System restund_process Execution failed | Does... Process ntp_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Execution failed | Does... Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process bono_process Running Process poll_restund Wait parent Program monit_uptime Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Wait parent Program poll_etcd_cluster Wait parent Program poll_etcd Wait parent Program poll_bono Status failed Program [bono]ubuntu@test-2:~$ [sprout]ubuntu@test-3:~$ sudo monit summary Monit 5.18.1 uptime: 15h 24m Service Name Status Type node-test-3 Running System sprout_process Execution failed | Does... Process ntp_process Running Process nginx_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Execution failed | Does... Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process sprout_uptime Wait parent Program poll_sprout_sip Wait parent Program poll_sprout_http Wait parent Program nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Wait parent Program poll_etcd_cluster Wait parent Program poll_etcd Wait parent Program [sprout]ubuntu@test-3:~$ [homer]ubuntu@test-4:~$ sudo monit summary Monit 5.18.1 uptime: 15h 24m Service Name Status Type node-test-4 Running System ntp_process Running Process nginx_process Running Process homer_process Execution failed | Does... Process clearwater_queue_manager_pro... Running Process etcd_process Execution failed | Does... Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program poll_homer Wait parent Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Wait parent Program poll_etcd_cluster Wait parent Program poll_etcd Wait parent Program [homer]ubuntu@test-4:~$ [dime]ubuntu@test-5:~$ sudo monit summary Monit 5.18.1 uptime: 15h 24m Service Name Status Type node-test-5 Running System snmpd_process Running Process ralf_process Execution failed | Does... Process ntp_process Running Process nginx_process Running Process homestead_process Execution failed | Does... Process homestead-prov_process Execution failed | Does... Process clearwater_queue_manager_pro... Running Process etcd_process Execution failed | Does... Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process ralf_uptime Wait parent Program poll_ralf Wait parent Program nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program homestead_uptime Wait parent Program poll_homestead Wait parent Program check_cx_health Wait parent Program poll_homestead-prov Wait parent Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Wait parent Program poll_etcd_cluster Wait parent Program poll_etcd Wait parent Program [dime]ubuntu@test-5:~$ [vellum]ubuntu@test-6:~$ sudo monit summary Monit 5.18.1 uptime: 0m Service Name Status Type node-test-6 Running System snmpd_process Running Process ntp_process Running Process memcached_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process cassandra_process Running Process chronos_process Running Process astaire_process Running Process monit_uptime Status ok Program memcached_uptime Status ok Program poll_memcached Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status failed Program poll_etcd Status failed Program cassandra_uptime Status ok Program poll_cassandra Status ok Program poll_cqlsh Status ok Program chronos_uptime Status ok Program poll_chronos Status ok Program astaire_uptime Status failed Program [vellum]ubuntu@test-6:~$ And “sudo service clearwater-etcd start” it doesn't work. Please find attached to this message the logs, shared config and local conifg. Thanks, Bader 2017-06-16 15:01 GMT+02:00 Andrew Edmonds <[email protected]<mailto:[email protected]>>: Hi Bader, Thank you for your question. Clearwater-etcd is the component responsible for forming a cluster across your Clearwater nodes to allow configuration to be shared. If there is a problem with uploading shared config, clearwater-etcd is usually a good place to start diagnosing the issue. A good first step is to check that the clearwater-etcd process is running, if you enter the command “sudo monit summary” you should see etcd_process reported as having status Running. If the etcd_process it is not running you can try using the command “sudo service clearwater-etcd start”. If this does not work could you please send me the /var/log/clearwater-etcd/clearwater-etcd.log log file. If the etcd_process is running you can you can use some of the commands here<http://clearwater.readthedocs.io/en/stable/Troubleshooting_and_Recovery.html?#deployment-management> to inspect the state of the underlying etcd cluster used by the management system. In particular “clearwater-etcdctl cluster-health” and “clearwater-etcdctl member list” might be useful. If none of the above suggestions help you upload shared config could you please also send me the contents of your shared config and local config files. Thanks, Andrew From: Clearwater [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Bader LAMTI Sent: Monday, June 12, 2017 9:31 AM To: [email protected]<mailto:[email protected]> Subject: [Project Clearwater] [Clearwater] Manual Install Instructions Shared Configuration error Hi, I am trying to install clearwater on openstack. I have created 6 VMs on OpenStack and I have followed Manual Install Instructions. The problem is the following : in the step “Provide Shared Configuration“. When I configure “/etc/clearwater/shared_config” and I run the command “cw-upload_shared_config” It remains blocked like this ubuntu@test-6:~$ ubuntu@test-6:~$ sudo cw-upload_shared_config Upload shared configuration failed to http://10.0.0.8:4000/v2/keys/clearwater/site1/configuration/shared_config % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 743 0 0 100 743 0 7 0:01:46 0:01:44 0:00:02 0 curl: (56) Recv failure: Connection reset by peer I would appreciate if you can help me to solve this issue. Thanking you + best regards, -- Bader _______________________________________________ Clearwater mailing list [email protected]<mailto:[email protected]> http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org --
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
