Hi,
I have installed clearwater manually using single node. I also scale up
using 2 more node for sprout 1 for vellum and dime. After some time my ip
is lost and i reset the ip in local_config file restart the services and the*
scale up node is no more (they are deleted)*
*Please reply, i want to fix it. I am waiting for your reply,, thanks in
advance*
*i have doubt that the scale up node is not remove properly. I just deleted
them*
I am getting the following errors in every node:
*[vellum]ubuntu@vellum:~$ cw-config download shared_config*
Unable to contact the etcd cluster.
*[bono]ubuntu@bono:~$ sudo monit summary*
[sudo] password for ubuntu:
Monit 5.18.1 uptime: 2h 38m
Service Name Status Type
node-bono Running System
restund_process Running Process
ntp_process Running Process
clearwater_queue_manager_pro... Running Process
etcd_process Execution failed | Does... Process
clearwater_diags_monitor_pro... Running Process
clearwater_config_manager_pr... Running Process
clearwater_cluster_manager_p... Running Process
bono_process Running Process
poll_restund Status ok Program
monit_uptime Status ok Program
clearwater_queue_manager_uptime Status ok Program
* etcd_uptime Wait parent Program*
* poll_etcd_cluster Wait parent Program*
* poll_etcd Wait parent Program*
poll_bono Status ok Program
*[vellum]ubuntu@vellum:~$ sudo cw-check_config_sync*
[sudo] password for ubuntu:
Traceback (most recent call last):
File
"/usr/share/clearwater/clearwater-config-manager/scripts/check_config_sync.py",
line 29, in <module>
result = client.get("/" + etcd_key + "/" + site + "/configuration/" +
plugin.key())
File "/usr/share/clearwater/clearwater-config-manager/env/local/
lib/python2.7/site-packages/etcd/client.py", line 679, in get
return self.read(key)
File "/usr/share/clearwater/clearwater-config-manager/env/local/
lib/python2.7/site-packages/etcd/client.py", line 536, in read
timeout=timeout)
File "/usr/share/clearwater/clearwater-config-manager/env/local/
lib/python2.7/site-packages/etcd/client.py", line 834, in wrapper
cause=e
etcd.EtcdConnectionFailed: Connection to etcd failed due to
MaxRetryError("HTTPConnectionPool(host='10.224.61.45', port=4000): Max
retries exceeded with url: /v2/keys/clearwater/site1/configuration/dns_json
(Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7f479c1d3910>: Failed to establish a new connection: [Errno 111]
Connection refused',))",)
*[vellum]ubuntu@vellum:~$ clearwater-etcdctl cluster-health*
cluster may be unhealthy: failed to list members
Error: client: etcd cluster is unavailable or misconfigured; error #0:
dial tcp 10.224.61.45:4000: getsockopt: connection refused
error #0: dial tcp 10.224.61.45:4000: getsockopt: connection refused
*[vellum]ubuntu@vellum:~$ clearwater-etcdctl member list*
Error: client: etcd cluster is unavailable or misconfigured; error #0:
dial tcp 10.224.61.45:4000: getsockopt: connection refused
error #0: dial tcp 10.224.61.45:4000: getsockopt: connection refused
*[bono]ubuntu@bono:/var/log/clearwater-etcd$ cat clearwater-etcd-initd.log*
context deadline exceeded
2018-04-19 06:25:52.190441844 etcdctl returned 1
2018-04-19 06:25:52.203217708 Joining existing cluster...
2018-04-19 06:26:21.211523618 Configured ETCDCTL_PEERS: 10.224.61.112:4000,
10.224.61.8:4000,10.224.61.109:4000,10.224.61.19:4000,10.224.61.47:4000,
10.224.61.24:4000,10.224.61.22:4000,10.224.61.115:4000,10.224.61.27:4000,
10.224.61.45:4000,
2018-04-19 06:26:21.213445229 Check cluster is healthy
2018-04-19 06:26:21.217376667 Running etcdctl cluster-health
2018-04-19 06:26:22.331785696 Found leaked etcd 2414 (correct is ) -
killing 2414
2018-04-19 06:26:22.335431738 etcdctl returned 137
2018-04-19 06:26:22.337320589 Restarting etcd clearwater-etcd
2018-04-19 06:26:22.343212899 Configured ETCDCTL_PEERS: 10.224.61.112:4000,
10.224.61.8:4000,10.224.61.109:4000,10.224.61.19:4000,10.224.61.47:4000,
10.224.61.24:4000,10.224.61.22:4000,10.224.61.115:4000,10.224.61.27:4000,
10.224.61.45:4000,
2018-04-19 06:26:22.343830818 Check for previous failed startup attempt
2018-04-19 06:26:22.344751074 Running etcdctl member list
context deadline exceeded
2018-04-19 06:26:32.357759475 etcdctl returned 1
2018-04-19 06:26:32.371886200 Joining existing cluster...
2018-04-19 06:26:37.382282245 Configured ETCDCTL_PEERS: 10.224.61.112:4000,
10.224.61.8:4000,10.224.61.109:4000,10.224.61.19:4000,10.224.61.47:4000,
10.224.61.24:4000,10.224.61.22:4000,10.224.61.115:4000,10.224.61.27:4000,
10.224.61.45:4000,
2018-04-19 06:26:37.383949088 Check cluster is healthy
2018-04-19 06:26:37.386734717 Running etcdctl cluster-health
cluster may be unhealthy: failed to list members
Error: client: etcd cluster is unavailable or misconfigured; error #0:
client: endpoint http://10.224.61.109:4000 exceeded header timeout
; error #1: dial tcp 10.224.61.19:4000: getsockopt: connection refused
; error #2: dial tcp 10.224.61.47:4000: getsockopt: connection refused
; error #3: http: no Host in request URL
; error #4: dial tcp 10.224.61.24:4000: getsockopt: connection refused
; error #5: dial tcp 10.224.61.45:4000: getsockopt: connection refused
; error #6: dial tcp 10.224.61.8:4000: getsockopt: connection refused
; error #7: client: endpoint http://10.224.61.115:4000 exceeded header
timeout
; error #8: client: endpoint http://10.224.61.112:4000 exceeded header
timeout
; error #9: dial tcp 10.224.61.27:4000: getsockopt: connection refused
; error #10: dial tcp 10.224.61.22:4000: getsockopt: connection refused
error #0: client: endpoint http://10.224.61.109:4000 exceeded header timeout
error #1: dial tcp 10.224.61.19:4000: getsockopt: connection refused
error #2: dial tcp 10.224.61.47:4000: getsockopt: connection refused
error #3: http: no Host in request URL
error #4: dial tcp 10.224.61.24:4000: getsockopt: connection refused
error #5: dial tcp 10.224.61.45:4000: getsockopt: connection refused
error #6: dial tcp 10.224.61.8:4000: getsockopt: connection refused
error #7: client: endpoint http://10.224.61.115:4000 exceeded header timeout
error #8: client: endpoint http://10.224.61.112:4000 exceeded header timeout
error #9: dial tcp 10.224.61.27:4000: getsockopt: connection refused
error #10: dial tcp 10.224.61.22:4000: getsockopt: connection refused
2018-04-19 06:26:43.409826012 etcdctl returned 4
2018-04-19 06:26:43.412523362 Not joining an unhealthy cluster
2018-04-19 06:27:03.126255654 Restarting etcd clearwater-etcd
2018-04-19 06:27:03.133102368 Configured ETCDCTL_PEERS: 10.224.61.112:4000,
10.224.61.8:4000,10.224.61.109:4000,10.224.61.19:4000,10.224.61.47:4000,
10.224.61.24:4000,10.224.61.22:4000,10.224.61.115:4000,10.224.61.27:4000,
10.224.61.45:4000,
2018-04-19 06:27:03.133904517 Check for previous failed startup attempt
2018-04-19 06:27:03.135105646 Running etcdctl member list
context deadline exceeded
2018-04-19 06:27:13.148679161 etcdctl returned 1
2018-04-19 06:27:13.162058102 Joining existing cluster...
2018-04-19 06:27:34.170718991 Configured ETCDCTL_PEERS: 10.224.61.112:4000,
10.224.61.8:4000,10.224.61.109:4000,10.224.61.19:4000,10.224.61.47:4000,
10.224.61.24:4000,10.224.61.22:4000,10.224.61.115:4000,10.224.61.27:4000,
10.224.61.45:4000,
2018-04-19 06:27:34.172721201 Check cluster is healthy
2018-04-19 06:27:34.176441659 Running etcdctl cluster-health
cluster may be unhealthy: failed to list members
Error: client: etcd cluster is unavailable or misconfigured; error #0:
client: endpoint http://10.224.61.109:4000 exceeded header timeout
; error #1: dial tcp 10.224.61.19:4000: getsockopt: connection refused
; error #2: client: endpoint http://10.224.61.112:4000 exceeded header
timeout
; error #3: dial tcp 10.224.61.47:4000: getsockopt: connection refused
; error #4: dial tcp 10.224.61.45:4000: getsockopt: connection refused
; error #5: dial tcp 10.224.61.8:4000: getsockopt: connection refused
; error #6: dial tcp 10.224.61.27:4000: getsockopt: connection refused
; error #7: client: endpoint http://10.224.61.115:4000 exceeded header
timeout
; error #8: dial tcp 10.224.61.22:4000: getsockopt: connection refused
; error #9: http: no Host in request URL
; error #10: dial tcp 10.224.61.24:4000: getsockopt: connection refused
error #0: client: endpoint http://10.224.61.109:4000 exceeded header timeout
error #1: dial tcp 10.224.61.19:4000: getsockopt: connection refused
error #2: client: endpoint http://10.224.61.112:4000 exceeded header timeout
error #3: dial tcp 10.224.61.47:4000: getsockopt: connection refused
error #4: dial tcp 10.224.61.45:4000: getsockopt: connection refused
error #5: dial tcp 10.224.61.8:4000: getsockopt: connection refused
error #6: dial tcp 10.224.61.27:4000: getsockopt: connection refused
error #7: client: endpoint http://10.224.61.115:4000 exceeded header timeout
error #8: dial tcp 10.224.61.22:4000: getsockopt: connection refused
error #9: http: no Host in request URL
error #10: dial tcp 10.224.61.24:4000: getsockopt: connection refused
thanks,
sunil
_______________________________________________
Clearwater mailing list
[email protected]
http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
