Hey Chris, followup, I re installed and everything is clustering correctly now. I think my mistake was installing all the machines first with all peers in etcd_cluster in local config. I reinstalled and setup each node one by one and everything appears to be good now.
Thanks, Trey -----Original Message-----From: "Ormsbee, Trey" <[email protected]> To: Chris Elford (projectclearwater.org) <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [Project Clearwater] New Cluster build etcd issues creating clusters. Date: Thu, 28 Apr 2016 12:05:49 -0400 thanks for the info, after a normal install only one node from each cluster showed in etcd. this is after following the manual install guide, what specifically tells etcd a new sprout/homestead node is being added to a cluster? I have no problem reinstalling to test this out again Hi Trey, Thanks for getting in touch. I think the issue is with the following commands: /usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_chronos_cluster sprout /usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_memcached_cluster sprout Recent versions of Project Clearwater will create their clustering files automatically. Both of those commands are used for upgrading an old Project Clearwater deployment without using etcd to a newer software level that does use etcd. There is no need to run them when deploying a new cluster, and they are likely to break your system by over-writing your clustering files. Please let us know if you have this issue following the latest install instructions: http://clearwater.readthedocs.io/en/stable/Manual_Install.html#manual-install-instructions. Yours, Chris -----Original Message----- From: Clearwater [mailto:[email protected]] On Behalf Of Trey Ormsbee Sent: 28 April 2016 15:58 To: [email protected] Subject: [Project Clearwater] New Cluster build etcd issues creating clusters. I have a new build running the latest release (as of friday last week). I am having difficulties getting etcd to create the clustering files. root@sprout01:/var/log/clearwater-etcd# clearwater-etcdctl cluster-health member e5b97d6d560689f is healthy: got healthy result from http://xxx.xxx.xxx.52:4000 member 6779e4ae1661cf04 is healthy: got healthy result from http://xxx.xxx.xxx.51:4000 member c2e2642d31ccf0da is healthy: got healthy result from http://xxx.xxx.xxx.42:4000 member cc8e3ee72cf33a87 is healthy: got healthy result from http://xxx.xxx.xxx.41:4000 cluster is healthy .41, .42 are the sprout nodes and have a management and signaling network. .51, .52 are homestead nodes and just have a management networks. After building the nodes and getting the initial etcd cluster healthy I ran the following scripts: /usr/share/clearwater/clearwater-config-manager/scripts/upload_enum_json /usr/share/clearwater/clearwater-config-manager/scripts/upload_bgcf_json /usr/share/clearwater/clearwater-config-manager/scripts/upload_scscf_json /usr/share/clearwater/clearwater-config-manager/scripts/upload_json and the check_config_sync_check shows: root@sprout01:/var/log/clearwater-etcd# /usr/share/clearwater/clearwater-config-manager/scripts/check_config_sync - /etc/clearwater/s-cscf.json is up to date - /etc/clearwater/shared_config is up to date - /etc/clearwater/enum.json is up to date - /etc/clearwater/bgcf.json is up to date The problem now lies with adding in cluster, I sprout01 and sprout02 I ran: /usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_chronos_cluster sprout /usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_memcached_cluster sprout After a full day: this is the state I see in etcdctl root@sprout01:~# clearwater-etcdctl get /clearwater/site1/sprout/clustering/chronos {"xxx.xx1.xxx.41": "joining, acknowledged change", "xxx.xx1.xxx.42": "normal"} root@sprout01:~# clearwater-etcdctl get /clearwater/site1/sprout/clustering/memcached {"xxx.xx1.xxx.41": "joining, acknowledged change", "xxx.xx1.xxx.42": "normal"} I ran the same on homestead01, homestead02 /usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_cassandra_cluster homestead root@sprout01:/var/log/clearwater-etcd# clearwater-etcdctl get /clearwater/homestead/clustering/cassandra {"xxx.xxx.xxx.51": "joining, acknowledged change", "xxx.xxx.xxx.52": "normal"} >From what I have read 30 seconds of joining, acknowledged change the configs should be rewritten. This is not happening in my case. The documentation on this seems to be a little lacking for a manual install so I am not sure if I am missing steps or if I misconfigured something does anyone have any idea what I need to do to get this to work correctly? Any logs I should be looking at, I dont see anything useful in the clearwater-etcd logs. Thanks, Trey _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
