I have re-installed the homestead node from scratch and after that the etcd cluster is no longer showing the homestead part of it's list.
I have issued following command on ellis and see homestead is missing: sudo clearwater-etcdctl member list 176d77d99abd6922: name=192-168-0-32 peerURLs=http://192.168.0.32:2380 clientURLs=http://192.168.0.32:4000 2c677e440fa18894: name=192-168-0-33 peerURLs=http://192.168.0.33:2380 clientURLs=http://192.168.0.33:4000 40ae69e640183094: name=192-168-0-34 peerURLs=http://192.168.0.34:2380 clientURLs=http://192.168.0.34:4000 722329d972aa3204: name=192-168-0-37 peerURLs=http://192.168.0.37:2380 clientURLs=http://192.168.0.37:4000 a9a4e28f06b02bd6: name=192-168-0-35 peerURLs=http://192.168.0.35:2380 clientURLs=http://192.168.0.35:4000 The local_config file do have the etcd cluster with "192.168.0.36", but it is missing from the member list. [ellis]ubuntu@ellis:~$ cat /etc/clearwater/local_config local_ip=192.168.0.32 public_ip=10.145.253.142 public_hostname=10.145.253.142 etcd_cluster="192.168.0.32,192.168.0.33,192.168.0.34,192.168.0.35,192.168.0.36,192.168.0.37" When I logged in on homestead I also noticed that homestead process is not started: [homestead]ubuntu@homestead:~$ netstat -an | grep LIST tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:2812 0.0.0.0:* LISTEN tcp6 0 0 :::80 :::* LISTEN tcp6 0 0 :::53 :::* LISTEN tcp6 0 0 :::22 :::* LISTEN unix 2 [ ACC ] STREAM LISTENING 9138 /tmp/clearwater_signaling_namespace_socket unix 2 [ ACC ] SEQPACKET LISTENING 7222 /run/udev/control unix 2 [ ACC ] STREAM LISTENING 8766 /var/run/dbus/system_bus_socket unix 2 [ ACC ] STREAM LISTENING 6525 @/com/ubuntu/upstart unix 2 [ ACC ] STREAM LISTENING 9164 /var/run/acpid.socket unix 2 [ ACC ] STREAM LISTENING 9091 /tmp/clearwater_management_namespace_socket unix 2 [ ACC ] STREAM LISTENING 9726 /var/agentx/master It looks like that Casandra is failing to start with the error: 12-04-2017 17:30:02.492 UTC Status load_monitor.cpp:105: Constructing LoadMonitor 12-04-2017 17:30:02.492 UTC Status load_monitor.cpp:106: Target latency (usecs) : 100000 12-04-2017 17:30:02.492 UTC Status load_monitor.cpp:107: Max bucket size : 1000 12-04-2017 17:30:02.492 UTC Status load_monitor.cpp:108: Initial token fill rate/s: 100.000000 12-04-2017 17:30:02.492 UTC Status load_monitor.cpp:109: Min token fill rate/s : 10.000000 12-04-2017 17:30:02.492 UTC Status dnscachedresolver.cpp:150: Creating Cached Resolver using servers: 12-04-2017 17:30:02.492 UTC Status dnscachedresolver.cpp:160: 127.0.0.1 12-04-2017 17:30:02.492 UTC Status a_record_resolver.cpp:54: Created ARecordResolver 12-04-2017 17:30:02.492 UTC Status a_record_resolver.cpp:54: Created ARecordResolver 12-04-2017 17:30:02.492 UTC Status cassandra_store.cpp:181: Configuring store connection 12-04-2017 17:30:02.492 UTC Status cassandra_store.cpp:182: Hostname: 127.0.0.1 12-04-2017 17:30:02.492 UTC Status cassandra_store.cpp:183: Port: 9160 12-04-2017 17:30:02.492 UTC Status cassandra_store.cpp:211: Configuring store worker pool 12-04-2017 17:30:02.492 UTC Status cassandra_store.cpp:212: Threads: 10 12-04-2017 17:30:02.492 UTC Status cassandra_store.cpp:213: Max Queue: 0 12-04-2017 17:30:02.493 UTC Error main.cpp:756: Failed to initialize the Cassandra cache with error code 3. 12-04-2017 17:30:02.493 UTC Status main.cpp:757: Homestead is shutting down [homestead]ubuntu@homestead:/var/log/homestead$ sudo monit summary Monit 5.18.1 uptime: 16h 5m Service Name Status Type node-homestead Running System snmpd_process Running Process ntp_process Running Process nginx_process Running Process homestead_process Execution failed | Does... Process homestead-prov_process Execution failed | Does... Process clearwater_queue_manager_pro... Running Process etcd_process Execution failed | Does... Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process cassandra_process Running Process nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program homestead_uptime Wait parent Program poll_homestead Wait parent Program check_cx_health Wait parent Program poll_homestead-prov Wait parent Program clearwater_queue_manager_uptime Status failed Program etcd_uptime Wait parent Program poll_etcd_cluster Wait parent Program poll_etcd Wait parent Program cassandra_uptime Status failed Program poll_cassandra Status ok Program poll_cqlsh Status ok Program In homestead_err log I also noticed the following error: Thrift: Wed Apr 12 17:09:25 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:09:25 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:11:09 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:11:09 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:12:51 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:12:51 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:14:35 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:14:35 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:16:17 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:16:17 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:18:00 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:18:00 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:19:44 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:19:44 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:21:27 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:21:27 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:23:09 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:23:09 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:24:52 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:24:52 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:26:35 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:26:35 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:28:19 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:28:19 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:30:02 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:30:02 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:31:45 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused Thrift: Wed Apr 12 17:31:45 2017 TSocket::open() error on socket (after THRIFT_POLL) <Host: 127.0.0.1 Port: 9160>Connection refused ---------------------------------------------------------- Salman Shaikh NFV Open Lab Architect Huawei R&D USA FutureWei Technologies, Inc Email: [email protected]<mailto:[email protected]> Mobile: +1 (408) 421-6210 ----------------------------------------------------------
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
