Hi CW Team,
* We are deploying Clearwater using Kubernetes pods, container and services environment. * We adapted the environment of Kubernetes from docker compose environment as provided under https://github.com/Metaswitch/clearwater-docker. * We have written yaml templates for each of the 10 modules. Each module is getting deployed as one container per pod. * Services are exposed for each of the pod as needed according to the docker-file exposed ports. * We have observed few of the containers are not working. These containers are homestead, homer and bono. * After debugging in to logs we found that the homestead is not able to resolve Cassandra DNS and thus not able to contact cassandra. Hence homestead service keeps on restarting. Below is the error messages (/var/log/homestead/homestead_current)which keeps on repeating 05-12-2016 12:59:59.426 UTC Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL]): 05-12-2016 12:59:59.499 UTC Status load_monitor.cpp:105: Constructing LoadMonitor 05-12-2016 12:59:59.500 UTC Status load_monitor.cpp:106: Target latency (usecs) : 100000 05-12-2016 12:59:59.500 UTC Status load_monitor.cpp:107: Max bucket size : 1000 05-12-2016 12:59:59.500 UTC Status load_monitor.cpp:108: Initial token fill rate/s: 100.000000 05-12-2016 12:59:59.500 UTC Status load_monitor.cpp:109: Min token fill rate/s : 10.000000 05-12-2016 12:59:59.500 UTC Status dnscachedresolver.cpp:150: Creating Cached Resolver using servers: 05-12-2016 12:59:59.500 UTC Status dnscachedresolver.cpp:160: 10.96.0.10 05-12-2016 12:59:59.500 UTC Status a_record_resolver.cpp:54: Created ARecordResolver 05-12-2016 12:59:59.500 UTC Status a_record_resolver.cpp:54: Created ARecordResolver 05-12-2016 12:59:59.500 UTC Status cassandra_store.cpp:181: Configuring store connection 05-12-2016 12:59:59.500 UTC Status cassandra_store.cpp:182: Hostname: cassandra 05-12-2016 12:59:59.500 UTC Status cassandra_store.cpp:183: Port: 9160 05-12-2016 12:59:59.500 UTC Status cassandra_store.cpp:211: Configuring store worker pool 05-12-2016 12:59:59.500 UTC Status cassandra_store.cpp:212: Threads: 10 05-12-2016 12:59:59.500 UTC Status cassandra_store.cpp:213: Max Queue: 0 05-12-2016 12:59:59.573 UTC Error dnscachedresolver.cpp:846: Failed to retrieve record for cassandra: Could not contact DNS servers 05-12-2016 12:59:59.578 UTC Error main.cpp:744: Failed to initialize the Cassandra cache with error code 5. 05-12-2016 12:59:59.578 UTC Status main.cpp:745: Homestead is shutting down >From the tcpump we could see the first DNS request(between Homestead pod and >Kube DNS pod) somehow gets failed. But subsequent messages are able to resolve >the response but still this messages are coming up. It seems as they caught in >some loop. As a workaround we tried to use Cassandra Ip directly in /etc/Clearwater/shared_config unser cassandra_hostname. Now the error gets removed. However sometimes it keeps on giving the Failed to initialize the Cassandra cache with error code 3. So the behavior of working (homestead and Cassandra connection) is very random. # Deployment definitions home_domain=example.com sprout_hostname=sprout hs_hostname=homestead:8888 hs_provisioning_hostname=homestead:8889 xdms_hostname=homer:7888 ralf_hostname=ralf:10888 chronos_hostname=chronos cassandra_hostname= 10.108.6.190 # Email server configuration smtp_smarthost=127.0.0.1 smtp_username=username smtp_password=password [email protected]<mailto:[email protected]> # I-CSCF/S-CSCF configuration upstream_hostname=sprout # Keys signup_key=secret turn_workaround=secret ellis_api_key=secret ellis_cookie_key=secret reduce_cassandra_mem_usage=Y signaling_dns_server=10.96.0.10 BR Parwinder "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
