Re: [opencontrail-dev] cassandra server switchover
Hi Robert van Leeuwen, Its not for production. I am using to for development test setup. Yes, cassandra 2 node cluster works with contrail. Thanks suresh. From: "Robert Van Leeuwen" <rovanleeu...@ebay.com> To: "Suresh Kumar S" <sureshkuma...@altencalsoftlabs.com>, "Megh Bhatt" <me...@juniper.net> Cc: "Dev@lists.opencontrail.org" <dev@lists.opencontrail.org> Sent: Wednesday, November 30, 2016 8:12:05 AM Subject: Re: [opencontrail-dev] cassandra server switchover > cassandra_server_list = 10.10.10.5:9160 10.10.10.4:9160 As with all clustered software: you would want to run with at least 3 (Cassandra) nodes for quorum reasons. There are probably some Cassandra settings you can tweak to make it functioning when doing a 2 node cluster but I would not recommend using that in production. Cheers, Robert van Leeuwen ___ Dev mailing list Dev@lists.opencontrail.org http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
Re: [opencontrail-dev] cassandra server switchover
Hi Raj, I am using cloudwatt specific version. One more observation, In the DISCOVERY (http://localhost:5998/services), Collector service shows down for 30 - 60 seconds during the switchover. Is that expected behavior? Thanks Suresh. From: "Raj Reddy" <rajre...@juniper.net> To: "Suresh Kumar S" <sureshkuma...@altencalsoftlabs.com>, "Megh Bhatt" <me...@juniper.net> Cc: Dev@lists.opencontrail.org Sent: Tuesday, November 29, 2016 5:58:57 PM Subject: Re: [opencontrail-dev] cassandra server switchover Is there any reason why you are using 2.21, better to be with newer releases. we have multiple stable releases after that. Regards Raj From: Dev <dev-boun...@lists.opencontrail.org> on behalf of Suresh Kumar S <sureshkuma...@altencalsoftlabs.com> Date: Tuesday, November 29, 2016 at 1:33 AM To: Megh Bhatt <me...@juniper.net> Cc: "Dev@lists.opencontrail.org" <Dev@lists.opencontrail.org> Subject: Re: [opencontrail-dev] cassandra server switchover Hi Megh, Thanks for the response. Yes, netstat -anp shows thats collector connects to the another server. Reg TimedOutException , Let me recheck my cassandra cluster configuration/setup. Thanks suresh. From: "Megh Bhatt" <me...@juniper.net> To: "Suresh Kumar S" <sureshkuma...@altencalsoftlabs.com> Cc: Dev@lists.opencontrail.org Sent: Tuesday, November 29, 2016 9:52:02 AM Subject: Re: [opencontrail-dev] cassandra server switchover In 2.21 the architecture is different, collector will establish one connection to one cassandra server per generator. If that fails, it will retry the next server. This is evident from the collector logs. It seems that you are getting TimedOutException which suggests your single remaining cassandra node might be getting overloaded. If you check using netstat -anp for contrail-collector pid you should be seeing connections to the 10.10.10.5 cassandra node. Thanks Megh On Nov 29, 2016, at 12:04 AM, Suresh Kumar S < sureshkuma...@altencalsoftlabs.com > wrote: Hi Megh, My scenario is very simple, My setup : Devstack - Mitaka Contrail : 2.21.x I have configured the cassandra cluster with two nodes, and specified those in the contrail-collector.conf file cloud@contrail:/etc/cassandra$ cat /etc/contrail/contrail-collector.conf [DEFAULT] log_local = 1 log_level = SYS_DEBUG cassandra_server_list = 10.10.10.5:9160 10.10.10.4:9160 hostname = contrail hostip = 10.10.10.6 [DISCOVERY] port = 5998 server = 10.10.10.6 cloud@contrail:/etc/cassandra$ Collector establishes the communication with 10.10.10.4, and works well. I purposefully down the 10.10.10.4 cassandra server. I expect contrail-control continue to work with 10.10.10.5 cassandra server. But contrail-control keeps trying to connect with 10.10.10.4 and fails to connect. Note: 1. Collector establishes the communication with only one server(10.10.10.4) out of two mentioned. collector logs are attached for debugging. Thanks suresh From: "Megh Bhatt" < me...@juniper.net > To: "Suresh Kumar S" < sureshkuma...@altencalsoftlabs.com > Cc: Dev@lists.opencontrail.org Sent: Tuesday, November 29, 2016 8:48:20 AM Subject: Re: [opencontrail-dev] cassandra server switchover Hi Suresh, cassandra supports active/active or cluster consisting of multiple active nodes and hence we can provide list of IP addresses to contrail-collector. contrail-collector simultaneously connects to all the cassandra servers and will handle cassandra server failures. Thanks Megh BQ_BEGIN On Nov 28, 2016, at 11:43 PM, Suresh Kumar S < sureshkuma...@altencalsoftlabs.com > wrote: Hi, In the contrail-collector config file, we can specify multiple cassandra servers. What is the purpose of it ? contrail-collecter.conf snippet # IP address and port to be used to connect to cassandra. # Multiple IP:port strings separated by space can be provided # cassandra_server_list=127.0.0.1:9042 Does contrail support server switchover, I mean when the current server is not reachable, will it switch to the new server(mentioned in the config) ? Thanks suresh ___ Dev mailing list Dev@lists.opencontrail.org http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org BQ_END ___ Dev mailing list Dev@lists.opencontrail.org http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
Re: [opencontrail-dev] cassandra server switchover
> cassandra_server_list = 10.10.10.5:9160 10.10.10.4:9160 As with all clustered software: you would want to run with at least 3 (Cassandra) nodes for quorum reasons. There are probably some Cassandra settings you can tweak to make it functioning when doing a 2 node cluster but I would not recommend using that in production. Cheers, Robert van Leeuwen ___ Dev mailing list Dev@lists.opencontrail.org http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
Re: [opencontrail-dev] cassandra server switchover
Is there any reason why you are using 2.21, better to be with newer releases. we have multiple stable releases after that. Regards Raj From: Dev <dev-boun...@lists.opencontrail.org> on behalf of Suresh Kumar S <sureshkuma...@altencalsoftlabs.com> Date: Tuesday, November 29, 2016 at 1:33 AM To: Megh Bhatt <me...@juniper.net> Cc: "Dev@lists.opencontrail.org" <Dev@lists.opencontrail.org> Subject: Re: [opencontrail-dev] cassandra server switchover Hi Megh, Thanks for the response. Yes, netstat -anp shows thats collector connects to the another server. Reg TimedOutException , Let me recheck my cassandra cluster configuration/setup. Thanks suresh. From: "Megh Bhatt" <me...@juniper.net> To: "Suresh Kumar S" <sureshkuma...@altencalsoftlabs.com> Cc: Dev@lists.opencontrail.org Sent: Tuesday, November 29, 2016 9:52:02 AM Subject: Re: [opencontrail-dev] cassandra server switchover In 2.21 the architecture is different, collector will establish one connection to one cassandra server per generator. If that fails, it will retry the next server. This is evident from the collector logs. It seems that you are getting TimedOutException which suggests your single remaining cassandra node might be getting overloaded. If you check using netstat -anp for contrail-collector pid you should be seeing connections to the 10.10.10.5 cassandra node. Thanks Megh On Nov 29, 2016, at 12:04 AM, Suresh Kumar S <sureshkuma...@altencalsoftlabs.com<mailto:sureshkuma...@altencalsoftlabs.com>> wrote: Hi Megh, My scenario is very simple, My setup : Devstack - Mitaka Contrail : 2.21.x I have configured the cassandra cluster with two nodes, and specified those in the contrail-collector.conf file cloud@contrail:/etc/cassandra$ cat /etc/contrail/contrail-collector.conf [DEFAULT] log_local = 1 log_level = SYS_DEBUG cassandra_server_list = 10.10.10.5:9160 10.10.10.4:9160 hostname = contrail hostip = 10.10.10.6 [DISCOVERY] port = 5998 server = 10.10.10.6 cloud@contrail:/etc/cassandra$ Collector establishes the communication with 10.10.10.4, and works well. I purposefully down the 10.10.10.4 cassandra server. I expect contrail-control continue to work with 10.10.10.5 cassandra server. But contrail-control keeps trying to connect with 10.10.10.4 and fails to connect. Note: 1. Collector establishes the communication with only one server(10.10.10.4) out of two mentioned. collector logs are attached for debugging. Thanks suresh From: "Megh Bhatt" <me...@juniper.net<mailto:me...@juniper.net>> To: "Suresh Kumar S" <sureshkuma...@altencalsoftlabs.com<mailto:sureshkuma...@altencalsoftlabs.com>> Cc: Dev@lists.opencontrail.org<mailto:Dev@lists.opencontrail.org> Sent: Tuesday, November 29, 2016 8:48:20 AM Subject: Re: [opencontrail-dev] cassandra server switchover Hi Suresh, cassandra supports active/active or cluster consisting of multiple active nodes and hence we can provide list of IP addresses to contrail-collector. contrail-collector simultaneously connects to all the cassandra servers and will handle cassandra server failures. Thanks Megh On Nov 28, 2016, at 11:43 PM, Suresh Kumar S <sureshkuma...@altencalsoftlabs.com<mailto:sureshkuma...@altencalsoftlabs.com>> wrote: Hi, In the contrail-collector config file, we can specify multiple cassandra servers. What is the purpose of it ? contrail-collecter.conf snippet # IP address and port to be used to connect to cassandra. # Multiple IP:port strings separated by space can be provided # cassandra_server_list=127.0.0.1:9042 Does contrail support server switchover, I mean when the current server is not reachable, will it switch to the new server(mentioned in the config) ? Thanks suresh ___ Dev mailing list Dev@lists.opencontrail.org<mailto:Dev@lists.opencontrail.org> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org ___ Dev mailing list Dev@lists.opencontrail.org http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
Re: [opencontrail-dev] cassandra server switchover
Hi Megh, Thanks for the response. Yes, netstat -anp shows thats collector connects to the another server. Reg TimedOutException , Let me recheck my cassandra cluster configuration/setup. Thanks suresh. From: "Megh Bhatt" <me...@juniper.net> To: "Suresh Kumar S" <sureshkuma...@altencalsoftlabs.com> Cc: Dev@lists.opencontrail.org Sent: Tuesday, November 29, 2016 9:52:02 AM Subject: Re: [opencontrail-dev] cassandra server switchover In 2.21 the architecture is different, collector will establish one connection to one cassandra server per generator. If that fails, it will retry the next server. This is evident from the collector logs. It seems that you are getting TimedOutException which suggests your single remaining cassandra node might be getting overloaded. If you check using netstat -anp for contrail-collector pid you should be seeing connections to the 10.10.10.5 cassandra node. Thanks Megh On Nov 29, 2016, at 12:04 AM, Suresh Kumar S < sureshkuma...@altencalsoftlabs.com > wrote: Hi Megh, My scenario is very simple, My setup : Devstack - Mitaka Contrail : 2.21.x I have configured the cassandra cluster with two nodes, and specified those in the contrail-collector.conf file cloud@contrail:/etc/cassandra$ cat /etc/contrail/contrail-collector.conf [DEFAULT] log_local = 1 log_level = SYS_DEBUG cassandra_server_list = 10.10.10.5:9160 10.10.10.4:9160 hostname = contrail hostip = 10.10.10.6 [DISCOVERY] port = 5998 server = 10.10.10.6 cloud@contrail:/etc/cassandra$ Collector establishes the communication with 10.10.10.4, and works well. I purposefully down the 10.10.10.4 cassandra server. I expect contrail-control continue to work with 10.10.10.5 cassandra server. But contrail-control keeps trying to connect with 10.10.10.4 and fails to connect. Note: 1. Collector establishes the communication with only one server(10.10.10.4) out of two mentioned. collector logs are attached for debugging. Thanks suresh From: "Megh Bhatt" < me...@juniper.net > To: "Suresh Kumar S" < sureshkuma...@altencalsoftlabs.com > Cc: Dev@lists.opencontrail.org Sent: Tuesday, November 29, 2016 8:48:20 AM Subject: Re: [opencontrail-dev] cassandra server switchover Hi Suresh, cassandra supports active/active or cluster consisting of multiple active nodes and hence we can provide list of IP addresses to contrail-collector. contrail-collector simultaneously connects to all the cassandra servers and will handle cassandra server failures. Thanks Megh BQ_BEGIN On Nov 28, 2016, at 11:43 PM, Suresh Kumar S < sureshkuma...@altencalsoftlabs.com > wrote: Hi, In the contrail-collector config file, we can specify multiple cassandra servers. What is the purpose of it ? contrail-collecter.conf snippet # IP address and port to be used to connect to cassandra. # Multiple IP:port strings separated by space can be provided # cassandra_server_list=127.0.0.1:9042 Does contrail support server switchover, I mean when the current server is not reachable, will it switch to the new server(mentioned in the config) ? Thanks suresh ___ Dev mailing list Dev@lists.opencontrail.org http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org BQ_END ___ Dev mailing list Dev@lists.opencontrail.org http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
Re: [opencontrail-dev] cassandra server switchover
In 2.21 the architecture is different, collector will establish one connection to one cassandra server per generator. If that fails, it will retry the next server. This is evident from the collector logs. It seems that you are getting TimedOutException which suggests your single remaining cassandra node might be getting overloaded. If you check using netstat -anp for contrail-collector pid you should be seeing connections to the 10.10.10.5 cassandra node. Thanks Megh On Nov 29, 2016, at 12:04 AM, Suresh Kumar S <sureshkuma...@altencalsoftlabs.com<mailto:sureshkuma...@altencalsoftlabs.com>> wrote: Hi Megh, My scenario is very simple, My setup : Devstack - Mitaka Contrail : 2.21.x I have configured the cassandra cluster with two nodes, and specified those in the contrail-collector.conf file cloud@contrail:/etc/cassandra$ cat /etc/contrail/contrail-collector.conf [DEFAULT] log_local = 1 log_level = SYS_DEBUG cassandra_server_list = 10.10.10.5:9160 10.10.10.4:9160 hostname = contrail hostip = 10.10.10.6 [DISCOVERY] port = 5998 server = 10.10.10.6 cloud@contrail:/etc/cassandra$ Collector establishes the communication with 10.10.10.4, and works well. I purposefully down the 10.10.10.4 cassandra server. I expect contrail-control continue to work with 10.10.10.5 cassandra server. But contrail-control keeps trying to connect with 10.10.10.4 and fails to connect. Note: 1. Collector establishes the communication with only one server(10.10.10.4) out of two mentioned. collector logs are attached for debugging. Thanks suresh From: "Megh Bhatt" <me...@juniper.net<mailto:me...@juniper.net>> To: "Suresh Kumar S" <sureshkuma...@altencalsoftlabs.com<mailto:sureshkuma...@altencalsoftlabs.com>> Cc: Dev@lists.opencontrail.org<mailto:Dev@lists.opencontrail.org> Sent: Tuesday, November 29, 2016 8:48:20 AM Subject: Re: [opencontrail-dev] cassandra server switchover Hi Suresh, cassandra supports active/active or cluster consisting of multiple active nodes and hence we can provide list of IP addresses to contrail-collector. contrail-collector simultaneously connects to all the cassandra servers and will handle cassandra server failures. Thanks Megh On Nov 28, 2016, at 11:43 PM, Suresh Kumar S <sureshkuma...@altencalsoftlabs.com<mailto:sureshkuma...@altencalsoftlabs.com>> wrote: Hi, In the contrail-collector config file, we can specify multiple cassandra servers. What is the purpose of it ? contrail-collecter.conf snippet # IP address and port to be used to connect to cassandra. # Multiple IP:port strings separated by space can be provided # cassandra_server_list=127.0.0.1:9042 Does contrail support server switchover, I mean when the current server is not reachable, will it switch to the new server(mentioned in the config) ? Thanks suresh ___ Dev mailing list Dev@lists.opencontrail.org<mailto:Dev@lists.opencontrail.org> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org ___ Dev mailing list Dev@lists.opencontrail.org http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
