I would recommend to have the non-preemtive mode set such that re-connections will not occur. I can send you the config by Monday. In addition, move VIP / LB outside of the controller with a separate small L2 subnet (/30 or /29) that has L3 connectivity with the controllers.
We have nova on VIP because of the async nature of the conductor processing messages from from RMQ and writing to mysql DB. The instance that pulls messages from Q needs to update mysql. If we have a local mysql connection in nova and that mysql failed after the messages are pulled from Q, then those updates to DB will not be successful, since messages are acked (on read) there is no re-transmission for those. Thanks, Sanju From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, August 27, 2015 at 2:50 AM To: Sanju Abraham <[email protected]<mailto:[email protected]>>, Ranjeet R <[email protected]<mailto:[email protected]>>, dev <[email protected]<mailto:[email protected]>> Subject: Re: Re: [opencontrail-dev] the mysql cfg in keystone.conf of openstack ha env set to localhost:3306, why not use vip:33306 ? hi Sanju, Thx for your reply, it's helpful to me. And I have two more questions 1. other components which connect to vip,like nova, when vip flap,is there also need time to reconnect 2. in my env, the master rabbit node and the master-vip node is the same,so when I poweron this node,the vip would back to it and it would become the master rabbit node again. it seems that services also need time to reconnect to the rabbitmq.rabbitmq cluster also use vip. ________________________________________ [email protected]<mailto:[email protected]> From: Sanju Abraham <mailto:[email protected]> Date: 2015-08-27 16:44 To: [email protected]<mailto:[email protected]>; Ranjeet R <mailto:[email protected]>; dev <mailto:[email protected]> Subject: Re: [opencontrail-dev] the mysql cfg in keystone.conf of openstack ha env set to localhost:3306, why not use vip:33306 ? Keyang, Openstack-HA in Contrail is not supported on centos / redhat distributions as of yet. We have full support for Ubuntu precise and trusty on Canonical Icehouse and Juno distributions. In the contrail-HA setup for mysql using galera we have the following to get around the issue with deadlocks that happen as a result of optimistic locking mechanism in galera. 1-> Tuning Innodb locks, buffers and read and write threads. 2-> HAP config with additional probes for API services that checks mysql state. Eg. If mysql on the local controller running KS has issue then requests will not be routed to that DIP, best effort is made to not send 5xx to client, instead HAP will keep updating the hash based on probes and will route the request to the DIP that is healthy. With connection on VIP for Keystone, there will be an impact to keystone service in following cases: 1-> Mysql on which the connections land fails. If connections land on the a particular mysql instance which has issues then the whole identity cluster will be impacted. 2-> VIP flap, mysql leading all ESTABLISHED long lived TCP connections to reconnect. This is what you might be seeing on your setup with VIP being pre-emptied (non-preemption will be supported in the next release), connections flap. Thanks, Sanju From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, August 26, 2015 at 11:00 PM To: Ranjeet R <[email protected]<mailto:[email protected]>>, dev <[email protected]<mailto:[email protected]>> Subject: Re: [opencontrail-dev] the mysql cfg in keystone.conf of openstack ha env set to localhost:3306, why not use vip:33306 ? hi Ranjeet in this page : https://docs.mirantis.com/openstack/fuel/fuel-6.1/reference-architecture.html#openstack-environment-architecture it said that : OpenStack support for multi-node writing to MySQL/Galera nodes is not production ready yet <http://lists.openstack.org/pipermail/openstack-operators/2014-September/005166.html>. "The simplest way to overcome this issue from the operator's point of view is to use only one writer node for these types of transactions". That is why Fuel configures HAProxy <https://docs.mirantis.com/openstack/fuel/fuel-6.1/terminology.html#haproxy-term>frontend for MySQL/Galera to use only one active node, while the other nodes in the cluster are retained standby (passive) state. my env is centos6.5 and use MariaDB-Galera-server for db I found that when keystone connected the local db,often encountered the 401 issue. the keystone.log show the msg: Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}} so I changed the keystone's connection to vip:33306 it seems that the issue solved. but the keystone connected the vip db,there is another iusse. poweroff the controller node which hold vip and when poweron it again it is said that when the vip back to the controller node again the keystone would lose the connection of db and it need about 10min to recovery the keystone.log show the msg: Database server has gone away: (2013, 'Lost connection to MySQL server during query') and the cmd like 'nova list' would timeout I changed the keystone's connection to local db this issue seems solved. can anyone explain the two issues above ? and which should keystone connect to,local or vip? thx. Keyang Li ________________________________________ [email protected]<mailto:[email protected]> From: Ranjeet R <mailto:[email protected]> Date: 2015-08-27 01:18 To: [email protected]<mailto:[email protected]>; dev <mailto:[email protected]> Subject: RE: [opencontrail-dev] the mysql cfg in keystone.conf of openstack ha env set to localhost:3306, why not use vip:33306 ? Hello Keylang Keystone being the auth endpoint for all the Openstack components, it accesses the DB(both reads/writes) more often than other components. Given that, we did not want to introduce another level of redirection for Keystone token creation/queries (VIP goes through HAProxy and load balances into a physical MySQL server) and hence used the localhost for MySQL connection. It is a small optimization, but if you change the connection parameter to VIP:33306, it should still work. Hope that helps! Ranjeet From: Dev [mailto:[email protected]] On Behalf Of [email protected]<mailto:[email protected]> Sent: Wednesday, August 26, 2015 1:46 AM To: dev <[email protected]<mailto:[email protected]>> Subject: [opencontrail-dev] the mysql cfg in keystone.conf of openstack ha env set to localhost:3306, why not use vip:33306 ? in keystone-server-setup.sh, there are: if [ "$INTERNAL_VIP" != "none" ]; then # Openstack HA specific config openstack-config --set /etc/keystone/keystone.conf sql connection mysql://keystone:keystone@$CONTROLLER:3306/keystone else openstack-config --set /etc/keystone/keystone.conf sql connection mysql://keystone:[email protected]/keystone<mailto:[email protected]/keystone> fi why not use vip:33306 like other component such as nova ? any help will be appreciated thx Keyang Li ________________________________________ [email protected]<mailto:[email protected]>
_______________________________________________ Dev mailing list [email protected] http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
