Wiadomość napisana przez Bogdan Dobrelya <[email protected]> w dniu 16 maj 2014, o godz. 11:14:
> On 05/16/2014 10:57 AM, Bartosz Kupidura wrote: >> Hello guys! >> I would like to sugest a few changes to Fuel HA/scalability features. >> >> 1. [HA] Ensure public/management VIP is running on node where HAproxy is >> working. >> >> Now if HAproxy dies, VIP is not moved to another node in a cluster. >> Simple way to check this is (HAProxy can die after segfault, wrong config, >> uninstalled package...): >> # echo deadbeef >> /etc/haproxy/haproxy.cfg >> # /etc/init.d/haproxy stop >> >> What happens: >> - Corosync can not start HAproxy >> - Corosync will NOT move VIP to another node >> - ALL connections to VIPs got 'connection refused' >> >> What should happen: >> - Corosync can not start HAproxy >> - Corosync will move VIP to another node >> >> Gerrit change: http://gerrit.vm.mirantis.net:8080/#/c/15617/ Gerrit change in review.openstack.org: https://review.openstack.org/#/c/93884/ > > Hello. Thank you for such a great feedback. > > It would be nice to provide an LP bugs for this patch as well as for all > patches below and submit them as a public openstack gerrit ones. Please > don't hesitate to submit, I could also help you to address it. > > As far as we are discussing here the issues related not only to the Fuel > HA but to the Oslo.messaging, RabbitMQ and Nova configuration, I've > added an openstack-dev tag as well. Perhaps, some of these changes could > be also contributed to the Nova and Oslo. > >> >> Now ocf:mirantis:haproxy check only if haproxy is running, in future we can >> implement more sophisticated health checks (backend timeouts, current >> connections limit...) >> >> 2. [HA] Tune TCP keepalive sysctl. >> >> Now we use default ubuntu/centos value (7200+9*75). >> This mean kernel will notice ‘silent’ (not RST, not FIN) connection failure >> after >2h. > > Yes, the defaults are (always) poor :-) > Here is a list for an existing issues (and patches, if any were > submitted already) https://etherpad.openstack.org/p/fuel-ha-rabbitmq > > That document also is a kind of a brainstorm, feel free to participate. > Personally I like the ideas to put rabbit cluster management onto the > Pacemaker and consider two only rabbits cluster in order to address > https://bugs.launchpad.net/oslo.messaging/+bug/856764/comments/22, if > there are indeed such a strange things are happening with cluster size 3+ > >> >> From my experience good value for HA systems is 180s: >> net.ipv4.tcp_keepalive_time = 120 >> net.ipv4.tcp_keepalive_probes = 3 >> net.ipv4.tcp_keepalive_intvl = 20 >> >> Gerrit change: http://gerrit.vm.mirantis.net:8080/#/c/15618/ > > Looks like your choice is better. There is a patch > https://review.openstack.org/#/c/93815/ related to the > https://bugs.launchpad.net/oslo.messaging/+bug/856764/comments/19 as > well. We could discuss which exactly TCPKA parameters are fitting better. > And one more related patch for RabbitMQ cluster > https://review.openstack.org/#/c/93411/ We can abbandon my patch, and stick to https://review.openstack.org/#/c/93815/ > >> >> 3. [Scalability] shuffle amqp nodes in Openstack configs. >> >> Now each Openstack node (compute, cinder, ...) connect to #1 controller, >> after failure it reconnects to #2, after that to #3 controller. >> >> In this case, ALL AMQP traffic is served by #1. >> >> We can shuffle 'rabbit_hosts' on each node. >> >> Gerrit change: http://gerrit.vm.mirantis.net:8080/#/c/15619/ Gerrit change in review.openstack.org: https://review.openstack.org/#/c/93883/ > > That is a brilliant idea. I was investigating the related things > recently and googled this > http://rabbitmq.1065348.n5.nabble.com/Correct-way-of-determining-which-node-is-master-td91.html. > According to this thread, there could be a good performance benefit in > spreading the queue masters around. > > But actually, we already have it in the recent amqp-nodes patches > accepted for Fuel 5.0. Let me elaborate. > We configure rabbit hosts for all controllers as: > > rabbit_nodes = 127.0.0.1:5673, rabbit1:5673, ... rabbitX:5673. > > As you can see, the initial connection point for new queues will always > be the node itself, hence all master queues would be automatically > "shuffled" as well. Yes, this is why for controllers i only remote duplicated entry (we already add 127.0.0.1, so we dont need $::internal_address in $amqp_nodes. But for other services (compute, cinder,…), patch add shuffle(). Previously rabbit_hosts was always '#controller1, #controller2, #controller3' > > Basically, I see this as a main reason of why we shouldn't use VIP for > rabbit cluster ever. > >> >> >> Best Regards, >> Bartosz Kupidura >> > > > -- > Best regards, > Bogdan Dobrelya, > Skype #bogdando_at_yahoo.com > Irc #bogdando -- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
