On 28.04.2015 15:15, Bogdan Dobrelya wrote: > > Hello, Zhou > > > Yes, this is a known issue [0]. Note, there were many bugfixes, like > [1],[2],[3], merged for MQ OCF script, so you may want to try to > backport them as well by the following guide [4] > > [0] https://bugs.launchpad.net/fuel/+bug/1432603 > [1] https://review.openstack.org/#/c/175460/ > [2] https://review.openstack.org/#/c/175457/ > [3] https://review.openstack.org/#/c/175371/ > [4] https://review.openstack.org/#/c/170476/ > > > Could you please elaborate the what is the same/different batches for MQ > and DB? Note, there is a MQ clustering logic flow charts available here > [5] and we're planning to release a dedicated technical bulletin for this. > > [5] http://goo.gl/PPNrw7 > > > This is very interesting, thank you! I believe all commands for MySQL RA > OCF script should be as well wrapped with timeout -SIGTERM or -SIGKILL > as we did for MQ RA OCF. And there should no be any sleep calls. I > created a bug for this [6]. > > [6] https://bugs.launchpad.net/fuel/+bug/1449542 > > > Yes, something like that. As I mentioned, there were several bug fixes > in the 6.1 dev, and you can also check the MQ clustering flow charts. > > after > > Not exactly. There is no master in mirrored MQ cluster. We define the > rabbit_hosts configuration option from Oslo.messaging. What ensures all > queue masters will be spread around all of MQ nodes in a long run. And > we use a master abstraction only for the Pacemaker RA clustering layer. > Here, a "master" is the MQ node what joins the rest of the MQ nodes. > > > We do erase the node master attribute in CIB for such cases. This should > not bring problems into the master election logic. > > > (Note, the RabbitMQ documentation mentions *queue* masters and slaves, > which are not the case for the Pacemaker RA clustering abstraction layer.) > > > We made an assumption what the node with the highest MQ uptime should > know the most about recent cluster state, so other nodes must join it. > RA OCF does not work with queue masters directly. > > > The full MQ cluster reassemble logic is far from the perfect state, > indeed. This might erase all mnesia files, hence any custom entities, > like users or vhosts, would be removed as well. Note, we do not > configure durable queues for Openstack so there is nothing to care about > here - the full cluster downtime assumes there will be no AMQP messages > stored at all. > > > Yes, this option is only supported for newest RabbitMQ versions. But we > definitely should look how this could help. > > > Indeed, there are cases when MQ's autoheal can do nothing with existing > partitions and remains partitioned for ever, for example: > > Masters: [ node-1 ] > Slaves: [ node-2 node-3 ] > root@node-1:~# rabbitmqctl cluster_status > Cluster status of node 'rabbit@node-1' ... > [{nodes,[{disc,['rabbit@node-1','rabbit@node-2']}]}, > {running_nodes,['rabbit@node-1']}, > {cluster_name,<<"rabbit@node-2">>}, > {partitions,[]}] > ...done. > root@node-2:~# rabbitmqctl cluster_status > Cluster status of node 'rabbit@node-2' ... > [{nodes,[{disc,['rabbit@node-2']}]}] > ...done. > root@node-3:~# rabbitmqctl cluster_status > Cluster status of node 'rabbit@node-3' ... > [{nodes,[{disc,['rabbit@node-1','rabbit@node-2','rabbit@node-3']}]}, > {running_nodes,['rabbit@node-3']}, > {cluster_name,<<"rabbit@node-2">>}, > {partitions,[]}]
Sorry, here is the correct one [0] ! [0] http://pastebin.com/m3fDdMA6 > > So we should test the pause-minority value as well. > But I strongly believe we should make MQ multi-state clone to support > many masters, related bp [7] > > [7] > https://blueprints.launchpad.net/fuel/+spec/rabbitmq-pacemaker-multimaster-clone > > > Well, we should not mess the queue masters and multi-clone master for MQ > resource in the pacemaker. > As I said, pacemaker RA has nothing to do with queue masters. And we > introduced this "master" mostly in order to support the full cluster > reassemble case - there must be a node promoted and other nodes should join. > > > This is a very good point, thank you. > > > Thank you for a thorough feedback! This was a really great job. > > > -- Best regards, Bogdan Dobrelya, Skype #bogdando_at_yahoo.com Irc #bogdando __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
