Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-20 Thread Vladimir Kuklin
Actually, we are not skipping 'Started' state - we just consider resource as started when beam is powered up and rabbitmq start_app/stop_app action succeeds. Such a node is considered as a good one that can be marked as 'Master' to which the nodes should connect and then all the cluster join/leave

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-19 Thread Andrew Beekhof
> On 20 May 2015, at 6:05 am, Andrew Woodward wrote: > > > > On Thu, May 7, 2015 at 5:01 PM Andrew Beekhof wrote: > > > On 5 May 2015, at 1:19 pm, Zhou Zheng Sheng / 周征晟 > > wrote: > > > > Thank you Andrew. > > > > on 2015/05/05 08:03, Andrew Beekhof wrote: > >>> On 28 Apr 2015, at 11:15 p

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-19 Thread Andrew Woodward
On Thu, May 7, 2015 at 5:01 PM Andrew Beekhof wrote: > > > On 5 May 2015, at 1:19 pm, Zhou Zheng Sheng / 周征晟 < > zhengsh...@awcloud.com> wrote: > > > > Thank you Andrew. > > > > on 2015/05/05 08:03, Andrew Beekhof wrote: > >>> On 28 Apr 2015, at 11:15 pm, Bogdan Dobrelya > wrote: > >>> > He

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-07 Thread Andrew Beekhof
> On 5 May 2015, at 7:52 pm, Bogdan Dobrelya wrote: > > On 05.05.2015 04:32, Andrew Beekhof wrote: >> >> >> [snip] >> >> >> Technically it calculates an ordered graph of actions that need to be >> performed for a set of related resources. >> You can see an example of the kinds of graphs it

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-07 Thread Andrew Beekhof
> On 5 May 2015, at 9:30 pm, Zhou Zheng Sheng / 周征晟 > wrote: > > Thank you Andrew. Sorry for misspell your name in the previous email. > > on 2015/05/05 14:25, Andrew Beekhof wrote: >>> On 5 May 2015, at 2:31 pm, Zhou Zheng Sheng / 周征晟 >>> wrote: >>> >>> Thank you Bogdan for clearing the pa

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-07 Thread Andrew Beekhof
> On 5 May 2015, at 1:19 pm, Zhou Zheng Sheng / 周征晟 > wrote: > > Thank you Andrew. > > on 2015/05/05 08:03, Andrew Beekhof wrote: >>> On 28 Apr 2015, at 11:15 pm, Bogdan Dobrelya wrote: >>> Hello, >>> Hello, Zhou >>> I using Fuel 6.0.1 and find that RabbitMQ recover time is long a

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-05 Thread Zhou Zheng Sheng / 周征晟
Thank you Andrew. Sorry for misspell your name in the previous email. on 2015/05/05 14:25, Andrew Beekhof wrote: >> On 5 May 2015, at 2:31 pm, Zhou Zheng Sheng / 周征晟 >> wrote: >> >> Thank you Bogdan for clearing the pacemaker promotion process for me. >> >> on 2015/05/05 10:32, Andrew Beekhof wr

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-05 Thread Bogdan Dobrelya
On 05.05.2015 04:32, Andrew Beekhof wrote: > > > [snip] > > > Technically it calculates an ordered graph of actions that need to be > performed for a set of related resources. > You can see an example of the kinds of graphs it produces at: > > > http://clusterlabs.org/doc/en-US/Pacemaker/

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-04 Thread Andrew Beekhof
> On 5 May 2015, at 2:31 pm, Zhou Zheng Sheng / 周征晟 > wrote: > > Thank you Bogdan for clearing the pacemaker promotion process for me. > > on 2015/05/05 10:32, Andrew Beekhof wrote: >>> On 29 Apr 2015, at 5:38 pm, Zhou Zheng Sheng / 周征晟 >>> wrote: >> [snip] >> >>> Batch is a pacemaker conce

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-04 Thread Zhou Zheng Sheng / 周征晟
Thank you Bogdan for clearing the pacemaker promotion process for me. on 2015/05/05 10:32, Andrew Beekhof wrote: >> On 29 Apr 2015, at 5:38 pm, Zhou Zheng Sheng / 周征晟 >> wrote: > [snip] > >> Batch is a pacemaker concept I found when I was reading its >> documentation and code. There is a "batch-

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-04 Thread Zhou Zheng Sheng / 周征晟
Thank you Andrew. on 2015/05/05 08:03, Andrew Beekhof wrote: >> On 28 Apr 2015, at 11:15 pm, Bogdan Dobrelya wrote: >> >>> Hello, >> Hello, Zhou >> >>> I using Fuel 6.0.1 and find that RabbitMQ recover time is long after >>> power failure. I have a running HA environment, then I reset power of >>

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-04 Thread Andrew Beekhof
> On 29 Apr 2015, at 5:38 pm, Zhou Zheng Sheng / 周征晟 > wrote: [snip] > Batch is a pacemaker concept I found when I was reading its > documentation and code. There is a "batch-limit: 30" in the output of > "pcs property list --all". The pacemaker official documentation > explanation is that it'

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-04 Thread Andrew Beekhof
> On 28 Apr 2015, at 11:15 pm, Bogdan Dobrelya wrote: > >> Hello, > > Hello, Zhou > >> >> I using Fuel 6.0.1 and find that RabbitMQ recover time is long after >> power failure. I have a running HA environment, then I reset power of >> all the machines at the same time. I observe that after re

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-03 Thread Zhou Zheng Sheng / 周征晟
Hello Sergii, Thank you for the great explanation on Galera OCF script. I replied your question inline. on 2015/05/03 04:49, Sergii Golovatiuk wrote: > Hi Zhou, > > Galera OCF script is a bit special. Since MySQL keeps the most > important data we should find the most recent data on all nodes acr

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-05-02 Thread Sergii Golovatiuk
Hi Zhou, Galera OCF script is a bit special. Since MySQL keeps the most important data we should find the most recent data on all nodes across the cluster. check_if_galera_pc is specially designed for that. Every server registers the latest status from grastate.dat to CIB. Once all nodes are regis

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-04-29 Thread Zhou Zheng Sheng / 周征晟
Hi! Thank you very much Vladimir and Bogdan! Thanks for the fast respond and rich information. I backported MySQL and RabbitMQ ocf patches from stable/6.0 and tested again. A full reassemble takes about 5mins, this improves a lot. Adding the "force_load" trick I mentioned in the previous email, i

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-04-28 Thread Bogdan Dobrelya
On 28.04.2015 15:15, Bogdan Dobrelya wrote: > > Hello, Zhou > > > Yes, this is a known issue [0]. Note, there were many bugfixes, like > [1],[2],[3], merged for MQ OCF script, so you may want to try to > backport them as well by the following guide [4] > > [0] https://bugs.launchpad.net/fuel/+b

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-04-28 Thread Bogdan Dobrelya
> Hello, Hello, Zhou > > I using Fuel 6.0.1 and find that RabbitMQ recover time is long after > power failure. I have a running HA environment, then I reset power of > all the machines at the same time. I observe that after reboot it > usually takes 10 minutes for RabittMQ cluster to appear runni

Re: [openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-04-28 Thread Vladimir Kuklin
Hi, Zhou Thank you for writing these awesome recommendations. We will look into them and see whether they provide significant impact. BTW, we have found a bunch of issues with our 5.1 and 6.0 RabbitMQ OCF script and fixed them in current master. Would you be so kind as to check out the newest ver

[openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

2015-04-27 Thread Zhou Zheng Sheng / 周征晟
Hello, I using Fuel 6.0.1 and find that RabbitMQ recover time is long after power failure. I have a running HA environment, then I reset power of all the machines at the same time. I observe that after reboot it usually takes 10 minutes for RabittMQ cluster to appear running master-slave mode in p