If the reason to avoid leader election is because it is complicated and error 
prone, this argument may not be true. Leader election is complicated in a pure 
distributed system in which there is no centralized storage. However, Magnum 
has a centralized database, so it is possible to implement a very simple leader 
election algorithm. For example, we can let each conductor register itself in 
the DB and elect the first registered conductor to be the leader. How about 

Best regards,

From: Adrian Otto []
Sent: August-03-15 12:49 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Magnum]horizontal scalability

On Aug 2, 2015, at 7:40 PM, 王华 
<<>> wrote:

Hi all,

As discussed in the Vancouver Summit, we are going to drop the bay lock 
implementation. Instead, each conductor will call Heat concurrently and rely on 
heat for concurrency control. However, I think we need an approach for state 
convergence from heat to magnum. Either periodic task [1] or heat notification 
[2] looks like a candidate.
If we use periodic task to sync state from heat to magnum, I think we should 
make periodic task a independent process and magnum-conductor only operates 
heat stack.
How to make periodic task high available?
1.We can run several periodic tasks.
2.Or we can use a leader selection mechanism to make only a periodic task 
and other periodic tasks waiting.
Shall we make periodic task a independent process? How to make periodic task 
high available?
Good question. The traditional solution for handling this in a distributed 
system is to hold a leader election. The elected leader is responsible for 
dispatching the job to a queue that one available worker will pick up and run. 
However, that may not actually be needed in our case. Consider the question:

What harm will come if all master nodes in the cluster perform the same 
periodic task?

One drawback is that more resources will be consumed than necessary 
(efficiency). As long as updates to the Magnum database are transactional, 
having concurrent updates to the same bay is actually not something we expect 
would result in data corruption. Worst case the same update would be processed 
multiple times. The advantage of using this approach is that we would not need 
to implement any form of leader selection. This would keep our implementation 
simpler, and less error prone.

We could still supervise each periodic task process so if they end up crashing 
on each node, they would be restarted. This is simple to do from a patent 
process that calls os.wait() on the child task.


OpenStack Development Mailing List (not for usage questions)

Reply via email to