Hi, I understood the problem within dynamic master/slave configuration. In my approach, when a deployment request is routed to a slave node there will not be a deployment. I suggest two options to avoid it. 1) Have static master/slave configuration only for deploy process 2) Modify the deployment web service to complie and verify the process and then copy it to the deploy folder irrespective of whether its a master or slave, then deployment poller should take care of the deployment
On 28 May 2015 at 14:43, Sathwik B P <sathwik...@gmail.com> wrote: > Sudharma, > > We definitely need a master/slave in the hazelcast cluster. This is > probably needed for the job migration in the Scheduler to migrate the jobs > associated with a down node. Let hold on this topic for future discussion. > > Going by the explanation where the master/slave nodes have certain > predefined tasks to perform is perfectly fine. > > I have this scenario, > > I am using HAProxy as my load balancer and configured 3 nodes in the > cluster. > > Node1 - Active > Node2 - Active > Node3 - Backup > > Load balancing algorithm: RoundRobin > > A Backup node (Node3) is one which the load balancer will not route > requests to, until one of the Active node i.e either Node1 or Node2 has > gone down. > > All these 3 nodes are also part of the hazelcast cluster as well. > > In the hazelcast cluster, assume Node1 is elected as the leader/master and > Node2,Node3 as slaves. > > I initiate the deploy operation on the DeploymentWebService which the load > balancer routes it to one of the Active nodes in the cluster, lets say it's > the Node1. Since Node1 is also the master in the hazelcast cluster, > deployment is a success. > > I initiate another deploy operation on the DeploymentWebService which the > load balancer routes it to the next active node which is Node2. Since Node2 > is a slave in the Hazelcast cluster, What happens to the deployment? > > regards, > sathwik > > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe < > suba...@cse.mrt.ac.lk > > wrote: > > > Hi, > > > > I will explain my approach as much as possible. The oldest node in the > > hazelcast cluster is elected as the master node. In the failure of the > > master node, next oldest node will be elected as the master node. This > > master-slave configuration is just for deployment. When the hazelcast > > cluster elected the master node, that node becomes a master node for > > deploying process. So it will do the deploying artifacts. If you want to > > get the idea of electing master node please refer the code which I have > > located in the github. ( > > https://github.com/Subasinghe/ode/tree/ode_clustering) > > > > I identified separated actions which should be followed by the master and > > salve nodes. > > Actions which are followed by master node only > > 1) create deployment unit > > 2) set the version nu to deployment unit > > 3) compile deployment unit > > 4) scan deployment unit > > 5) retire previous versions > > Master node and slave nodes should create _processes which stores > > ProcessConfImpl > > Only master node will write the version nu to database, create .deployed > > file > > > > So there are some actions which should be followed only by master node > > while other actions should be followed by all the nodes.The idea of > having > > a master node is deploying artifacts and avoid others from writing the > > version nu to database. > > Whether a node is active or passive, all nodes should do the > > deployment.Master > > and slaves will follow necessary actions as in above. > > > > > > > > > > > > On 27 May 2015 at 15:49, Sathwik B P <sathwik...@gmail.com> wrote: > > > > > Nandika, > > > > > > I very well understand what you have put across, but it's secondary to > me > > > now. > > > > > > Sudharma, > > > My primary concern is to understand at a high level the deployment > > > architecture and how would master-slave configuration fit in. Are there > > any > > > restrictions imposed by the in-progress design? > > > > > > Firstly, how would ODE process deployment work under these cluster > > > configurations? > > > > > > Sample Cluster configurations: A load balancer is frontending the > > servers. > > > 1) Cluster consisting of 2 nodes all Active-Active. > > > 2) Cluster consisting of 2 nodes Active-Passive. > > > 3) Cluster with 2+ nodes with additional nodes either in Active or > > Passive. > > > > > > regards, > > > sathwik > > > > > > > > > > > > > > > > > > > > > > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana < > jayaw...@gmail.com > > > > > > wrote: > > > > > > > Hi Sathwik, > > > > > > > > According to my understanding, in the clustering scenario, the master > > > node > > > > should perform all the deployment actions and the slave nodes also > need > > > to > > > > perform some deployment actions. For example, the slave nodes also > > should > > > > handle the process ACTIVATED event so that the process configuration > is > > > > added to the engine and necessary web services are created so that > when > > > the > > > > load balancer send requests to any node in the cluster, it is ready > to > > > > accept those requests. > > > > > > > > Regards > > > > Nandika > > > > > > > > > > > > > > > > > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <sathwik...@gmail.com> > > > > wrote: > > > > > > > > > Sudharma, > > > > > > > > > > Where are you going to configure the master-slaves, is it in the > web > > > > > application level or at the load balancer? > > > > > > > > > > regards, > > > > > sathwik > > > > > > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe < > > > > > suba...@cse.mrt.ac.lk> > > > > > wrote: > > > > > > > > > > > Hi Tammo, > > > > > > > > > > > > Can you suggest the best method from these to implement? As > first I > > > > > > suggested the master-slaves scenario I think it is easy to > > implement > > > > than > > > > > > distributed lock scenario. However if you can suggest one from > > these > > > > two, > > > > > > then I can think about it. > > > > > > > > > > > > Thank you > > > > > > > > > > > > On 21 May 2015 at 12:40, Sathwik B P <sathwik...@gmail.com> > wrote: > > > > > > > > > > > > > With respect to the hotdeployment, > > > > > > > > > > > > > > We can drop the deployment archive onto the deployment folder. > > > Since > > > > > the > > > > > > > DeploymentPoller are acquiring the distributed lock for the > > > > > > DeploymentUnit, > > > > > > > only one of the nodes will get the lock and initiate the > > > deployment. > > > > > > > DeploymentPollers on other nodes will fail in acquiring the > lock > > > and > > > > > > hence > > > > > > > will silently ignore it. > > > > > > > > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P < > > > sathwik...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Tammo, > > > > > > > > > > > > > > > > The distributed lock acquisition on the DeploymentUnit should > > be > > > > > added > > > > > > to > > > > > > > > both DeploymentWebService and DeploymentPoller. > > > > > > > > > > > > > > > > When a deployment operation is initiated through the > > > > > > > DeploymentWebService, > > > > > > > > The load balancer routes it to any of the available nodes. > > > > > > > > > > > > > > > > On the routed node, the DeploymentWebService acquires the > > > > Distributed > > > > > > > > lock. On the remaining nodes the DeploymentPoller will try to > > > > acquire > > > > > > the > > > > > > > > distributed lock and will not get it and hence will silently > > > ignore > > > > > it. > > > > > > > > > > > > > > > > Once the routed node completes the deployment, it will > release > > > the > > > > > > lock. > > > > > > > > This way we don't have to stall the DeploymentPoller in other > > > > nodes. > > > > > > > > > > > > > > > > Does it answer the concerns? > > > > > > > > > > > > > > > > > > > > > > > > Now, if we give the responsibility of identifying the master > > node > > > > to > > > > > > the > > > > > > > > hazelcast, how do we plan to intimate the load balancer to > > change > > > > > it's > > > > > > > > configuration about the master node? > > > > > > > > Assuming there are 3 nodes in the cluster, > > > > > > > > node1 -master > > > > > > > > node2 - slave > > > > > > > > node3 - slave > > > > > > > > > > > > > > > > Node1 goes down, the LB will promote Node2 as master node, > but > > > > > > hazelcast > > > > > > > > might promote Node3 as master node. They are out of sync. > > > > > > > > > > > > > > > > Is this argument valid? > > > > > > > > > > > > > > > > regards, > > > > > > > > sathwik > > > > > > > > > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen < > > > > > > tvanles...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > >> Hi Sudharma, > > > > > > > >> > > > > > > > >> what do you expect from the "other nodes deployment"? > > > Compilation > > > > is > > > > > > not > > > > > > > >> needed since the CBP file is written to the (shared) FS. > > > > > Registration > > > > > > is > > > > > > > >> also not needed, since it is done via the shared database. > So > > > the > > > > > only > > > > > > > >> thing that might be needed is to tell the engine that there > > is a > > > > new > > > > > > > >> deployment. I'd need to check that. If this is needed, I > > revert > > > my > > > > > > last > > > > > > > >> statement, then it is perhaps better to just send an event > > over > > > > > > > Hazelcast > > > > > > > >> to all nodes that the deployment has changed. > > > > > > > >> > > > > > > > >> Best, > > > > > > > >> Tammo > > > > > > > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe < > > > > > > > >> suba...@cse.mrt.ac.lk > > > > > > > >> > wrote: > > > > > > > >> > > > > > > > >> > Hi Tammo, > > > > > > > >> > > > > > > > > >> > The master node writes meta data. But runtime information > > must > > > > be > > > > > > > >> available > > > > > > > >> > in all nodes.Since the folder is shared, all nodes will > see > > > the > > > > > > > >> > availability of a new process. My idea is for master node > to > > > > write > > > > > > the > > > > > > > >> meta > > > > > > > >> > data and other nodes to just read the meta data and load > > > > > process.So > > > > > > we > > > > > > > >> need > > > > > > > >> > a small delay between master node deployment and other > nodes > > > > > > > deployment. > > > > > > > >> > > > > > > > > >> > Is there anyway to set the delay between master node and > > > slaves > > > > > > until > > > > > > > >> > master node finish the deployment? > > > > > > > >> > > > > > > > > >> > Thank you > > > > > > > >> > Sudharma > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen < > > > tvanles...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > >> > > > > > > > > >> > > Hi Sathwik, > > > > > > > >> > > > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik B P < > > > > > > sathwik...@gmail.com> > > > > > > > >> > wrote: > > > > > > > >> > > > > > > > > > >> > > > Sudharma/Tammo, > > > > > > > >> > > > > > > > > > > >> > > > 1) How do we plan to decide which is the master node > in > > > the > > > > > > > cluster? > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > I think the easiest approach is to always elect the > oldest > > > > node > > > > > in > > > > > > > the > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast can easily > asked > > > for > > > > > > this > > > > > > > >> > > information. > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > 2) Don't we need to stall the Deployment Pollers in > the > > > > slave > > > > > > > nodes? > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > Absolutely. > > > > > > > >> > > > > > > > > > >> > > Suggestion: > > > > > > > >> > > > I am not sure whether do we need Master-SLaves. Why > not > > > give > > > > > > every > > > > > > > >> node > > > > > > > >> > > in > > > > > > > >> > > > the cluster the same status (Active-Active). > > > > > > > >> > > > > > > > > > > >> > > > When a new deployment is made, the load balancer can > > push > > > it > > > > > to > > > > > > > any > > > > > > > >> of > > > > > > > >> > > the > > > > > > > >> > > > available nodes. That node will probably acquire a > > > > distributed > > > > > > > lock > > > > > > > >> on > > > > > > > >> > > the > > > > > > > >> > > > deployment unit and acts as master for that > deployment. > > > This > > > > > > > ensures > > > > > > > >> > > > optimum usage of the cluster nodes. Probably no static > > > > > > > >> configuration of > > > > > > > >> > > > Master-Slave in the load balancer nor in the > hazelcast. > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > But this would not allow to have the hotdeployment via > > > > > filesystem > > > > > > > >> still > > > > > > > >> > > enabled, right? > > > > > > > >> > > > > > > > > > >> > > Best, > > > > > > > >> > > Tammo > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > -- > > > > > > > >> > > Tammo van Lessen - http://www.taval.de > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> -- > > > > > > > >> Tammo van Lessen - http://www.taval.de > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
