Hi Sathwik,
I tried to implement your logic also. Following is the link for committed
code upto now.
https://github.com/Subasinghe/ode/tree/ODECluster
I used info msgs to debugging as it can be observed easily in console. When
releasing the lock acquired by poller or web service the state is set to
false as in my code. But sometimes it is set to true and at that time
second server's value is also true. Following is the related log for each
server.
Server 1
02:12:02,957 INFO [DeploymentPoller] Trying to access the lock for
MagicSession
02:12:02,962 INFO [HazelcastClusterImpl] ThreadID:63 duLocked value for
MagicSession file after locking: true
02:12:03,838 INFO [BpelServerImpl] Registered process {
http://ode/bpel/unit-test}MagicSessionMain-3.
02:12:04,015 INFO [BpelServerImpl] Registered process {
http://ode/bpel/responder}MagicSessionResponder-3.
02:12:04,015 INFO [DeploymentPoller] Deployment of artifact MagicSession
successful: [{http://ode/bpel/unit-test}MagicSessionMain-3, {
http://ode/bpel/responder}MagicSessionResponder-3]
02:12:04,015 INFO [DeploymentPoller] Trying to release the lock for
MagicSession
02:12:04,017 INFO [HazelcastClusterImpl] ThreadID:63 duLocked value for
MagicSession file after unlocking: true
Server 2
02:12:03,119 INFO [DeploymentPoller] Trying to access the lock for
MagicSession
02:12:04,017 INFO [HazelcastClusterImpl] ThreadID:61 duLocked value for
MagicSession file after locking: true
02:12:04,017 INFO [DeploymentPoller] Trying to release the lock for
MagicSession
02:12:04,019 INFO [HazelcastClusterImpl] ThreadID:61 duLocked value for
MagicSession file after unlocking: false
Is this possible? But there were not any conflicts while deploying. It
worked perfectly.
Thank you
On 7 June 2015 at 11:10, sudharma subasinghe <suba...@cse.mrt.ac.lk> wrote:
> Hi Sathwik,
>
> I will concern your approach. Thank you for your effort.
>
> On 6 June 2015 at 18:43, Sathwik B P <sathwik...@gmail.com> wrote:
>
>> I have attached to the JIRA https://issues.apache.org/jira/browse/ODE-563
>>
>> On Sat, Jun 6, 2015 at 5:55 PM, sudharma subasinghe <
>> suba...@cse.mrt.ac.lk>
>> wrote:
>>
>> > Hi Sathwik,
>> >
>> > I can't find the attached document. Please kind be enough to resend it.
>> >
>> > On 6 June 2015 at 15:42, Sathwik B P <sathwik...@gmail.com> wrote:
>> >
>> > > Refer this one as I have corrected the numbering of steps.
>> > >
>> > > On Sat, Jun 6, 2015 at 3:33 PM, Sathwik B P <sathwik...@gmail.com>
>> > wrote:
>> > >
>> > >> Hi Sudharma/Tammo,
>> > >>
>> > >> Kindly review attached document on my proposed approach. Let me know
>> if
>> > >> you have any concerns or doubts on it.
>> > >>
>> > >> regards,
>> > >> sathwik
>> > >>
>> > >> On Fri, Jun 5, 2015 at 6:56 PM, Sathwik B P <sathwik...@gmail.com>
>> > wrote:
>> > >>
>> > >>> find response inline
>> > >>>
>> > >>> On Wed, Jun 3, 2015 at 10:22 PM, sudharma subasinghe <
>> > >>> suba...@cse.mrt.ac.lk> wrote:
>> > >>>
>> > >>>> Hi,
>> > >>>>
>> > >>>> Sorry for the late reply. I was trying to achieve a proper
>> solution.
>> > >>>> Following is my approach.
>> > >>>>
>> > >>>> 1) I elected master node for deploying purpose. So when a master
>> node
>> > >>>> goes
>> > >>>> down hazelcast will elect the next oldest node for master.
>> > >>>>
>> > >>>
>> > >>> Perfect
>> > >>>
>> > >>>
>> > >>>> 2) ODEServer can identify whether clustering is enabled or not by
>> > >>>> getting
>> > >>>> the property value in ode-axis2.properties file. So I introduced
>> new
>> > >>>> property called "ode-axis2.hazelcast.clustering.enabled". If there
>> is
>> > no
>> > >>>> clustering enabled server will work as it is. If clustering is
>> > enabled,
>> > >>>> cluster will be initialized.
>> > >>>>
>> > >>>
>> > >>> Perfect
>> > >>>
>> > >>>
>> > >>>> 3) In manual deployment its responsibility is taken by the
>> > >>>> DeploymentPoller. So I give the deployment capability to master
>> node,s
>> > >>>> poller by setting "isDeploymentFromODEFileSystemAllowed()" to
>> true.So
>> > >>>> others will not be able to go into check() method in
>> DeploymentPoller.
>> > >>>> So
>> > >>>> deployment will be done by only the master node.
>> > >>>>
>> > >>>
>> > >>> Perfect
>> > >>>
>> > >>>
>> > >>>>
>> > >>>> 4) In DeploymentWebService, I had to consider few cases.If the
>> deploy
>> > >>>> request goes to the master node, it will deploy the process through
>> > web
>> > >>>> service.Others pollers will not go into check() method as they are
>> not
>> > >>>> masters. So master can continue without any involvement of others.
>> > >>>>
>> > >>>>
>> > >>> Perfect
>> > >>>
>> > >>>
>> > >>>> 5) If the deploy request goes to a slave node, it will do up to
>> file
>> > >>>> creation in the file system.Slave will be stopped at that point. As
>> > only
>> > >>>> master poller is checking, master can continue from created files
>> in
>> > the
>> > >>>> file system.
>> > >>>>
>> > >>>
>> > >>> DeploymentWebService provides synchronous operations. The
>> status of
>> > >>> the operation should be communicated to the calling client in the
>> same
>> > call.
>> > >>> DeploymentPoller is a backend thread that goes over each and
>> every
>> > >>> directory under the Deployment directory checking for any changes to
>> > >>> deploy.xml in existing processes and deploy newly added processes.
>> This
>> > >>> process is sequential and time consuming. As the process
>> directories
>> > >>> grows, so does the time taken for execution of the thread.
>> > >>>
>> > >>> Since the request is on a slave node and the processing is done
>> on
>> > >>> master node, how do you check for the completion of the
>> > >>> deployment/undeployment of processes and respond back to the client
>> > since
>> > >>> the web service call is a synchronous operation. As
>> DeploymentPoller is
>> > >>> taking a lot of time in processing, your request will time out
>> right.
>> > >>>
>> > >>>
>> > >>>>
>> > >>>> 6) But there was problem with _deploymentUnits in ProcessStoreImpl.
>> > Each
>> > >>>> _deploymentUnits stores only what its server has deployed. So
>> think,
>> > >>>> that a
>> > >>>> master node goes down another master node appears.But its
>> > >>>> __deploymentUnits
>> > >>>> does not have dus which has deployed by the earlier master node.
>> Hence
>> > >>>> it
>> > >>>> will not be able retire earlier version of the process which is
>> > >>>> deployed by
>> > >>>> previous master. So there will two process which are in "ACTIVE"
>> state
>> > >>>
>> > >>>
>> > >>>> 7) To avoid this, I add the ODEServer as an Observer to check when
>> a
>> > new
>> > >>>> master is electing, then load all the deployment units out of the
>> > >>>> store. So
>> > >>>> new master node can have all the dus and can retire appropriate
>> > version.
>> > >>>> Usually loadAll() is called at the server start-up time. But there
>> is
>> > no
>> > >>>> other way to solve this. I tried to use Hazelcast IMap to store all
>> > dus
>> > >>>> among all nodes. But it wasn't success as du is not serializable
>> > >>>> object.
>> > >>>>
>> > >>>
>> > >>>> 8) I figured out that we do not need send cluster message to
>> others as
>> > >>>> all
>> > >>>> the dus' data are persisted to the shared DB. So each node can take
>> > the
>> > >>>> du
>> > >>>> and retrieve necessary data using already implemented methods in
>> > Process
>> > >>>> Store.
>> > >>>>
>> > >>>> 9) But there is an another problem.The axis2 service corresponding
>> to
>> > a
>> > >>>> deployed process does not appear on all nodes of the cluster. That
>> is
>> > >>>> because each server add du which is deployed by it to the process
>> > >>>> store.That is why I had use loadAll() when masters are changing.
>> How
>> > to
>> > >>>> solve this?
>> > >>>>
>> > >>>
>> > >>> I do appriciate your efforts in understanding the implmentation
>> and
>> > >>> changes that need to be done. You are bang on it.
>> > >>> fireEvent(..) is the method that triggers process activation and
>> > >>> necessary service creation.
>> > >>>
>> > >>>
>> > >>> But with this given apporach from steps 6 to 8, ODE cannot have
>> > >>> atleast 2 Active servers for Load balancing. You are concentrating
>> on
>> > only
>> > >>> one active node that will do deployments and cater to process
>> > invocations.
>> > >>> We should also think about scaling ODE to multiple servers to
>> > handle
>> > >>> load.
>> > >>>
>> > >>> What do you think.
>> > >>>
>> > >>>
>> > >>>> Thank you,
>> > >>>> Sudharma
>> > >>>>
>> > >>>> On 2 June 2015 at 08:51, Sathwik B P <sathwik...@gmail.com> wrote:
>> > >>>>
>> > >>>> > Sudharma,
>> > >>>> >
>> > >>>> > Any updates?
>> > >>>> >
>> > >>>> > regards,
>> > >>>> > sathwik
>> > >>>> >
>> > >>>> > On Fri, May 29, 2015 at 5:26 PM, Sathwik B P <
>> sathwik...@gmail.com>
>> > >>>> wrote:
>> > >>>> >
>> > >>>> > > Sudharma,
>> > >>>> > >
>> > >>>> > > Can you elaborate on your option 1).
>> > >>>> > >
>> > >>>> > > Response to your option 2).
>> > >>>> > >
>> > >>>> > > Process Store is the component that handles process
>> metadata,
>> > >>>> > > compilation and deployment in ODE. Integration layers in ODE
>> > >>>> (Axis2, JBI)
>> > >>>> > > use the process store.
>> > >>>> > > Future implementations of IL for ODE will also use the
>> process
>> > >>>> store.
>> > >>>> > > We should not be thinking of moving the process store
>> > functionality
>> > >>>> to
>> > >>>> > the
>> > >>>> > > integration layers.
>> > >>>> > >
>> > >>>> > >
>> > >>>> > > On Thu, May 28, 2015 at 9:33 PM, sudharma subasinghe <
>> > >>>> > > suba...@cse.mrt.ac.lk> wrote:
>> > >>>> > >
>> > >>>> > >> Hi,
>> > >>>> > >>
>> > >>>> > >> I understood the problem within dynamic master/slave
>> > >>>> configuration. In
>> > >>>> > my
>> > >>>> > >> approach, when a deployment request is routed to a slave node
>> > >>>> there will
>> > >>>> > >> not be a deployment. I suggest two options to avoid it.
>> > >>>> > >> 1) Have static master/slave configuration only for deploy
>> process
>> > >>>> > >>
>> > >>>> > > 2) Modify the deployment web service to complie and verify the
>> > >>>> process
>> > >>>> > and
>> > >>>> > >> then copy it to the deploy folder irrespective of whether its
>> a
>> > >>>> master
>> > >>>> > or
>> > >>>> > >> slave, then deployment poller should take care of the
>> deployment
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> > >
>> > >>>> > >>
>> > >>>> > >> On 28 May 2015 at 14:43, Sathwik B P <sathwik...@gmail.com>
>> > wrote:
>> > >>>> > >>
>> > >>>> > >> > Sudharma,
>> > >>>> > >> >
>> > >>>> > >> > We definitely need a master/slave in the hazelcast cluster.
>> > This
>> > >>>> is
>> > >>>> > >> > probably needed for the job migration in the Scheduler to
>> > >>>> migrate the
>> > >>>> > >> jobs
>> > >>>> > >> > associated with a down node. Let hold on this topic for
>> future
>> > >>>> > >> discussion.
>> > >>>> > >> >
>> > >>>> > >> > Going by the explanation where the master/slave nodes have
>> > >>>> certain
>> > >>>> > >> > predefined tasks to perform is perfectly fine.
>> > >>>> > >> >
>> > >>>> > >> > I have this scenario,
>> > >>>> > >> >
>> > >>>> > >> > I am using HAProxy as my load balancer and configured 3
>> nodes
>> > in
>> > >>>> the
>> > >>>> > >> > cluster.
>> > >>>> > >> >
>> > >>>> > >> > Node1 - Active
>> > >>>> > >> > Node2 - Active
>> > >>>> > >> > Node3 - Backup
>> > >>>> > >> >
>> > >>>> > >> > Load balancing algorithm: RoundRobin
>> > >>>> > >> >
>> > >>>> > >> > A Backup node (Node3) is one which the load balancer will
>> not
>> > >>>> route
>> > >>>> > >> > requests to, until one of the Active node i.e either Node1
>> or
>> > >>>> Node2
>> > >>>> > has
>> > >>>> > >> > gone down.
>> > >>>> > >> >
>> > >>>> > >> > All these 3 nodes are also part of the hazelcast cluster as
>> > well.
>> > >>>> > >> >
>> > >>>> > >> > In the hazelcast cluster, assume Node1 is elected as the
>> > >>>> leader/master
>> > >>>> > >> and
>> > >>>> > >> > Node2,Node3 as slaves.
>> > >>>> > >> >
>> > >>>> > >> > I initiate the deploy operation on the DeploymentWebService
>> > >>>> which the
>> > >>>> > >> load
>> > >>>> > >> > balancer routes it to one of the Active nodes in the
>> cluster,
>> > >>>> lets say
>> > >>>> > >> it's
>> > >>>> > >> > the Node1. Since Node1 is also the master in the hazelcast
>> > >>>> cluster,
>> > >>>> > >> > deployment is a success.
>> > >>>> > >> >
>> > >>>> > >> > I initiate another deploy operation on the
>> DeploymentWebService
>> > >>>> which
>> > >>>> > >> the
>> > >>>> > >> > load balancer routes it to the next active node which is
>> Node2.
>> > >>>> Since
>> > >>>> > >> Node2
>> > >>>> > >> > is a slave in the Hazelcast cluster, What happens to the
>> > >>>> deployment?
>> > >>>> > >> >
>> > >>>> > >> > regards,
>> > >>>> > >> > sathwik
>> > >>>> > >> >
>> > >>>> > >> > On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
>> > >>>> > >> > suba...@cse.mrt.ac.lk
>> > >>>> > >> > > wrote:
>> > >>>> > >> >
>> > >>>> > >> > > Hi,
>> > >>>> > >> > >
>> > >>>> > >> > > I will explain my approach as much as possible. The oldest
>> > >>>> node in
>> > >>>> > the
>> > >>>> > >> > > hazelcast cluster is elected as the master node. In the
>> > >>>> failure of
>> > >>>> > the
>> > >>>> > >> > > master node, next oldest node will be elected as the
>> master
>> > >>>> node.
>> > >>>> > This
>> > >>>> > >> > > master-slave configuration is just for deployment. When
>> the
>> > >>>> > hazelcast
>> > >>>> > >> > > cluster elected the master node, that node becomes a
>> master
>> > >>>> node for
>> > >>>> > >> > > deploying process. So it will do the deploying artifacts.
>> If
>> > >>>> you
>> > >>>> > want
>> > >>>> > >> to
>> > >>>> > >> > > get the idea of electing master node please refer the code
>> > >>>> which I
>> > >>>> > >> have
>> > >>>> > >> > > located in the github. (
>> > >>>> > >> > > https://github.com/Subasinghe/ode/tree/ode_clustering)
>> > >>>> > >> > >
>> > >>>> > >> > > I identified separated actions which should be followed by
>> > the
>> > >>>> > master
>> > >>>> > >> and
>> > >>>> > >> > > salve nodes.
>> > >>>> > >> > > Actions which are followed by master node only
>> > >>>> > >> > > 1) create deployment unit
>> > >>>> > >> > > 2) set the version nu to deployment unit
>> > >>>> > >> > > 3) compile deployment unit
>> > >>>> > >> > > 4) scan deployment unit
>> > >>>> > >> > > 5) retire previous versions
>> > >>>> > >> > > Master node and slave nodes should create _processes which
>> > >>>> stores
>> > >>>> > >> > > ProcessConfImpl
>> > >>>> > >> > > Only master node will write the version nu to database,
>> > create
>> > >>>> > >> .deployed
>> > >>>> > >> > > file
>> > >>>> > >> > >
>> > >>>> > >> > > So there are some actions which should be followed only by
>> > >>>> master
>> > >>>> > node
>> > >>>> > >> > > while other actions should be followed by all the
>> nodes.The
>> > >>>> idea of
>> > >>>> > >> > having
>> > >>>> > >> > > a master node is deploying artifacts and avoid others from
>> > >>>> writing
>> > >>>> > the
>> > >>>> > >> > > version nu to database.
>> > >>>> > >> > > Whether a node is active or passive, all nodes should do
>> the
>> > >>>> > >> > > deployment.Master
>> > >>>> > >> > > and slaves will follow necessary actions as in above.
>> > >>>> > >> > >
>> > >>>> > >> > >
>> > >>>> > >> > >
>> > >>>> > >> > >
>> > >>>> > >> > >
>> > >>>> > >> > > On 27 May 2015 at 15:49, Sathwik B P <
>> sathwik...@gmail.com>
>> > >>>> wrote:
>> > >>>> > >> > >
>> > >>>> > >> > > > Nandika,
>> > >>>> > >> > > >
>> > >>>> > >> > > > I very well understand what you have put across, but
>> it's
>> > >>>> > secondary
>> > >>>> > >> to
>> > >>>> > >> > me
>> > >>>> > >> > > > now.
>> > >>>> > >> > > >
>> > >>>> > >> > > > Sudharma,
>> > >>>> > >> > > > My primary concern is to understand at a high level the
>> > >>>> deployment
>> > >>>> > >> > > > architecture and how would master-slave configuration
>> fit
>> > >>>> in. Are
>> > >>>> > >> there
>> > >>>> > >> > > any
>> > >>>> > >> > > > restrictions imposed by the in-progress design?
>> > >>>> > >> > > >
>> > >>>> > >> > > > Firstly, how would ODE process deployment work under
>> these
>> > >>>> cluster
>> > >>>> > >> > > > configurations?
>> > >>>> > >> > > >
>> > >>>> > >> > > > Sample Cluster configurations: A load balancer is
>> > >>>> frontending the
>> > >>>> > >> > > servers.
>> > >>>> > >> > > > 1) Cluster consisting of 2 nodes all Active-Active.
>> > >>>> > >> > > > 2) Cluster consisting of 2 nodes Active-Passive.
>> > >>>> > >> > > > 3) Cluster with 2+ nodes with additional nodes either in
>> > >>>> Active or
>> > >>>> > >> > > Passive.
>> > >>>> > >> > > >
>> > >>>> > >> > > > regards,
>> > >>>> > >> > > > sathwik
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > >
>> > >>>> > >> > > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
>> > >>>> > >> > jayaw...@gmail.com
>> > >>>> > >> > > >
>> > >>>> > >> > > > wrote:
>> > >>>> > >> > > >
>> > >>>> > >> > > > > Hi Sathwik,
>> > >>>> > >> > > > >
>> > >>>> > >> > > > > According to my understanding, in the clustering
>> > scenario,
>> > >>>> the
>> > >>>> > >> master
>> > >>>> > >> > > > node
>> > >>>> > >> > > > > should perform all the deployment actions and the
>> slave
>> > >>>> nodes
>> > >>>> > also
>> > >>>> > >> > need
>> > >>>> > >> > > > to
>> > >>>> > >> > > > > perform some deployment actions. For example, the
>> slave
>> > >>>> nodes
>> > >>>> > also
>> > >>>> > >> > > should
>> > >>>> > >> > > > > handle the process ACTIVATED event so that the process
>> > >>>> > >> configuration
>> > >>>> > >> > is
>> > >>>> > >> > > > > added to the engine and necessary web services are
>> > created
>> > >>>> so
>> > >>>> > that
>> > >>>> > >> > when
>> > >>>> > >> > > > the
>> > >>>> > >> > > > > load balancer send requests to any node in the
>> cluster,
>> > it
>> > >>>> is
>> > >>>> > >> ready
>> > >>>> > >> > to
>> > >>>> > >> > > > > accept those requests.
>> > >>>> > >> > > > >
>> > >>>> > >> > > > > Regards
>> > >>>> > >> > > > > Nandika
>> > >>>> > >> > > > >
>> > >>>> > >> > > > >
>> > >>>> > >> > > > >
>> > >>>> > >> > > > >
>> > >>>> > >> > > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <
>> > >>>> > >> sathwik...@gmail.com>
>> > >>>> > >> > > > > wrote:
>> > >>>> > >> > > > >
>> > >>>> > >> > > > > > Sudharma,
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > > > Where are you going to configure the master-slaves,
>> is
>> > >>>> it in
>> > >>>> > the
>> > >>>> > >> > web
>> > >>>> > >> > > > > > application level or at the load balancer?
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > > > regards,
>> > >>>> > >> > > > > > sathwik
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma
>> subasinghe <
>> > >>>> > >> > > > > > suba...@cse.mrt.ac.lk>
>> > >>>> > >> > > > > > wrote:
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > > > > Hi Tammo,
>> > >>>> > >> > > > > > >
>> > >>>> > >> > > > > > > Can you suggest the best method from these to
>> > >>>> implement? As
>> > >>>> > >> > first I
>> > >>>> > >> > > > > > > suggested the master-slaves scenario I think it is
>> > >>>> easy to
>> > >>>> > >> > > implement
>> > >>>> > >> > > > > than
>> > >>>> > >> > > > > > > distributed lock scenario. However if you can
>> suggest
>> > >>>> one
>> > >>>> > from
>> > >>>> > >> > > these
>> > >>>> > >> > > > > two,
>> > >>>> > >> > > > > > > then I can think about it.
>> > >>>> > >> > > > > > >
>> > >>>> > >> > > > > > > Thank you
>> > >>>> > >> > > > > > >
>> > >>>> > >> > > > > > > On 21 May 2015 at 12:40, Sathwik B P <
>> > >>>> sathwik...@gmail.com>
>> > >>>> > >> > wrote:
>> > >>>> > >> > > > > > >
>> > >>>> > >> > > > > > > > With respect to the hotdeployment,
>> > >>>> > >> > > > > > > >
>> > >>>> > >> > > > > > > > We can drop the deployment archive onto the
>> > >>>> deployment
>> > >>>> > >> folder.
>> > >>>> > >> > > > Since
>> > >>>> > >> > > > > > the
>> > >>>> > >> > > > > > > > DeploymentPoller are acquiring the distributed
>> lock
>> > >>>> for
>> > >>>> > the
>> > >>>> > >> > > > > > > DeploymentUnit,
>> > >>>> > >> > > > > > > > only one of the nodes will get the lock and
>> > initiate
>> > >>>> the
>> > >>>> > >> > > > deployment.
>> > >>>> > >> > > > > > > > DeploymentPollers on other nodes will fail in
>> > >>>> acquiring
>> > >>>> > the
>> > >>>> > >> > lock
>> > >>>> > >> > > > and
>> > >>>> > >> > > > > > > hence
>> > >>>> > >> > > > > > > > will silently ignore it.
>> > >>>> > >> > > > > > > >
>> > >>>> > >> > > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
>> > >>>> > >> > > > sathwik...@gmail.com>
>> > >>>> > >> > > > > > > > wrote:
>> > >>>> > >> > > > > > > >
>> > >>>> > >> > > > > > > > > Hi Tammo,
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > The distributed lock acquisition on the
>> > >>>> DeploymentUnit
>> > >>>> > >> should
>> > >>>> > >> > > be
>> > >>>> > >> > > > > > added
>> > >>>> > >> > > > > > > to
>> > >>>> > >> > > > > > > > > both DeploymentWebService and
>> DeploymentPoller.
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > When a deployment operation is initiated
>> through
>> > >>>> the
>> > >>>> > >> > > > > > > > DeploymentWebService,
>> > >>>> > >> > > > > > > > > The load balancer routes it to any of the
>> > available
>> > >>>> > nodes.
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > On the routed node, the DeploymentWebService
>> > >>>> acquires
>> > >>>> > the
>> > >>>> > >> > > > > Distributed
>> > >>>> > >> > > > > > > > > lock. On the remaining nodes the
>> DeploymentPoller
>> > >>>> will
>> > >>>> > >> try to
>> > >>>> > >> > > > > acquire
>> > >>>> > >> > > > > > > the
>> > >>>> > >> > > > > > > > > distributed lock and will not get it and hence
>> > will
>> > >>>> > >> silently
>> > >>>> > >> > > > ignore
>> > >>>> > >> > > > > > it.
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > Once the routed node completes the
>> deployment, it
>> > >>>> will
>> > >>>> > >> > release
>> > >>>> > >> > > > the
>> > >>>> > >> > > > > > > lock.
>> > >>>> > >> > > > > > > > > This way we don't have to stall the
>> > >>>> DeploymentPoller in
>> > >>>> > >> other
>> > >>>> > >> > > > > nodes.
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > Does it answer the concerns?
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > Now, if we give the responsibility of
>> identifying
>> > >>>> the
>> > >>>> > >> master
>> > >>>> > >> > > node
>> > >>>> > >> > > > > to
>> > >>>> > >> > > > > > > the
>> > >>>> > >> > > > > > > > > hazelcast, how do we plan to intimate the load
>> > >>>> balancer
>> > >>>> > to
>> > >>>> > >> > > change
>> > >>>> > >> > > > > > it's
>> > >>>> > >> > > > > > > > > configuration about the master node?
>> > >>>> > >> > > > > > > > > Assuming there are 3 nodes in the cluster,
>> > >>>> > >> > > > > > > > > node1 -master
>> > >>>> > >> > > > > > > > > node2 - slave
>> > >>>> > >> > > > > > > > > node3 - slave
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > Node1 goes down, the LB will promote Node2 as
>> > >>>> master
>> > >>>> > node,
>> > >>>> > >> > but
>> > >>>> > >> > > > > > > hazelcast
>> > >>>> > >> > > > > > > > > might promote Node3 as master node. They are
>> out
>> > of
>> > >>>> > sync.
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > Is this argument valid?
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > regards,
>> > >>>> > >> > > > > > > > > sathwik
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van
>> > Lessen <
>> > >>>> > >> > > > > > > tvanles...@gmail.com>
>> > >>>> > >> > > > > > > > > wrote:
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > >> Hi Sudharma,
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >> what do you expect from the "other nodes
>> > >>>> deployment"?
>> > >>>> > >> > > > Compilation
>> > >>>> > >> > > > > is
>> > >>>> > >> > > > > > > not
>> > >>>> > >> > > > > > > > >> needed since the CBP file is written to the
>> > >>>> (shared)
>> > >>>> > FS.
>> > >>>> > >> > > > > > Registration
>> > >>>> > >> > > > > > > is
>> > >>>> > >> > > > > > > > >> also not needed, since it is done via the
>> shared
>> > >>>> > >> database.
>> > >>>> > >> > So
>> > >>>> > >> > > > the
>> > >>>> > >> > > > > > only
>> > >>>> > >> > > > > > > > >> thing that might be needed is to tell the
>> engine
>> > >>>> that
>> > >>>> > >> there
>> > >>>> > >> > > is a
>> > >>>> > >> > > > > new
>> > >>>> > >> > > > > > > > >> deployment. I'd need to check that. If this
>> is
>> > >>>> needed,
>> > >>>> > I
>> > >>>> > >> > > revert
>> > >>>> > >> > > > my
>> > >>>> > >> > > > > > > last
>> > >>>> > >> > > > > > > > >> statement, then it is perhaps better to just
>> > send
>> > >>>> an
>> > >>>> > >> event
>> > >>>> > >> > > over
>> > >>>> > >> > > > > > > > Hazelcast
>> > >>>> > >> > > > > > > > >> to all nodes that the deployment has changed.
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >> Best,
>> > >>>> > >> > > > > > > > >> Tammo
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma
>> > >>>> subasinghe <
>> > >>>> > >> > > > > > > > >> suba...@cse.mrt.ac.lk
>> > >>>> > >> > > > > > > > >> > wrote:
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >> > Hi Tammo,
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> > The master node writes meta data. But
>> runtime
>> > >>>> > >> information
>> > >>>> > >> > > must
>> > >>>> > >> > > > > be
>> > >>>> > >> > > > > > > > >> available
>> > >>>> > >> > > > > > > > >> > in all nodes.Since the folder is shared,
>> all
>> > >>>> nodes
>> > >>>> > will
>> > >>>> > >> > see
>> > >>>> > >> > > > the
>> > >>>> > >> > > > > > > > >> > availability of a new process. My idea is
>> for
>> > >>>> master
>> > >>>> > >> node
>> > >>>> > >> > to
>> > >>>> > >> > > > > write
>> > >>>> > >> > > > > > > the
>> > >>>> > >> > > > > > > > >> meta
>> > >>>> > >> > > > > > > > >> > data and other nodes to just read the meta
>> > data
>> > >>>> and
>> > >>>> > >> load
>> > >>>> > >> > > > > > process.So
>> > >>>> > >> > > > > > > we
>> > >>>> > >> > > > > > > > >> need
>> > >>>> > >> > > > > > > > >> > a small delay between master node
>> deployment
>> > and
>> > >>>> > other
>> > >>>> > >> > nodes
>> > >>>> > >> > > > > > > > deployment.
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> > Is there anyway to set the delay between
>> > master
>> > >>>> node
>> > >>>> > >> and
>> > >>>> > >> > > > slaves
>> > >>>> > >> > > > > > > until
>> > >>>> > >> > > > > > > > >> > master node finish the deployment?
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> > Thank you
>> > >>>> > >> > > > > > > > >> > Sudharma
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen <
>> > >>>> > >> > > > tvanles...@gmail.com
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > > > > > wrote:
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >> > > Hi Sathwik,
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik
>> B
>> > P <
>> > >>>> > >> > > > > > > sathwik...@gmail.com>
>> > >>>> > >> > > > > > > > >> > wrote:
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > > Sudharma/Tammo,
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > > > 1) How do we plan to decide which is
>> the
>> > >>>> master
>> > >>>> > >> node
>> > >>>> > >> > in
>> > >>>> > >> > > > the
>> > >>>> > >> > > > > > > > cluster?
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > I think the easiest approach is to always
>> > >>>> elect the
>> > >>>> > >> > oldest
>> > >>>> > >> > > > > node
>> > >>>> > >> > > > > > in
>> > >>>> > >> > > > > > > > the
>> > >>>> > >> > > > > > > > >> > > cluster to be the master. AFAIK Hazelcast
>> > can
>> > >>>> > easily
>> > >>>> > >> > asked
>> > >>>> > >> > > > for
>> > >>>> > >> > > > > > > this
>> > >>>> > >> > > > > > > > >> > > information.
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > > 2) Don't we need to stall the
>> Deployment
>> > >>>> Pollers
>> > >>>> > in
>> > >>>> > >> > the
>> > >>>> > >> > > > > slave
>> > >>>> > >> > > > > > > > nodes?
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > > Absolutely.
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > Suggestion:
>> > >>>> > >> > > > > > > > >> > > > I am not sure whether do we need
>> > >>>> Master-SLaves.
>> > >>>> > Why
>> > >>>> > >> > not
>> > >>>> > >> > > > give
>> > >>>> > >> > > > > > > every
>> > >>>> > >> > > > > > > > >> node
>> > >>>> > >> > > > > > > > >> > > in
>> > >>>> > >> > > > > > > > >> > > > the cluster the same status
>> > (Active-Active).
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > > > When a new deployment is made, the load
>> > >>>> balancer
>> > >>>> > >> can
>> > >>>> > >> > > push
>> > >>>> > >> > > > it
>> > >>>> > >> > > > > > to
>> > >>>> > >> > > > > > > > any
>> > >>>> > >> > > > > > > > >> of
>> > >>>> > >> > > > > > > > >> > > the
>> > >>>> > >> > > > > > > > >> > > > available nodes. That node will
>> probably
>> > >>>> acquire
>> > >>>> > a
>> > >>>> > >> > > > > distributed
>> > >>>> > >> > > > > > > > lock
>> > >>>> > >> > > > > > > > >> on
>> > >>>> > >> > > > > > > > >> > > the
>> > >>>> > >> > > > > > > > >> > > > deployment unit and acts as master for
>> > that
>> > >>>> > >> > deployment.
>> > >>>> > >> > > > This
>> > >>>> > >> > > > > > > > ensures
>> > >>>> > >> > > > > > > > >> > > > optimum usage of the cluster nodes.
>> > >>>> Probably no
>> > >>>> > >> static
>> > >>>> > >> > > > > > > > >> configuration of
>> > >>>> > >> > > > > > > > >> > > > Master-Slave in the load balancer nor
>> in
>> > the
>> > >>>> > >> > hazelcast.
>> > >>>> > >> > > > > > > > >> > > >
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > But this would not allow to have the
>> > >>>> hotdeployment
>> > >>>> > >> via
>> > >>>> > >> > > > > > filesystem
>> > >>>> > >> > > > > > > > >> still
>> > >>>> > >> > > > > > > > >> > > enabled, right?
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > Best,
>> > >>>> > >> > > > > > > > >> > > Tammo
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> > > --
>> > >>>> > >> > > > > > > > >> > > Tammo van Lessen - http://www.taval.de
>> > >>>> > >> > > > > > > > >> > >
>> > >>>> > >> > > > > > > > >> >
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >> --
>> > >>>> > >> > > > > > > > >> Tammo van Lessen - http://www.taval.de
>> > >>>> > >> > > > > > > > >>
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > > >
>> > >>>> > >> > > > > > > >
>> > >>>> > >> > > > > > >
>> > >>>> > >> > > > > >
>> > >>>> > >> > > > >
>> > >>>> > >> > > >
>> > >>>> > >> > >
>> > >>>> > >> >
>> > >>>> > >>
>> > >>>> > >
>> > >>>> > >
>> > >>>> >
>> > >>>>
>> > >>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>
