Hi guys,

So the generalized coordination component was done by SameeraR, and the
discussions for that can be seen at [1] and [2]. We've identified some
improvements that also can be done to it, to make it more stable.

[1] "Updated Invitation: Code Review for the implementation of RDBMS based
coordin... @ Thu Dec 1, 2016 2pm - 3pm (IST) (WSO2 Engineering Group)"
[2] "Implementing a RDBMS based leader election mechanism"

Cheers,
Anjana.

On Wed, Nov 30, 2016 at 4:21 PM, Asanka Abeyweera <asank...@wso2.com> wrote:

> Hi all,
>
> Following is the PR[1] related the implementation. This is currently
> merged to the master branch and will be a feature of the next MB release
> (3.2.0).
>
> [1] https://github.com/wso2/andes/pull/668
>
> On Mon, Nov 7, 2016 at 12:45 PM, Ramith Jayasinghe <ram...@wso2.com>
> wrote:
>
>> +1
>>
>> On Mon, Nov 7, 2016 at 12:40 PM, Anjana Fernando <anj...@wso2.com> wrote:
>>
>>> Hi Ramith,
>>>
>>> Sure. Actually, I was talking with SameeraR to take over this and create
>>> a common component which has the required coordination functionality. The
>>> idea is to create a component, where the providers can be plugged in, such
>>> as the RDBMS based one, ZK, or any other container specific provider that
>>> maybe out there.
>>>
>>> Cheers,
>>> Anjana.
>>>
>>> On Mon, Nov 7, 2016 at 12:38 PM, Ramith Jayasinghe <ram...@wso2.com>
>>> wrote:
>>>
>>>> this might require some work.. shall we have a chat?
>>>>
>>>> On Thu, Nov 3, 2016 at 3:52 PM, Anjana Fernando <anj...@wso2.com>
>>>> wrote:
>>>>
>>>>> Ping! ..
>>>>>
>>>>> On Wed, Nov 2, 2016 at 5:03 PM, Anjana Fernando <anj...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Wed, Nov 2, 2016 at 3:14 PM, Asanka Abeyweera <asank...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Anjana,
>>>>>>>
>>>>>>> Currently, the implementation is part of the MB code (not a common
>>>>>>> component).
>>>>>>>
>>>>>>
>>>>>> Okay, can we please get it as a common component.
>>>>>>
>>>>>> Cheers,
>>>>>> Anjana.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 2, 2016 at 3:00 PM, Anjana Fernando <anj...@wso2.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Asanka/Ramith,
>>>>>>>>
>>>>>>>> So for C5 based Streaming Analytics solution, we need coordination
>>>>>>>> functionality there as well. Is the functionality mentioned here 
>>>>>>>> created as
>>>>>>>> a common component or baked in to the MB code? .. if so, can we please 
>>>>>>>> get
>>>>>>>> it implemented it as a generic component, so other products can also 
>>>>>>>> use
>>>>>>>> it.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Anjana.
>>>>>>>>
>>>>>>>> On Tue, Aug 9, 2016 at 3:10 PM, Anjana Fernando <anj...@wso2.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Great! ..
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Anjana.
>>>>>>>>>
>>>>>>>>> On Tue, Aug 9, 2016 at 1:49 PM, Asanka Abeyweera <
>>>>>>>>> asank...@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Anjana,
>>>>>>>>>>
>>>>>>>>>> Thank you for the suggestion. We have already done a similar
>>>>>>>>>> thing. We have added a backoff time after creating the leader entry 
>>>>>>>>>> and
>>>>>>>>>> check if the leader entry is the entry created by self before 
>>>>>>>>>> informing the
>>>>>>>>>> leader change.
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 9, 2016 at 12:27 PM, Anjana Fernando <anj...@wso2.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> I see, thanks for the clarification, looks good! .. I think
>>>>>>>>>>> small thing to consider is, to avoid the situation where, the 
>>>>>>>>>>> current
>>>>>>>>>>> leader goes away, and two other competes to become the leader, and 
>>>>>>>>>>> the
>>>>>>>>>>> first one and the second one checks (reads) the table to check the 
>>>>>>>>>>> last
>>>>>>>>>>> heartbeat and figures out that the leader is outdated at the same 
>>>>>>>>>>> time, and
>>>>>>>>>>> then first one delete the entry and puts his one, and after that, 
>>>>>>>>>>> second
>>>>>>>>>>> one will also delete the existing one and put his one, so both will 
>>>>>>>>>>> think
>>>>>>>>>>> they became the leader, due to the condition that both succeeded in 
>>>>>>>>>>> adding
>>>>>>>>>>> the entry without an error. So this can probably be fixed by 
>>>>>>>>>>> checking back
>>>>>>>>>>> after a bit of time if the current node is actually me, which
>>>>>>>>>>> probabilistically will work well, if that time period is sufficient 
>>>>>>>>>>> big
>>>>>>>>>>> enough than a typical database transaction required by a node to do 
>>>>>>>>>>> the
>>>>>>>>>>> earlier operations. Or else, we should make sure the database 
>>>>>>>>>>> transaction
>>>>>>>>>>> level used in this scenario is at least REPEATABLE_READ, where when 
>>>>>>>>>>> we read
>>>>>>>>>>> the record, it will lock it throughout the transaction. So some 
>>>>>>>>>>> DBMSs does
>>>>>>>>>>> not support REPEATABLE_READ, where in that case, we should be able 
>>>>>>>>>>> to use
>>>>>>>>>>> SERIALIZABLE, which most of them support.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Anjana.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 9, 2016 at 11:11 AM, Maninda Edirisooriya <
>>>>>>>>>>> mani...@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Anjana,
>>>>>>>>>>>>
>>>>>>>>>>>> After having an offline chat with Asanka what I understood was
>>>>>>>>>>>> that the leader election was done completely via the database but 
>>>>>>>>>>>> with no
>>>>>>>>>>>> network communication. The leader is mentioned in the database 
>>>>>>>>>>>> first. Then
>>>>>>>>>>>> the leader updates the node data periodically in the database. If 
>>>>>>>>>>>> some node
>>>>>>>>>>>> realizes the data in the DB are outdated that means the leader was
>>>>>>>>>>>> disconnected. Then that node will look at the created timestamp of 
>>>>>>>>>>>> the
>>>>>>>>>>>> leader entry. If that is not very recent that means there was no 
>>>>>>>>>>>> new leader
>>>>>>>>>>>> elected recently. So he will try to update the leader entry with 
>>>>>>>>>>>> his ID. As
>>>>>>>>>>>> I understand there the leader entry is using the leader ID and the
>>>>>>>>>>>> timestamp as the primary key. Even several nodes try to do it
>>>>>>>>>>>> simultaneously only one node will successfully be able to update 
>>>>>>>>>>>> the entry
>>>>>>>>>>>> with the help of atomicity provided by the DB. Others members will 
>>>>>>>>>>>> note the
>>>>>>>>>>>> timestamp of the leader was updated so will accept the first one 
>>>>>>>>>>>> who
>>>>>>>>>>>> updates as the leader. Even after the leader is elected, the 
>>>>>>>>>>>> leader will
>>>>>>>>>>>> only notify node data via updating DB instead of network calls. 
>>>>>>>>>>>> Other nodes
>>>>>>>>>>>> will just observe it and check the latest timestmps of the entry.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *Maninda Edirisooriya*
>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>
>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware.
>>>>>>>>>>>>
>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/
>>>>>>>>>>>> *E-mail* : mani...@wso2.com
>>>>>>>>>>>> *Skype* : @manindae
>>>>>>>>>>>> *Twitter* : @maninda
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 9, 2016 at 10:13 AM, Anjana Fernando <
>>>>>>>>>>>> anj...@wso2.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I just noticed this thread. I've some concerns on this
>>>>>>>>>>>>> implementations. First of all, I don't think the statement 
>>>>>>>>>>>>> mentioned here
>>>>>>>>>>>>> saying an external service such as ZooKeeper doesn't work, is 
>>>>>>>>>>>>> correct.
>>>>>>>>>>>>> Because, if you have a ZK cluster (it is suppose to be used as a 
>>>>>>>>>>>>> cluster),
>>>>>>>>>>>>> you will not have any issues. All the nodes have a list of 
>>>>>>>>>>>>> endpoints to all
>>>>>>>>>>>>> the ZK nodes and they connect to those, and ZK has a quorum based 
>>>>>>>>>>>>> mechanism
>>>>>>>>>>>>> in keeping its state. So this makes sure, all the users have a 
>>>>>>>>>>>>> single
>>>>>>>>>>>>> version of the ZK data.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, I guess the fundamental problem here in the split brain
>>>>>>>>>>>>> situation is, we need one external entity taking the decision 
>>>>>>>>>>>>> (e.g. ZK
>>>>>>>>>>>>> cluster), because it should have oversight to the whole 
>>>>>>>>>>>>> environment. I
>>>>>>>>>>>>> don't see how this RDBMS mechanism would solve that. Because, 
>>>>>>>>>>>>> what it gives
>>>>>>>>>>>>> is a central location of state persistence. But the decisions of 
>>>>>>>>>>>>> making who
>>>>>>>>>>>>> is the leader is taken by the users, which can be problematic. 
>>>>>>>>>>>>> Where when
>>>>>>>>>>>>> we have a network partition scenario in that occasion, two groups 
>>>>>>>>>>>>> of users
>>>>>>>>>>>>> will be overriding each other in the centralized RDBMS data 
>>>>>>>>>>>>> repeatedly and
>>>>>>>>>>>>> it will go on forever, where in the ZK situation, there will be 
>>>>>>>>>>>>> only one
>>>>>>>>>>>>> leader, and the guys in the other partition simply won't be able 
>>>>>>>>>>>>> to reach
>>>>>>>>>>>>> the leader, until its network issues are sorted.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I also think, as Imesh mentioned, creating a coordination
>>>>>>>>>>>>> algorithm from scratch may not be a wise decision, and we should 
>>>>>>>>>>>>> use proven
>>>>>>>>>>>>> technology/libraries to do that. And on a side note, the main 
>>>>>>>>>>>>> reason for
>>>>>>>>>>>>> not using ZK for this earlier was because of the hassle of 
>>>>>>>>>>>>> bringing up
>>>>>>>>>>>>> another set of servers when our products are clustered, and we 
>>>>>>>>>>>>> knew that
>>>>>>>>>>>>> the split brain scenario will occur in HZ, but maybe now we 
>>>>>>>>>>>>> should give an
>>>>>>>>>>>>> extension point probably to plug into an external service if for 
>>>>>>>>>>>>> some
>>>>>>>>>>>>> applications the split brain scenario is a show stopper.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Anjana.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 9, 2016 at 4:45 AM, Kasun Indrasiri <
>>>>>>>>>>>>> ka...@wso2.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Ramith/Asanka,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ESB/DSS natask impl is also based on HZ. I guess if this
>>>>>>>>>>>>>> model works for the MB, we should make it generic for all such 
>>>>>>>>>>>>>> coordination
>>>>>>>>>>>>>> requirements. (Thinking about using this in ESB 5.1)?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 3:58 AM, Sajini De Silva <
>>>>>>>>>>>>>> saj...@wso2.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Maninda,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Locking the  database will  be supported by some databases
>>>>>>>>>>>>>>> but there will be huge performance impact. So  we  cannot use 
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> approach. If this approach  cannot be adapted the only thing we 
>>>>>>>>>>>>>>> can do is
>>>>>>>>>>>>>>> queue wise load balancing through slot coordinator. But in this 
>>>>>>>>>>>>>>> case we
>>>>>>>>>>>>>>> cannot guarantee that load balance will be equally distributed 
>>>>>>>>>>>>>>> since some
>>>>>>>>>>>>>>> queues can be  loaded while some will be idle. Also we cannot 
>>>>>>>>>>>>>>> have multiple
>>>>>>>>>>>>>>> slot coordinators having same queue as it may cause several 
>>>>>>>>>>>>>>> complications
>>>>>>>>>>>>>>> such as, same slot is assigned to two nodes by different 
>>>>>>>>>>>>>>> subscribers,
>>>>>>>>>>>>>>> message duplication etc. Actually this slot architecture was 
>>>>>>>>>>>>>>> discussed in a
>>>>>>>>>>>>>>> separate mail thread before it is implemented.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 3:12 PM, Maninda Edirisooriya <
>>>>>>>>>>>>>>> mani...@wso2.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Sajini,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes that is what I meant. As the number of slots are
>>>>>>>>>>>>>>>> proportional to the number of messages passing through the 
>>>>>>>>>>>>>>>> cluster, slot
>>>>>>>>>>>>>>>> delivery should not be handled by the coordinator when there 
>>>>>>>>>>>>>>>> is only one
>>>>>>>>>>>>>>>> coordinator in the cluster which is a bottleneck for scaling 
>>>>>>>>>>>>>>>> messages
>>>>>>>>>>>>>>>> passing through the cluster. If there is only a single 
>>>>>>>>>>>>>>>> coordinator, it
>>>>>>>>>>>>>>>> should handle operations that are not proportional to messages 
>>>>>>>>>>>>>>>> throughput
>>>>>>>>>>>>>>>> of the cluster. Then only the tasks like subscriber adding / 
>>>>>>>>>>>>>>>> removing
>>>>>>>>>>>>>>>> should be handled by the coordinator. As this is not the 
>>>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>> implementation, we can consider multiple coordinator approach. 
>>>>>>>>>>>>>>>> Then the
>>>>>>>>>>>>>>>> number of coordinators should be scalable with the message 
>>>>>>>>>>>>>>>> throughout. I am
>>>>>>>>>>>>>>>> not sure whether locking the database per transaction would 
>>>>>>>>>>>>>>>> achieve this
>>>>>>>>>>>>>>>> coordinator scalability in the multiple coordinator 
>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Maninda Edirisooriya*
>>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/
>>>>>>>>>>>>>>>> *E-mail* : mani...@wso2.com
>>>>>>>>>>>>>>>> *Skype* : @manindae
>>>>>>>>>>>>>>>> *Twitter* : @maninda
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:42 PM, Sajini De Silva <
>>>>>>>>>>>>>>>> saj...@wso2.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Maninda,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:28 PM, Maninda Edirisooriya <
>>>>>>>>>>>>>>>>> mani...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> @Sajini,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> But the number of slots are proportional to the number of
>>>>>>>>>>>>>>>>>> messages pass through the MB which needs to be handled by 
>>>>>>>>>>>>>>>>>> the coordinator.
>>>>>>>>>>>>>>>>>> That is what I meant by "information related to meta data of 
>>>>>>>>>>>>>>>>>> messages pass
>>>>>>>>>>>>>>>>>> through a single coordinator". Ideally after the senders and 
>>>>>>>>>>>>>>>>>> receivers are
>>>>>>>>>>>>>>>>>> subscribed to the cluster, coordinator should have nothing 
>>>>>>>>>>>>>>>>>> to do until they
>>>>>>>>>>>>>>>>>> are removed or changed.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Even though it is possible to have multiple coordinators
>>>>>>>>>>>>>>>>> after having en effort (Lock the database for a whole 
>>>>>>>>>>>>>>>>> transaction or the
>>>>>>>>>>>>>>>>> work load distribution as described by Ramith) , coordinator 
>>>>>>>>>>>>>>>>> may have
>>>>>>>>>>>>>>>>> different work to do other than subscriber adding and 
>>>>>>>>>>>>>>>>> removing. As I said
>>>>>>>>>>>>>>>>> earlier our MB message distribution system is based on slot 
>>>>>>>>>>>>>>>>> architecture
>>>>>>>>>>>>>>>>> and slots will managed by the coordinator. You can read [1] 
>>>>>>>>>>>>>>>>> to understand
>>>>>>>>>>>>>>>>> more about slot architecture in MB.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1] http://sajinid.blogspot.co
>>>>>>>>>>>>>>>>> m/2015/03/wso2-message-broker-300-slot-based.html
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> @Ramith,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +1 for multiple coordinators by partitioning the cluster
>>>>>>>>>>>>>>>>>> which maintains the simplicity and correctness of the 
>>>>>>>>>>>>>>>>>> algorithm than
>>>>>>>>>>>>>>>>>> compromising simplicity with a less important factor like 
>>>>>>>>>>>>>>>>>> "delivering a
>>>>>>>>>>>>>>>>>> good mix of messages".
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Maninda Edirisooriya*
>>>>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/
>>>>>>>>>>>>>>>>>> *E-mail* : mani...@wso2.com
>>>>>>>>>>>>>>>>>> *Skype* : @manindae
>>>>>>>>>>>>>>>>>> *Twitter* : @maninda
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:05 PM, Ramith Jayasinghe <
>>>>>>>>>>>>>>>>>> ram...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> @Imesh,
>>>>>>>>>>>>>>>>>>>  We can prove that doing leader election using
>>>>>>>>>>>>>>>>>>> a lib (where we maintain cluster state in another place, 
>>>>>>>>>>>>>>>>>>> a.k.a DB) will not
>>>>>>>>>>>>>>>>>>> solve our original problem (this also relates to our past 
>>>>>>>>>>>>>>>>>>> experience with
>>>>>>>>>>>>>>>>>>> both the zookeeper and hazelcast).
>>>>>>>>>>>>>>>>>>>  We can make this implementation a common component if
>>>>>>>>>>>>>>>>>>> other products have a use of it. BPS might be able to use 
>>>>>>>>>>>>>>>>>>> it since their
>>>>>>>>>>>>>>>>>>> data is also in the database.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> @Malaka:
>>>>>>>>>>>>>>>>>>>  VFS scenario can't be solved by relying on this
>>>>>>>>>>>>>>>>>>> implementation. why? you can have the access to DB but not 
>>>>>>>>>>>>>>>>>>> VFS
>>>>>>>>>>>>>>>>>>> resources/file (and vice versa). this is the same point we 
>>>>>>>>>>>>>>>>>>> explained before.
>>>>>>>>>>>>>>>>>>>  in Ntask implementation,  if tasks are stored in the
>>>>>>>>>>>>>>>>>>> database then using this implementation makes sense.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> @Akila,
>>>>>>>>>>>>>>>>>>>  implementing (distributed) a queue algorithm is
>>>>>>>>>>>>>>>>>>> non-trivial. Having one coordinator (single source of 
>>>>>>>>>>>>>>>>>>> truth) keeps things
>>>>>>>>>>>>>>>>>>> simple hence it's a conscious design decision we agreed 
>>>>>>>>>>>>>>>>>>> during the initial
>>>>>>>>>>>>>>>>>>> stages. However, possible extension to this scheme is to 
>>>>>>>>>>>>>>>>>>> have multiple
>>>>>>>>>>>>>>>>>>> coordinators ( each responsible for coordinating a subset 
>>>>>>>>>>>>>>>>>>> of  queues in the
>>>>>>>>>>>>>>>>>>> cluster), that will be some what similar to kafka.
>>>>>>>>>>>>>>>>>>> Even if its preferable to have no
>>>>>>>>>>>>>>>>>>> coordinator at-all, (to decide how messages are 
>>>>>>>>>>>>>>>>>>> disseminated in the
>>>>>>>>>>>>>>>>>>> cluster)  that will make us give up desired behaviour such 
>>>>>>>>>>>>>>>>>>> as delivering
>>>>>>>>>>>>>>>>>>> a good mix of messages (from different publishers) to 
>>>>>>>>>>>>>>>>>>> consumers in a
>>>>>>>>>>>>>>>>>>> cluster. having said this, we have an ongoing research on 
>>>>>>>>>>>>>>>>>>> how to improve
>>>>>>>>>>>>>>>>>>> the algorithm and we like to try out both these approaches.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 1:31 PM, Malaka Silva <
>>>>>>>>>>>>>>>>>>> mal...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The same issue with Hazelcast can be experienced with
>>>>>>>>>>>>>>>>>>>> ESB inbounds (running on top of NTASK) and VFS 
>>>>>>>>>>>>>>>>>>>> distribution locks.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The idea of only single worker works at a given time
>>>>>>>>>>>>>>>>>>>> breaks if there is a Hazelcast heart beat fails. This will 
>>>>>>>>>>>>>>>>>>>> make two workers
>>>>>>>>>>>>>>>>>>>> to work in parallel.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Also with distributed locking there is no guarantee
>>>>>>>>>>>>>>>>>>>> that file is only process only by one worker.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> So in the case of network fail
>>>>>>>>>>>>>>>>>>>> ​ with DB​
>>>>>>>>>>>>>>>>>>>> make sense to stop processing until it's recovered.
>>>>>>>>>>>>>>>>>>>> ​ Also making this component generic ESB can reuse.​
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 9:21 AM, Asitha Nanayakkara <
>>>>>>>>>>>>>>>>>>>> asi...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Imesh,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:33 AM, Imesh Gunaratne <
>>>>>>>>>>>>>>>>>>>>> im...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:31 AM, Imesh Gunaratne <
>>>>>>>>>>>>>>>>>>>>>> im...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> You can see here [3] how K8S has implemented leader
>>>>>>>>>>>>>>>>>>>>>>> election feature for the products deployed on top of 
>>>>>>>>>>>>>>>>>>>>>>> that to utilize.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ​Correction: Please refer [4].​
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:27 PM, Asanka Abeyweera <
>>>>>>>>>>>>>>>>>>>>>>>> asank...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Imesh,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> We are not implementing this to overcome a
>>>>>>>>>>>>>>>>>>>>>>>>> limitation in the coordination algorithm available in 
>>>>>>>>>>>>>>>>>>>>>>>>> the Hazlecast. We are
>>>>>>>>>>>>>>>>>>>>>>>>> implementing this since we need an RDBMS based 
>>>>>>>>>>>>>>>>>>>>>>>>> coordination algorithm (not
>>>>>>>>>>>>>>>>>>>>>>>>> a network based algorithm).
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ​Are you saying that database connections do not use
>>>>>>>>>>>>>>>>>>>>>>> the same network used by Hazelcast?
>>>>>>>>>>>>>>>>>>>>>>> ​
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The reason is, a network based election algorithm
>>>>>>>>>>>>>>>>>>>>>>>>> will always elect multiple leaders when the network 
>>>>>>>>>>>>>>>>>>>>>>>>> is partitioned. But if
>>>>>>>>>>>>>>>>>>>>>>>>> we use a RDBMS based algorithm this will not happen.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ​I do not think your argument is correct. If there
>>>>>>>>>>>>>>>>>>>>>>> is a problem with the network, i​t may apply to both 
>>>>>>>>>>>>>>>>>>>>>>> Hazelcast based
>>>>>>>>>>>>>>>>>>>>>>> solution and database based solution.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yes, if the same network interface is used network
>>>>>>>>>>>>>>>>>>>>> partion will cause all types of connections to be 
>>>>>>>>>>>>>>>>>>>>> partitioned. But user can
>>>>>>>>>>>>>>>>>>>>> use multiple network interfaces for database, Hazelcast 
>>>>>>>>>>>>>>>>>>>>> and thrift.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Following is the scenario we are trying to solve in MB.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> In MB all the details related to messages,
>>>>>>>>>>>>>>>>>>>>> subscriptions, queues, topics etc are stored in database. 
>>>>>>>>>>>>>>>>>>>>> And we operate
>>>>>>>>>>>>>>>>>>>>> depending on that information. If the MB node can't 
>>>>>>>>>>>>>>>>>>>>> connect to the database
>>>>>>>>>>>>>>>>>>>>> that means the node is ineffective in the cluster until 
>>>>>>>>>>>>>>>>>>>>> it can make a
>>>>>>>>>>>>>>>>>>>>> database connection.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We have seen instances where Hazelcast cluster get
>>>>>>>>>>>>>>>>>>>>> partitioned for some time period in networks, Reasons 
>>>>>>>>>>>>>>>>>>>>> were,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    1. Due to heavy load Hazelcast couldn't process or
>>>>>>>>>>>>>>>>>>>>>    send (some times both) hearbeats, hence a network 
>>>>>>>>>>>>>>>>>>>>> partition for Hazelcast
>>>>>>>>>>>>>>>>>>>>>    cluster
>>>>>>>>>>>>>>>>>>>>>    2. An actual network partition of Hazelcast
>>>>>>>>>>>>>>>>>>>>>    cluster
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> In both scenarios the database connection was working.
>>>>>>>>>>>>>>>>>>>>> In that case we get two coordinators elected through 
>>>>>>>>>>>>>>>>>>>>> Hazelcast and working
>>>>>>>>>>>>>>>>>>>>> on the same database to deliver the messages. this leads 
>>>>>>>>>>>>>>>>>>>>> to inconsistencies
>>>>>>>>>>>>>>>>>>>>> in the cluster behavior (for instances duplicate message 
>>>>>>>>>>>>>>>>>>>>> delivery,
>>>>>>>>>>>>>>>>>>>>> corrupred subscription states etc) .
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Since the point of interest for MB is the database, we
>>>>>>>>>>>>>>>>>>>>> decided to do the coordinator election through database 
>>>>>>>>>>>>>>>>>>>>> as well. If the
>>>>>>>>>>>>>>>>>>>>> node can't connect to the database, then the MB won't 
>>>>>>>>>>>>>>>>>>>>> operate anyway.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>> Asitha
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [4] http://blog.kubernetes.io/
>>>>>>>>>>>>>>>>>>>>>>> 2016/01/simple-leader-election-with-Kubernetes.html
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ​Thanks​
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:16 PM, Imesh Gunaratne <
>>>>>>>>>>>>>>>>>>>>>>>>> im...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Asanka,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need to implement a leader election
>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm on our own? AFAIU this is a complex 
>>>>>>>>>>>>>>>>>>>>>>>>>> problem which has been
>>>>>>>>>>>>>>>>>>>>>>>>>> already solved by several algorithms [1]. IMO it 
>>>>>>>>>>>>>>>>>>>>>>>>>> would be better to go
>>>>>>>>>>>>>>>>>>>>>>>>>> ahead with an existing well established 
>>>>>>>>>>>>>>>>>>>>>>>>>> implementation on etcd [1] or
>>>>>>>>>>>>>>>>>>>>>>>>>> Consul [2].
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Those provide HTTP APIs for clients to make
>>>>>>>>>>>>>>>>>>>>>>>>>> leader election calls. [3] is a client library 
>>>>>>>>>>>>>>>>>>>>>>>>>> written in Node.js for etcd
>>>>>>>>>>>>>>>>>>>>>>>>>> based leader election.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://www.projectcalico.
>>>>>>>>>>>>>>>>>>>>>>>>>> org/using-etcd-for-elections
>>>>>>>>>>>>>>>>>>>>>>>>>> [2] https://www.consul.io/docs
>>>>>>>>>>>>>>>>>>>>>>>>>> /guides/leader-election.html
>>>>>>>>>>>>>>>>>>>>>>>>>> [3] https://www.npmjs.com/package/etcd-leader
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 3, 2016 at 5:12 PM, Asanka Abeyweera
>>>>>>>>>>>>>>>>>>>>>>>>>> <asank...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Maninda,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Since we are using RDBMS to poll the node
>>>>>>>>>>>>>>>>>>>>>>>>>>> status, the cluster will not end up in situation 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1,2 or 3. With this
>>>>>>>>>>>>>>>>>>>>>>>>>>> approach we consider a node unreachable when it 
>>>>>>>>>>>>>>>>>>>>>>>>>>> cannot access the database.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore an unreachable node can never be the 
>>>>>>>>>>>>>>>>>>>>>>>>>>> leader.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> As you have mentioned, we are currently using
>>>>>>>>>>>>>>>>>>>>>>>>>>> the RDBMS as an atomic global variable to create 
>>>>>>>>>>>>>>>>>>>>>>>>>>> the coordinator entry.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 2, 2016 at 5:22 PM, Maninda
>>>>>>>>>>>>>>>>>>>>>>>>>>> Edirisooriya <mani...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Asanka,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> As I understand the accuracy of electing the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> leader correctly is dependent on the election 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mechanism with RDBMS because
>>>>>>>>>>>>>>>>>>>>>>>>>>>> there can be edge cases like,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Unreachable leader activates during the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> election process: Then who becomes the leader?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. The elected leader becomes unreachable
>>>>>>>>>>>>>>>>>>>>>>>>>>>> before the election is completed: Then will there 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> be a situation where
>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is no leader?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. A leader and a set of nodes are disconnected
>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the other part of the cluster and while the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> leader is trying to remove
>>>>>>>>>>>>>>>>>>>>>>>>>>>> unreachable members other part is calling an 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> election to make a leader: Who
>>>>>>>>>>>>>>>>>>>>>>>>>>>> will win?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> RDBMS based election algorithm should handle
>>>>>>>>>>>>>>>>>>>>>>>>>>>> such cases without bringing the cluster to an 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent state or dead
>>>>>>>>>>>>>>>>>>>>>>>>>>>> lock in all concurrent cases. If all these kind of 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cases cannot be handled
>>>>>>>>>>>>>>>>>>>>>>>>>>>> isn't it better to keep the current hazelcast 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clustering and use the RDBMS
>>>>>>>>>>>>>>>>>>>>>>>>>>>> only to handle the split brain scenario? In other 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> words when a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> hazelcast leader is elected it should be updated 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the RDBMS. If another
>>>>>>>>>>>>>>>>>>>>>>>>>>>> split party has already elected a leader, the node 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> who is going to write it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to RDBMS should avoid updating it. Simply, the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> RDBMS can be used as an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> atomic global variable to keep the leader name by 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> modifying the hazelcast
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clustering. WDYT?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Maninda Edirisooriya*
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> *E-mail* : mani...@wso2.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Skype* : @manindae
>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Twitter* : @maninda
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 4:38 PM, Asanka
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Abeyweera <asank...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Akila,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let me explain the issue in a different way.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let's assume the MB nodes are using two different 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network interfaces for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication and database 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> communication. With such a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration, there can be failures only in the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network interface used for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication in some nodes. When this 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> happens, there will be two
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or more Hazelcast clusters due to the network 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation, and as a result
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there will be multiple coordinators. Since every 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> node still have access to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the database, multiple coordinators can affect 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the correctness of the data
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stored in the DB. But if we used a RDBMS based 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach we won't have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple coordinators due to a network partition 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Hazelcast. This is one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> advantage we get from this approach.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Even when we use Zookeeper or RAFT the same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue will be there since we are using different 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interfaces for Hazelcast
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> communication and DB communication.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 2:56 PM, Akila
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ravihansa Perera <raviha...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What's the advantage of using RDBMS (even as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an alternative) to implement a 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leader/coordinator election? If the network
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connection to DB fails then this will be a 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> single point of failure. I don't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think we can scale RDBMS instances and expect 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the election algorithm to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. That would be reducing this problem to 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> another problem (electing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator RDBMS instance).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IMHO it would be better to look at Zookeeper
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Atomic Broadcast (ZAB) [1] or RAFT leader 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> election [2] algorithms which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have already proven results.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://cwiki.apache.org/c
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> onfluence/display/ZOOKEEPER/Zab1.0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] http://libraft.io/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:42 PM, Nandika
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jayawardana <nand...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 to make it a common component . We have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the clustering implementation for BPEL 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> component based on hazelcast.  If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the coordination is available at RDBMS level, 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we can remove hazelcast
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dependancy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nandika
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:28 PM, Hasitha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Aravinda <hasi...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we make it a common component, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not hard coupled with MB. BPS has the same 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 9:47 AM, Asanka
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Abeyweera <asank...@wso2.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In MB, we have used a coordinator based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach to manage distributed messaging 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm in the cluster.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently Hazelcast is used to elect the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator. But one issue we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> faced with Hazelcast is, during a network 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation (split brain),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast can elect two or more coordinators 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cluster. This affects
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the correctness of the distributed messaging 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm since there are some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tables in the database that should only be 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> edited by a single node (i.e.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As a solution to this problem we have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented minimum node count based approach 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] to deactivate set of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitioned nodes to stop multiple nodes 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> becoming coordinators until the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network segmentation issue is fixed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As an alternative solution, we are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thinking of implementing an RDBMS based 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach to elect the coordinator
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> node in the cluster. By doing this we can 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make sure that even during a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network segmentation only one node will be 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> elected as the coordinator node
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> since the election is happening through the 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> database.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The algorithm will use a polling mechanism
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to check the validity of the nodes. To make 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the election algorithm
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalable, only the coordinator node will be 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> checking status of all the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nodes in the cluster and it will inform other 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nodes through database when a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> member is added/left. The nodes will be only 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> checking for the status of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator node. When a node detect that 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator is invalid it will go
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a election to elect a new coordinator.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We are currently working on a POC to test
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how this works with MB's slot based messaging 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://wso2.org/jira/browse/MB-1664
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha Aravinda,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Associate Technical Lead,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Email: hasi...@wso2.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mobile : +94 718 210 200
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nandika Jayawardana
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc ; http://wso2.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Akila Ravihansa Perera
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.;  http://wso2.com/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera
>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648
>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne*
>>>>>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com
>>>>>>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057
>>>>>>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh
>>>>>>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera
>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648
>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Ramith Jayasinghe
>>>>>>>>>>>>>>>>>>>>>>>> Technical Lead
>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> E: ram...@wso2.com
>>>>>>>>>>>>>>>>>>>>>>>> P: +94 772534930
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne*
>>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com
>>>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057
>>>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh
>>>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne*
>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com
>>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057
>>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh
>>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> *Asitha Nanayakkara* <http://asitha.github.io/>
>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>>>>>>>> WSO2, Inc. <http://wso2.com/>
>>>>>>>>>>>>>>>>>>>>> Mob: +94 77 853 0682
>>>>>>>>>>>>>>>>>>>>> [image: https://wso2.com/signature]
>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Malaka Silva
>>>>>>>>>>>>>>>>>>>> Senior Technical Lead
>>>>>>>>>>>>>>>>>>>> M: +94 777 219 791
>>>>>>>>>>>>>>>>>>>> Tel : 94 11 214 5345
>>>>>>>>>>>>>>>>>>>> Fax :94 11 2145300
>>>>>>>>>>>>>>>>>>>> Skype : malaka.sampath.silva
>>>>>>>>>>>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/
>>>>>>>>>>>>>>>>>>>> malaka-silva/6/33/77
>>>>>>>>>>>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> WSO2, Inc.
>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>>>>>>>> https://wso2.com/signature
>>>>>>>>>>>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/
>>>>>>>>>>>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/>
>>>>>>>>>>>>>>>>>>>> https://store.wso2.com/store/
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Don't make Trees rare, we should keep them with care
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Ramith Jayasinghe
>>>>>>>>>>>>>>>>>>> Technical Lead
>>>>>>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> E: ram...@wso2.com
>>>>>>>>>>>>>>>>>>> P: +94 772534930
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/
>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Sajini De SIlva
>>>>>>>>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com ,
>>>>>>>>>>>>>>>>> Email: saj...@wso2.com
>>>>>>>>>>>>>>>>> Blog: http://sajinid.blogspot.com/
>>>>>>>>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Phone: +94 712797729
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Sajini De SIlva
>>>>>>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com ,
>>>>>>>>>>>>>>> Email: saj...@wso2.com
>>>>>>>>>>>>>>> Blog: http://sajinid.blogspot.com/
>>>>>>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Phone: +94 712797729
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Kasun Indrasiri
>>>>>>>>>>>>>> Director, Integration Technologies
>>>>>>>>>>>>>> WSO2, Inc.; http://wso2.com
>>>>>>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> cell: +1 650 450 2293
>>>>>>>>>>>>>> Blog : http://kasunpanorama.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> *Anjana Fernando*
>>>>>>>>>>>>> Associate Director / Architect
>>>>>>>>>>>>> WSO2 Inc. | http://wso2.com
>>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Architecture mailing list
>>>>>>>>>>>>> Architecture@wso2.org
>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Anjana Fernando*
>>>>>>>>>>> Associate Director / Architect
>>>>>>>>>>> WSO2 Inc. | http://wso2.com
>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Asanka Abeyweera
>>>>>>>>>> Senior Software Engineer
>>>>>>>>>> WSO2 Inc.
>>>>>>>>>>
>>>>>>>>>> Phone: +94 712228648
>>>>>>>>>> Blog: a5anka.github.io
>>>>>>>>>>
>>>>>>>>>> <https://wso2.com/signature>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Anjana Fernando*
>>>>>>>>> Associate Director / Architect
>>>>>>>>> WSO2 Inc. | http://wso2.com
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Anjana Fernando*
>>>>>>>> Associate Director / Architect
>>>>>>>> WSO2 Inc. | http://wso2.com
>>>>>>>> lean . enterprise . middleware
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Asanka Abeyweera
>>>>>>> Senior Software Engineer
>>>>>>> WSO2 Inc.
>>>>>>>
>>>>>>> Phone: +94 712228648
>>>>>>> Blog: a5anka.github.io
>>>>>>>
>>>>>>> <https://wso2.com/signature>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Anjana Fernando*
>>>>>> Associate Director / Architect
>>>>>> WSO2 Inc. | http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Anjana Fernando*
>>>>> Associate Director / Architect
>>>>> WSO2 Inc. | http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ramith Jayasinghe
>>>> Technical Lead
>>>> WSO2 Inc., http://wso2.com
>>>> lean.enterprise.middleware
>>>>
>>>> E: ram...@wso2.com
>>>> P: +94 772534930
>>>>
>>>
>>>
>>>
>>> --
>>> *Anjana Fernando*
>>> Associate Director / Architect
>>> WSO2 Inc. | http://wso2.com
>>> lean . enterprise . middleware
>>>
>>
>>
>>
>> --
>> Ramith Jayasinghe
>> Technical Lead
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> E: ram...@wso2.com
>> P: +94 772534930 <+94%2077%20253%204930>
>>
>
>
>
> --
> Asanka Abeyweera
> Senior Software Engineer
> WSO2 Inc.
>
> Phone: +94 712228648 <+94%2071%20222%208648>
> Blog: a5anka.github.io
>
> <https://wso2.com/signature>
>



-- 
*Anjana Fernando*
Associate Director / Architect
WSO2 Inc. | http://wso2.com
lean . enterprise . middleware
_______________________________________________
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Reply via email to