Cool. When are you guys planning to release the generalized component? On Fri, Dec 2, 2016 at 10:57 AM, Anjana Fernando <anj...@wso2.com> wrote:
> Hi guys, > > So the generalized coordination component was done by SameeraR, and the > discussions for that can be seen at [1] and [2]. We've identified some > improvements that also can be done to it, to make it more stable. > > [1] "Updated Invitation: Code Review for the implementation of RDBMS based > coordin... @ Thu Dec 1, 2016 2pm - 3pm (IST) (WSO2 Engineering Group)" > [2] "Implementing a RDBMS based leader election mechanism" > > Cheers, > Anjana. > > On Wed, Nov 30, 2016 at 4:21 PM, Asanka Abeyweera <asank...@wso2.com> > wrote: > >> Hi all, >> >> Following is the PR[1] related the implementation. This is currently >> merged to the master branch and will be a feature of the next MB release >> (3.2.0). >> >> [1] https://github.com/wso2/andes/pull/668 >> >> On Mon, Nov 7, 2016 at 12:45 PM, Ramith Jayasinghe <ram...@wso2.com> >> wrote: >> >>> +1 >>> >>> On Mon, Nov 7, 2016 at 12:40 PM, Anjana Fernando <anj...@wso2.com> >>> wrote: >>> >>>> Hi Ramith, >>>> >>>> Sure. Actually, I was talking with SameeraR to take over this and >>>> create a common component which has the required coordination >>>> functionality. The idea is to create a component, where the providers can >>>> be plugged in, such as the RDBMS based one, ZK, or any other container >>>> specific provider that maybe out there. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> On Mon, Nov 7, 2016 at 12:38 PM, Ramith Jayasinghe <ram...@wso2.com> >>>> wrote: >>>> >>>>> this might require some work.. shall we have a chat? >>>>> >>>>> On Thu, Nov 3, 2016 at 3:52 PM, Anjana Fernando <anj...@wso2.com> >>>>> wrote: >>>>> >>>>>> Ping! .. >>>>>> >>>>>> On Wed, Nov 2, 2016 at 5:03 PM, Anjana Fernando <anj...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> On Wed, Nov 2, 2016 at 3:14 PM, Asanka Abeyweera <asank...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Anjana, >>>>>>>> >>>>>>>> Currently, the implementation is part of the MB code (not a common >>>>>>>> component). >>>>>>>> >>>>>>> >>>>>>> Okay, can we please get it as a common component. >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 2, 2016 at 3:00 PM, Anjana Fernando <anj...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Asanka/Ramith, >>>>>>>>> >>>>>>>>> So for C5 based Streaming Analytics solution, we need coordination >>>>>>>>> functionality there as well. Is the functionality mentioned here >>>>>>>>> created as >>>>>>>>> a common component or baked in to the MB code? .. if so, can we >>>>>>>>> please get >>>>>>>>> it implemented it as a generic component, so other products can also >>>>>>>>> use >>>>>>>>> it. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Anjana. >>>>>>>>> >>>>>>>>> On Tue, Aug 9, 2016 at 3:10 PM, Anjana Fernando <anj...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Great! .. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Anjana. >>>>>>>>>> >>>>>>>>>> On Tue, Aug 9, 2016 at 1:49 PM, Asanka Abeyweera < >>>>>>>>>> asank...@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Anjana, >>>>>>>>>>> >>>>>>>>>>> Thank you for the suggestion. We have already done a similar >>>>>>>>>>> thing. We have added a backoff time after creating the leader entry >>>>>>>>>>> and >>>>>>>>>>> check if the leader entry is the entry created by self before >>>>>>>>>>> informing the >>>>>>>>>>> leader change. >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 9, 2016 at 12:27 PM, Anjana Fernando < >>>>>>>>>>> anj...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I see, thanks for the clarification, looks good! .. I think >>>>>>>>>>>> small thing to consider is, to avoid the situation where, the >>>>>>>>>>>> current >>>>>>>>>>>> leader goes away, and two other competes to become the leader, and >>>>>>>>>>>> the >>>>>>>>>>>> first one and the second one checks (reads) the table to check the >>>>>>>>>>>> last >>>>>>>>>>>> heartbeat and figures out that the leader is outdated at the same >>>>>>>>>>>> time, and >>>>>>>>>>>> then first one delete the entry and puts his one, and after that, >>>>>>>>>>>> second >>>>>>>>>>>> one will also delete the existing one and put his one, so both >>>>>>>>>>>> will think >>>>>>>>>>>> they became the leader, due to the condition that both succeeded >>>>>>>>>>>> in adding >>>>>>>>>>>> the entry without an error. So this can probably be fixed by >>>>>>>>>>>> checking back >>>>>>>>>>>> after a bit of time if the current node is actually me, which >>>>>>>>>>>> probabilistically will work well, if that time period is >>>>>>>>>>>> sufficient big >>>>>>>>>>>> enough than a typical database transaction required by a node to >>>>>>>>>>>> do the >>>>>>>>>>>> earlier operations. Or else, we should make sure the database >>>>>>>>>>>> transaction >>>>>>>>>>>> level used in this scenario is at least REPEATABLE_READ, where >>>>>>>>>>>> when we read >>>>>>>>>>>> the record, it will lock it throughout the transaction. So some >>>>>>>>>>>> DBMSs does >>>>>>>>>>>> not support REPEATABLE_READ, where in that case, we should be able >>>>>>>>>>>> to use >>>>>>>>>>>> SERIALIZABLE, which most of them support. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Anjana. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 9, 2016 at 11:11 AM, Maninda Edirisooriya < >>>>>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Anjana, >>>>>>>>>>>>> >>>>>>>>>>>>> After having an offline chat with Asanka what I understood was >>>>>>>>>>>>> that the leader election was done completely via the database but >>>>>>>>>>>>> with no >>>>>>>>>>>>> network communication. The leader is mentioned in the database >>>>>>>>>>>>> first. Then >>>>>>>>>>>>> the leader updates the node data periodically in the database. If >>>>>>>>>>>>> some node >>>>>>>>>>>>> realizes the data in the DB are outdated that means the leader was >>>>>>>>>>>>> disconnected. Then that node will look at the created timestamp >>>>>>>>>>>>> of the >>>>>>>>>>>>> leader entry. If that is not very recent that means there was no >>>>>>>>>>>>> new leader >>>>>>>>>>>>> elected recently. So he will try to update the leader entry with >>>>>>>>>>>>> his ID. As >>>>>>>>>>>>> I understand there the leader entry is using the leader ID and the >>>>>>>>>>>>> timestamp as the primary key. Even several nodes try to do it >>>>>>>>>>>>> simultaneously only one node will successfully be able to update >>>>>>>>>>>>> the entry >>>>>>>>>>>>> with the help of atomicity provided by the DB. Others members >>>>>>>>>>>>> will note the >>>>>>>>>>>>> timestamp of the leader was updated so will accept the first one >>>>>>>>>>>>> who >>>>>>>>>>>>> updates as the leader. Even after the leader is elected, the >>>>>>>>>>>>> leader will >>>>>>>>>>>>> only notify node data via updating DB instead of network calls. >>>>>>>>>>>>> Other nodes >>>>>>>>>>>>> will just observe it and check the latest timestmps of the entry. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>> >>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>> >>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Aug 9, 2016 at 10:13 AM, Anjana Fernando < >>>>>>>>>>>>> anj...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I just noticed this thread. I've some concerns on this >>>>>>>>>>>>>> implementations. First of all, I don't think the statement >>>>>>>>>>>>>> mentioned here >>>>>>>>>>>>>> saying an external service such as ZooKeeper doesn't work, is >>>>>>>>>>>>>> correct. >>>>>>>>>>>>>> Because, if you have a ZK cluster (it is suppose to be used as a >>>>>>>>>>>>>> cluster), >>>>>>>>>>>>>> you will not have any issues. All the nodes have a list of >>>>>>>>>>>>>> endpoints to all >>>>>>>>>>>>>> the ZK nodes and they connect to those, and ZK has a quorum >>>>>>>>>>>>>> based mechanism >>>>>>>>>>>>>> in keeping its state. So this makes sure, all the users have a >>>>>>>>>>>>>> single >>>>>>>>>>>>>> version of the ZK data. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also, I guess the fundamental problem here in the split brain >>>>>>>>>>>>>> situation is, we need one external entity taking the decision >>>>>>>>>>>>>> (e.g. ZK >>>>>>>>>>>>>> cluster), because it should have oversight to the whole >>>>>>>>>>>>>> environment. I >>>>>>>>>>>>>> don't see how this RDBMS mechanism would solve that. Because, >>>>>>>>>>>>>> what it gives >>>>>>>>>>>>>> is a central location of state persistence. But the decisions of >>>>>>>>>>>>>> making who >>>>>>>>>>>>>> is the leader is taken by the users, which can be problematic. >>>>>>>>>>>>>> Where when >>>>>>>>>>>>>> we have a network partition scenario in that occasion, two >>>>>>>>>>>>>> groups of users >>>>>>>>>>>>>> will be overriding each other in the centralized RDBMS data >>>>>>>>>>>>>> repeatedly and >>>>>>>>>>>>>> it will go on forever, where in the ZK situation, there will be >>>>>>>>>>>>>> only one >>>>>>>>>>>>>> leader, and the guys in the other partition simply won't be able >>>>>>>>>>>>>> to reach >>>>>>>>>>>>>> the leader, until its network issues are sorted. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So I also think, as Imesh mentioned, creating a coordination >>>>>>>>>>>>>> algorithm from scratch may not be a wise decision, and we should >>>>>>>>>>>>>> use proven >>>>>>>>>>>>>> technology/libraries to do that. And on a side note, the main >>>>>>>>>>>>>> reason for >>>>>>>>>>>>>> not using ZK for this earlier was because of the hassle of >>>>>>>>>>>>>> bringing up >>>>>>>>>>>>>> another set of servers when our products are clustered, and we >>>>>>>>>>>>>> knew that >>>>>>>>>>>>>> the split brain scenario will occur in HZ, but maybe now we >>>>>>>>>>>>>> should give an >>>>>>>>>>>>>> extension point probably to plug into an external service if for >>>>>>>>>>>>>> some >>>>>>>>>>>>>> applications the split brain scenario is a show stopper. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Anjana. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Aug 9, 2016 at 4:45 AM, Kasun Indrasiri < >>>>>>>>>>>>>> ka...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Ramith/Asanka, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ESB/DSS natask impl is also based on HZ. I guess if this >>>>>>>>>>>>>>> model works for the MB, we should make it generic for all such >>>>>>>>>>>>>>> coordination >>>>>>>>>>>>>>> requirements. (Thinking about using this in ESB 5.1)? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 3:58 AM, Sajini De Silva < >>>>>>>>>>>>>>> saj...@wso2.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Locking the database will be supported by some databases >>>>>>>>>>>>>>>> but there will be huge performance impact. So we cannot use >>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>> approach. If this approach cannot be adapted the only thing >>>>>>>>>>>>>>>> we can do is >>>>>>>>>>>>>>>> queue wise load balancing through slot coordinator. But in >>>>>>>>>>>>>>>> this case we >>>>>>>>>>>>>>>> cannot guarantee that load balance will be equally distributed >>>>>>>>>>>>>>>> since some >>>>>>>>>>>>>>>> queues can be loaded while some will be idle. Also we cannot >>>>>>>>>>>>>>>> have multiple >>>>>>>>>>>>>>>> slot coordinators having same queue as it may cause several >>>>>>>>>>>>>>>> complications >>>>>>>>>>>>>>>> such as, same slot is assigned to two nodes by different >>>>>>>>>>>>>>>> subscribers, >>>>>>>>>>>>>>>> message duplication etc. Actually this slot architecture was >>>>>>>>>>>>>>>> discussed in a >>>>>>>>>>>>>>>> separate mail thread before it is implemented. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 3:12 PM, Maninda Edirisooriya < >>>>>>>>>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Sajini, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes that is what I meant. As the number of slots are >>>>>>>>>>>>>>>>> proportional to the number of messages passing through the >>>>>>>>>>>>>>>>> cluster, slot >>>>>>>>>>>>>>>>> delivery should not be handled by the coordinator when there >>>>>>>>>>>>>>>>> is only one >>>>>>>>>>>>>>>>> coordinator in the cluster which is a bottleneck for scaling >>>>>>>>>>>>>>>>> messages >>>>>>>>>>>>>>>>> passing through the cluster. If there is only a single >>>>>>>>>>>>>>>>> coordinator, it >>>>>>>>>>>>>>>>> should handle operations that are not proportional to >>>>>>>>>>>>>>>>> messages throughput >>>>>>>>>>>>>>>>> of the cluster. Then only the tasks like subscriber adding / >>>>>>>>>>>>>>>>> removing >>>>>>>>>>>>>>>>> should be handled by the coordinator. As this is not the >>>>>>>>>>>>>>>>> current >>>>>>>>>>>>>>>>> implementation, we can consider multiple coordinator >>>>>>>>>>>>>>>>> approach. Then the >>>>>>>>>>>>>>>>> number of coordinators should be scalable with the message >>>>>>>>>>>>>>>>> throughout. I am >>>>>>>>>>>>>>>>> not sure whether locking the database per transaction would >>>>>>>>>>>>>>>>> achieve this >>>>>>>>>>>>>>>>> coordinator scalability in the multiple coordinator >>>>>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:42 PM, Sajini De Silva < >>>>>>>>>>>>>>>>> saj...@wso2.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:28 PM, Maninda Edirisooriya < >>>>>>>>>>>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> @Sajini, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> But the number of slots are proportional to the number >>>>>>>>>>>>>>>>>>> of messages pass through the MB which needs to be handled >>>>>>>>>>>>>>>>>>> by the >>>>>>>>>>>>>>>>>>> coordinator. That is what I meant by "information related >>>>>>>>>>>>>>>>>>> to meta data of >>>>>>>>>>>>>>>>>>> messages pass through a single coordinator". Ideally after >>>>>>>>>>>>>>>>>>> the senders and >>>>>>>>>>>>>>>>>>> receivers are subscribed to the cluster, coordinator should >>>>>>>>>>>>>>>>>>> have nothing to >>>>>>>>>>>>>>>>>>> do until they are removed or changed. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Even though it is possible to have multiple coordinators >>>>>>>>>>>>>>>>>> after having en effort (Lock the database for a whole >>>>>>>>>>>>>>>>>> transaction or the >>>>>>>>>>>>>>>>>> work load distribution as described by Ramith) , coordinator >>>>>>>>>>>>>>>>>> may have >>>>>>>>>>>>>>>>>> different work to do other than subscriber adding and >>>>>>>>>>>>>>>>>> removing. As I said >>>>>>>>>>>>>>>>>> earlier our MB message distribution system is based on slot >>>>>>>>>>>>>>>>>> architecture >>>>>>>>>>>>>>>>>> and slots will managed by the coordinator. You can read [1] >>>>>>>>>>>>>>>>>> to understand >>>>>>>>>>>>>>>>>> more about slot architecture in MB. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [1] http://sajinid.blogspot.co >>>>>>>>>>>>>>>>>> m/2015/03/wso2-message-broker-300-slot-based.html >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> @Ramith, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> +1 for multiple coordinators by partitioning the cluster >>>>>>>>>>>>>>>>>>> which maintains the simplicity and correctness of the >>>>>>>>>>>>>>>>>>> algorithm than >>>>>>>>>>>>>>>>>>> compromising simplicity with a less important factor like >>>>>>>>>>>>>>>>>>> "delivering a >>>>>>>>>>>>>>>>>>> good mix of messages". >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:05 PM, Ramith Jayasinghe < >>>>>>>>>>>>>>>>>>> ram...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> @Imesh, >>>>>>>>>>>>>>>>>>>> We can prove that doing leader election using >>>>>>>>>>>>>>>>>>>> a lib (where we maintain cluster state in another place, >>>>>>>>>>>>>>>>>>>> a.k.a DB) will not >>>>>>>>>>>>>>>>>>>> solve our original problem (this also relates to our past >>>>>>>>>>>>>>>>>>>> experience with >>>>>>>>>>>>>>>>>>>> both the zookeeper and hazelcast). >>>>>>>>>>>>>>>>>>>> We can make this implementation a common component if >>>>>>>>>>>>>>>>>>>> other products have a use of it. BPS might be able to use >>>>>>>>>>>>>>>>>>>> it since their >>>>>>>>>>>>>>>>>>>> data is also in the database. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> @Malaka: >>>>>>>>>>>>>>>>>>>> VFS scenario can't be solved by relying on this >>>>>>>>>>>>>>>>>>>> implementation. why? you can have the access to DB but not >>>>>>>>>>>>>>>>>>>> VFS >>>>>>>>>>>>>>>>>>>> resources/file (and vice versa). this is the same point we >>>>>>>>>>>>>>>>>>>> explained before. >>>>>>>>>>>>>>>>>>>> in Ntask implementation, if tasks are stored in the >>>>>>>>>>>>>>>>>>>> database then using this implementation makes sense. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> @Akila, >>>>>>>>>>>>>>>>>>>> implementing (distributed) a queue algorithm is >>>>>>>>>>>>>>>>>>>> non-trivial. Having one coordinator (single source of >>>>>>>>>>>>>>>>>>>> truth) keeps things >>>>>>>>>>>>>>>>>>>> simple hence it's a conscious design decision we agreed >>>>>>>>>>>>>>>>>>>> during the initial >>>>>>>>>>>>>>>>>>>> stages. However, possible extension to this scheme is to >>>>>>>>>>>>>>>>>>>> have multiple >>>>>>>>>>>>>>>>>>>> coordinators ( each responsible for coordinating a subset >>>>>>>>>>>>>>>>>>>> of queues in the >>>>>>>>>>>>>>>>>>>> cluster), that will be some what similar to kafka. >>>>>>>>>>>>>>>>>>>> Even if its preferable to have no >>>>>>>>>>>>>>>>>>>> coordinator at-all, (to decide how messages are >>>>>>>>>>>>>>>>>>>> disseminated in the >>>>>>>>>>>>>>>>>>>> cluster) that will make us give up desired behaviour such >>>>>>>>>>>>>>>>>>>> as delivering >>>>>>>>>>>>>>>>>>>> a good mix of messages (from different publishers) to >>>>>>>>>>>>>>>>>>>> consumers in a >>>>>>>>>>>>>>>>>>>> cluster. having said this, we have an ongoing research on >>>>>>>>>>>>>>>>>>>> how to improve >>>>>>>>>>>>>>>>>>>> the algorithm and we like to try out both these approaches. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 1:31 PM, Malaka Silva < >>>>>>>>>>>>>>>>>>>> mal...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The same issue with Hazelcast can be experienced with >>>>>>>>>>>>>>>>>>>>> ESB inbounds (running on top of NTASK) and VFS >>>>>>>>>>>>>>>>>>>>> distribution locks. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The idea of only single worker works at a given time >>>>>>>>>>>>>>>>>>>>> breaks if there is a Hazelcast heart beat fails. This >>>>>>>>>>>>>>>>>>>>> will make two workers >>>>>>>>>>>>>>>>>>>>> to work in parallel. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Also with distributed locking there is no guarantee >>>>>>>>>>>>>>>>>>>>> that file is only process only by one worker. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> So in the case of network fail >>>>>>>>>>>>>>>>>>>>> with DB >>>>>>>>>>>>>>>>>>>>> make sense to stop processing until it's recovered. >>>>>>>>>>>>>>>>>>>>> Also making this component generic ESB can reuse. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 9:21 AM, Asitha Nanayakkara < >>>>>>>>>>>>>>>>>>>>> asi...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi Imesh, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:33 AM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>>>>> im...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:31 AM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>>>>>> im...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> You can see here [3] how K8S has implemented leader >>>>>>>>>>>>>>>>>>>>>>>> election feature for the products deployed on top of >>>>>>>>>>>>>>>>>>>>>>>> that to utilize. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Correction: Please refer [4]. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:27 PM, Asanka Abeyweera < >>>>>>>>>>>>>>>>>>>>>>>>> asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Imesh, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> We are not implementing this to overcome a >>>>>>>>>>>>>>>>>>>>>>>>>> limitation in the coordination algorithm available >>>>>>>>>>>>>>>>>>>>>>>>>> in the Hazlecast. We are >>>>>>>>>>>>>>>>>>>>>>>>>> implementing this since we need an RDBMS based >>>>>>>>>>>>>>>>>>>>>>>>>> coordination algorithm (not >>>>>>>>>>>>>>>>>>>>>>>>>> a network based algorithm). >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Are you saying that database connections do not >>>>>>>>>>>>>>>>>>>>>>>> use the same network used by Hazelcast? >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> The reason is, a network based election algorithm >>>>>>>>>>>>>>>>>>>>>>>>>> will always elect multiple leaders when the network >>>>>>>>>>>>>>>>>>>>>>>>>> is partitioned. But if >>>>>>>>>>>>>>>>>>>>>>>>>> we use a RDBMS based algorithm this will not happen. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I do not think your argument is correct. If there >>>>>>>>>>>>>>>>>>>>>>>> is a problem with the network, it may apply to both >>>>>>>>>>>>>>>>>>>>>>>> Hazelcast based >>>>>>>>>>>>>>>>>>>>>>>> solution and database based solution. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Yes, if the same network interface is used network >>>>>>>>>>>>>>>>>>>>>> partion will cause all types of connections to be >>>>>>>>>>>>>>>>>>>>>> partitioned. But user can >>>>>>>>>>>>>>>>>>>>>> use multiple network interfaces for database, Hazelcast >>>>>>>>>>>>>>>>>>>>>> and thrift. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Following is the scenario we are trying to solve in >>>>>>>>>>>>>>>>>>>>>> MB. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> In MB all the details related to messages, >>>>>>>>>>>>>>>>>>>>>> subscriptions, queues, topics etc are stored in >>>>>>>>>>>>>>>>>>>>>> database. And we operate >>>>>>>>>>>>>>>>>>>>>> depending on that information. If the MB node can't >>>>>>>>>>>>>>>>>>>>>> connect to the database >>>>>>>>>>>>>>>>>>>>>> that means the node is ineffective in the cluster until >>>>>>>>>>>>>>>>>>>>>> it can make a >>>>>>>>>>>>>>>>>>>>>> database connection. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> We have seen instances where Hazelcast cluster get >>>>>>>>>>>>>>>>>>>>>> partitioned for some time period in networks, Reasons >>>>>>>>>>>>>>>>>>>>>> were, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1. Due to heavy load Hazelcast couldn't process >>>>>>>>>>>>>>>>>>>>>> or send (some times both) hearbeats, hence a network >>>>>>>>>>>>>>>>>>>>>> partition for >>>>>>>>>>>>>>>>>>>>>> Hazelcast cluster >>>>>>>>>>>>>>>>>>>>>> 2. An actual network partition of Hazelcast >>>>>>>>>>>>>>>>>>>>>> cluster >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> In both scenarios the database connection was >>>>>>>>>>>>>>>>>>>>>> working. In that case we get two coordinators elected >>>>>>>>>>>>>>>>>>>>>> through Hazelcast and >>>>>>>>>>>>>>>>>>>>>> working on the same database to deliver the messages. >>>>>>>>>>>>>>>>>>>>>> this leads to >>>>>>>>>>>>>>>>>>>>>> inconsistencies in the cluster behavior (for instances >>>>>>>>>>>>>>>>>>>>>> duplicate message >>>>>>>>>>>>>>>>>>>>>> delivery, corrupred subscription states etc) . >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Since the point of interest for MB is the database, >>>>>>>>>>>>>>>>>>>>>> we decided to do the coordinator election through >>>>>>>>>>>>>>>>>>>>>> database as well. If the >>>>>>>>>>>>>>>>>>>>>> node can't connect to the database, then the MB won't >>>>>>>>>>>>>>>>>>>>>> operate anyway. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>>> Asitha >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [4] http://blog.kubernetes.io/ >>>>>>>>>>>>>>>>>>>>>>>> 2016/01/simple-leader-election-with-Kubernetes.html >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:16 PM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>>>>>>>>> im...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need to implement a leader election >>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm on our own? AFAIU this is a complex >>>>>>>>>>>>>>>>>>>>>>>>>>> problem which has been >>>>>>>>>>>>>>>>>>>>>>>>>>> already solved by several algorithms [1]. IMO it >>>>>>>>>>>>>>>>>>>>>>>>>>> would be better to go >>>>>>>>>>>>>>>>>>>>>>>>>>> ahead with an existing well established >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation on etcd [1] or >>>>>>>>>>>>>>>>>>>>>>>>>>> Consul [2]. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Those provide HTTP APIs for clients to make >>>>>>>>>>>>>>>>>>>>>>>>>>> leader election calls. [3] is a client library >>>>>>>>>>>>>>>>>>>>>>>>>>> written in Node.js for etcd >>>>>>>>>>>>>>>>>>>>>>>>>>> based leader election. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://www.projectcalico. >>>>>>>>>>>>>>>>>>>>>>>>>>> org/using-etcd-for-elections >>>>>>>>>>>>>>>>>>>>>>>>>>> [2] https://www.consul.io/docs >>>>>>>>>>>>>>>>>>>>>>>>>>> /guides/leader-election.html >>>>>>>>>>>>>>>>>>>>>>>>>>> [3] https://www.npmjs.com/package/etcd-leader >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 3, 2016 at 5:12 PM, Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>> <asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Since we are using RDBMS to poll the node >>>>>>>>>>>>>>>>>>>>>>>>>>>> status, the cluster will not end up in situation >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1,2 or 3. With this >>>>>>>>>>>>>>>>>>>>>>>>>>>> approach we consider a node unreachable when it >>>>>>>>>>>>>>>>>>>>>>>>>>>> cannot access the database. >>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore an unreachable node can never be the >>>>>>>>>>>>>>>>>>>>>>>>>>>> leader. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> As you have mentioned, we are currently using >>>>>>>>>>>>>>>>>>>>>>>>>>>> the RDBMS as an atomic global variable to create >>>>>>>>>>>>>>>>>>>>>>>>>>>> the coordinator entry. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 2, 2016 at 5:22 PM, Maninda >>>>>>>>>>>>>>>>>>>>>>>>>>>> Edirisooriya <mani...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> As I understand the accuracy of electing the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> leader correctly is dependent on the election >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mechanism with RDBMS because >>>>>>>>>>>>>>>>>>>>>>>>>>>>> there can be edge cases like, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Unreachable leader activates during the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> election process: Then who becomes the leader? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. The elected leader becomes unreachable >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before the election is completed: Then will there >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be a situation where >>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is no leader? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. A leader and a set of nodes are >>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnected from the other part of the cluster >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and while the leader is >>>>>>>>>>>>>>>>>>>>>>>>>>>>> trying to remove unreachable members other part >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is calling an election to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> make a leader: Who will win? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> RDBMS based election algorithm should handle >>>>>>>>>>>>>>>>>>>>>>>>>>>>> such cases without bringing the cluster to an >>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent state or dead >>>>>>>>>>>>>>>>>>>>>>>>>>>>> lock in all concurrent cases. If all these kind >>>>>>>>>>>>>>>>>>>>>>>>>>>>> of cases cannot be handled >>>>>>>>>>>>>>>>>>>>>>>>>>>>> isn't it better to keep the current hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>>>> clustering and use the RDBMS >>>>>>>>>>>>>>>>>>>>>>>>>>>>> only to handle the split brain scenario? In other >>>>>>>>>>>>>>>>>>>>>>>>>>>>> words when a new >>>>>>>>>>>>>>>>>>>>>>>>>>>>> hazelcast leader is elected it should be updated >>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the RDBMS. If another >>>>>>>>>>>>>>>>>>>>>>>>>>>>> split party has already elected a leader, the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> node who is going to write it >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to RDBMS should avoid updating it. Simply, the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> RDBMS can be used as an >>>>>>>>>>>>>>>>>>>>>>>>>>>>> atomic global variable to keep the leader name by >>>>>>>>>>>>>>>>>>>>>>>>>>>>> modifying the hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>>>> clustering. WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 4:38 PM, Asanka >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Abeyweera <asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Akila, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let me explain the issue in a different way. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let's assume the MB nodes are using two >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different network interfaces for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication and database >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> communication. With such a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration, there can be failures only in the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network interface used for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication in some nodes. When this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> happens, there will be two >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or more Hazelcast clusters due to the network >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation, and as a result >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there will be multiple coordinators. Since every >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> node still have access to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the database, multiple coordinators can affect >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the correctness of the data >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stored in the DB. But if we used a RDBMS based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach we won't have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple coordinators due to a network partition >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Hazelcast. This is one >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> advantage we get from this approach. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Even when we use Zookeeper or RAFT the same >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue will be there since we are using different >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interfaces for Hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> communication and DB communication. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 2:56 PM, Akila >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ravihansa Perera <raviha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What's the advantage of using RDBMS (even as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an alternative) to implement a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leader/coordinator election? If the network >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connection to DB fails then this will be a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> single point of failure. I don't >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think we can scale RDBMS instances and expect >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the election algorithm to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. That would be reducing this problem to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> another problem (electing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator RDBMS instance). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IMHO it would be better to look at Zookeeper >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Atomic Broadcast (ZAB) [1] or RAFT leader >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> election [2] algorithms which >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have already proven results. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://cwiki.apache.org/c >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> onfluence/display/ZOOKEEPER/Zab1.0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] http://libraft.io/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:42 PM, Nandika >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jayawardana <nand...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 to make it a common component . We have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the clustering implementation for BPEL >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> component based on hazelcast. If >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the coordination is available at RDBMS level, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we can remove hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dependancy. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nandika >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:28 PM, Hasitha >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Aravinda <hasi...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we make it a common component, which >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is not hard coupled with MB. BPS has the same >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 9:47 AM, Asanka >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Abeyweera <asank...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In MB, we have used a coordinator based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach to manage distributed messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm in the cluster. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently Hazelcast is used to elect the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator. But one issue we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> faced with Hazelcast is, during a network >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation (split brain), >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast can elect two or more coordinators >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cluster. This affects >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the correctness of the distributed messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm since there are some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tables in the database that should only be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> edited by a single node (i.e. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As a solution to this problem we have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented minimum node count based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach [1] to deactivate set of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitioned nodes to stop multiple nodes >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> becoming coordinators until the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network segmentation issue is fixed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As an alternative solution, we are >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thinking of implementing an RDBMS based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach to elect the coordinator >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> node in the cluster. By doing this we can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make sure that even during a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network segmentation only one node will be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> elected as the coordinator node >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> since the election is happening through the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> database. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The algorithm will use a polling >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mechanism to check the validity of the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nodes. To make the election >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm scalable, only the coordinator >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> node will be checking status of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all the nodes in the cluster and it will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inform other nodes through >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> database when a member is added/left. The >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nodes will be only checking for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the status of the coordinator node. When a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> node detect that coordinator is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid it will go for a election to elect a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new coordinator. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We are currently working on a POC to test >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how this works with MB's slot based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> messaging algorithm. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://wso2.org/jira/browse/MB-1664 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha Aravinda, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Associate Technical Lead, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Email: hasi...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mobile : +94 718 210 200 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nandika Jayawardana >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc ; http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>> Ramith Jayasinghe >>>>>>>>>>>>>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> E: ram...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>> P: +94 772534930 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>> *Asitha Nanayakkara* <http://asitha.github.io/> >>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>> WSO2, Inc. <http://wso2.com/> >>>>>>>>>>>>>>>>>>>>>> Mob: +94 77 853 0682 >>>>>>>>>>>>>>>>>>>>>> [image: https://wso2.com/signature] >>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Malaka Silva >>>>>>>>>>>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>>>>>>>>>>> M: +94 777 219 791 >>>>>>>>>>>>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>>>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>>>>>>>>>>>> Skype : malaka.sampath.silva >>>>>>>>>>>>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/ >>>>>>>>>>>>>>>>>>>>> malaka-silva/6/33/77 >>>>>>>>>>>>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>>>>>>>>> https://wso2.com/signature >>>>>>>>>>>>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/ >>>>>>>>>>>>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/> >>>>>>>>>>>>>>>>>>>>> https://store.wso2.com/store/ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Don't make Trees rare, we should keep them with care >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> Ramith Jayasinghe >>>>>>>>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> E: ram...@wso2.com >>>>>>>>>>>>>>>>>>>> P: +94 772534930 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Sajini De SIlva >>>>>>>>>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>>>>>>>>>>>>>> Email: saj...@wso2.com >>>>>>>>>>>>>>>>>> Blog: http://sajinid.blogspot.com/ >>>>>>>>>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Phone: +94 712797729 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Sajini De SIlva >>>>>>>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>>>>>>>>>>>> Email: saj...@wso2.com >>>>>>>>>>>>>>>> Blog: http://sajinid.blogspot.com/ >>>>>>>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Phone: +94 712797729 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Kasun Indrasiri >>>>>>>>>>>>>>> Director, Integration Technologies >>>>>>>>>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cell: +1 650 450 2293 >>>>>>>>>>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> *Anjana Fernando* >>>>>>>>>>>>>> Associate Director / Architect >>>>>>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Anjana Fernando* >>>>>>>>>>>> Associate Director / Architect >>>>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>> Senior Software Engineer >>>>>>>>>>> WSO2 Inc. >>>>>>>>>>> >>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>> >>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Anjana Fernando* >>>>>>>>>> Associate Director / Architect >>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>> lean . enterprise . middleware >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Anjana Fernando* >>>>>>>>> Associate Director / Architect >>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>> lean . enterprise . middleware >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Asanka Abeyweera >>>>>>>> Senior Software Engineer >>>>>>>> WSO2 Inc. >>>>>>>> >>>>>>>> Phone: +94 712228648 >>>>>>>> Blog: a5anka.github.io >>>>>>>> >>>>>>>> <https://wso2.com/signature> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Associate Director / Architect >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Associate Director / Architect >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Ramith Jayasinghe >>>>> Technical Lead >>>>> WSO2 Inc., http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> E: ram...@wso2.com >>>>> P: +94 772534930 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Associate Director / Architect >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> Ramith Jayasinghe >>> Technical Lead >>> WSO2 Inc., http://wso2.com >>> lean.enterprise.middleware >>> >>> E: ram...@wso2.com >>> P: +94 772534930 <+94%2077%20253%204930> >>> >> >> >> >> -- >> Asanka Abeyweera >> Senior Software Engineer >> WSO2 Inc. >> >> Phone: +94 712228648 <+94%2071%20222%208648> >> Blog: a5anka.github.io >> >> <https://wso2.com/signature> >> > > > > -- > *Anjana Fernando* > Associate Director / Architect > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- Asanka Abeyweera Senior Software Engineer WSO2 Inc. Phone: +94 712228648 Blog: a5anka.github.io <https://wso2.com/signature>
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
