Hi guys, So the generalized coordination component was done by SameeraR, and the discussions for that can be seen at [1] and [2]. We've identified some improvements that also can be done to it, to make it more stable.
[1] "Updated Invitation: Code Review for the implementation of RDBMS based coordin... @ Thu Dec 1, 2016 2pm - 3pm (IST) (WSO2 Engineering Group)" [2] "Implementing a RDBMS based leader election mechanism" Cheers, Anjana. On Wed, Nov 30, 2016 at 4:21 PM, Asanka Abeyweera <[email protected]> wrote: > Hi all, > > Following is the PR[1] related the implementation. This is currently > merged to the master branch and will be a feature of the next MB release > (3.2.0). > > [1] https://github.com/wso2/andes/pull/668 > > On Mon, Nov 7, 2016 at 12:45 PM, Ramith Jayasinghe <[email protected]> > wrote: > >> +1 >> >> On Mon, Nov 7, 2016 at 12:40 PM, Anjana Fernando <[email protected]> wrote: >> >>> Hi Ramith, >>> >>> Sure. Actually, I was talking with SameeraR to take over this and create >>> a common component which has the required coordination functionality. The >>> idea is to create a component, where the providers can be plugged in, such >>> as the RDBMS based one, ZK, or any other container specific provider that >>> maybe out there. >>> >>> Cheers, >>> Anjana. >>> >>> On Mon, Nov 7, 2016 at 12:38 PM, Ramith Jayasinghe <[email protected]> >>> wrote: >>> >>>> this might require some work.. shall we have a chat? >>>> >>>> On Thu, Nov 3, 2016 at 3:52 PM, Anjana Fernando <[email protected]> >>>> wrote: >>>> >>>>> Ping! .. >>>>> >>>>> On Wed, Nov 2, 2016 at 5:03 PM, Anjana Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> On Wed, Nov 2, 2016 at 3:14 PM, Asanka Abeyweera <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Anjana, >>>>>>> >>>>>>> Currently, the implementation is part of the MB code (not a common >>>>>>> component). >>>>>>> >>>>>> >>>>>> Okay, can we please get it as a common component. >>>>>> >>>>>> Cheers, >>>>>> Anjana. >>>>>> >>>>>> >>>>>>> >>>>>>> On Wed, Nov 2, 2016 at 3:00 PM, Anjana Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Asanka/Ramith, >>>>>>>> >>>>>>>> So for C5 based Streaming Analytics solution, we need coordination >>>>>>>> functionality there as well. Is the functionality mentioned here >>>>>>>> created as >>>>>>>> a common component or baked in to the MB code? .. if so, can we please >>>>>>>> get >>>>>>>> it implemented it as a generic component, so other products can also >>>>>>>> use >>>>>>>> it. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Anjana. >>>>>>>> >>>>>>>> On Tue, Aug 9, 2016 at 3:10 PM, Anjana Fernando <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Great! .. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Anjana. >>>>>>>>> >>>>>>>>> On Tue, Aug 9, 2016 at 1:49 PM, Asanka Abeyweera < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Anjana, >>>>>>>>>> >>>>>>>>>> Thank you for the suggestion. We have already done a similar >>>>>>>>>> thing. We have added a backoff time after creating the leader entry >>>>>>>>>> and >>>>>>>>>> check if the leader entry is the entry created by self before >>>>>>>>>> informing the >>>>>>>>>> leader change. >>>>>>>>>> >>>>>>>>>> On Tue, Aug 9, 2016 at 12:27 PM, Anjana Fernando <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> I see, thanks for the clarification, looks good! .. I think >>>>>>>>>>> small thing to consider is, to avoid the situation where, the >>>>>>>>>>> current >>>>>>>>>>> leader goes away, and two other competes to become the leader, and >>>>>>>>>>> the >>>>>>>>>>> first one and the second one checks (reads) the table to check the >>>>>>>>>>> last >>>>>>>>>>> heartbeat and figures out that the leader is outdated at the same >>>>>>>>>>> time, and >>>>>>>>>>> then first one delete the entry and puts his one, and after that, >>>>>>>>>>> second >>>>>>>>>>> one will also delete the existing one and put his one, so both will >>>>>>>>>>> think >>>>>>>>>>> they became the leader, due to the condition that both succeeded in >>>>>>>>>>> adding >>>>>>>>>>> the entry without an error. So this can probably be fixed by >>>>>>>>>>> checking back >>>>>>>>>>> after a bit of time if the current node is actually me, which >>>>>>>>>>> probabilistically will work well, if that time period is sufficient >>>>>>>>>>> big >>>>>>>>>>> enough than a typical database transaction required by a node to do >>>>>>>>>>> the >>>>>>>>>>> earlier operations. Or else, we should make sure the database >>>>>>>>>>> transaction >>>>>>>>>>> level used in this scenario is at least REPEATABLE_READ, where when >>>>>>>>>>> we read >>>>>>>>>>> the record, it will lock it throughout the transaction. So some >>>>>>>>>>> DBMSs does >>>>>>>>>>> not support REPEATABLE_READ, where in that case, we should be able >>>>>>>>>>> to use >>>>>>>>>>> SERIALIZABLE, which most of them support. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Anjana. >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 9, 2016 at 11:11 AM, Maninda Edirisooriya < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Anjana, >>>>>>>>>>>> >>>>>>>>>>>> After having an offline chat with Asanka what I understood was >>>>>>>>>>>> that the leader election was done completely via the database but >>>>>>>>>>>> with no >>>>>>>>>>>> network communication. The leader is mentioned in the database >>>>>>>>>>>> first. Then >>>>>>>>>>>> the leader updates the node data periodically in the database. If >>>>>>>>>>>> some node >>>>>>>>>>>> realizes the data in the DB are outdated that means the leader was >>>>>>>>>>>> disconnected. Then that node will look at the created timestamp of >>>>>>>>>>>> the >>>>>>>>>>>> leader entry. If that is not very recent that means there was no >>>>>>>>>>>> new leader >>>>>>>>>>>> elected recently. So he will try to update the leader entry with >>>>>>>>>>>> his ID. As >>>>>>>>>>>> I understand there the leader entry is using the leader ID and the >>>>>>>>>>>> timestamp as the primary key. Even several nodes try to do it >>>>>>>>>>>> simultaneously only one node will successfully be able to update >>>>>>>>>>>> the entry >>>>>>>>>>>> with the help of atomicity provided by the DB. Others members will >>>>>>>>>>>> note the >>>>>>>>>>>> timestamp of the leader was updated so will accept the first one >>>>>>>>>>>> who >>>>>>>>>>>> updates as the leader. Even after the leader is elected, the >>>>>>>>>>>> leader will >>>>>>>>>>>> only notify node data via updating DB instead of network calls. >>>>>>>>>>>> Other nodes >>>>>>>>>>>> will just observe it and check the latest timestmps of the entry. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>> >>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>> >>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 9, 2016 at 10:13 AM, Anjana Fernando < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> I just noticed this thread. I've some concerns on this >>>>>>>>>>>>> implementations. First of all, I don't think the statement >>>>>>>>>>>>> mentioned here >>>>>>>>>>>>> saying an external service such as ZooKeeper doesn't work, is >>>>>>>>>>>>> correct. >>>>>>>>>>>>> Because, if you have a ZK cluster (it is suppose to be used as a >>>>>>>>>>>>> cluster), >>>>>>>>>>>>> you will not have any issues. All the nodes have a list of >>>>>>>>>>>>> endpoints to all >>>>>>>>>>>>> the ZK nodes and they connect to those, and ZK has a quorum based >>>>>>>>>>>>> mechanism >>>>>>>>>>>>> in keeping its state. So this makes sure, all the users have a >>>>>>>>>>>>> single >>>>>>>>>>>>> version of the ZK data. >>>>>>>>>>>>> >>>>>>>>>>>>> Also, I guess the fundamental problem here in the split brain >>>>>>>>>>>>> situation is, we need one external entity taking the decision >>>>>>>>>>>>> (e.g. ZK >>>>>>>>>>>>> cluster), because it should have oversight to the whole >>>>>>>>>>>>> environment. I >>>>>>>>>>>>> don't see how this RDBMS mechanism would solve that. Because, >>>>>>>>>>>>> what it gives >>>>>>>>>>>>> is a central location of state persistence. But the decisions of >>>>>>>>>>>>> making who >>>>>>>>>>>>> is the leader is taken by the users, which can be problematic. >>>>>>>>>>>>> Where when >>>>>>>>>>>>> we have a network partition scenario in that occasion, two groups >>>>>>>>>>>>> of users >>>>>>>>>>>>> will be overriding each other in the centralized RDBMS data >>>>>>>>>>>>> repeatedly and >>>>>>>>>>>>> it will go on forever, where in the ZK situation, there will be >>>>>>>>>>>>> only one >>>>>>>>>>>>> leader, and the guys in the other partition simply won't be able >>>>>>>>>>>>> to reach >>>>>>>>>>>>> the leader, until its network issues are sorted. >>>>>>>>>>>>> >>>>>>>>>>>>> So I also think, as Imesh mentioned, creating a coordination >>>>>>>>>>>>> algorithm from scratch may not be a wise decision, and we should >>>>>>>>>>>>> use proven >>>>>>>>>>>>> technology/libraries to do that. And on a side note, the main >>>>>>>>>>>>> reason for >>>>>>>>>>>>> not using ZK for this earlier was because of the hassle of >>>>>>>>>>>>> bringing up >>>>>>>>>>>>> another set of servers when our products are clustered, and we >>>>>>>>>>>>> knew that >>>>>>>>>>>>> the split brain scenario will occur in HZ, but maybe now we >>>>>>>>>>>>> should give an >>>>>>>>>>>>> extension point probably to plug into an external service if for >>>>>>>>>>>>> some >>>>>>>>>>>>> applications the split brain scenario is a show stopper. >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> Anjana. >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Aug 9, 2016 at 4:45 AM, Kasun Indrasiri < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Ramith/Asanka, >>>>>>>>>>>>>> >>>>>>>>>>>>>> ESB/DSS natask impl is also based on HZ. I guess if this >>>>>>>>>>>>>> model works for the MB, we should make it generic for all such >>>>>>>>>>>>>> coordination >>>>>>>>>>>>>> requirements. (Thinking about using this in ESB 5.1)? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 3:58 AM, Sajini De Silva < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Locking the database will be supported by some databases >>>>>>>>>>>>>>> but there will be huge performance impact. So we cannot use >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>> approach. If this approach cannot be adapted the only thing we >>>>>>>>>>>>>>> can do is >>>>>>>>>>>>>>> queue wise load balancing through slot coordinator. But in this >>>>>>>>>>>>>>> case we >>>>>>>>>>>>>>> cannot guarantee that load balance will be equally distributed >>>>>>>>>>>>>>> since some >>>>>>>>>>>>>>> queues can be loaded while some will be idle. Also we cannot >>>>>>>>>>>>>>> have multiple >>>>>>>>>>>>>>> slot coordinators having same queue as it may cause several >>>>>>>>>>>>>>> complications >>>>>>>>>>>>>>> such as, same slot is assigned to two nodes by different >>>>>>>>>>>>>>> subscribers, >>>>>>>>>>>>>>> message duplication etc. Actually this slot architecture was >>>>>>>>>>>>>>> discussed in a >>>>>>>>>>>>>>> separate mail thread before it is implemented. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 3:12 PM, Maninda Edirisooriya < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Sajini, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes that is what I meant. As the number of slots are >>>>>>>>>>>>>>>> proportional to the number of messages passing through the >>>>>>>>>>>>>>>> cluster, slot >>>>>>>>>>>>>>>> delivery should not be handled by the coordinator when there >>>>>>>>>>>>>>>> is only one >>>>>>>>>>>>>>>> coordinator in the cluster which is a bottleneck for scaling >>>>>>>>>>>>>>>> messages >>>>>>>>>>>>>>>> passing through the cluster. If there is only a single >>>>>>>>>>>>>>>> coordinator, it >>>>>>>>>>>>>>>> should handle operations that are not proportional to messages >>>>>>>>>>>>>>>> throughput >>>>>>>>>>>>>>>> of the cluster. Then only the tasks like subscriber adding / >>>>>>>>>>>>>>>> removing >>>>>>>>>>>>>>>> should be handled by the coordinator. As this is not the >>>>>>>>>>>>>>>> current >>>>>>>>>>>>>>>> implementation, we can consider multiple coordinator approach. >>>>>>>>>>>>>>>> Then the >>>>>>>>>>>>>>>> number of coordinators should be scalable with the message >>>>>>>>>>>>>>>> throughout. I am >>>>>>>>>>>>>>>> not sure whether locking the database per transaction would >>>>>>>>>>>>>>>> achieve this >>>>>>>>>>>>>>>> coordinator scalability in the multiple coordinator >>>>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:42 PM, Sajini De Silva < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:28 PM, Maninda Edirisooriya < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> @Sajini, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> But the number of slots are proportional to the number of >>>>>>>>>>>>>>>>>> messages pass through the MB which needs to be handled by >>>>>>>>>>>>>>>>>> the coordinator. >>>>>>>>>>>>>>>>>> That is what I meant by "information related to meta data of >>>>>>>>>>>>>>>>>> messages pass >>>>>>>>>>>>>>>>>> through a single coordinator". Ideally after the senders and >>>>>>>>>>>>>>>>>> receivers are >>>>>>>>>>>>>>>>>> subscribed to the cluster, coordinator should have nothing >>>>>>>>>>>>>>>>>> to do until they >>>>>>>>>>>>>>>>>> are removed or changed. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Even though it is possible to have multiple coordinators >>>>>>>>>>>>>>>>> after having en effort (Lock the database for a whole >>>>>>>>>>>>>>>>> transaction or the >>>>>>>>>>>>>>>>> work load distribution as described by Ramith) , coordinator >>>>>>>>>>>>>>>>> may have >>>>>>>>>>>>>>>>> different work to do other than subscriber adding and >>>>>>>>>>>>>>>>> removing. As I said >>>>>>>>>>>>>>>>> earlier our MB message distribution system is based on slot >>>>>>>>>>>>>>>>> architecture >>>>>>>>>>>>>>>>> and slots will managed by the coordinator. You can read [1] >>>>>>>>>>>>>>>>> to understand >>>>>>>>>>>>>>>>> more about slot architecture in MB. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1] http://sajinid.blogspot.co >>>>>>>>>>>>>>>>> m/2015/03/wso2-message-broker-300-slot-based.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> @Ramith, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> +1 for multiple coordinators by partitioning the cluster >>>>>>>>>>>>>>>>>> which maintains the simplicity and correctness of the >>>>>>>>>>>>>>>>>> algorithm than >>>>>>>>>>>>>>>>>> compromising simplicity with a less important factor like >>>>>>>>>>>>>>>>>> "delivering a >>>>>>>>>>>>>>>>>> good mix of messages". >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:05 PM, Ramith Jayasinghe < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> @Imesh, >>>>>>>>>>>>>>>>>>> We can prove that doing leader election using >>>>>>>>>>>>>>>>>>> a lib (where we maintain cluster state in another place, >>>>>>>>>>>>>>>>>>> a.k.a DB) will not >>>>>>>>>>>>>>>>>>> solve our original problem (this also relates to our past >>>>>>>>>>>>>>>>>>> experience with >>>>>>>>>>>>>>>>>>> both the zookeeper and hazelcast). >>>>>>>>>>>>>>>>>>> We can make this implementation a common component if >>>>>>>>>>>>>>>>>>> other products have a use of it. BPS might be able to use >>>>>>>>>>>>>>>>>>> it since their >>>>>>>>>>>>>>>>>>> data is also in the database. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> @Malaka: >>>>>>>>>>>>>>>>>>> VFS scenario can't be solved by relying on this >>>>>>>>>>>>>>>>>>> implementation. why? you can have the access to DB but not >>>>>>>>>>>>>>>>>>> VFS >>>>>>>>>>>>>>>>>>> resources/file (and vice versa). this is the same point we >>>>>>>>>>>>>>>>>>> explained before. >>>>>>>>>>>>>>>>>>> in Ntask implementation, if tasks are stored in the >>>>>>>>>>>>>>>>>>> database then using this implementation makes sense. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> @Akila, >>>>>>>>>>>>>>>>>>> implementing (distributed) a queue algorithm is >>>>>>>>>>>>>>>>>>> non-trivial. Having one coordinator (single source of >>>>>>>>>>>>>>>>>>> truth) keeps things >>>>>>>>>>>>>>>>>>> simple hence it's a conscious design decision we agreed >>>>>>>>>>>>>>>>>>> during the initial >>>>>>>>>>>>>>>>>>> stages. However, possible extension to this scheme is to >>>>>>>>>>>>>>>>>>> have multiple >>>>>>>>>>>>>>>>>>> coordinators ( each responsible for coordinating a subset >>>>>>>>>>>>>>>>>>> of queues in the >>>>>>>>>>>>>>>>>>> cluster), that will be some what similar to kafka. >>>>>>>>>>>>>>>>>>> Even if its preferable to have no >>>>>>>>>>>>>>>>>>> coordinator at-all, (to decide how messages are >>>>>>>>>>>>>>>>>>> disseminated in the >>>>>>>>>>>>>>>>>>> cluster) that will make us give up desired behaviour such >>>>>>>>>>>>>>>>>>> as delivering >>>>>>>>>>>>>>>>>>> a good mix of messages (from different publishers) to >>>>>>>>>>>>>>>>>>> consumers in a >>>>>>>>>>>>>>>>>>> cluster. having said this, we have an ongoing research on >>>>>>>>>>>>>>>>>>> how to improve >>>>>>>>>>>>>>>>>>> the algorithm and we like to try out both these approaches. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 1:31 PM, Malaka Silva < >>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The same issue with Hazelcast can be experienced with >>>>>>>>>>>>>>>>>>>> ESB inbounds (running on top of NTASK) and VFS >>>>>>>>>>>>>>>>>>>> distribution locks. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The idea of only single worker works at a given time >>>>>>>>>>>>>>>>>>>> breaks if there is a Hazelcast heart beat fails. This will >>>>>>>>>>>>>>>>>>>> make two workers >>>>>>>>>>>>>>>>>>>> to work in parallel. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Also with distributed locking there is no guarantee >>>>>>>>>>>>>>>>>>>> that file is only process only by one worker. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> So in the case of network fail >>>>>>>>>>>>>>>>>>>> with DB >>>>>>>>>>>>>>>>>>>> make sense to stop processing until it's recovered. >>>>>>>>>>>>>>>>>>>> Also making this component generic ESB can reuse. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 9:21 AM, Asitha Nanayakkara < >>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Imesh, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:33 AM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:31 AM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> You can see here [3] how K8S has implemented leader >>>>>>>>>>>>>>>>>>>>>>> election feature for the products deployed on top of >>>>>>>>>>>>>>>>>>>>>>> that to utilize. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Correction: Please refer [4]. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:27 PM, Asanka Abeyweera < >>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Imesh, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> We are not implementing this to overcome a >>>>>>>>>>>>>>>>>>>>>>>>> limitation in the coordination algorithm available in >>>>>>>>>>>>>>>>>>>>>>>>> the Hazlecast. We are >>>>>>>>>>>>>>>>>>>>>>>>> implementing this since we need an RDBMS based >>>>>>>>>>>>>>>>>>>>>>>>> coordination algorithm (not >>>>>>>>>>>>>>>>>>>>>>>>> a network based algorithm). >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Are you saying that database connections do not use >>>>>>>>>>>>>>>>>>>>>>> the same network used by Hazelcast? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> The reason is, a network based election algorithm >>>>>>>>>>>>>>>>>>>>>>>>> will always elect multiple leaders when the network >>>>>>>>>>>>>>>>>>>>>>>>> is partitioned. But if >>>>>>>>>>>>>>>>>>>>>>>>> we use a RDBMS based algorithm this will not happen. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I do not think your argument is correct. If there >>>>>>>>>>>>>>>>>>>>>>> is a problem with the network, it may apply to both >>>>>>>>>>>>>>>>>>>>>>> Hazelcast based >>>>>>>>>>>>>>>>>>>>>>> solution and database based solution. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Yes, if the same network interface is used network >>>>>>>>>>>>>>>>>>>>> partion will cause all types of connections to be >>>>>>>>>>>>>>>>>>>>> partitioned. But user can >>>>>>>>>>>>>>>>>>>>> use multiple network interfaces for database, Hazelcast >>>>>>>>>>>>>>>>>>>>> and thrift. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Following is the scenario we are trying to solve in MB. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> In MB all the details related to messages, >>>>>>>>>>>>>>>>>>>>> subscriptions, queues, topics etc are stored in database. >>>>>>>>>>>>>>>>>>>>> And we operate >>>>>>>>>>>>>>>>>>>>> depending on that information. If the MB node can't >>>>>>>>>>>>>>>>>>>>> connect to the database >>>>>>>>>>>>>>>>>>>>> that means the node is ineffective in the cluster until >>>>>>>>>>>>>>>>>>>>> it can make a >>>>>>>>>>>>>>>>>>>>> database connection. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> We have seen instances where Hazelcast cluster get >>>>>>>>>>>>>>>>>>>>> partitioned for some time period in networks, Reasons >>>>>>>>>>>>>>>>>>>>> were, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 1. Due to heavy load Hazelcast couldn't process or >>>>>>>>>>>>>>>>>>>>> send (some times both) hearbeats, hence a network >>>>>>>>>>>>>>>>>>>>> partition for Hazelcast >>>>>>>>>>>>>>>>>>>>> cluster >>>>>>>>>>>>>>>>>>>>> 2. An actual network partition of Hazelcast >>>>>>>>>>>>>>>>>>>>> cluster >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> In both scenarios the database connection was working. >>>>>>>>>>>>>>>>>>>>> In that case we get two coordinators elected through >>>>>>>>>>>>>>>>>>>>> Hazelcast and working >>>>>>>>>>>>>>>>>>>>> on the same database to deliver the messages. this leads >>>>>>>>>>>>>>>>>>>>> to inconsistencies >>>>>>>>>>>>>>>>>>>>> in the cluster behavior (for instances duplicate message >>>>>>>>>>>>>>>>>>>>> delivery, >>>>>>>>>>>>>>>>>>>>> corrupred subscription states etc) . >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Since the point of interest for MB is the database, we >>>>>>>>>>>>>>>>>>>>> decided to do the coordinator election through database >>>>>>>>>>>>>>>>>>>>> as well. If the >>>>>>>>>>>>>>>>>>>>> node can't connect to the database, then the MB won't >>>>>>>>>>>>>>>>>>>>> operate anyway. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>> Asitha >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [4] http://blog.kubernetes.io/ >>>>>>>>>>>>>>>>>>>>>>> 2016/01/simple-leader-election-with-Kubernetes.html >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:16 PM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need to implement a leader election >>>>>>>>>>>>>>>>>>>>>>>>>> algorithm on our own? AFAIU this is a complex >>>>>>>>>>>>>>>>>>>>>>>>>> problem which has been >>>>>>>>>>>>>>>>>>>>>>>>>> already solved by several algorithms [1]. IMO it >>>>>>>>>>>>>>>>>>>>>>>>>> would be better to go >>>>>>>>>>>>>>>>>>>>>>>>>> ahead with an existing well established >>>>>>>>>>>>>>>>>>>>>>>>>> implementation on etcd [1] or >>>>>>>>>>>>>>>>>>>>>>>>>> Consul [2]. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Those provide HTTP APIs for clients to make >>>>>>>>>>>>>>>>>>>>>>>>>> leader election calls. [3] is a client library >>>>>>>>>>>>>>>>>>>>>>>>>> written in Node.js for etcd >>>>>>>>>>>>>>>>>>>>>>>>>> based leader election. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://www.projectcalico. >>>>>>>>>>>>>>>>>>>>>>>>>> org/using-etcd-for-elections >>>>>>>>>>>>>>>>>>>>>>>>>> [2] https://www.consul.io/docs >>>>>>>>>>>>>>>>>>>>>>>>>> /guides/leader-election.html >>>>>>>>>>>>>>>>>>>>>>>>>> [3] https://www.npmjs.com/package/etcd-leader >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 3, 2016 at 5:12 PM, Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Since we are using RDBMS to poll the node >>>>>>>>>>>>>>>>>>>>>>>>>>> status, the cluster will not end up in situation >>>>>>>>>>>>>>>>>>>>>>>>>>> 1,2 or 3. With this >>>>>>>>>>>>>>>>>>>>>>>>>>> approach we consider a node unreachable when it >>>>>>>>>>>>>>>>>>>>>>>>>>> cannot access the database. >>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore an unreachable node can never be the >>>>>>>>>>>>>>>>>>>>>>>>>>> leader. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> As you have mentioned, we are currently using >>>>>>>>>>>>>>>>>>>>>>>>>>> the RDBMS as an atomic global variable to create >>>>>>>>>>>>>>>>>>>>>>>>>>> the coordinator entry. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 2, 2016 at 5:22 PM, Maninda >>>>>>>>>>>>>>>>>>>>>>>>>>> Edirisooriya <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> As I understand the accuracy of electing the >>>>>>>>>>>>>>>>>>>>>>>>>>>> leader correctly is dependent on the election >>>>>>>>>>>>>>>>>>>>>>>>>>>> mechanism with RDBMS because >>>>>>>>>>>>>>>>>>>>>>>>>>>> there can be edge cases like, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Unreachable leader activates during the >>>>>>>>>>>>>>>>>>>>>>>>>>>> election process: Then who becomes the leader? >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. The elected leader becomes unreachable >>>>>>>>>>>>>>>>>>>>>>>>>>>> before the election is completed: Then will there >>>>>>>>>>>>>>>>>>>>>>>>>>>> be a situation where >>>>>>>>>>>>>>>>>>>>>>>>>>>> there is no leader? >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. A leader and a set of nodes are disconnected >>>>>>>>>>>>>>>>>>>>>>>>>>>> from the other part of the cluster and while the >>>>>>>>>>>>>>>>>>>>>>>>>>>> leader is trying to remove >>>>>>>>>>>>>>>>>>>>>>>>>>>> unreachable members other part is calling an >>>>>>>>>>>>>>>>>>>>>>>>>>>> election to make a leader: Who >>>>>>>>>>>>>>>>>>>>>>>>>>>> will win? >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> RDBMS based election algorithm should handle >>>>>>>>>>>>>>>>>>>>>>>>>>>> such cases without bringing the cluster to an >>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent state or dead >>>>>>>>>>>>>>>>>>>>>>>>>>>> lock in all concurrent cases. If all these kind of >>>>>>>>>>>>>>>>>>>>>>>>>>>> cases cannot be handled >>>>>>>>>>>>>>>>>>>>>>>>>>>> isn't it better to keep the current hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>>> clustering and use the RDBMS >>>>>>>>>>>>>>>>>>>>>>>>>>>> only to handle the split brain scenario? In other >>>>>>>>>>>>>>>>>>>>>>>>>>>> words when a new >>>>>>>>>>>>>>>>>>>>>>>>>>>> hazelcast leader is elected it should be updated >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the RDBMS. If another >>>>>>>>>>>>>>>>>>>>>>>>>>>> split party has already elected a leader, the node >>>>>>>>>>>>>>>>>>>>>>>>>>>> who is going to write it >>>>>>>>>>>>>>>>>>>>>>>>>>>> to RDBMS should avoid updating it. Simply, the >>>>>>>>>>>>>>>>>>>>>>>>>>>> RDBMS can be used as an >>>>>>>>>>>>>>>>>>>>>>>>>>>> atomic global variable to keep the leader name by >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifying the hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>>> clustering. WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>>>>>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 4:38 PM, Asanka >>>>>>>>>>>>>>>>>>>>>>>>>>>> Abeyweera <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Akila, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let me explain the issue in a different way. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let's assume the MB nodes are using two different >>>>>>>>>>>>>>>>>>>>>>>>>>>>> network interfaces for >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication and database >>>>>>>>>>>>>>>>>>>>>>>>>>>>> communication. With such a >>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration, there can be failures only in the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> network interface used for >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication in some nodes. When this >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happens, there will be two >>>>>>>>>>>>>>>>>>>>>>>>>>>>> or more Hazelcast clusters due to the network >>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation, and as a result >>>>>>>>>>>>>>>>>>>>>>>>>>>>> there will be multiple coordinators. Since every >>>>>>>>>>>>>>>>>>>>>>>>>>>>> node still have access to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the database, multiple coordinators can affect >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the correctness of the data >>>>>>>>>>>>>>>>>>>>>>>>>>>>> stored in the DB. But if we used a RDBMS based >>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach we won't have >>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple coordinators due to a network partition >>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Hazelcast. This is one >>>>>>>>>>>>>>>>>>>>>>>>>>>>> advantage we get from this approach. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Even when we use Zookeeper or RAFT the same >>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue will be there since we are using different >>>>>>>>>>>>>>>>>>>>>>>>>>>>> interfaces for Hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>>>> communication and DB communication. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 2:56 PM, Akila >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ravihansa Perera <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What's the advantage of using RDBMS (even as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an alternative) to implement a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leader/coordinator election? If the network >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connection to DB fails then this will be a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> single point of failure. I don't >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think we can scale RDBMS instances and expect >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the election algorithm to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work. That would be reducing this problem to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> another problem (electing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator RDBMS instance). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IMHO it would be better to look at Zookeeper >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Atomic Broadcast (ZAB) [1] or RAFT leader >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> election [2] algorithms which >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have already proven results. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://cwiki.apache.org/c >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> onfluence/display/ZOOKEEPER/Zab1.0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] http://libraft.io/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:42 PM, Nandika >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jayawardana <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 to make it a common component . We have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the clustering implementation for BPEL >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> component based on hazelcast. If >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the coordination is available at RDBMS level, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we can remove hazelcast >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dependancy. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nandika >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:28 PM, Hasitha >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Aravinda <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can we make it a common component, which is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not hard coupled with MB. BPS has the same >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 9:47 AM, Asanka >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Abeyweera <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In MB, we have used a coordinator based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach to manage distributed messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm in the cluster. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently Hazelcast is used to elect the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator. But one issue we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> faced with Hazelcast is, during a network >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation (split brain), >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast can elect two or more coordinators >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cluster. This affects >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the correctness of the distributed messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm since there are some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tables in the database that should only be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> edited by a single node (i.e. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As a solution to this problem we have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented minimum node count based approach >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] to deactivate set of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitioned nodes to stop multiple nodes >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> becoming coordinators until the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network segmentation issue is fixed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As an alternative solution, we are >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thinking of implementing an RDBMS based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach to elect the coordinator >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> node in the cluster. By doing this we can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make sure that even during a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network segmentation only one node will be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> elected as the coordinator node >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> since the election is happening through the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> database. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The algorithm will use a polling mechanism >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to check the validity of the nodes. To make >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the election algorithm >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalable, only the coordinator node will be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> checking status of all the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nodes in the cluster and it will inform other >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nodes through database when a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> member is added/left. The nodes will be only >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> checking for the status of the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator node. When a node detect that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator is invalid it will go >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a election to elect a new coordinator. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We are currently working on a POC to test >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how this works with MB's slot based messaging >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> algorithm. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://wso2.org/jira/browse/MB-1664 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha Aravinda, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Associate Technical Lead, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mobile : +94 718 210 200 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nandika Jayawardana >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc ; http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> _________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>> Ramith Jayasinghe >>>>>>>>>>>>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> E: [email protected] >>>>>>>>>>>>>>>>>>>>>>>> P: +94 772534930 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> *Asitha Nanayakkara* <http://asitha.github.io/> >>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>> WSO2, Inc. <http://wso2.com/> >>>>>>>>>>>>>>>>>>>>> Mob: +94 77 853 0682 >>>>>>>>>>>>>>>>>>>>> [image: https://wso2.com/signature] >>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Malaka Silva >>>>>>>>>>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>>>>>>>>>> M: +94 777 219 791 >>>>>>>>>>>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>>>>>>>>>>> Skype : malaka.sampath.silva >>>>>>>>>>>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/ >>>>>>>>>>>>>>>>>>>> malaka-silva/6/33/77 >>>>>>>>>>>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/ >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>>>>>>>> https://wso2.com/signature >>>>>>>>>>>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/ >>>>>>>>>>>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/> >>>>>>>>>>>>>>>>>>>> https://store.wso2.com/store/ >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Don't make Trees rare, we should keep them with care >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Ramith Jayasinghe >>>>>>>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> E: [email protected] >>>>>>>>>>>>>>>>>>> P: +94 772534930 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Sajini De SIlva >>>>>>>>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>>>>>> Blog: http://sajinid.blogspot.com/ >>>>>>>>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Phone: +94 712797729 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Sajini De SIlva >>>>>>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>>>> Blog: http://sajinid.blogspot.com/ >>>>>>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Phone: +94 712797729 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Kasun Indrasiri >>>>>>>>>>>>>> Director, Integration Technologies >>>>>>>>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>> >>>>>>>>>>>>>> cell: +1 650 450 2293 >>>>>>>>>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Anjana Fernando* >>>>>>>>>>>>> Associate Director / Architect >>>>>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Anjana Fernando* >>>>>>>>>>> Associate Director / Architect >>>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Asanka Abeyweera >>>>>>>>>> Senior Software Engineer >>>>>>>>>> WSO2 Inc. >>>>>>>>>> >>>>>>>>>> Phone: +94 712228648 >>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>> >>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Anjana Fernando* >>>>>>>>> Associate Director / Architect >>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>> lean . enterprise . middleware >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Anjana Fernando* >>>>>>>> Associate Director / Architect >>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>> lean . enterprise . middleware >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Asanka Abeyweera >>>>>>> Senior Software Engineer >>>>>>> WSO2 Inc. >>>>>>> >>>>>>> Phone: +94 712228648 >>>>>>> Blog: a5anka.github.io >>>>>>> >>>>>>> <https://wso2.com/signature> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Associate Director / Architect >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Anjana Fernando* >>>>> Associate Director / Architect >>>>> WSO2 Inc. | http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>> >>>> >>>> >>>> -- >>>> Ramith Jayasinghe >>>> Technical Lead >>>> WSO2 Inc., http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> E: [email protected] >>>> P: +94 772534930 >>>> >>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Associate Director / Architect >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> Ramith Jayasinghe >> Technical Lead >> WSO2 Inc., http://wso2.com >> lean.enterprise.middleware >> >> E: [email protected] >> P: +94 772534930 <+94%2077%20253%204930> >> > > > > -- > Asanka Abeyweera > Senior Software Engineer > WSO2 Inc. > > Phone: +94 712228648 <+94%2071%20222%208648> > Blog: a5anka.github.io > > <https://wso2.com/signature> > -- *Anjana Fernando* Associate Director / Architect WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
