Hi, On Wed, Nov 2, 2016 at 3:14 PM, Asanka Abeyweera <[email protected]> wrote:
> Hi Anjana, > > Currently, the implementation is part of the MB code (not a common > component). > Okay, can we please get it as a common component. Cheers, Anjana. > > On Wed, Nov 2, 2016 at 3:00 PM, Anjana Fernando <[email protected]> wrote: > >> Hi Asanka/Ramith, >> >> So for C5 based Streaming Analytics solution, we need coordination >> functionality there as well. Is the functionality mentioned here created as >> a common component or baked in to the MB code? .. if so, can we please get >> it implemented it as a generic component, so other products can also use >> it. >> >> Cheers, >> Anjana. >> >> On Tue, Aug 9, 2016 at 3:10 PM, Anjana Fernando <[email protected]> wrote: >> >>> Great! .. >>> >>> Cheers, >>> Anjana. >>> >>> On Tue, Aug 9, 2016 at 1:49 PM, Asanka Abeyweera <[email protected]> >>> wrote: >>> >>>> Hi Anjana, >>>> >>>> Thank you for the suggestion. We have already done a similar thing. We >>>> have added a backoff time after creating the leader entry and check if the >>>> leader entry is the entry created by self before informing the leader >>>> change. >>>> >>>> On Tue, Aug 9, 2016 at 12:27 PM, Anjana Fernando <[email protected]> >>>> wrote: >>>> >>>>> I see, thanks for the clarification, looks good! .. I think small >>>>> thing to consider is, to avoid the situation where, the current leader >>>>> goes >>>>> away, and two other competes to become the leader, and the first one and >>>>> the second one checks (reads) the table to check the last heartbeat and >>>>> figures out that the leader is outdated at the same time, and then first >>>>> one delete the entry and puts his one, and after that, second one will >>>>> also >>>>> delete the existing one and put his one, so both will think they became >>>>> the >>>>> leader, due to the condition that both succeeded in adding the entry >>>>> without an error. So this can probably be fixed by checking back after a >>>>> bit of time if the current node is actually me, which probabilistically >>>>> will work well, if that time period is sufficient big enough than a >>>>> typical >>>>> database transaction required by a node to do the earlier operations. Or >>>>> else, we should make sure the database transaction level used in this >>>>> scenario is at least REPEATABLE_READ, where when we read the record, it >>>>> will lock it throughout the transaction. So some DBMSs does not support >>>>> REPEATABLE_READ, where in that case, we should be able to use >>>>> SERIALIZABLE, >>>>> which most of them support. >>>>> >>>>> Cheers, >>>>> Anjana. >>>>> >>>>> On Tue, Aug 9, 2016 at 11:11 AM, Maninda Edirisooriya < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Anjana, >>>>>> >>>>>> After having an offline chat with Asanka what I understood was that >>>>>> the leader election was done completely via the database but with no >>>>>> network communication. The leader is mentioned in the database first. >>>>>> Then >>>>>> the leader updates the node data periodically in the database. If some >>>>>> node >>>>>> realizes the data in the DB are outdated that means the leader was >>>>>> disconnected. Then that node will look at the created timestamp of the >>>>>> leader entry. If that is not very recent that means there was no new >>>>>> leader >>>>>> elected recently. So he will try to update the leader entry with his ID. >>>>>> As >>>>>> I understand there the leader entry is using the leader ID and the >>>>>> timestamp as the primary key. Even several nodes try to do it >>>>>> simultaneously only one node will successfully be able to update the >>>>>> entry >>>>>> with the help of atomicity provided by the DB. Others members will note >>>>>> the >>>>>> timestamp of the leader was updated so will accept the first one who >>>>>> updates as the leader. Even after the leader is elected, the leader will >>>>>> only notify node data via updating DB instead of network calls. Other >>>>>> nodes >>>>>> will just observe it and check the latest timestmps of the entry. >>>>>> >>>>>> >>>>>> *Maninda Edirisooriya* >>>>>> Senior Software Engineer >>>>>> >>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>> >>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>> *E-mail* : [email protected] >>>>>> *Skype* : @manindae >>>>>> *Twitter* : @maninda >>>>>> >>>>>> On Tue, Aug 9, 2016 at 10:13 AM, Anjana Fernando <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I just noticed this thread. I've some concerns on this >>>>>>> implementations. First of all, I don't think the statement mentioned >>>>>>> here >>>>>>> saying an external service such as ZooKeeper doesn't work, is correct. >>>>>>> Because, if you have a ZK cluster (it is suppose to be used as a >>>>>>> cluster), >>>>>>> you will not have any issues. All the nodes have a list of endpoints to >>>>>>> all >>>>>>> the ZK nodes and they connect to those, and ZK has a quorum based >>>>>>> mechanism >>>>>>> in keeping its state. So this makes sure, all the users have a single >>>>>>> version of the ZK data. >>>>>>> >>>>>>> Also, I guess the fundamental problem here in the split brain >>>>>>> situation is, we need one external entity taking the decision (e.g. ZK >>>>>>> cluster), because it should have oversight to the whole environment. I >>>>>>> don't see how this RDBMS mechanism would solve that. Because, what it >>>>>>> gives >>>>>>> is a central location of state persistence. But the decisions of making >>>>>>> who >>>>>>> is the leader is taken by the users, which can be problematic. Where >>>>>>> when >>>>>>> we have a network partition scenario in that occasion, two groups of >>>>>>> users >>>>>>> will be overriding each other in the centralized RDBMS data repeatedly >>>>>>> and >>>>>>> it will go on forever, where in the ZK situation, there will be only one >>>>>>> leader, and the guys in the other partition simply won't be able to >>>>>>> reach >>>>>>> the leader, until its network issues are sorted. >>>>>>> >>>>>>> So I also think, as Imesh mentioned, creating a coordination >>>>>>> algorithm from scratch may not be a wise decision, and we should use >>>>>>> proven >>>>>>> technology/libraries to do that. And on a side note, the main reason for >>>>>>> not using ZK for this earlier was because of the hassle of bringing up >>>>>>> another set of servers when our products are clustered, and we knew that >>>>>>> the split brain scenario will occur in HZ, but maybe now we should give >>>>>>> an >>>>>>> extension point probably to plug into an external service if for some >>>>>>> applications the split brain scenario is a show stopper. >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> On Tue, Aug 9, 2016 at 4:45 AM, Kasun Indrasiri <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Ramith/Asanka, >>>>>>>> >>>>>>>> ESB/DSS natask impl is also based on HZ. I guess if this model >>>>>>>> works for the MB, we should make it generic for all such coordination >>>>>>>> requirements. (Thinking about using this in ESB 5.1)? >>>>>>>> >>>>>>>> On Fri, Aug 5, 2016 at 3:58 AM, Sajini De Silva <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Maninda, >>>>>>>>> >>>>>>>>> Locking the database will be supported by some databases but >>>>>>>>> there will be huge performance impact. So we cannot use that >>>>>>>>> approach. If >>>>>>>>> this approach cannot be adapted the only thing we can do is queue >>>>>>>>> wise >>>>>>>>> load balancing through slot coordinator. But in this case we cannot >>>>>>>>> guarantee that load balance will be equally distributed since some >>>>>>>>> queues >>>>>>>>> can be loaded while some will be idle. Also we cannot have multiple >>>>>>>>> slot >>>>>>>>> coordinators having same queue as it may cause several complications >>>>>>>>> such >>>>>>>>> as, same slot is assigned to two nodes by different subscribers, >>>>>>>>> message >>>>>>>>> duplication etc. Actually this slot architecture was discussed in a >>>>>>>>> separate mail thread before it is implemented. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> On Fri, Aug 5, 2016 at 3:12 PM, Maninda Edirisooriya < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Sajini, >>>>>>>>>> >>>>>>>>>> Yes that is what I meant. As the number of slots are proportional >>>>>>>>>> to the number of messages passing through the cluster, slot delivery >>>>>>>>>> should >>>>>>>>>> not be handled by the coordinator when there is only one coordinator >>>>>>>>>> in the >>>>>>>>>> cluster which is a bottleneck for scaling messages passing through >>>>>>>>>> the >>>>>>>>>> cluster. If there is only a single coordinator, it should handle >>>>>>>>>> operations >>>>>>>>>> that are not proportional to messages throughput of the cluster. >>>>>>>>>> Then only >>>>>>>>>> the tasks like subscriber adding / removing should be handled by the >>>>>>>>>> coordinator. As this is not the current implementation, we can >>>>>>>>>> consider >>>>>>>>>> multiple coordinator approach. Then the number of coordinators >>>>>>>>>> should be >>>>>>>>>> scalable with the message throughout. I am not sure whether locking >>>>>>>>>> the >>>>>>>>>> database per transaction would achieve this coordinator scalability >>>>>>>>>> in the >>>>>>>>>> multiple coordinator implementation. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>> Senior Software Engineer >>>>>>>>>> >>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>> >>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>> *Skype* : @manindae >>>>>>>>>> *Twitter* : @maninda >>>>>>>>>> >>>>>>>>>> On Fri, Aug 5, 2016 at 2:42 PM, Sajini De Silva <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Maninda, >>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 5, 2016 at 2:28 PM, Maninda Edirisooriya < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> @Sajini, >>>>>>>>>>>> >>>>>>>>>>>> But the number of slots are proportional to the number of >>>>>>>>>>>> messages pass through the MB which needs to be handled by the >>>>>>>>>>>> coordinator. >>>>>>>>>>>> That is what I meant by "information related to meta data of >>>>>>>>>>>> messages pass >>>>>>>>>>>> through a single coordinator". Ideally after the senders and >>>>>>>>>>>> receivers are >>>>>>>>>>>> subscribed to the cluster, coordinator should have nothing to do >>>>>>>>>>>> until they >>>>>>>>>>>> are removed or changed. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Even though it is possible to have multiple coordinators after >>>>>>>>>>> having en effort (Lock the database for a whole transaction or the >>>>>>>>>>> work >>>>>>>>>>> load distribution as described by Ramith) , coordinator may have >>>>>>>>>>> different >>>>>>>>>>> work to do other than subscriber adding and removing. As I said >>>>>>>>>>> earlier our >>>>>>>>>>> MB message distribution system is based on slot architecture and >>>>>>>>>>> slots will >>>>>>>>>>> managed by the coordinator. You can read [1] to understand more >>>>>>>>>>> about slot >>>>>>>>>>> architecture in MB. >>>>>>>>>>> >>>>>>>>>>> [1] http://sajinid.blogspot.com/2015/03/wso2-message-broker- >>>>>>>>>>> 300-slot-based.html >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> @Ramith, >>>>>>>>>>>> >>>>>>>>>>>> +1 for multiple coordinators by partitioning the cluster which >>>>>>>>>>>> maintains the simplicity and correctness of the algorithm than >>>>>>>>>>>> compromising >>>>>>>>>>>> simplicity with a less important factor like "delivering a good >>>>>>>>>>>> mix of >>>>>>>>>>>> messages". >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>> >>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>> >>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 5, 2016 at 2:05 PM, Ramith Jayasinghe < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> @Imesh, >>>>>>>>>>>>> We can prove that doing leader election using a lib (where we >>>>>>>>>>>>> maintain cluster state in another place, a.k.a DB) will not solve >>>>>>>>>>>>> our >>>>>>>>>>>>> original problem (this also relates to our past experience with >>>>>>>>>>>>> both the >>>>>>>>>>>>> zookeeper and hazelcast). >>>>>>>>>>>>> We can make this implementation a common component if other >>>>>>>>>>>>> products have a use of it. BPS might be able to use it since >>>>>>>>>>>>> their data is >>>>>>>>>>>>> also in the database. >>>>>>>>>>>>> >>>>>>>>>>>>> @Malaka: >>>>>>>>>>>>> VFS scenario can't be solved by relying on this >>>>>>>>>>>>> implementation. why? you can have the access to DB but not VFS >>>>>>>>>>>>> resources/file (and vice versa). this is the same point we >>>>>>>>>>>>> explained before. >>>>>>>>>>>>> in Ntask implementation, if tasks are stored in the database >>>>>>>>>>>>> then using this implementation makes sense. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> @Akila, >>>>>>>>>>>>> implementing (distributed) a queue algorithm is non-trivial. >>>>>>>>>>>>> Having one coordinator (single source of truth) keeps things >>>>>>>>>>>>> simple hence >>>>>>>>>>>>> it's a conscious design decision we agreed during the initial >>>>>>>>>>>>> stages. >>>>>>>>>>>>> However, possible extension to this scheme is to have multiple >>>>>>>>>>>>> coordinators >>>>>>>>>>>>> ( each responsible for coordinating a subset of queues in the >>>>>>>>>>>>> cluster), >>>>>>>>>>>>> that will be some what similar to kafka. >>>>>>>>>>>>> Even if its preferable to have no coordinator at-all, (to >>>>>>>>>>>>> decide how messages are disseminated in the cluster) that will >>>>>>>>>>>>> make us >>>>>>>>>>>>> give up desired behaviour such as delivering a good mix of >>>>>>>>>>>>> messages (from >>>>>>>>>>>>> different publishers) to consumers in a cluster. having said >>>>>>>>>>>>> this, we have >>>>>>>>>>>>> an ongoing research on how to improve the algorithm and we like >>>>>>>>>>>>> to try out >>>>>>>>>>>>> both these approaches. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Aug 5, 2016 at 1:31 PM, Malaka Silva <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> The same issue with Hazelcast can be experienced with ESB >>>>>>>>>>>>>> inbounds (running on top of NTASK) and VFS distribution locks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The idea of only single worker works at a given time breaks >>>>>>>>>>>>>> if there is a Hazelcast heart beat fails. This will make two >>>>>>>>>>>>>> workers to >>>>>>>>>>>>>> work in parallel. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also with distributed locking there is no guarantee that file >>>>>>>>>>>>>> is only process only by one worker. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So in the case of network fail >>>>>>>>>>>>>> with DB >>>>>>>>>>>>>> make sense to stop processing until it's recovered. >>>>>>>>>>>>>> Also making this component generic ESB can reuse. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 9:21 AM, Asitha Nanayakkara < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Imesh, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:33 AM, Imesh Gunaratne < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Aug 5, 2016 at 7:31 AM, Imesh Gunaratne < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> You can see here [3] how K8S has implemented leader >>>>>>>>>>>>>>>>> election feature for the products deployed on top of that to >>>>>>>>>>>>>>>>> utilize. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Correction: Please refer [4]. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:27 PM, Asanka Abeyweera < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Imesh, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> We are not implementing this to overcome a limitation in >>>>>>>>>>>>>>>>>>> the coordination algorithm available in the Hazlecast. We >>>>>>>>>>>>>>>>>>> are implementing >>>>>>>>>>>>>>>>>>> this since we need an RDBMS based coordination algorithm >>>>>>>>>>>>>>>>>>> (not a network >>>>>>>>>>>>>>>>>>> based algorithm). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Are you saying that database connections do not use the >>>>>>>>>>>>>>>>> same network used by Hazelcast? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The reason is, a network based election algorithm will >>>>>>>>>>>>>>>>>>> always elect multiple leaders when the network is >>>>>>>>>>>>>>>>>>> partitioned. But if we >>>>>>>>>>>>>>>>>>> use a RDBMS based algorithm this will not happen. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I do not think your argument is correct. If there is a >>>>>>>>>>>>>>>>> problem with the network, it may apply to both Hazelcast >>>>>>>>>>>>>>>>> based solution >>>>>>>>>>>>>>>>> and database based solution. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, if the same network interface is used network partion >>>>>>>>>>>>>>> will cause all types of connections to be partitioned. But user >>>>>>>>>>>>>>> can use >>>>>>>>>>>>>>> multiple network interfaces for database, Hazelcast and thrift. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Following is the scenario we are trying to solve in MB. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In MB all the details related to messages, subscriptions, >>>>>>>>>>>>>>> queues, topics etc are stored in database. And we operate >>>>>>>>>>>>>>> depending on that >>>>>>>>>>>>>>> information. If the MB node can't connect to the database that >>>>>>>>>>>>>>> means the >>>>>>>>>>>>>>> node is ineffective in the cluster until it can make a database >>>>>>>>>>>>>>> connection. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We have seen instances where Hazelcast cluster get >>>>>>>>>>>>>>> partitioned for some time period in networks, Reasons were, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Due to heavy load Hazelcast couldn't process or send >>>>>>>>>>>>>>> (some times both) hearbeats, hence a network partition for >>>>>>>>>>>>>>> Hazelcast cluster >>>>>>>>>>>>>>> 2. An actual network partition of Hazelcast cluster >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In both scenarios the database connection was working. In >>>>>>>>>>>>>>> that case we get two coordinators elected through Hazelcast and >>>>>>>>>>>>>>> working on >>>>>>>>>>>>>>> the same database to deliver the messages. this leads to >>>>>>>>>>>>>>> inconsistencies in >>>>>>>>>>>>>>> the cluster behavior (for instances duplicate message delivery, >>>>>>>>>>>>>>> corrupred >>>>>>>>>>>>>>> subscription states etc) . >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Since the point of interest for MB is the database, we >>>>>>>>>>>>>>> decided to do the coordinator election through database as >>>>>>>>>>>>>>> well. If the >>>>>>>>>>>>>>> node can't connect to the database, then the MB won't operate >>>>>>>>>>>>>>> anyway. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Asitha >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [4] http://blog.kubernetes.io/ >>>>>>>>>>>>>>>>> 2016/01/simple-leader-election-with-Kubernetes.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Thu, Aug 4, 2016 at 7:16 PM, Imesh Gunaratne < >>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Do we really need to implement a leader election >>>>>>>>>>>>>>>>>>>> algorithm on our own? AFAIU this is a complex problem >>>>>>>>>>>>>>>>>>>> which has been >>>>>>>>>>>>>>>>>>>> already solved by several algorithms [1]. IMO it would be >>>>>>>>>>>>>>>>>>>> better to go >>>>>>>>>>>>>>>>>>>> ahead with an existing well established implementation on >>>>>>>>>>>>>>>>>>>> etcd [1] or >>>>>>>>>>>>>>>>>>>> Consul [2]. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Those provide HTTP APIs for clients to make leader >>>>>>>>>>>>>>>>>>>> election calls. [3] is a client library written in Node.js >>>>>>>>>>>>>>>>>>>> for etcd based >>>>>>>>>>>>>>>>>>>> leader election. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [1] https://www.projectcalico. >>>>>>>>>>>>>>>>>>>> org/using-etcd-for-elections >>>>>>>>>>>>>>>>>>>> [2] https://www.consul.io/docs >>>>>>>>>>>>>>>>>>>> /guides/leader-election.html >>>>>>>>>>>>>>>>>>>> [3] https://www.npmjs.com/package/etcd-leader >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Wed, Aug 3, 2016 at 5:12 PM, Asanka Abeyweera < >>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Maninda, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Since we are using RDBMS to poll the node status, the >>>>>>>>>>>>>>>>>>>>> cluster will not end up in situation 1,2 or 3. With this >>>>>>>>>>>>>>>>>>>>> approach we >>>>>>>>>>>>>>>>>>>>> consider a node unreachable when it cannot access the >>>>>>>>>>>>>>>>>>>>> database. Therefore >>>>>>>>>>>>>>>>>>>>> an unreachable node can never be the leader. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> As you have mentioned, we are currently using the >>>>>>>>>>>>>>>>>>>>> RDBMS as an atomic global variable to create the >>>>>>>>>>>>>>>>>>>>> coordinator entry. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Tue, Aug 2, 2016 at 5:22 PM, Maninda Edirisooriya < >>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi Asanka, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> As I understand the accuracy of electing the leader >>>>>>>>>>>>>>>>>>>>>> correctly is dependent on the election mechanism with >>>>>>>>>>>>>>>>>>>>>> RDBMS because there >>>>>>>>>>>>>>>>>>>>>> can be edge cases like, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1. Unreachable leader activates during the election >>>>>>>>>>>>>>>>>>>>>> process: Then who becomes the leader? >>>>>>>>>>>>>>>>>>>>>> 2. The elected leader becomes unreachable before the >>>>>>>>>>>>>>>>>>>>>> election is completed: Then will there be a situation >>>>>>>>>>>>>>>>>>>>>> where there is no >>>>>>>>>>>>>>>>>>>>>> leader? >>>>>>>>>>>>>>>>>>>>>> 3. A leader and a set of nodes are disconnected from >>>>>>>>>>>>>>>>>>>>>> the other part of the cluster and while the leader is >>>>>>>>>>>>>>>>>>>>>> trying to remove >>>>>>>>>>>>>>>>>>>>>> unreachable members other part is calling an election to >>>>>>>>>>>>>>>>>>>>>> make a leader: Who >>>>>>>>>>>>>>>>>>>>>> will win? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> RDBMS based election algorithm should handle such >>>>>>>>>>>>>>>>>>>>>> cases without bringing the cluster to an inconsistent >>>>>>>>>>>>>>>>>>>>>> state or dead lock in >>>>>>>>>>>>>>>>>>>>>> all concurrent cases. If all these kind of cases cannot >>>>>>>>>>>>>>>>>>>>>> be handled isn't it >>>>>>>>>>>>>>>>>>>>>> better to keep the current hazelcast clustering and use >>>>>>>>>>>>>>>>>>>>>> the RDBMS only to >>>>>>>>>>>>>>>>>>>>>> handle the split brain scenario? In other words when a >>>>>>>>>>>>>>>>>>>>>> new hazelcast leader >>>>>>>>>>>>>>>>>>>>>> is elected it should be updated in the RDBMS. If another >>>>>>>>>>>>>>>>>>>>>> split party has >>>>>>>>>>>>>>>>>>>>>> already elected a leader, the node who is going to write >>>>>>>>>>>>>>>>>>>>>> it to RDBMS should >>>>>>>>>>>>>>>>>>>>>> avoid updating it. Simply, the RDBMS can be used as an >>>>>>>>>>>>>>>>>>>>>> atomic global >>>>>>>>>>>>>>>>>>>>>> variable to keep the leader name by modifying the >>>>>>>>>>>>>>>>>>>>>> hazelcast clustering. >>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 4:38 PM, Asanka Abeyweera < >>>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Akila, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Let me explain the issue in a different way. Let's >>>>>>>>>>>>>>>>>>>>>>> assume the MB nodes are using two different network >>>>>>>>>>>>>>>>>>>>>>> interfaces for >>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication and database communication. >>>>>>>>>>>>>>>>>>>>>>> With such a >>>>>>>>>>>>>>>>>>>>>>> configuration, there can be failures only in the >>>>>>>>>>>>>>>>>>>>>>> network interface used for >>>>>>>>>>>>>>>>>>>>>>> Hazelcast communication in some nodes. When this >>>>>>>>>>>>>>>>>>>>>>> happens, there will be two >>>>>>>>>>>>>>>>>>>>>>> or more Hazelcast clusters due to the network >>>>>>>>>>>>>>>>>>>>>>> segmentation, and as a result >>>>>>>>>>>>>>>>>>>>>>> there will be multiple coordinators. Since every node >>>>>>>>>>>>>>>>>>>>>>> still have access to >>>>>>>>>>>>>>>>>>>>>>> the database, multiple coordinators can affect the >>>>>>>>>>>>>>>>>>>>>>> correctness of the data >>>>>>>>>>>>>>>>>>>>>>> stored in the DB. But if we used a RDBMS based approach >>>>>>>>>>>>>>>>>>>>>>> we won't have >>>>>>>>>>>>>>>>>>>>>>> multiple coordinators due to a network partition in >>>>>>>>>>>>>>>>>>>>>>> Hazelcast. This is one >>>>>>>>>>>>>>>>>>>>>>> advantage we get from this approach. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Even when we use Zookeeper or RAFT the same issue >>>>>>>>>>>>>>>>>>>>>>> will be there since we are using different interfaces >>>>>>>>>>>>>>>>>>>>>>> for Hazelcast >>>>>>>>>>>>>>>>>>>>>>> communication and DB communication. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 2:56 PM, Akila Ravihansa >>>>>>>>>>>>>>>>>>>>>>> Perera <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> What's the advantage of using RDBMS (even as an >>>>>>>>>>>>>>>>>>>>>>>> alternative) to implement a leader/coordinator >>>>>>>>>>>>>>>>>>>>>>>> election? If the network >>>>>>>>>>>>>>>>>>>>>>>> connection to DB fails then this will be a single >>>>>>>>>>>>>>>>>>>>>>>> point of failure. I don't >>>>>>>>>>>>>>>>>>>>>>>> think we can scale RDBMS instances and expect the >>>>>>>>>>>>>>>>>>>>>>>> election algorithm to >>>>>>>>>>>>>>>>>>>>>>>> work. That would be reducing this problem to another >>>>>>>>>>>>>>>>>>>>>>>> problem (electing >>>>>>>>>>>>>>>>>>>>>>>> coordinator RDBMS instance). >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> IMHO it would be better to look at Zookeeper Atomic >>>>>>>>>>>>>>>>>>>>>>>> Broadcast (ZAB) [1] or RAFT leader election [2] >>>>>>>>>>>>>>>>>>>>>>>> algorithms which have >>>>>>>>>>>>>>>>>>>>>>>> already proven results. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [1] https://cwiki.apache.org/c >>>>>>>>>>>>>>>>>>>>>>>> onfluence/display/ZOOKEEPER/Zab1.0 >>>>>>>>>>>>>>>>>>>>>>>> [2] http://libraft.io/ >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:42 PM, Nandika >>>>>>>>>>>>>>>>>>>>>>>> Jayawardana <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> +1 to make it a common component . We have the >>>>>>>>>>>>>>>>>>>>>>>>> clustering implementation for BPEL component based on >>>>>>>>>>>>>>>>>>>>>>>>> hazelcast. If the >>>>>>>>>>>>>>>>>>>>>>>>> coordination is available at RDBMS level, we can >>>>>>>>>>>>>>>>>>>>>>>>> remove hazelcast >>>>>>>>>>>>>>>>>>>>>>>>> dependancy. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>>>>>>> Nandika >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:28 PM, Hasitha Aravinda >>>>>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Can we make it a common component, which is not >>>>>>>>>>>>>>>>>>>>>>>>>> hard coupled with MB. BPS has the same requirement. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 9:47 AM, Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> In MB, we have used a coordinator based approach >>>>>>>>>>>>>>>>>>>>>>>>>>> to manage distributed messaging algorithm in the >>>>>>>>>>>>>>>>>>>>>>>>>>> cluster. Currently >>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast is used to elect the coordinator. But one >>>>>>>>>>>>>>>>>>>>>>>>>>> issue we faced with >>>>>>>>>>>>>>>>>>>>>>>>>>> Hazelcast is, during a network segmentation (split >>>>>>>>>>>>>>>>>>>>>>>>>>> brain), Hazelcast can >>>>>>>>>>>>>>>>>>>>>>>>>>> elect two or more coordinators in the cluster. This >>>>>>>>>>>>>>>>>>>>>>>>>>> affects the correctness >>>>>>>>>>>>>>>>>>>>>>>>>>> of the distributed messaging algorithm since there >>>>>>>>>>>>>>>>>>>>>>>>>>> are some tables in the >>>>>>>>>>>>>>>>>>>>>>>>>>> database that should only be edited by a single >>>>>>>>>>>>>>>>>>>>>>>>>>> node (i.e. coordinator). >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> As a solution to this problem we have >>>>>>>>>>>>>>>>>>>>>>>>>>> implemented minimum node count based approach [1] >>>>>>>>>>>>>>>>>>>>>>>>>>> to deactivate set of >>>>>>>>>>>>>>>>>>>>>>>>>>> partitioned nodes to stop multiple nodes becoming >>>>>>>>>>>>>>>>>>>>>>>>>>> coordinators until the >>>>>>>>>>>>>>>>>>>>>>>>>>> network segmentation issue is fixed. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> As an alternative solution, we are thinking of >>>>>>>>>>>>>>>>>>>>>>>>>>> implementing an RDBMS based approach to elect the >>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator node in the >>>>>>>>>>>>>>>>>>>>>>>>>>> cluster. By doing this we can make sure that even >>>>>>>>>>>>>>>>>>>>>>>>>>> during a network >>>>>>>>>>>>>>>>>>>>>>>>>>> segmentation only one node will be elected as the >>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator node since >>>>>>>>>>>>>>>>>>>>>>>>>>> the election is happening through the database. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> The algorithm will use a polling mechanism to >>>>>>>>>>>>>>>>>>>>>>>>>>> check the validity of the nodes. To make the >>>>>>>>>>>>>>>>>>>>>>>>>>> election algorithm scalable, >>>>>>>>>>>>>>>>>>>>>>>>>>> only the coordinator node will be checking status >>>>>>>>>>>>>>>>>>>>>>>>>>> of all the nodes in the >>>>>>>>>>>>>>>>>>>>>>>>>>> cluster and it will inform other nodes through >>>>>>>>>>>>>>>>>>>>>>>>>>> database when a member is >>>>>>>>>>>>>>>>>>>>>>>>>>> added/left. The nodes will be only checking for the >>>>>>>>>>>>>>>>>>>>>>>>>>> status of the >>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator node. When a node detect that >>>>>>>>>>>>>>>>>>>>>>>>>>> coordinator is invalid it will go >>>>>>>>>>>>>>>>>>>>>>>>>>> for a election to elect a new coordinator. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> We are currently working on a POC to test how >>>>>>>>>>>>>>>>>>>>>>>>>>> this works with MB's slot based messaging algorithm. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts? >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> [1] https://wso2.org/jira/browse/MB-1664 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>> Hasitha Aravinda, >>>>>>>>>>>>>>>>>>>>>>>>>> Associate Technical Lead, >>>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>> Mobile : +94 718 210 200 >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>> Nandika Jayawardana >>>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc ; http://wso2.com >>>>>>>>>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Ramith Jayasinghe >>>>>>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> E: [email protected] >>>>>>>>>>>>>>>>>> P: +94 772534930 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/ >>>>>>>>>>>>>>>>>> mailman/listinfo/architecture >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> *Imesh Gunaratne* >>>>>>>>>>>>>>>> Software Architect >>>>>>>>>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>>>>>>>>> lean. enterprise. middleware >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> *Asitha Nanayakkara* <http://asitha.github.io/> >>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>> WSO2, Inc. <http://wso2.com/> >>>>>>>>>>>>>>> Mob: +94 77 853 0682 >>>>>>>>>>>>>>> [image: https://wso2.com/signature] >>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Malaka Silva >>>>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>>>> M: +94 777 219 791 >>>>>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>>>>> Skype : malaka.sampath.silva >>>>>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 >>>>>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>> https://wso2.com/signature >>>>>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/ >>>>>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/> >>>>>>>>>>>>>> https://store.wso2.com/store/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> Don't make Trees rare, we should keep them with care >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Ramith Jayasinghe >>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>> >>>>>>>>>>>>> E: [email protected] >>>>>>>>>>>>> P: +94 772534930 >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Sajini De SIlva >>>>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>>>>>>> Email: [email protected] >>>>>>>>>>> Blog: http://sajinid.blogspot.com/ >>>>>>>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>>>>>>> >>>>>>>>>>> Phone: +94 712797729 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Sajini De SIlva >>>>>>>>> Senior Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>>>>> Email: [email protected] >>>>>>>>> Blog: http://sajinid.blogspot.com/ >>>>>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>>>>> >>>>>>>>> Phone: +94 712797729 >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Kasun Indrasiri >>>>>>>> Director, Integration Technologies >>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> >>>>>>>> cell: +1 650 450 2293 >>>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Associate Director / Architect >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Anjana Fernando* >>>>> Associate Director / Architect >>>>> WSO2 Inc. | http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>> >>>> >>>> >>>> -- >>>> Asanka Abeyweera >>>> Senior Software Engineer >>>> WSO2 Inc. >>>> >>>> Phone: +94 712228648 >>>> Blog: a5anka.github.io >>>> >>>> <https://wso2.com/signature> >>>> >>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Associate Director / Architect >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> *Anjana Fernando* >> Associate Director / Architect >> WSO2 Inc. | http://wso2.com >> lean . enterprise . middleware >> > > > > -- > Asanka Abeyweera > Senior Software Engineer > WSO2 Inc. > > Phone: +94 712228648 > Blog: a5anka.github.io > > <https://wso2.com/signature> > -- *Anjana Fernando* Associate Director / Architect WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
