Hi Maninda, On Fri, Aug 5, 2016 at 2:28 PM, Maninda Edirisooriya <[email protected]> wrote:
> @Sajini, > > But the number of slots are proportional to the number of messages pass > through the MB which needs to be handled by the coordinator. That is what I > meant by "information related to meta data of messages pass through a > single coordinator". Ideally after the senders and receivers are subscribed > to the cluster, coordinator should have nothing to do until they are > removed or changed. > Even though it is possible to have multiple coordinators after having en effort (Lock the database for a whole transaction or the work load distribution as described by Ramith) , coordinator may have different work to do other than subscriber adding and removing. As I said earlier our MB message distribution system is based on slot architecture and slots will managed by the coordinator. You can read [1] to understand more about slot architecture in MB. [1] http://sajinid.blogspot.com/2015/03/wso2-message-broker-300-slot-based.html Thanks > > @Ramith, > > +1 for multiple coordinators by partitioning the cluster which maintains > the simplicity and correctness of the algorithm than compromising > simplicity with a less important factor like "delivering a good mix of > messages". > > Thanks. > > > *Maninda Edirisooriya* > Senior Software Engineer > > *WSO2, Inc.*lean.enterprise.middleware. > > *Blog* : http://maninda.blogspot.com/ > *E-mail* : [email protected] > *Skype* : @manindae > *Twitter* : @maninda > > On Fri, Aug 5, 2016 at 2:05 PM, Ramith Jayasinghe <[email protected]> wrote: > >> @Imesh, >> We can prove that doing leader election using a lib (where we maintain >> cluster state in another place, a.k.a DB) will not solve our original >> problem (this also relates to our past experience with both the zookeeper >> and hazelcast). >> We can make this implementation a common component if other products >> have a use of it. BPS might be able to use it since their data is also in >> the database. >> >> @Malaka: >> VFS scenario can't be solved by relying on this implementation. why? you >> can have the access to DB but not VFS resources/file (and vice versa). this >> is the same point we explained before. >> in Ntask implementation, if tasks are stored in the database then using >> this implementation makes sense. >> >> >> @Akila, >> implementing (distributed) a queue algorithm is non-trivial. Having one >> coordinator (single source of truth) keeps things simple hence it's a >> conscious design decision we agreed during the initial stages. However, >> possible extension to this scheme is to have multiple coordinators ( each >> responsible for coordinating a subset of queues in the cluster), that will >> be some what similar to kafka. >> Even if its preferable to have no coordinator at-all, (to decide how >> messages are disseminated in the cluster) that will make us give up >> desired behaviour such as delivering a good mix of messages (from different >> publishers) to consumers in a cluster. having said this, we have an ongoing >> research on how to improve the algorithm and we like to try out both these >> approaches. >> >> >> On Fri, Aug 5, 2016 at 1:31 PM, Malaka Silva <[email protected]> wrote: >> >>> The same issue with Hazelcast can be experienced with ESB inbounds >>> (running on top of NTASK) and VFS distribution locks. >>> >>> The idea of only single worker works at a given time breaks if there is >>> a Hazelcast heart beat fails. This will make two workers to work in >>> parallel. >>> >>> Also with distributed locking there is no guarantee that file is only >>> process only by one worker. >>> >>> So in the case of network fail >>> with DB >>> make sense to stop processing until it's recovered. >>> Also making this component generic ESB can reuse. >>> >>> On Fri, Aug 5, 2016 at 9:21 AM, Asitha Nanayakkara <[email protected]> >>> wrote: >>> >>>> Hi Imesh, >>>> >>>> On Fri, Aug 5, 2016 at 7:33 AM, Imesh Gunaratne <[email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Fri, Aug 5, 2016 at 7:31 AM, Imesh Gunaratne <[email protected]> >>>>> wrote: >>>>>> >>>>>> >>>>>> You can see here [3] how K8S has implemented leader election feature >>>>>> for the products deployed on top of that to utilize. >>>>>> >>>>> >>>>> Correction: Please refer [4]. >>>>> >>>>> >>>>>> >>>>>> >>>>>>> On Thu, Aug 4, 2016 at 7:27 PM, Asanka Abeyweera <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Imesh, >>>>>>>> >>>>>>>> We are not implementing this to overcome a limitation in the >>>>>>>> coordination algorithm available in the Hazlecast. We are implementing >>>>>>>> this >>>>>>>> since we need an RDBMS based coordination algorithm (not a network >>>>>>>> based >>>>>>>> algorithm). >>>>>>>> >>>>>>> >>>>>> Are you saying that database connections do not use the same network >>>>>> used by Hazelcast? >>>>>> >>>>>> >>>>>> >>>>>>> The reason is, a network based election algorithm will always elect >>>>>>>> multiple leaders when the network is partitioned. But if we use a RDBMS >>>>>>>> based algorithm this will not happen. >>>>>>>> >>>>>>> >>>>>> I do not think your argument is correct. If there is a problem with >>>>>> the network, it may apply to both Hazelcast based solution and database >>>>>> based solution. >>>>>> >>>>> >>>> Yes, if the same network interface is used network partion will cause >>>> all types of connections to be partitioned. But user can use multiple >>>> network interfaces for database, Hazelcast and thrift. >>>> >>>> Following is the scenario we are trying to solve in MB. >>>> >>>> In MB all the details related to messages, subscriptions, queues, >>>> topics etc are stored in database. And we operate depending on that >>>> information. If the MB node can't connect to the database that means the >>>> node is ineffective in the cluster until it can make a database connection. >>>> >>>> We have seen instances where Hazelcast cluster get partitioned for some >>>> time period in networks, Reasons were, >>>> >>>> 1. Due to heavy load Hazelcast couldn't process or send (some times >>>> both) hearbeats, hence a network partition for Hazelcast cluster >>>> 2. An actual network partition of Hazelcast cluster >>>> >>>> In both scenarios the database connection was working. In that case we >>>> get two coordinators elected through Hazelcast and working on the same >>>> database to deliver the messages. this leads to inconsistencies in the >>>> cluster behavior (for instances duplicate message delivery, corrupred >>>> subscription states etc) . >>>> >>>> Since the point of interest for MB is the database, we decided to do >>>> the coordinator election through database as well. If the node can't >>>> connect to the database, then the MB won't operate anyway. >>>> >>>> Regards, >>>> Asitha >>>> >>>> >>>>>> [4] http://blog.kubernetes.io/2016/01/simple-leader-election >>>>>> -with-Kubernetes.html >>>>>> >>>>>> Thanks >>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Thu, Aug 4, 2016 at 7:16 PM, Imesh Gunaratne <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Asanka, >>>>>>>>> >>>>>>>>> Do we really need to implement a leader election algorithm on our >>>>>>>>> own? AFAIU this is a complex problem which has been already solved by >>>>>>>>> several algorithms [1]. IMO it would be better to go ahead with an >>>>>>>>> existing >>>>>>>>> well established implementation on etcd [1] or Consul [2]. >>>>>>>>> >>>>>>>>> Those provide HTTP APIs for clients to make leader election calls. >>>>>>>>> [3] is a client library written in Node.js for etcd based leader >>>>>>>>> election. >>>>>>>>> >>>>>>>>> [1] https://www.projectcalico.org/using-etcd-for-elections >>>>>>>>> [2] https://www.consul.io/docs/guides/leader-election.html >>>>>>>>> [3] https://www.npmjs.com/package/etcd-leader >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> On Wed, Aug 3, 2016 at 5:12 PM, Asanka Abeyweera < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Maninda, >>>>>>>>>> >>>>>>>>>> Since we are using RDBMS to poll the node status, the cluster >>>>>>>>>> will not end up in situation 1,2 or 3. With this approach we >>>>>>>>>> consider a >>>>>>>>>> node unreachable when it cannot access the database. Therefore an >>>>>>>>>> unreachable node can never be the leader. >>>>>>>>>> >>>>>>>>>> As you have mentioned, we are currently using the RDBMS as an >>>>>>>>>> atomic global variable to create the coordinator entry. >>>>>>>>>> >>>>>>>>>> On Tue, Aug 2, 2016 at 5:22 PM, Maninda Edirisooriya < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Asanka, >>>>>>>>>>> >>>>>>>>>>> As I understand the accuracy of electing the leader correctly is >>>>>>>>>>> dependent on the election mechanism with RDBMS because there can be >>>>>>>>>>> edge >>>>>>>>>>> cases like, >>>>>>>>>>> >>>>>>>>>>> 1. Unreachable leader activates during the election process: >>>>>>>>>>> Then who becomes the leader? >>>>>>>>>>> 2. The elected leader becomes unreachable before the election is >>>>>>>>>>> completed: Then will there be a situation where there is no leader? >>>>>>>>>>> 3. A leader and a set of nodes are disconnected from the other >>>>>>>>>>> part of the cluster and while the leader is trying to remove >>>>>>>>>>> unreachable >>>>>>>>>>> members other part is calling an election to make a leader: Who >>>>>>>>>>> will win? >>>>>>>>>>> >>>>>>>>>>> RDBMS based election algorithm should handle such cases without >>>>>>>>>>> bringing the cluster to an inconsistent state or dead lock in all >>>>>>>>>>> concurrent cases. If all these kind of cases cannot be handled >>>>>>>>>>> isn't it >>>>>>>>>>> better to keep the current hazelcast clustering and use the RDBMS >>>>>>>>>>> only to >>>>>>>>>>> handle the split brain scenario? In other words when a new >>>>>>>>>>> hazelcast leader >>>>>>>>>>> is elected it should be updated in the RDBMS. If another split >>>>>>>>>>> party has >>>>>>>>>>> already elected a leader, the node who is going to write it to >>>>>>>>>>> RDBMS should >>>>>>>>>>> avoid updating it. Simply, the RDBMS can be used as an atomic global >>>>>>>>>>> variable to keep the leader name by modifying the hazelcast >>>>>>>>>>> clustering. >>>>>>>>>>> WDYT? >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>> Senior Software Engineer >>>>>>>>>>> >>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>> >>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>> >>>>>>>>>>> On Thu, Jul 28, 2016 at 4:38 PM, Asanka Abeyweera < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Akila, >>>>>>>>>>>> >>>>>>>>>>>> Let me explain the issue in a different way. Let's assume the >>>>>>>>>>>> MB nodes are using two different network interfaces for Hazelcast >>>>>>>>>>>> communication and database communication. With such a >>>>>>>>>>>> configuration, there >>>>>>>>>>>> can be failures only in the network interface used for Hazelcast >>>>>>>>>>>> communication in some nodes. When this happens, there will be two >>>>>>>>>>>> or more >>>>>>>>>>>> Hazelcast clusters due to the network segmentation, and as a >>>>>>>>>>>> result there >>>>>>>>>>>> will be multiple coordinators. Since every node still have access >>>>>>>>>>>> to the >>>>>>>>>>>> database, multiple coordinators can affect the correctness of the >>>>>>>>>>>> data >>>>>>>>>>>> stored in the DB. But if we used a RDBMS based approach we won't >>>>>>>>>>>> have >>>>>>>>>>>> multiple coordinators due to a network partition in Hazelcast. >>>>>>>>>>>> This is one >>>>>>>>>>>> advantage we get from this approach. >>>>>>>>>>>> >>>>>>>>>>>> Even when we use Zookeeper or RAFT the same issue will be there >>>>>>>>>>>> since we are using different interfaces for Hazelcast >>>>>>>>>>>> communication and DB >>>>>>>>>>>> communication. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jul 28, 2016 at 2:56 PM, Akila Ravihansa Perera < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> What's the advantage of using RDBMS (even as an alternative) >>>>>>>>>>>>> to implement a leader/coordinator election? If the network >>>>>>>>>>>>> connection to DB >>>>>>>>>>>>> fails then this will be a single point of failure. I don't think >>>>>>>>>>>>> we can >>>>>>>>>>>>> scale RDBMS instances and expect the election algorithm to work. >>>>>>>>>>>>> That would >>>>>>>>>>>>> be reducing this problem to another problem (electing coordinator >>>>>>>>>>>>> RDBMS >>>>>>>>>>>>> instance). >>>>>>>>>>>>> >>>>>>>>>>>>> IMHO it would be better to look at Zookeeper Atomic Broadcast >>>>>>>>>>>>> (ZAB) [1] or RAFT leader election [2] algorithms which have >>>>>>>>>>>>> already proven >>>>>>>>>>>>> results. >>>>>>>>>>>>> >>>>>>>>>>>>> [1] https://cwiki.apache.org/confluence/display/ZOOKEEPER/Za >>>>>>>>>>>>> b1.0 >>>>>>>>>>>>> [2] http://libraft.io/ >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:42 PM, Nandika Jayawardana < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> +1 to make it a common component . We have the clustering >>>>>>>>>>>>>> implementation for BPEL component based on hazelcast. If the >>>>>>>>>>>>>> coordination >>>>>>>>>>>>>> is available at RDBMS level, we can remove hazelcast dependancy. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards >>>>>>>>>>>>>> Nandika >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 1:28 PM, Hasitha Aravinda < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can we make it a common component, which is not hard coupled >>>>>>>>>>>>>>> with MB. BPS has the same requirement. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Hasitha. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Jul 28, 2016 at 9:47 AM, Asanka Abeyweera < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In MB, we have used a coordinator based approach to manage >>>>>>>>>>>>>>>> distributed messaging algorithm in the cluster. Currently >>>>>>>>>>>>>>>> Hazelcast is used >>>>>>>>>>>>>>>> to elect the coordinator. But one issue we faced with >>>>>>>>>>>>>>>> Hazelcast is, during >>>>>>>>>>>>>>>> a network segmentation (split brain), Hazelcast can elect two >>>>>>>>>>>>>>>> or more >>>>>>>>>>>>>>>> coordinators in the cluster. This affects the correctness of >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> distributed messaging algorithm since there are some tables in >>>>>>>>>>>>>>>> the database >>>>>>>>>>>>>>>> that should only be edited by a single node (i.e. coordinator). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As a solution to this problem we have implemented minimum >>>>>>>>>>>>>>>> node count based approach [1] to deactivate set of partitioned >>>>>>>>>>>>>>>> nodes to >>>>>>>>>>>>>>>> stop multiple nodes becoming coordinators until the network >>>>>>>>>>>>>>>> segmentation >>>>>>>>>>>>>>>> issue is fixed. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As an alternative solution, we are thinking of implementing >>>>>>>>>>>>>>>> an RDBMS based approach to elect the coordinator node in the >>>>>>>>>>>>>>>> cluster. By >>>>>>>>>>>>>>>> doing this we can make sure that even during a network >>>>>>>>>>>>>>>> segmentation only >>>>>>>>>>>>>>>> one node will be elected as the coordinator node since the >>>>>>>>>>>>>>>> election is >>>>>>>>>>>>>>>> happening through the database. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The algorithm will use a polling mechanism to check the >>>>>>>>>>>>>>>> validity of the nodes. To make the election algorithm >>>>>>>>>>>>>>>> scalable, only the >>>>>>>>>>>>>>>> coordinator node will be checking status of all the nodes in >>>>>>>>>>>>>>>> the cluster >>>>>>>>>>>>>>>> and it will inform other nodes through database when a member >>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>> added/left. The nodes will be only checking for the status of >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> coordinator node. When a node detect that coordinator is >>>>>>>>>>>>>>>> invalid it will go >>>>>>>>>>>>>>>> for a election to elect a new coordinator. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We are currently working on a POC to test how this works >>>>>>>>>>>>>>>> with MB's slot based messaging algorithm. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> thoughts? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] https://wso2.org/jira/browse/MB-1664 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Hasitha Aravinda, >>>>>>>>>>>>>>> Associate Technical Lead, >>>>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>>>> Mobile : +94 718 210 200 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Nandika Jayawardana >>>>>>>>>>>>>> WSO2 Inc ; http://wso2.com >>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>>>>> >>>>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Asanka Abeyweera >>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>> >>>>>>>>>>>> Phone: +94 712228648 >>>>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>>>> >>>>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Asanka Abeyweera >>>>>>>>>> Senior Software Engineer >>>>>>>>>> WSO2 Inc. >>>>>>>>>> >>>>>>>>>> Phone: +94 712228648 >>>>>>>>>> Blog: a5anka.github.io >>>>>>>>>> >>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Imesh Gunaratne* >>>>>>>>> Software Architect >>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>> lean. enterprise. middleware >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Asanka Abeyweera >>>>>>>> Senior Software Engineer >>>>>>>> WSO2 Inc. >>>>>>>> >>>>>>>> Phone: +94 712228648 >>>>>>>> Blog: a5anka.github.io >>>>>>>> >>>>>>>> <https://wso2.com/signature> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Ramith Jayasinghe >>>>>>> Technical Lead >>>>>>> WSO2 Inc., http://wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> >>>>>>> E: [email protected] >>>>>>> P: +94 772534930 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Imesh Gunaratne* >>>>>> Software Architect >>>>>> WSO2 Inc: http://wso2.com >>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>> lean. enterprise. middleware >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Imesh Gunaratne* >>>>> Software Architect >>>>> WSO2 Inc: http://wso2.com >>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>> W: https://medium.com/@imesh TW: @imesh >>>>> lean. enterprise. middleware >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Asitha Nanayakkara* <http://asitha.github.io/> >>>> Senior Software Engineer >>>> WSO2, Inc. <http://wso2.com/> >>>> Mob: +94 77 853 0682 >>>> [image: https://wso2.com/signature] <https://wso2.com/signature> >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> >>> Best Regards, >>> >>> Malaka Silva >>> Senior Technical Lead >>> M: +94 777 219 791 >>> Tel : 94 11 214 5345 >>> Fax :94 11 2145300 >>> Skype : malaka.sampath.silva >>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 >>> Blog : http://mrmalakasilva.blogspot.com/ >>> >>> WSO2, Inc. >>> lean . enterprise . middleware >>> https://wso2.com/signature >>> http://www.wso2.com/about/team/malaka-silva/ >>> <http://wso2.com/about/team/malaka-silva/> >>> https://store.wso2.com/store/ >>> >>> Don't make Trees rare, we should keep them with care >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Ramith Jayasinghe >> Technical Lead >> WSO2 Inc., http://wso2.com >> lean.enterprise.middleware >> >> E: [email protected] >> P: +94 772534930 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > -- Sajini De SIlva Senior Software Engineer; WSO2 Inc.; http://wso2.com , Email: [email protected] Blog: http://sajinid.blogspot.com/ Git hub profile: https://github.com/sajinidesilva Phone: +94 712797729
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
