@Anoukh, We can configure a LB in front to handle failover. Actually that is the correct solution rather than publishing to both. But I think we need to figure out how to handle incoming databridge traffic as it cannot be load balanced. Yes we can utilize client side load balancing and directly publish to SP nodes. But then we will need to shutdown thrift/binary receiving servers of passive node for the failover to happen.
@Damith, I have below questions/suggestions. 1. What is the expected behavior when queue is full in active node? First of all I suppose this is a in-memory queue. This is something we should think clearly and implement as even now we are facing quite a number of issues where server is stalled because queues are getting filled. When queue in active node is full - We can't let receiving threads to get block on the queue as it will cause server to freeze. - At the same time We can't also drop the event if exactly once processing or zero event drop is enforced. - Also we can't keep adding stuff to queue as it will go OOM. So IMO we need a way to persist the queue. In other words we need a proper broker queue to solve this issue. If user is fine with event loss then We can live with in-memory queue. 2. We need a automated purging task to clean queue2 to after every state persistance done by active node. /Tishan On Thu, Jul 12, 2018 at 10:41 PM, Anoukh Jayawardena <[email protected]> wrote: > Hi Damith, > > Just a few clarifications on the new architecture. > I'm assuming that "queue1" and "queue2" are actually per Source? Which > means if we have 3 sources, we'll have 3 queues in active node and 3 queues > in passive node? Does that mean there will be 3 threads working in async? > Also, the main difference with the current HA architecture is that both > nodes will not receive all the events. So at the moment the active node > goes down, how does the passive node start receiving events? Will we be > handling this or should the cluster be fronted by a separate load balancer? > > Thanks, > Anoukh > > On Thu, Jul 12, 2018 at 8:34 PM Damith Wickramasinghe <[email protected]> > wrote: > >> Hi all, >> >> We are in the process of refactor/improve the existing HA architecture >> due to various concerns found. >> >> Below is the high level design came up with. We will provide more >> in-depth details as the implementation carries on. >> >> >> >> >> As per above at a given point of time there will only be a one active >> node. Passive node will not consume any events. Electing active node will >> work as per current architecture via cluster coordination. >> A thread will work in the active node to put the data into a >> queue(queue1) and same thread will then publish the events to >> outside.Reason for having a queue here because to send events >> asynchronously to passive node. Here we are going to send events to passive >> node via TCP. (This we need to decide) Active node will persist the state >> periodically to the database. >> >> When active node goes down via cluster coordination passive node will >> become active. When active it will get the state from database(to sync the >> state) with latest event timestamp , and filter events from queue 2 (in >> order to stop processing already processed events) and send them out. Then >> open the ports in the newly active node and start receiving events from >> sources. In this architecture also at least one processing will be done. >> >> Nisala please add anything I missed. >> >> Regards, >> Damith >> >> >> >> >> >> >> -- >> Senior Software Engineer >> WSO2 Inc.; http://wso2.com >> <http://www.google.com/url?q=http%3A%2F%2Fwso2.com&sa=D&sntz=1&usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg> >> lean.enterprise.middleware >> >> mobile: *+94728671315* >> >> > > -- > *Anoukh Jayawardena* > *Software Engineer* > > *WSO2 Lanka (Private) Limited: http://wso2.com > <http://wso2.com/>lean.enterprise.middle-ware* > > > *phone: (+94) 77 99 28932* > <https://wso2.com/signature> > -- Tishan Dahanayakage Associate Technical Lead WSO2, Inc. Mobile:+94 716481328 Disclaimer: This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, re-transmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
