According to this architecture, the passive node will only fetch event
states and filter events only once it is active. This model might exhaust
the queue 2 very soon depending on the rate of events. What if we could add
another async task to check the state and filter events even if the node is
*WSO2, Inc.:** wso2.com <http://wso2.com/>*
On Fri, Jul 13, 2018 at 9:01 AM Tishan Dahanayakage <tis...@wso2.com> wrote:
> We can configure a LB in front to handle failover. Actually that is the
> correct solution rather than publishing to both. But I think we need to
> figure out how to handle incoming databridge traffic as it cannot be load
> balanced. Yes we can utilize client side load balancing and directly
> publish to SP nodes. But then we will need to shutdown thrift/binary
> receiving servers of passive node for the failover to happen.
> I have below questions/suggestions.
> 1. What is the expected behavior when queue is full in active node?
> First of all I suppose this is a in-memory queue. This is something we
> should think clearly and implement as even now we are facing quite a number
> of issues where server is stalled because queues are getting filled. When
> queue in active node is full
> - We can't let receiving threads to get block on the queue as it will
> cause server to freeze.
> - At the same time We can't also drop the event if exactly once processing
> or zero event drop is enforced.
> - Also we can't keep adding stuff to queue as it will go OOM.
> So IMO we need a way to persist the queue. In other words we need a proper
> broker queue to solve this issue. If user is fine with event loss then We
> can live with in-memory queue.
> 2. We need a automated purging task to clean queue2 to after every state
> persistance done by active node.
> On Thu, Jul 12, 2018 at 10:41 PM, Anoukh Jayawardena <ano...@wso2.com>
>> Hi Damith,
>> Just a few clarifications on the new architecture.
>> I'm assuming that "queue1" and "queue2" are actually per Source? Which
>> means if we have 3 sources, we'll have 3 queues in active node and 3 queues
>> in passive node? Does that mean there will be 3 threads working in async?
>> Also, the main difference with the current HA architecture is that both
>> nodes will not receive all the events. So at the moment the active node
>> goes down, how does the passive node start receiving events? Will we be
>> handling this or should the cluster be fronted by a separate load balancer?
>> On Thu, Jul 12, 2018 at 8:34 PM Damith Wickramasinghe <dami...@wso2.com>
>>> Hi all,
>>> We are in the process of refactor/improve the existing HA architecture
>>> due to various concerns found.
>>> Below is the high level design came up with. We will provide more
>>> in-depth details as the implementation carries on.
>>> As per above at a given point of time there will only be a one active
>>> node. Passive node will not consume any events. Electing active node will
>>> work as per current architecture via cluster coordination.
>>> A thread will work in the active node to put the data into a
>>> queue(queue1) and same thread will then publish the events to
>>> outside.Reason for having a queue here because to send events
>>> asynchronously to passive node. Here we are going to send events to passive
>>> node via TCP. (This we need to decide) Active node will persist the state
>>> periodically to the database.
>>> When active node goes down via cluster coordination passive node will
>>> become active. When active it will get the state from database(to sync the
>>> state) with latest event timestamp , and filter events from queue 2 (in
>>> order to stop processing already processed events) and send them out. Then
>>> open the ports in the newly active node and start receiving events from
>>> sources. In this architecture also at least one processing will be done.
>>> Nisala please add anything I missed.
>>> Senior Software Engineer
>>> WSO2 Inc.; http://wso2.com
>>> mobile: *+94728671315*
>> *Anoukh Jayawardena*
>> *Software Engineer*
>> *WSO2 Lanka (Private) Limited: http://wso2.com
>> *phone: (+94) 77 99 28932*
> Tishan Dahanayakage
> Associate Technical Lead
> WSO2, Inc.
> Mobile:+94 716481328
> Disclaimer: This communication may contain privileged or other
> confidential information and is intended exclusively for the addressee/s.
> If you are not the intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender indicating
> that fact and delete the copy you received and in addition, you should not
> print, copy, re-transmit, disseminate, or otherwise use the information
> contained in this communication. Internet communications cannot be
> guaranteed to be timely, secure, error or virus-free. The sender does not
> accept liability for any errors or omissions.
> Architecture mailing list
Architecture mailing list