Hi Sumedha, I did a research on possibility of applying flow control patterns in this scenario. We cannot apply general flow control patterns here since this is not a general flow controlling problem. I studied about RabbitMQ and Qpid flow controlling mechanisms which are similar to this approach.
I had an offline chat with PrabathA about this matter and we decided to go with the proposed architecture for the simplicity. Thank you, Sajini. On Wed, Jul 23, 2014 at 4:35 PM, Sumedha Rubasinghe <[email protected]> wrote: > On a related note, there are generic flow control patterns to deal with > different scenarios. For example there are flow controls to decide what to > happen when in-memory store runs out of memory, or a particular queue runs > out of space or disk runs out of allocated quota. > > So it might be a good idea to think of pluggable aspect of this and > implement the high/low watermark pattern using that extension. > > > > > > > On Wed, Jul 23, 2014 at 2:24 AM, Sajini De Silva <[email protected]> wrote: > >> Hi, >> >> This is the proposed design for implementing this feature. >> >> GlobalQueueWorker(GQW): >> >> >> - Maintain a HashMap of number of messages copied to each node queue. >> Key of this HashMap is NodeQueue+queueName. >> - Before Copying messages to nodeQueues GQW checks the message copied >> count in the Hashmap and copies only if that value is lower than >> HighWaterMark(HWM) threshold. >> - If the value is equal to HWM level GQW notifies the related >> NodeQueue(NQ) that it has reached the HWM level and cease sending >> messages further to that NQ. After that GQW selects another subscriber NQ >> from the rest of the nodeQueues randomly. >> - If QDW received a notification from NQ that it has reached its >> LowWaterMark >> (LWM) level, it reset the value related to that NQ in the hashmap. >> >> QueueDiliveryWorker(QDW): >> >> - When QDW is notified by the GQW that it has reached the HWM level >> it keeps a count on number of messages to be delivered. When it reaches to >> the LWM level it notifies the QDW that it has reached to its LWM level. >> >> Message notification is done using Hazelcast. >> >> Any suggestions will be appreciated. >> >> >> Thank you, >> >> Sajini. >> >> >> On Tue, Jul 22, 2014 at 2:00 PM, Sajini De Silva <[email protected]> wrote: >> >>> Adding to architecture@. >>> >>> >>> On Mon, Jul 21, 2014 at 3:09 PM, Sajini De Silva <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> I'm implementing the $subject in MB 3.0.0. Currently in an MB cluster >>>> if there is only one subscriber for a particular queue >>>> GlobalQueueWorker(GQW) copy all the messages which come to that particular >>>> queue, to node queue in the subscriber's MB node. If another subscriber >>>> joins from another node after GQW copied its all messages to another node >>>> queue there is no way the new subscriber can get any messages even though >>>> most of the messages are not delivered to the previous subscriber. >>>> >>>> The idea here is to introduce two threshold values known as >>>> HighWaterMark(HWM) level and LowWaterMark(LWM) level in order to avoid the >>>> above scenario. So that GQW copies messages to a node queue only up to HWM >>>> even though there are many more messages. If another subscriber joins to >>>> the cluster later he will not be starved. When a nodeQueue hits the LWM >>>> threshold if there are more messages in the global queue, node queue will >>>> be filled up to HWM by the GQW. >>>> >>>> I'll update the thread as I progress. >>>> >>>> Thank you, >>>> Sajini. >>>> >>>> -- >>>> Sajini De SIlva >>>> Software Engineer; WSO2 Inc.; http://wso2.com , >>>> Email: [email protected] >>>> Blog: http://sajinid.blogspot.com/ >>>> Git hub profile: https://github.com/sajinidesilva >>>> >>>> >>> >>> >>> -- >>> Sajini De SIlva >>> Software Engineer; WSO2 Inc.; http://wso2.com , >>> Email: [email protected] >>> Blog: http://sajinid.blogspot.com/ >>> Git hub profile: https://github.com/sajinidesilva >>> >>> >> >> >> -- >> Sajini De SIlva >> Software Engineer; WSO2 Inc.; http://wso2.com , >> Email: [email protected] >> Blog: http://sajinid.blogspot.com/ >> Git hub profile: https://github.com/sajinidesilva >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > /sumedha > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Sajini De SIlva Software Engineer; WSO2 Inc.; http://wso2.com , Email: [email protected] Blog: http://sajinid.blogspot.com/ Git hub profile: https://github.com/sajinidesilva
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
