On a related note, there are generic flow control patterns to deal with different scenarios. For example there are flow controls to decide what to happen when in-memory store runs out of memory, or a particular queue runs out of space or disk runs out of allocated quota.
So it might be a good idea to think of pluggable aspect of this and implement the high/low watermark pattern using that extension. On Wed, Jul 23, 2014 at 2:24 AM, Sajini De Silva <[email protected]> wrote: > Hi, > > This is the proposed design for implementing this feature. > > GlobalQueueWorker(GQW): > > > - Maintain a HashMap of number of messages copied to each node queue. > Key of this HashMap is NodeQueue+queueName. > - Before Copying messages to nodeQueues GQW checks the message copied > count in the Hashmap and copies only if that value is lower than > HighWaterMark(HWM) threshold. > - If the value is equal to HWM level GQW notifies the related > NodeQueue(NQ) that it has reached the HWM level and cease sending > messages further to that NQ. After that GQW selects another subscriber NQ > from the rest of the nodeQueues randomly. > - If QDW received a notification from NQ that it has reached its > LowWaterMark > (LWM) level, it reset the value related to that NQ in the hashmap. > > QueueDiliveryWorker(QDW): > > - When QDW is notified by the GQW that it has reached the HWM level it > keeps a count on number of messages to be delivered. When it reaches to the > LWM level it notifies the QDW that it has reached to its LWM level. > > Message notification is done using Hazelcast. > > Any suggestions will be appreciated. > > > Thank you, > > Sajini. > > > On Tue, Jul 22, 2014 at 2:00 PM, Sajini De Silva <[email protected]> wrote: > >> Adding to architecture@. >> >> >> On Mon, Jul 21, 2014 at 3:09 PM, Sajini De Silva <[email protected]> wrote: >> >>> Hi, >>> >>> I'm implementing the $subject in MB 3.0.0. Currently in an MB cluster if >>> there is only one subscriber for a particular queue GlobalQueueWorker(GQW) >>> copy all the messages which come to that particular queue, to node queue in >>> the subscriber's MB node. If another subscriber joins from another node >>> after GQW copied its all messages to another node queue there is no way the >>> new subscriber can get any messages even though most of the messages are >>> not delivered to the previous subscriber. >>> >>> The idea here is to introduce two threshold values known as >>> HighWaterMark(HWM) level and LowWaterMark(LWM) level in order to avoid the >>> above scenario. So that GQW copies messages to a node queue only up to HWM >>> even though there are many more messages. If another subscriber joins to >>> the cluster later he will not be starved. When a nodeQueue hits the LWM >>> threshold if there are more messages in the global queue, node queue will >>> be filled up to HWM by the GQW. >>> >>> I'll update the thread as I progress. >>> >>> Thank you, >>> Sajini. >>> >>> -- >>> Sajini De SIlva >>> Software Engineer; WSO2 Inc.; http://wso2.com , >>> Email: [email protected] >>> Blog: http://sajinid.blogspot.com/ >>> Git hub profile: https://github.com/sajinidesilva >>> >>> >> >> >> -- >> Sajini De SIlva >> Software Engineer; WSO2 Inc.; http://wso2.com , >> Email: [email protected] >> Blog: http://sajinid.blogspot.com/ >> Git hub profile: https://github.com/sajinidesilva >> >> > > > -- > Sajini De SIlva > Software Engineer; WSO2 Inc.; http://wso2.com , > Email: [email protected] > Blog: http://sajinid.blogspot.com/ > Git hub profile: https://github.com/sajinidesilva > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- /sumedha
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
