Sajini, I just sent a mail to arch@ "Removing Global Queue from MB" for feedback.
if we did that I think we do not need a watermark. Please chat with Shammi before continue. --Srinath On Thu, Jul 24, 2014 at 1:05 PM, Sajini De Silva <[email protected]> wrote: > Hi Sumedha, > > I did a research on possibility of applying flow control patterns in this > scenario. We cannot apply general flow control patterns here since this is > not a general flow controlling problem. I studied about RabbitMQ and Qpid > flow controlling mechanisms which are similar to this approach. > > I had an offline chat with PrabathA about this matter and we decided to > go with the proposed architecture for the simplicity. > > Thank you, > Sajini. > > > > On Wed, Jul 23, 2014 at 4:35 PM, Sumedha Rubasinghe <[email protected]> > wrote: > >> On a related note, there are generic flow control patterns to deal with >> different scenarios. For example there are flow controls to decide what to >> happen when in-memory store runs out of memory, or a particular queue runs >> out of space or disk runs out of allocated quota. >> >> So it might be a good idea to think of pluggable aspect of this and >> implement the high/low watermark pattern using that extension. >> >> >> >> >> >> >> On Wed, Jul 23, 2014 at 2:24 AM, Sajini De Silva <[email protected]> wrote: >> >>> Hi, >>> >>> This is the proposed design for implementing this feature. >>> >>> GlobalQueueWorker(GQW): >>> >>> >>> - Maintain a HashMap of number of messages copied to each node >>> queue. Key of this HashMap is NodeQueue+queueName. >>> - Before Copying messages to nodeQueues GQW checks the message >>> copied count in the Hashmap and copies only if that value is lower than >>> HighWaterMark(HWM) threshold. >>> - If the value is equal to HWM level GQW notifies the related >>> NodeQueue(NQ) that it has reached the HWM level and cease sending >>> messages further to that NQ. After that GQW selects another subscriber NQ >>> from the rest of the nodeQueues randomly. >>> - If QDW received a notification from NQ that it has reached its >>> LowWaterMark >>> (LWM) level, it reset the value related to that NQ in the hashmap. >>> >>> QueueDiliveryWorker(QDW): >>> >>> - When QDW is notified by the GQW that it has reached the HWM level >>> it keeps a count on number of messages to be delivered. When it reaches >>> to >>> the LWM level it notifies the QDW that it has reached to its LWM level. >>> >>> Message notification is done using Hazelcast. >>> >>> Any suggestions will be appreciated. >>> >>> >>> Thank you, >>> >>> Sajini. >>> >>> >>> On Tue, Jul 22, 2014 at 2:00 PM, Sajini De Silva <[email protected]> >>> wrote: >>> >>>> Adding to architecture@. >>>> >>>> >>>> On Mon, Jul 21, 2014 at 3:09 PM, Sajini De Silva <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm implementing the $subject in MB 3.0.0. Currently in an MB cluster >>>>> if there is only one subscriber for a particular queue >>>>> GlobalQueueWorker(GQW) copy all the messages which come to that particular >>>>> queue, to node queue in the subscriber's MB node. If another subscriber >>>>> joins from another node after GQW copied its all messages to another node >>>>> queue there is no way the new subscriber can get any messages even though >>>>> most of the messages are not delivered to the previous subscriber. >>>>> >>>>> The idea here is to introduce two threshold values known as >>>>> HighWaterMark(HWM) level and LowWaterMark(LWM) level in order to avoid the >>>>> above scenario. So that GQW copies messages to a node queue only up to >>>>> HWM >>>>> even though there are many more messages. If another subscriber joins to >>>>> the cluster later he will not be starved. When a nodeQueue hits the LWM >>>>> threshold if there are more messages in the global queue, node queue will >>>>> be filled up to HWM by the GQW. >>>>> >>>>> I'll update the thread as I progress. >>>>> >>>>> Thank you, >>>>> Sajini. >>>>> >>>>> -- >>>>> Sajini De SIlva >>>>> Software Engineer; WSO2 Inc.; http://wso2.com , >>>>> Email: [email protected] >>>>> Blog: http://sajinid.blogspot.com/ >>>>> Git hub profile: https://github.com/sajinidesilva >>>>> >>>>> >>>> >>>> >>>> -- >>>> Sajini De SIlva >>>> Software Engineer; WSO2 Inc.; http://wso2.com , >>>> Email: [email protected] >>>> Blog: http://sajinid.blogspot.com/ >>>> Git hub profile: https://github.com/sajinidesilva >>>> >>>> >>> >>> >>> -- >>> Sajini De SIlva >>> Software Engineer; WSO2 Inc.; http://wso2.com , >>> Email: [email protected] >>> Blog: http://sajinid.blogspot.com/ >>> Git hub profile: https://github.com/sajinidesilva >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> /sumedha >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Sajini De SIlva > Software Engineer; WSO2 Inc.; http://wso2.com , > Email: [email protected] > Blog: http://sajinid.blogspot.com/ > Git hub profile: https://github.com/sajinidesilva > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- ============================ Srinath Perera, Ph.D. http://people.apache.org/~hemapani/ http://srinathsview.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
