Hi, According to the offline chat with Shammi, we are not going to implement this feature with this design since the global queues are going to be removed from MB.
Thanks, Sajini. On Thu, Jul 24, 2014 at 3:52 PM, Srinath Perera <[email protected]> wrote: > Sajini, I just sent a mail to arch@ "Removing Global Queue from MB" for > feedback. > > if we did that I think we do not need a watermark. Please chat with Shammi > before continue. > > --Srinath > > > On Thu, Jul 24, 2014 at 1:05 PM, Sajini De Silva <[email protected]> wrote: > >> Hi Sumedha, >> >> I did a research on possibility of applying flow control patterns in this >> scenario. We cannot apply general flow control patterns here since this is >> not a general flow controlling problem. I studied about RabbitMQ and Qpid >> flow controlling mechanisms which are similar to this approach. >> >> I had an offline chat with PrabathA about this matter and we decided to >> go with the proposed architecture for the simplicity. >> >> Thank you, >> Sajini. >> >> >> >> On Wed, Jul 23, 2014 at 4:35 PM, Sumedha Rubasinghe <[email protected]> >> wrote: >> >>> On a related note, there are generic flow control patterns to deal with >>> different scenarios. For example there are flow controls to decide what to >>> happen when in-memory store runs out of memory, or a particular queue runs >>> out of space or disk runs out of allocated quota. >>> >>> So it might be a good idea to think of pluggable aspect of this and >>> implement the high/low watermark pattern using that extension. >>> >>> >>> >>> >>> >>> >>> On Wed, Jul 23, 2014 at 2:24 AM, Sajini De Silva <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> This is the proposed design for implementing this feature. >>>> >>>> GlobalQueueWorker(GQW): >>>> >>>> >>>> - Maintain a HashMap of number of messages copied to each node >>>> queue. Key of this HashMap is NodeQueue+queueName. >>>> - Before Copying messages to nodeQueues GQW checks the message >>>> copied count in the Hashmap and copies only if that value is lower than >>>> HighWaterMark(HWM) threshold. >>>> - If the value is equal to HWM level GQW notifies the related >>>> NodeQueue(NQ) that it has reached the HWM level and cease sending >>>> messages further to that NQ. After that GQW selects another subscriber >>>> NQ >>>> from the rest of the nodeQueues randomly. >>>> - If QDW received a notification from NQ that it has reached its >>>> LowWaterMark >>>> (LWM) level, it reset the value related to that NQ in the hashmap. >>>> >>>> QueueDiliveryWorker(QDW): >>>> >>>> - When QDW is notified by the GQW that it has reached the HWM level >>>> it keeps a count on number of messages to be delivered. When it reaches >>>> to >>>> the LWM level it notifies the QDW that it has reached to its LWM level. >>>> >>>> Message notification is done using Hazelcast. >>>> >>>> Any suggestions will be appreciated. >>>> >>>> >>>> Thank you, >>>> >>>> Sajini. >>>> >>>> >>>> On Tue, Jul 22, 2014 at 2:00 PM, Sajini De Silva <[email protected]> >>>> wrote: >>>> >>>>> Adding to architecture@. >>>>> >>>>> >>>>> On Mon, Jul 21, 2014 at 3:09 PM, Sajini De Silva <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm implementing the $subject in MB 3.0.0. Currently in an MB cluster >>>>>> if there is only one subscriber for a particular queue >>>>>> GlobalQueueWorker(GQW) copy all the messages which come to that >>>>>> particular >>>>>> queue, to node queue in the subscriber's MB node. If another subscriber >>>>>> joins from another node after GQW copied its all messages to another node >>>>>> queue there is no way the new subscriber can get any messages even though >>>>>> most of the messages are not delivered to the previous subscriber. >>>>>> >>>>>> The idea here is to introduce two threshold values known as >>>>>> HighWaterMark(HWM) level and LowWaterMark(LWM) level in order to avoid >>>>>> the >>>>>> above scenario. So that GQW copies messages to a node queue only up to >>>>>> HWM >>>>>> even though there are many more messages. If another subscriber joins to >>>>>> the cluster later he will not be starved. When a nodeQueue hits the LWM >>>>>> threshold if there are more messages in the global queue, node queue will >>>>>> be filled up to HWM by the GQW. >>>>>> >>>>>> I'll update the thread as I progress. >>>>>> >>>>>> Thank you, >>>>>> Sajini. >>>>>> >>>>>> -- >>>>>> Sajini De SIlva >>>>>> Software Engineer; WSO2 Inc.; http://wso2.com , >>>>>> Email: [email protected] >>>>>> Blog: http://sajinid.blogspot.com/ >>>>>> Git hub profile: https://github.com/sajinidesilva >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Sajini De SIlva >>>>> Software Engineer; WSO2 Inc.; http://wso2.com , >>>>> Email: [email protected] >>>>> Blog: http://sajinid.blogspot.com/ >>>>> Git hub profile: https://github.com/sajinidesilva >>>>> >>>>> >>>> >>>> >>>> -- >>>> Sajini De SIlva >>>> Software Engineer; WSO2 Inc.; http://wso2.com , >>>> Email: [email protected] >>>> Blog: http://sajinid.blogspot.com/ >>>> Git hub profile: https://github.com/sajinidesilva >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> /sumedha >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Sajini De SIlva >> Software Engineer; WSO2 Inc.; http://wso2.com , >> Email: [email protected] >> Blog: http://sajinid.blogspot.com/ >> Git hub profile: https://github.com/sajinidesilva >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > ============================ > Srinath Perera, Ph.D. > http://people.apache.org/~hemapani/ > http://srinathsview.blogspot.com/ > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Sajini De SIlva Software Engineer; WSO2 Inc.; http://wso2.com , Email: [email protected] Blog: http://sajinid.blogspot.com/ Git hub profile: https://github.com/sajinidesilva
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
