OOoooooh - we don't make a distinction between guaranteed and not guaranteed delivery? But we do support guaranteed messaging - right? (sounds like *all messages* are guaranteed regardless of whether they should be or not ??).
If I've got the above correct then do we have a problem with messaging throughput in comparison to others in the market? John Hawkins Director: Solutions Architecture On Mon, Jul 20, 2015 at 10:32 AM, Ramith Jayasinghe <[email protected]> wrote: > Hi, > > Philippe: > yes your suggestions make sense. could you create a jira so we can > consider that in the roadmap. But as a start we will start printing > warnings on the log. > > > @John, > We don't make a distinction given that we don't support > clustered+in-memory mode (at least yet). > Each broker node will store messages in a (relational) database as > oppose to keep routing messages from broker-to-broker. > > Problem we are trying to solve here is a edge scenario which will lead > to a OOM. > and in my opinion, if the subscriber comparatively slow (than others > connecting to same MB node for the same topic) and > requires all messages published it can switch to durable subscription. > > regards > Ramith > > On Mon, Jul 20, 2015 at 1:42 PM, John Hawkins <[email protected]> wrote: > >> Hi Folks, >> >> What distinction are we making between guaranteed and not-guaranteed >> messaging here? >> If a topic or message is marked as guaranteed then sure we shouldn't lose >> it. However, if the topic or message is not marked as guaranteed then we >> are free to "do our best" not to lose it however - there are, literally, no >> guarantees. This would be a perfect example of where, if the message is not >> marked as guaranteed then we can lose the message? That would be my main >> strategy here. >> >> Also - when we say that we persist the message - what db is this in - is >> it a clustered in-memory db? Seems quite slow to me to start persisting >> messages when the user has not asked for a guarantee. I would have thought >> that an in-memory strategy with a spill-to-disk would make more sense ? >> >> >> regards, >> John. >> >> >> On Sunday, July 19, 2015, <[email protected]> wrote: >> >>> Send Architecture mailing list submissions to >>> [email protected] >>> >>> To subscribe or unsubscribe via the World Wide Web, visit >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> or, via email, send a message with subject or body 'help' to >>> [email protected] >>> >>> You can reach the person managing the list at >>> [email protected] >>> >>> When replying, please edit your Subject line so it is more specific >>> than "Re: Contents of Architecture digest..." >>> >>> >>> Today's Topics: >>> >>> 1. Re: WSO2 Message Broker - Slow topic consumers can make >>> broker OOM (Pamod Sylvester) >>> >>> >>> ---------------------------------------------------------------------- >>> >>> Message: 1 >>> Date: Sun, 19 Jul 2015 23:17:01 +0530 >>> From: Pamod Sylvester <[email protected]> >>> To: Ramith Jayasinghe <[email protected]> >>> Cc: architecture <[email protected]> >>> Subject: Re: [Architecture] WSO2 Message Broker - Slow topic consumers >>> can make broker OOM >>> Message-ID: >>> <CABoCC4=5SNKaLz_VeNtqUH133BK= >>> [email protected]> >>> Content-Type: text/plain; charset="utf-8" >>> >>> Yes that sounds good. Also when going with option (2) will it make sense >>> to >>> provide a config to allow users to specify a rate a subscription should >>> be >>> at (i.e min rate a subscription could consume ) >>> >>> So that if a subscription starts to consume a lower rate than the >>> specified >>> a broker could prompt a warning or if feasible remove that subscription >>> off >>> the node. WDYT ? >>> >>> On Sunday, July 19, 2015, Ramith Jayasinghe <[email protected]> wrote: >>> >>> > Hi, >>> > I can propose two solutions/workarounds: >>> > 1) I suppose if a particular subscriber is slow and requires >>> guaranteed >>> > delivery it can subscribe as a durable subscription. >>> > 2) Further, given that we duplicate topic messages per node ( - not >>> per >>> > subscriber) , users can to group subscribers based on rate subscribers >>> can >>> > consume messages. in other words, slow consumers can connect to a one >>> > message broker node , while fast consumers can connect to a another >>> broker >>> > node. >>> > >>> > regards >>> > Ramith >>> > >>> > >>> > >>> > On Sun, Jul 19, 2015 at 9:34 PM, Sriskandarajah Suhothayan < >>> [email protected] >>> > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>> > >>> >> +1, correct we cannot block fast subscribers. >>> >> >>> >> We also need to support guaranteed delivery for all subscribers, in >>> this >>> >> case we can use the DB based approach. During this process why can't >>> we use >>> >> a single table and sequence number pointer for each subscriber? >>> >> >>> >> Suho >>> >> >>> >> >>> >> >>> >> On Sun, Jul 19, 2015 at 7:58 PM, Hasitha Hiranya <[email protected] >>> >> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>> >> >>> >>> Yes. If we shutdown whole publisher channels because of a slow >>> consumer, >>> >>> it is not fair for other topic subscribers (queue/durable topic >>> >>> subscribers). We just corrupting the broker because of a few slow >>> >>> subscribers. >>> >>> >>> >>> >>> >>> On Sun, Jul 19, 2015 at 6:42 PM, Pamod Sylvester <[email protected] >>> >>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>> >>> >>> >>>> Hi Suho, >>> >>>> >>> >>>> We discussed on the possibility of applying back pressure on the >>> >>>> producers side as well. >>> >>>> >>> >>>> AFAIR we couldn't go with that option Since the situation here is >>> that >>> >>>> some of the consumers for a given topic will be fast and some will >>> be slow >>> >>>> and the fast consumers might starve because of one slow consumer if >>> >>>> publisher is throttled. (Hasitha, Team Correct me if i am wrong ). >>> >>>> >>> >>>> Thanks, >>> >>>> Pamod >>> >>>> >>> >>>> On Sunday, July 19, 2015, Sriskandarajah Suhothayan <[email protected] >>> >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>> >>>> >>> >>>>> I personally believe throttling the publisher before going OOM is a >>> >>>>> good option if thats possible. >>> >>>>> >>> >>>>> Regards >>> >>>>> Suho >>> >>>>> >>> >>>>> On Sun, Jul 19, 2015 at 4:30 PM, Hasitha Hiranya < >>> [email protected]> >>> >>>>> wrote: >>> >>>>> >>> >>>>>> We also did a evaluation how ActiveMQ behave on the situation. >>> >>>>>> >>> >>>>>> >>ActiveMQ did not loose any message to slow consumer. >>> >>>>>> >>All messages were received in order. >>> >>>>>> >>> >>>>>> But ActiveMQ went Out of Memory. Final state was that publisher >>> (who >>> >>>>>> publish in bursts) went Out of Memory as server was not >>> accepting any more >>> >>>>>> messages. But the written messages were delivering to the slow >>> subscribers >>> >>>>>> in order. >>> >>>>>> >>> >>>>>> >>> >>>>>> ? >>> >>> >>>>>> Thanks >>> >>>>>> >>> >>>>>> >>> >>>>>> On Sun, Jul 19, 2015 at 4:25 PM, Hasitha Hiranya < >>> [email protected]> >>> >>>>>> wrote: >>> >>>>>> >>> >>>>>>> Hi, >>> >>>>>>> >>> >>>>>>> Topic message subscription practically means to be real time. >>> Most >>> >>>>>>> brokers handle topic messages in-memory whereas WSO2 Message >>> Broker store >>> >>>>>>> each topic message as well to provide clustering support for >>> subscribers >>> >>>>>>> (publish to Node1 and subscribe from Node2). >>> >>>>>>> >>> >>>>>>> Think of a scenario where there are multiple topic subscribers. >>> We >>> >>>>>>> do not duplicate the message for each subscriber, rather do >>> reference >>> >>>>>>> counting. Message is considered as delivered if all topic >>> consumers sent >>> >>>>>>> the ACK to the server. >>> >>>>>>> >>> >>>>>>> Now, when there is a slow topic consumer, even though all other >>> >>>>>>> subscribers ACK the message, as there is no ACK from the slow >>> consumer, we >>> >>>>>>> need to keep the message metadata in memory for reference >>> counting. >>> >>>>>>> >>> >>>>>>> If the situation is kept for millions of messages MB server will >>> go >>> >>>>>>> Out of Memory because of the slow consumer. >>> >>>>>>> >>> >>>>>>> Thus solution to this problem is, >>> >>>>>>> >>> >>>>>>> >> duplicate messages per subscriber and put to the database. >>> This >>> >>>>>>> solution is not going to scale with subscription count. Also >>> will use many >>> >>>>>>> I/O, memory >>> >>>>>>> >>> >>>>>>> >> drop messages in the middle. This is not also a good solution >>> if >>> >>>>>>> data is mission critical. >>> >>>>>>> >>> >>>>>>> >> Give the choice to the user. He can either loose messages or >>> bare >>> >>>>>>> the risk of going server Out of Memory. >>> >>>>>>> >>> >>>>>>> We did a internal discussion and landed on the third option. >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> -- >>> >>>>>>> *Hasitha Abeykoon* >>> >>>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>> >>>>>>> *cell:* *+94 719363063* >>> >>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>> >>>>>>> >>> >>>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> -- >>> >>>>>> *Hasitha Abeykoon* >>> >>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>> >>>>>> *cell:* *+94 719363063* >>> >>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>> >>>>>> >>> >>>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> >>> >>>>> *S. Suhothayan* >>> >>>>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>> >>>>> *WSO2 Inc. *http://wso2.com >>> >>>>> * <http://wso2.com/>* >>> >>>>> lean . enterprise . middleware >>> >>>>> >>> >>>>> >>> >>>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>> >>>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/ >>> >twitter: >>> >>>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | >>> linked-in: >>> >>>>> http://lk.linkedin.com/in/suhothayan < >>> http://lk.linkedin.com/in/suhothayan>* >>> >>>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> *Pamod Sylvester * >>> >>>> >>> >>>> *WSO2 Inc.; http://wso2.com <http://wso2.com>* >>> >>>> cell: +94 77 7779495 >>> >>>> >>> >>>> >>> >>> >>> >>> >>> >>> -- >>> >>> *Hasitha Abeykoon* >>> >>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>> >>> *cell:* *+94 719363063* >>> >>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>> >>> >>> >>> >>> >> >>> >> >>> >> -- >>> >> >>> >> *S. Suhothayan* >>> >> Technical Lead & Team Lead of WSO2 Complex Event Processor >>> >> *WSO2 Inc. *http://wso2.com >>> >> * <http://wso2.com/>* >>> >> lean . enterprise . middleware >>> >> >>> >> >>> >> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>> >> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/ >>> >twitter: >>> >> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | >>> linked-in: >>> >> http://lk.linkedin.com/in/suhothayan < >>> http://lk.linkedin.com/in/suhothayan>* >>> >> >>> > >>> > >>> > >>> > -- >>> > Ramith Jayasinghe >>> > Technical Lead >>> > WSO2 Inc., http://wso2.com >>> > lean.enterprise.middleware >>> > >>> > >>> > >>> >>> -- >>> *Pamod Sylvester * >>> >>> *WSO2 Inc.; http://wso2.com <http://wso2.com>* >>> cell: +94 77 7779495 >>> -------------- next part -------------- >>> An HTML attachment was scrubbed... >>> URL: < >>> http://mail.wso2.org/pipermail/architecture/attachments/20150719/8dd9b6b5/attachment.html >>> > >>> -------------- next part -------------- >>> A non-text attachment was scrubbed... >>> Name: activemq-oom.png >>> Type: image/png >>> Size: 241605 bytes >>> Desc: not available >>> URL: < >>> http://mail.wso2.org/pipermail/architecture/attachments/20150719/8dd9b6b5/attachment.png >>> > >>> >>> ------------------------------ >>> >>> Subject: Digest Footer >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >>> ------------------------------ >>> >>> End of Architecture Digest, Vol 72, Issue 105 >>> ********************************************* >>> >> >> >> -- >> John Hawkins >> Director: Solutions Architecture >> >> >> > > > -- > Ramith Jayasinghe > Technical Lead > WSO2 Inc., http://wso2.com > lean.enterprise.middleware > > E: [email protected] > P: +94 777542851 > >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
