Hi Folks, What distinction are we making between guaranteed and not-guaranteed messaging here? If a topic or message is marked as guaranteed then sure we shouldn't lose it. However, if the topic or message is not marked as guaranteed then we are free to "do our best" not to lose it however - there are, literally, no guarantees. This would be a perfect example of where, if the message is not marked as guaranteed then we can lose the message? That would be my main strategy here.
Also - when we say that we persist the message - what db is this in - is it a clustered in-memory db? Seems quite slow to me to start persisting messages when the user has not asked for a guarantee. I would have thought that an in-memory strategy with a spill-to-disk would make more sense ? regards, John. On Sunday, July 19, 2015, <[email protected]> wrote: > Send Architecture mailing list submissions to > [email protected] <javascript:;> > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > or, via email, send a message with subject or body 'help' to > [email protected] <javascript:;> > > You can reach the person managing the list at > [email protected] <javascript:;> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Architecture digest..." > > > Today's Topics: > > 1. Re: WSO2 Message Broker - Slow topic consumers can make > broker OOM (Pamod Sylvester) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 19 Jul 2015 23:17:01 +0530 > From: Pamod Sylvester <[email protected] <javascript:;>> > To: Ramith Jayasinghe <[email protected] <javascript:;>> > Cc: architecture <[email protected] <javascript:;>> > Subject: Re: [Architecture] WSO2 Message Broker - Slow topic consumers > can make broker OOM > Message-ID: > <CABoCC4=5SNKaLz_VeNtqUH133BK= > [email protected] <javascript:;>> > Content-Type: text/plain; charset="utf-8" > > Yes that sounds good. Also when going with option (2) will it make sense to > provide a config to allow users to specify a rate a subscription should be > at (i.e min rate a subscription could consume ) > > So that if a subscription starts to consume a lower rate than the specified > a broker could prompt a warning or if feasible remove that subscription off > the node. WDYT ? > > On Sunday, July 19, 2015, Ramith Jayasinghe <[email protected] > <javascript:;>> wrote: > > > Hi, > > I can propose two solutions/workarounds: > > 1) I suppose if a particular subscriber is slow and requires guaranteed > > delivery it can subscribe as a durable subscription. > > 2) Further, given that we duplicate topic messages per node ( - not per > > subscriber) , users can to group subscribers based on rate subscribers > can > > consume messages. in other words, slow consumers can connect to a one > > message broker node , while fast consumers can connect to a another > broker > > node. > > > > regards > > Ramith > > > > > > > > On Sun, Jul 19, 2015 at 9:34 PM, Sriskandarajah Suhothayan < > [email protected] <javascript:;> > > <javascript:_e(%7B%7D,'cvml','[email protected] <javascript:;>');>> wrote: > > > >> +1, correct we cannot block fast subscribers. > >> > >> We also need to support guaranteed delivery for all subscribers, in this > >> case we can use the DB based approach. During this process why can't we > use > >> a single table and sequence number pointer for each subscriber? > >> > >> Suho > >> > >> > >> > >> On Sun, Jul 19, 2015 at 7:58 PM, Hasitha Hiranya <[email protected] > <javascript:;> > >> <javascript:_e(%7B%7D,'cvml','[email protected] <javascript:;>');>> > wrote: > >> > >>> Yes. If we shutdown whole publisher channels because of a slow > consumer, > >>> it is not fair for other topic subscribers (queue/durable topic > >>> subscribers). We just corrupting the broker because of a few slow > >>> subscribers. > >>> > >>> > >>> On Sun, Jul 19, 2015 at 6:42 PM, Pamod Sylvester <[email protected] > <javascript:;> > >>> <javascript:_e(%7B%7D,'cvml','[email protected] <javascript:;>');>> > wrote: > >>> > >>>> Hi Suho, > >>>> > >>>> We discussed on the possibility of applying back pressure on the > >>>> producers side as well. > >>>> > >>>> AFAIR we couldn't go with that option Since the situation here is that > >>>> some of the consumers for a given topic will be fast and some will be > slow > >>>> and the fast consumers might starve because of one slow consumer if > >>>> publisher is throttled. (Hasitha, Team Correct me if i am wrong ). > >>>> > >>>> Thanks, > >>>> Pamod > >>>> > >>>> On Sunday, July 19, 2015, Sriskandarajah Suhothayan <[email protected] > <javascript:;> > >>>> <javascript:_e(%7B%7D,'cvml','[email protected] <javascript:;>');>> > wrote: > >>>> > >>>>> I personally believe throttling the publisher before going OOM is a > >>>>> good option if thats possible. > >>>>> > >>>>> Regards > >>>>> Suho > >>>>> > >>>>> On Sun, Jul 19, 2015 at 4:30 PM, Hasitha Hiranya <[email protected] > <javascript:;>> > >>>>> wrote: > >>>>> > >>>>>> We also did a evaluation how ActiveMQ behave on the situation. > >>>>>> > >>>>>> >>ActiveMQ did not loose any message to slow consumer. > >>>>>> >>All messages were received in order. > >>>>>> > >>>>>> But ActiveMQ went Out of Memory. Final state was that publisher (who > >>>>>> publish in bursts) went Out of Memory as server was not accepting > any more > >>>>>> messages. But the written messages were delivering to the slow > subscribers > >>>>>> in order. > >>>>>> > >>>>>> > >>>>>> ? > >>>>>> Thanks > >>>>>> > >>>>>> > >>>>>> On Sun, Jul 19, 2015 at 4:25 PM, Hasitha Hiranya <[email protected] > <javascript:;>> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> Topic message subscription practically means to be real time. Most > >>>>>>> brokers handle topic messages in-memory whereas WSO2 Message > Broker store > >>>>>>> each topic message as well to provide clustering support for > subscribers > >>>>>>> (publish to Node1 and subscribe from Node2). > >>>>>>> > >>>>>>> Think of a scenario where there are multiple topic subscribers. We > >>>>>>> do not duplicate the message for each subscriber, rather do > reference > >>>>>>> counting. Message is considered as delivered if all topic > consumers sent > >>>>>>> the ACK to the server. > >>>>>>> > >>>>>>> Now, when there is a slow topic consumer, even though all other > >>>>>>> subscribers ACK the message, as there is no ACK from the slow > consumer, we > >>>>>>> need to keep the message metadata in memory for reference counting. > >>>>>>> > >>>>>>> If the situation is kept for millions of messages MB server will go > >>>>>>> Out of Memory because of the slow consumer. > >>>>>>> > >>>>>>> Thus solution to this problem is, > >>>>>>> > >>>>>>> >> duplicate messages per subscriber and put to the database. This > >>>>>>> solution is not going to scale with subscription count. Also will > use many > >>>>>>> I/O, memory > >>>>>>> > >>>>>>> >> drop messages in the middle. This is not also a good solution if > >>>>>>> data is mission critical. > >>>>>>> > >>>>>>> >> Give the choice to the user. He can either loose messages or > bare > >>>>>>> the risk of going server Out of Memory. > >>>>>>> > >>>>>>> We did a internal discussion and landed on the third option. > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> *Hasitha Abeykoon* > >>>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com > >>>>>>> *cell:* *+94 719363063* > >>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> *Hasitha Abeykoon* > >>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com > >>>>>> *cell:* *+94 719363063* > >>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> > >>>>> *S. Suhothayan* > >>>>> Technical Lead & Team Lead of WSO2 Complex Event Processor > >>>>> *WSO2 Inc. *http://wso2.com > >>>>> * <http://wso2.com/>* > >>>>> lean . enterprise . middleware > >>>>> > >>>>> > >>>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: > >>>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/ > >twitter: > >>>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | > linked-in: > >>>>> http://lk.linkedin.com/in/suhothayan < > http://lk.linkedin.com/in/suhothayan>* > >>>>> > >>>> > >>>> > >>>> -- > >>>> *Pamod Sylvester * > >>>> > >>>> *WSO2 Inc.; http://wso2.com <http://wso2.com>* > >>>> cell: +94 77 7779495 > >>>> > >>>> > >>> > >>> > >>> -- > >>> *Hasitha Abeykoon* > >>> Senior Software Engineer; WSO2, Inc.; http://wso2.com > >>> *cell:* *+94 719363063* > >>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> > >>> > >>> > >> > >> > >> -- > >> > >> *S. Suhothayan* > >> Technical Lead & Team Lead of WSO2 Complex Event Processor > >> *WSO2 Inc. *http://wso2.com > >> * <http://wso2.com/>* > >> lean . enterprise . middleware > >> > >> > >> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: > >> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/ > >twitter: > >> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | > linked-in: > >> http://lk.linkedin.com/in/suhothayan < > http://lk.linkedin.com/in/suhothayan>* > >> > > > > > > > > -- > > Ramith Jayasinghe > > Technical Lead > > WSO2 Inc., http://wso2.com > > lean.enterprise.middleware > > > > > > > > -- > *Pamod Sylvester * > > *WSO2 Inc.; http://wso2.com <http://wso2.com>* > cell: +94 77 7779495 > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.wso2.org/pipermail/architecture/attachments/20150719/8dd9b6b5/attachment.html > > > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: activemq-oom.png > Type: image/png > Size: 241605 bytes > Desc: not available > URL: < > http://mail.wso2.org/pipermail/architecture/attachments/20150719/8dd9b6b5/attachment.png > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Architecture mailing list > [email protected] <javascript:;> > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > > ------------------------------ > > End of Architecture Digest, Vol 72, Issue 105 > ********************************************* > -- John Hawkins Director: Solutions Architecture
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
