Hi Folks, As a newbie I don't have any history of what you are trying to achieve. Therefore, I realised that I didn't really understand what QOS you were going for and what was the mechanisms that you were using to achieve that QOS. Do you have a design doc you can share please?
many thanks, John. John Hawkins Director: Solutions Architecture On Mon, Jul 20, 2015 at 10:43 AM, John Hawkins <[email protected]> wrote: > OOoooooh - we don't make a distinction between guaranteed and not > guaranteed delivery? But we do support guaranteed messaging - right? > (sounds like *all messages* are guaranteed regardless of whether they > should be or not ??). > > If I've got the above correct then do we have a problem with messaging > throughput in comparison to others in the market? > > John Hawkins > Director: Solutions Architecture > > > On Mon, Jul 20, 2015 at 10:32 AM, Ramith Jayasinghe <[email protected]> > wrote: > >> Hi, >> >> Philippe: >> yes your suggestions make sense. could you create a jira so we can >> consider that in the roadmap. But as a start we will start printing >> warnings on the log. >> >> >> @John, >> We don't make a distinction given that we don't support >> clustered+in-memory mode (at least yet). >> Each broker node will store messages in a (relational) database as >> oppose to keep routing messages from broker-to-broker. >> >> Problem we are trying to solve here is a edge scenario which will lead >> to a OOM. >> and in my opinion, if the subscriber comparatively slow (than others >> connecting to same MB node for the same topic) and >> requires all messages published it can switch to durable subscription. >> >> regards >> Ramith >> >> On Mon, Jul 20, 2015 at 1:42 PM, John Hawkins <[email protected]> wrote: >> >>> Hi Folks, >>> >>> What distinction are we making between guaranteed and not-guaranteed >>> messaging here? >>> If a topic or message is marked as guaranteed then sure we shouldn't >>> lose it. However, if the topic or message is not marked as guaranteed then >>> we are free to "do our best" not to lose it however - there are, literally, >>> no guarantees. This would be a perfect example of where, if the message is >>> not marked as guaranteed then we can lose the message? That would be my >>> main strategy here. >>> >>> Also - when we say that we persist the message - what db is this in - is >>> it a clustered in-memory db? Seems quite slow to me to start persisting >>> messages when the user has not asked for a guarantee. I would have thought >>> that an in-memory strategy with a spill-to-disk would make more sense ? >>> >>> >>> regards, >>> John. >>> >>> >>> On Sunday, July 19, 2015, <[email protected]> wrote: >>> >>>> Send Architecture mailing list submissions to >>>> [email protected] >>>> >>>> To subscribe or unsubscribe via the World Wide Web, visit >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> or, via email, send a message with subject or body 'help' to >>>> [email protected] >>>> >>>> You can reach the person managing the list at >>>> [email protected] >>>> >>>> When replying, please edit your Subject line so it is more specific >>>> than "Re: Contents of Architecture digest..." >>>> >>>> >>>> Today's Topics: >>>> >>>> 1. Re: WSO2 Message Broker - Slow topic consumers can make >>>> broker OOM (Pamod Sylvester) >>>> >>>> >>>> ---------------------------------------------------------------------- >>>> >>>> Message: 1 >>>> Date: Sun, 19 Jul 2015 23:17:01 +0530 >>>> From: Pamod Sylvester <[email protected]> >>>> To: Ramith Jayasinghe <[email protected]> >>>> Cc: architecture <[email protected]> >>>> Subject: Re: [Architecture] WSO2 Message Broker - Slow topic consumers >>>> can make broker OOM >>>> Message-ID: >>>> <CABoCC4=5SNKaLz_VeNtqUH133BK= >>>> [email protected]> >>>> Content-Type: text/plain; charset="utf-8" >>>> >>>> Yes that sounds good. Also when going with option (2) will it make >>>> sense to >>>> provide a config to allow users to specify a rate a subscription should >>>> be >>>> at (i.e min rate a subscription could consume ) >>>> >>>> So that if a subscription starts to consume a lower rate than the >>>> specified >>>> a broker could prompt a warning or if feasible remove that subscription >>>> off >>>> the node. WDYT ? >>>> >>>> On Sunday, July 19, 2015, Ramith Jayasinghe <[email protected]> wrote: >>>> >>>> > Hi, >>>> > I can propose two solutions/workarounds: >>>> > 1) I suppose if a particular subscriber is slow and requires >>>> guaranteed >>>> > delivery it can subscribe as a durable subscription. >>>> > 2) Further, given that we duplicate topic messages per node ( - not >>>> per >>>> > subscriber) , users can to group subscribers based on rate >>>> subscribers can >>>> > consume messages. in other words, slow consumers can connect to a one >>>> > message broker node , while fast consumers can connect to a another >>>> broker >>>> > node. >>>> > >>>> > regards >>>> > Ramith >>>> > >>>> > >>>> > >>>> > On Sun, Jul 19, 2015 at 9:34 PM, Sriskandarajah Suhothayan < >>>> [email protected] >>>> > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> > >>>> >> +1, correct we cannot block fast subscribers. >>>> >> >>>> >> We also need to support guaranteed delivery for all subscribers, in >>>> this >>>> >> case we can use the DB based approach. During this process why can't >>>> we use >>>> >> a single table and sequence number pointer for each subscriber? >>>> >> >>>> >> Suho >>>> >> >>>> >> >>>> >> >>>> >> On Sun, Jul 19, 2015 at 7:58 PM, Hasitha Hiranya <[email protected] >>>> >> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> >> >>>> >>> Yes. If we shutdown whole publisher channels because of a slow >>>> consumer, >>>> >>> it is not fair for other topic subscribers (queue/durable topic >>>> >>> subscribers). We just corrupting the broker because of a few slow >>>> >>> subscribers. >>>> >>> >>>> >>> >>>> >>> On Sun, Jul 19, 2015 at 6:42 PM, Pamod Sylvester <[email protected] >>>> >>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> >>> >>>> >>>> Hi Suho, >>>> >>>> >>>> >>>> We discussed on the possibility of applying back pressure on the >>>> >>>> producers side as well. >>>> >>>> >>>> >>>> AFAIR we couldn't go with that option Since the situation here is >>>> that >>>> >>>> some of the consumers for a given topic will be fast and some will >>>> be slow >>>> >>>> and the fast consumers might starve because of one slow consumer if >>>> >>>> publisher is throttled. (Hasitha, Team Correct me if i am wrong ). >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Pamod >>>> >>>> >>>> >>>> On Sunday, July 19, 2015, Sriskandarajah Suhothayan <[email protected] >>>> >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> >>>> >>>> >>>>> I personally believe throttling the publisher before going OOM is >>>> a >>>> >>>>> good option if thats possible. >>>> >>>>> >>>> >>>>> Regards >>>> >>>>> Suho >>>> >>>>> >>>> >>>>> On Sun, Jul 19, 2015 at 4:30 PM, Hasitha Hiranya < >>>> [email protected]> >>>> >>>>> wrote: >>>> >>>>> >>>> >>>>>> We also did a evaluation how ActiveMQ behave on the situation. >>>> >>>>>> >>>> >>>>>> >>ActiveMQ did not loose any message to slow consumer. >>>> >>>>>> >>All messages were received in order. >>>> >>>>>> >>>> >>>>>> But ActiveMQ went Out of Memory. Final state was that publisher >>>> (who >>>> >>>>>> publish in bursts) went Out of Memory as server was not >>>> accepting any more >>>> >>>>>> messages. But the written messages were delivering to the slow >>>> subscribers >>>> >>>>>> in order. >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> ? >>>> >>>> >>>>>> Thanks >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> On Sun, Jul 19, 2015 at 4:25 PM, Hasitha Hiranya < >>>> [email protected]> >>>> >>>>>> wrote: >>>> >>>>>> >>>> >>>>>>> Hi, >>>> >>>>>>> >>>> >>>>>>> Topic message subscription practically means to be real time. >>>> Most >>>> >>>>>>> brokers handle topic messages in-memory whereas WSO2 Message >>>> Broker store >>>> >>>>>>> each topic message as well to provide clustering support for >>>> subscribers >>>> >>>>>>> (publish to Node1 and subscribe from Node2). >>>> >>>>>>> >>>> >>>>>>> Think of a scenario where there are multiple topic subscribers. >>>> We >>>> >>>>>>> do not duplicate the message for each subscriber, rather do >>>> reference >>>> >>>>>>> counting. Message is considered as delivered if all topic >>>> consumers sent >>>> >>>>>>> the ACK to the server. >>>> >>>>>>> >>>> >>>>>>> Now, when there is a slow topic consumer, even though all other >>>> >>>>>>> subscribers ACK the message, as there is no ACK from the slow >>>> consumer, we >>>> >>>>>>> need to keep the message metadata in memory for reference >>>> counting. >>>> >>>>>>> >>>> >>>>>>> If the situation is kept for millions of messages MB server >>>> will go >>>> >>>>>>> Out of Memory because of the slow consumer. >>>> >>>>>>> >>>> >>>>>>> Thus solution to this problem is, >>>> >>>>>>> >>>> >>>>>>> >> duplicate messages per subscriber and put to the database. >>>> This >>>> >>>>>>> solution is not going to scale with subscription count. Also >>>> will use many >>>> >>>>>>> I/O, memory >>>> >>>>>>> >>>> >>>>>>> >> drop messages in the middle. This is not also a good >>>> solution if >>>> >>>>>>> data is mission critical. >>>> >>>>>>> >>>> >>>>>>> >> Give the choice to the user. He can either loose messages or >>>> bare >>>> >>>>>>> the risk of going server Out of Memory. >>>> >>>>>>> >>>> >>>>>>> We did a internal discussion and landed on the third option. >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> -- >>>> >>>>>>> *Hasitha Abeykoon* >>>> >>>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>> >>>>>>> *cell:* *+94 719363063* >>>> >>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> -- >>>> >>>>>> *Hasitha Abeykoon* >>>> >>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>> >>>>>> *cell:* *+94 719363063* >>>> >>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>> >>>>>> >>>> >>>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> -- >>>> >>>>> >>>> >>>>> *S. Suhothayan* >>>> >>>>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>>> >>>>> *WSO2 Inc. *http://wso2.com >>>> >>>>> * <http://wso2.com/>* >>>> >>>>> lean . enterprise . middleware >>>> >>>>> >>>> >>>>> >>>> >>>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>> >>>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/ >>>> >twitter: >>>> >>>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | >>>> linked-in: >>>> >>>>> http://lk.linkedin.com/in/suhothayan < >>>> http://lk.linkedin.com/in/suhothayan>* >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> *Pamod Sylvester * >>>> >>>> >>>> >>>> *WSO2 Inc.; http://wso2.com <http://wso2.com>* >>>> >>>> cell: +94 77 7779495 >>>> >>>> >>>> >>>> >>>> >>> >>>> >>> >>>> >>> -- >>>> >>> *Hasitha Abeykoon* >>>> >>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>> >>> *cell:* *+94 719363063* >>>> >>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>> >>> >>>> >>> >>>> >> >>>> >> >>>> >> -- >>>> >> >>>> >> *S. Suhothayan* >>>> >> Technical Lead & Team Lead of WSO2 Complex Event Processor >>>> >> *WSO2 Inc. *http://wso2.com >>>> >> * <http://wso2.com/>* >>>> >> lean . enterprise . middleware >>>> >> >>>> >> >>>> >> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>> >> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/ >>>> >twitter: >>>> >> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | >>>> linked-in: >>>> >> http://lk.linkedin.com/in/suhothayan < >>>> http://lk.linkedin.com/in/suhothayan>* >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > Ramith Jayasinghe >>>> > Technical Lead >>>> > WSO2 Inc., http://wso2.com >>>> > lean.enterprise.middleware >>>> > >>>> > >>>> > >>>> >>>> -- >>>> *Pamod Sylvester * >>>> >>>> *WSO2 Inc.; http://wso2.com <http://wso2.com>* >>>> cell: +94 77 7779495 >>>> -------------- next part -------------- >>>> An HTML attachment was scrubbed... >>>> URL: < >>>> http://mail.wso2.org/pipermail/architecture/attachments/20150719/8dd9b6b5/attachment.html >>>> > >>>> -------------- next part -------------- >>>> A non-text attachment was scrubbed... >>>> Name: activemq-oom.png >>>> Type: image/png >>>> Size: 241605 bytes >>>> Desc: not available >>>> URL: < >>>> http://mail.wso2.org/pipermail/architecture/attachments/20150719/8dd9b6b5/attachment.png >>>> > >>>> >>>> ------------------------------ >>>> >>>> Subject: Digest Footer >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>>> ------------------------------ >>>> >>>> End of Architecture Digest, Vol 72, Issue 105 >>>> ********************************************* >>>> >>> >>> >>> -- >>> John Hawkins >>> Director: Solutions Architecture >>> >>> >>> >> >> >> -- >> Ramith Jayasinghe >> Technical Lead >> WSO2 Inc., http://wso2.com >> lean.enterprise.middleware >> >> E: [email protected] >> P: +94 777542851 >> >> >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
