Jakub, I see the difficulties. Looks like it might not be that simple.
Regards, Rajith On Mon, Jul 23, 2012 at 11:35 AM, Jakub Scholz <[email protected]> wrote: > Hi Rajith, > > Most of the messages are delivered as a broadcasts from one producer > to multiple receivers. And even when the queue of some receivers is > full because they are not consuming, we still need to deliver the > message for the rest of the receivers and at the same time to be aware > who didn't received the message to eventually resend it later. I'm > afraid that the producer flow control would not allow this so easily. > We would probably need to change our producers to send the messages > individually - N producers for N consumers. Only then we will be able > to use the producer flow control ... > > Regards > Jakub > > On Mon, Jul 23, 2012 at 3:27 PM, Rajith Attapattu <[email protected]> wrote: >> Jakub, >> >> I wonder if producer flow control can help here. >> If implemented properly this should (at least theoretically) prevent >> the broker from going out of memory due to queue growth. >> As you correctly point out, flow-2-disk just postpones it at best, in >> addition to the fact that it has a serious impact on perf. >> >> While there might be an impact on perf with producer flow control, I'm >> sure it's way better than flow-2-disk. >> >> Regards, >> >> Rajith >> >> On Mon, Jul 23, 2012 at 7:40 AM, Jakub Scholz <[email protected]> wrote: >>> Yes, the use of flow-to-disk queues unfortunately doesn't solve the >>> memory issue on 100%. It just decreases the memory consumption, so the >>> point when the broker runs out of memory is postponed a bit. >>> >>> Regards >>> Jakub >>> >>> On Mon, Jul 23, 2012 at 11:09 AM, Gordon Sim <[email protected]> wrote: >>>> On 07/22/2012 09:31 PM, Jakub Scholz wrote: >>>>> >>>>> We expect the brokers to deliver approximately hundreds of GB of >>>>> messages per day. Under normal circumstances, most of the messages >>>>> will be consumed by the clients almost immediately, but in some >>>>> exceptional situations, they may need to be stored on the broker. And >>>>> since the performance isn't the biggest issue for us (while on the >>>>> other hand reliability of the broker is), the flow-to-disk queues kind >>>>> of help (yes, the headers are still kept in memory). >>>> >>>> >>>> My point was that in such cases, the memory consumed by the queue is not in >>>> any way bounded. It will keep growing as the messages pile up. If the >>>> content is a significant part of that (i.e. if the messages are relatively >>>> large), then flow to disk can at least slow the growth. I assume that >>>> slowed >>>> growth is useful in your case? >>>> >>>> >>>>> Although you can always say that we can get HW with more memory or >>>>> split the load between multiple brokers if necessary, it would still >>>>> be better if the flow-to-disk queues are replaced by some better >>>>> alternative. >>>> >>>> >>>> I certainly agree a better solution is needed. One where the memory >>>> required >>>> for such queues can be bounded in a reliable fashion, regardless of their >>>> size. >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
