+1 for giving the control to protocol level. AMQP has its own way of flow controlling. Some protocols may not have it OOB.
On Fri, Jun 5, 2015 at 7:06 AM, Pamod Sylvester <[email protected]> wrote: > for AMQP isn't it possible to apply TCP back pressure and then release > each channel when invoking flow control for individual publishers. > > I am not sure whether the capability is provided out of the box in MINA, > > i.e what i am suggesting is somewhat a solution where all acceptance > handlers would be notified when global flow controlling is triggered at a > glance it would stop reading from all the channels. Then when we iterate > through each publisher channel to apply flow controlling we release each > corresponding individual channel for reading then trigger the flow > controlling operation. > > The goal is to prevent messages from being bombarded if the buffer is > full, db has timed out etc. Mainly to avoid broker from going OOM at such > situations. > > also, if the above functionality is not provided out of the box this will > be too complex to implement :) > > Thanks, > Pamod > > On Fri, Jun 5, 2015 at 5:04 PM, Asitha Nanayakkara <[email protected]> > wrote: > >> Yes. when we stop auto read we won't be able to read. But in MQTT there >> is no flow control mechanism. Hence, we can use stop auto read to do flow >> control (publishers) >> And I agree this may not work in AMQP since we have the flow control >> mechanism within the protocol itself. We may need to send some packets back >> and forth. >> >> What I suggest is to give that extra bit of information (global flow >> control is enabled or disabled) to the protocol so that at protocol level >> we can take advantage of that information if that is applicable. If not we >> may have to iterate through all the publishers. >> >> On Fri, Jun 5, 2015 at 4:51 PM, Asanka Abeyweera <[email protected]> >> wrote: >> >>> Hi Asitha, >>> >>> Thank you for the explanation. Say we go ahead an stop auto read in >>> Netty (or Mina) when global flow control triggered. Will it be an issue to >>> send the flow control enable messages to publishers since some packets have >>> to go back and forth? >>> >>> On Fri, Jun 5, 2015 at 4:35 PM, Asitha Nanayakkara <[email protected]> >>> wrote: >>> >>>> Hi Asanka, >>>> >>>> For AMQP yes we need to iterate. Since we need to block each channel >>>> separately. At the moment with TCP back pressure for MQTT, what Pamod is >>>> suggesting is to block all the publishers at once. With a separate listener >>>> for global flow control we can give the protocol developer the ability do >>>> his own implementation when Andes core notifies global flow control limit >>>> is reached. >>>> >>>> @Pamod correct me if I misunderstood the use-case. >>>> >>>> Thanks, >>>> Asitha >>>> >>>> >>>> On Fri, Jun 5, 2015 at 3:24 PM, Asanka Abeyweera <[email protected]> >>>> wrote: >>>> >>>>> Hi Asitha, >>>>> >>>>> Event with this approach, at the end doesn't it boils down to >>>>> iterating over a list to inform the listeners. >>>>> >>>>> >>>>> >>>>> On Fri, Jun 5, 2015 at 3:03 PM, Asitha Nanayakkara <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Pamod, >>>>>> >>>>>> In Andes core, we can have a listener to inform global flow control. >>>>>> Similar to FlowControlListner. With that when the global flow control >>>>>> limit >>>>>> is reached we can invoke this listener interface. In the protocol level >>>>>> implementation of this global flow control listener interface, we can >>>>>> either iterate or directly flow control all publishers depending on the >>>>>> protocol capability. WDYT? >>>>>> >>>>>> Thanks, >>>>>> Asitha >>>>>> >>>>>> On Fri, Jun 5, 2015 at 2:45 PM, Pamod Sylvester <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> Flow control is activated when the distruptor buffer is full, >>>>>>> database encounters an error etc. When its being applied, currently it >>>>>>> iterates through the list of publisher channels, notifying the >>>>>>> publishers >>>>>>> individually to flow control. >>>>>>> >>>>>>> in the FlowControlManager its illustrated through the following code >>>>>>> segment. >>>>>>> >>>>>>> private synchronized void >>>>>>> blockListenersOnBufferBasedFlowControl() { >>>>>>> if (!globalBufferBasedFlowControlEnabled) { >>>>>>> globalBufferBasedFlowControlEnabled = true; >>>>>>> >>>>>>> for (AndesChannel channel : channels) { >>>>>>> >>>>>>> channel.notifyGlobalBufferBasedFlowControlActivation(); >>>>>>> } >>>>>>> >>>>>>> scheduledBufferBasedFlowControlTimeoutFuture = >>>>>>> executor.schedule(flowControlTimeoutTask, 1, TimeUnit.MINUTES); >>>>>>> log.info("Global buffer based flow control enabled."); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> >>>>>>> Since its being iterated and flow controlled individually there >>>>>>> could be a slight latency when applying flow control to the last >>>>>>> publisher >>>>>>> in the list. (Until flow control is applied the publisher/s would >>>>>>> continue >>>>>>> to send messages) >>>>>>> >>>>>>> The question is, lets say there're > 10000 of publishers which need >>>>>>> to be flow controlled at a given time. Will there be a possibility for >>>>>>> the >>>>>>> broker to go OOM due to the above mentioned reason. i.e flow control is >>>>>>> applied due to buffer nearly became full ? >>>>>>> >>>>>>> If that's the case, what would be the best way to deal with it ? >>>>>>> >>>>>>> One suggestion would be to at a glance, stop the server from >>>>>>> accepting messages totally.( i.e in Netty server we could do it by >>>>>>> adding a >>>>>>> child option to set auto read 'false' in the ServerBootstrap). until the >>>>>>> whole publisher list is iterated and flow control is applied and then >>>>>>> retain accepting connections. >>>>>>> >>>>>>> Thanks, >>>>>>> Pamod >>>>>>> >>>>>>> -- >>>>>>> *Pamod Sylvester * >>>>>>> >>>>>>> *WSO2 Inc.; http://wso2.com <http://wso2.com>* >>>>>>> cell: +94 77 7779495 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Asitha Nanayakkara* >>>>>> Software Engineer >>>>>> WSO2, Inc. http://wso2.com/ >>>>>> Mob: + 94 77 85 30 682 >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Asanka Abeyweera >>>>> Software Engineer >>>>> WSO2 Inc. >>>>> >>>>> Phone: +94 712228648 >>>>> Blog: a5anka.github.io >>>>> >>>> >>>> >>>> >>>> -- >>>> *Asitha Nanayakkara* >>>> Software Engineer >>>> WSO2, Inc. http://wso2.com/ >>>> Mob: + 94 77 85 30 682 >>>> >>>> >>> >>> >>> -- >>> Asanka Abeyweera >>> Software Engineer >>> WSO2 Inc. >>> >>> Phone: +94 712228648 >>> Blog: a5anka.github.io >>> >> >> >> >> -- >> *Asitha Nanayakkara* >> Software Engineer >> WSO2, Inc. http://wso2.com/ >> Mob: + 94 77 85 30 682 >> >> > > > -- > *Pamod Sylvester * > > *WSO2 Inc.; http://wso2.com <http://wso2.com>* > cell: +94 77 7779495 > -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
