Hi, Here is a Stats Listener for doing this. Haven't tested it. Need some sample test data on hdfs to test. This is based on the input port queue size of Block reader.
https://github.com/chandnisingh/Malhar/blob/examples/apps/spitter/src/main/java/com/datatorrent/splitter/SplitterThresholdRegulator.java On Tue, Feb 23, 2016 at 8:45 PM, Thomas Weise <[email protected]> wrote: > That's one option, though stats listener may provide better context for > decision making. > > This is needed for better batch support. > > -- > sent from mobile > On Feb 23, 2016 8:36 PM, "Sandesh Hegde" <[email protected]> wrote: > > > Experimental, Can we use iteration to send the feedback and based on that > > adjust the bandwidth? > > > > On Tue, Feb 23, 2016 at 8:32 PM Thomas Weise <[email protected]> > > wrote: > > > >> Priyanka, that's right. In this case we want to process as fast as > >> possible without overwhelming the downstream operators. > >> > >> -- > >> sent from mobile > >> On Feb 23, 2016 8:25 PM, "Priyanka Gugale" <[email protected]> > >> wrote: > >> > >>> Hi, > >>> > >>> I have seen that code in Ingestion, and as I remember it had some > issues > >>> and wasn't working. > >>> I don't have exactly what Thomas asked, but we can use Bandwidth limit > >>> feature, where splitter itself emits based on configured limit, but > yes it > >>> isn't based on load on readers. > >>> > >>> -Priyanka > >>> > >>> On Wed, Feb 24, 2016 at 5:33 AM, Chandni Singh < > [email protected]> > >>> wrote: > >>> > >>>> Hi Thomas, > >>>> > >>>> I had written a StatsListener which when set on FileSplitter and > >>>> BlockReader would automatically throttle block emission rate. The > stats > >>>> listener will send a setProperty command to FileSplitter however if I > >>>> recall correctly, this changed the property only on the logical > operator > >>>> and wasn't propagated to the physical operator. > >>>> > >>>> It was a while back so I am not sure if that was still the issue or > >>>> there was something else wrong my implementation. > >>>> > >>>> Thanks, > >>>> Chandni > >>>> > >>>> On Tue, Feb 23, 2016 at 3:48 PM, Thomas Weise <[email protected] > > > >>>> wrote: > >>>> > >>>>> Is there any support for the splitter to automatically throttle block > >>>>> emission rate based on the readers throughput? > >>>>> > >>>>> -- > >>>>> sent from mobile > >>>>> > >>>> > >>>> > >>> >
