Thank you everyone for your support !!
On Tue, Dec 6, 2016 at 9:57 PM, Vlad Rozov <v.ro...@datatorrent.com> wrote: > I'd recommend to use additional output port solution outlined by Bhupesh. > There are few Apex applications on the field that leverage that solution. > > Thank you, > > Vlad > > > On 12/4/16 11:45, Vishal Agrawal wrote: > > Thank you Bhupesh and Ram. Appreciate your quick response. > > I see ThrottlingStatsListener.processStats() method gets called whenever > new stats are received from the operators. > How frequently these stats are sent by the operators? Is it end of every > window? > > > Thanks, > Vishal > > > > On Sat, Dec 3, 2016 at 1:46 PM, Munagala Ramanath <r...@datatorrent.com> > wrote: > > To further clarify Bhupesh's comment, suppose you determine in window N in > the input operator the > data reading phase is complete and send the control tuple on the dedicated > port to the output > operator in window N+1. If the downstream operators (including the output > operator) P_i are > processing respective windows W_i the output operator will > not actually see that control tuple until all the W_i have reached N+1. > > Another option is to use the OperatorRequest mechanism to communicate > among the operators > out-of-band; an example is at: > https://github.com/DataTorrent/examples/tree/master/tutorials/throttle > > That example shows how to modulate the speed of upstream operators but it > can be adapted for > your scenario by checking and recording the "completion status" of all the > operators. > > Ram > > On Sat, Dec 3, 2016 at 5:10 AM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > Hi Vishal, > > > A window is processed by an operator only when the previous window is > completely processed. When you send the control tuple in a new window, you > can be sure that all previous windows have been processed. > > > That is the reason I asked you to send the control tuple in a new window. > > > For shutdown, you can try throwing a ShutdownException() from the input > operator. This will propagate through the entire Dag and shutdown all the > operators in sequence. > > > ~ Bhupesh > > > > On Dec 3, 2016 18:15, "Vishal Agrawal" <vishal.agrawal...@gmail.com> > wrote: > > Thank you Bhupesh. > > Another catch is just because input operator has processed last record > doesn't mean all the intermediate operators have processed it as well. How > can I ensure that all the operators have processed all the records before > performing the write operation. > > Also is there a way to shutdown the dag programmatically once it has > performed the write operation. > > > Thanks, > Vishal > > > On Fri, Dec 2, 2016 at 11:11 PM Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > Hi Vishal, > > The support for such operations is currently being enhanced in Apex. > > For now, you can do the following: > - Have an additional output port in your input operator as well as an > input port in the "Writer" operator. > - Once the Input operator has read and emitted all the data that it > wanted to, you can send a tuple on the new port that you have created. This > tuple will act as your signal. Make sure to do this in a new window - > ideally if the input is done in window x, send this tuple in window x+1. > - When you receive this tuple on the Writer operator, you can perform the > write operation on the external system. > > ~ Bhupesh > > On Sat, Dec 3, 2016 at 3:56 AM, Vishal Agrawal < > vishal.agrawal...@gmail.com> wrote: > > Hi, > > I am performing a batch operation. My input operator is reading multiple > files line by line and then there are bunch of operators manipulating the > records to evaluate result. > My output operator is supposed to write the final result to external > system once all the records from each of the files are processed. > > On completion of reading all the files, how can I trigger an event which > will inform my output operator to perform the write operation on external > system. > > > Thanks, > Vishal > > > > > > > > > > > > > > > > > > > > >
