APEX-78 says a callback before a checkpoint, but I think what the DelayOperator needs is not a callback right before the checkpoint, but the number of windows till the next checkpoint.
On Wed, Nov 4, 2015 at 10:48 AM, Thomas Weise <tho...@datatorrent.com> wrote: > Is #5 same as https://malhar.atlassian.net/browse/APEX-78 > > It would be good as we can leverage that in other operators also. > > On Wed, Nov 4, 2015 at 10:41 AM, David Yan <da...@datatorrent.com> wrote: > > > Thanks Pramod and Sandesh for offering to help. > > > > Here's my rough plan: > > > > 1. Add DelayOperator interface. The DelayOperator will have a method > > firstWindow(long windowId). Implementations of this interface is > supposed > > to emit tuples for the first window of the execution of the operator > > (either the first window of the execution of the application or the first > > window after recovery). > > > > 2. Add SimpleDelayOperator that implements DelayOperator. It has one > input > > port and one output port. It simply passes the tuples from the input > port > > to the output port and does not do anything for firstWindow() call. > > > > 3. Engine (e.g. DAG validation) to support loops in case of > DelayOperator. > > > > 4. Implement the +1 delay in the engine for input ports that are > connected > > to output ports of DelayOperator. > > > > 5. Add capability in the engine to let the operator know when is the next > > checkpoint window. > > > > 6. Add DefaultDelayOperator that extends SimpleDelayOperator. It writes > > the tuples in the window before each checkpoint to a DFS-backed WAL > (using > > item 5 above), and it overrides firstWindow() to read from the WAL and > > emits the tuples at recovery. > > > > 7. Add +N delay capability (in addition to +1) in the > DefaultDelayOperator. > > > > Let me know whether this plan sounds good to you. > > > > I'm done with 1, 2, and 3. 4 is in progress. > > I think the bulk of the work is 5 and 6 and we can discuss how we can > > divide the work. > > 7 is a nice-to-have and does not have to be done unless there is a > demand. > > > > David > > > > > > On Wed, Nov 4, 2015 at 10:11 AM, Pramod Immaneni <pra...@datatorrent.com > > > > wrote: > > > > > I would like to help. I might be able to pick up some of the smaller > > tasks. > > > > > > On Wed, Nov 4, 2015 at 10:05 AM, David Yan <da...@datatorrent.com> > > wrote: > > > > > > > Thank you for all your feedback. Looks like option #2 wins. > > > > > > > > I will be working on this in November and please let me know if you'd > > > like > > > > to join the effort! > > > > > > > > On Wed, Nov 4, 2015 at 9:58 AM, Thomas Weise <tho...@datatorrent.com > > > > > > wrote: > > > > > > > > > Agreed, there is no ambiguity. > > > > > > > > > > #2 will also allow the user to tune locality as there are no > implicit > > > > > streams as opposed to the unifier like approach. > > > > > > > > > > On Wed, Nov 4, 2015 at 9:54 AM, David Yan <da...@datatorrent.com> > > > wrote: > > > > > > > > > > > On Tue, Nov 3, 2015 at 9:57 PM, Thomas Weise < > > tho...@datatorrent.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > #2 will address that. But if an operator with the delay > interface > > > has > > > > > > > multiple input ports, on which port will the engine perform the > > > > delay? > > > > > > > Maybe we will need to validate that a delay operator can only > > have > > > a > > > > > > single > > > > > > > input port? > > > > > > > > > > > > > > > > > > > My understanding is that the engine performs the +1 delay on the > > > input > > > > > > ports of operators that are connected to output ports of delay > > > > operators. > > > > > > So whether or not the delay operator has multiple input ports > > should > > > > not > > > > > > matter. > > > > > > > > > > > > > > > > > > > > >
