+1
It also can help to Shutdown the application gracefully.
Bright
> On Jun 24, 2016, at 1:35 PM, Siyuan Hua <[email protected]> wrote:
>
> +1
>
> I think it's good to have custom control tuple and I prefer the 1 option.
>
> Also I think we should think about couple different callbacks, that could
> be operator level(triggered when an operator receives an control tuple) or
> dag level(triggered when control tuple flow over the whole dag)
>
> Regards,
> Siyuan
>
>
>
>
> On Fri, Jun 24, 2016 at 12:42 PM, David Yan <[email protected]> wrote:
>
>> My initial thinking is that the custom control tuples, just like the
>> existing control tuples, will only be generated from the input operators
>> and will be propagated downstream to all operators in the DAG. So the NxM
>> partitioning scenario works just like how other control tuples work, i.e.
>> the callback will not be called unless all ports have received the control
>> tuple for a particular window. This creates a little bit of complication
>> with multiple input operators though.
>>
>> David
>>
>>
>> On Fri, Jun 24, 2016 at 12:03 PM, Tushar Gosavi <[email protected]>
>> wrote:
>>
>>> +1 for the feature
>>>
>>> I am in favor of option 1, but we may need an helper method to avoid
>>> compiler error on typed port, as calling port.emit(controlTuple) will
>>> be an error if type of control tuple and port does not match. or new
>>> method in outputPort object , emitControlTuple(ControlTuple).
>>>
>>> Can you give example of piggy backing tuple with current BEGIN_WINDOW
>>> and END_WINDOW control tuples?
>>>
>>> In case of NxM partitioning, each downstream operator will receive N
>>> control tuples. will it call user handler N times for each downstream
>>> operator or just once.
>>>
>>> Regards,
>>> - Tushar.
>>>
>>>
>>>
>>> On Fri, Jun 24, 2016 at 11:52 PM, David Yan <[email protected]>
>> wrote:
>>>> Hi all,
>>>>
>>>> I would like to propose a new feature to the Apex core engine -- the
>>>> support of custom control tuples. Currently, we have control tuples
>> such
>>> as
>>>> BEGIN_WINDOW, END_WINDOW, CHECKPOINT, and so on, but we don't have the
>>>> support for applications to insert their own control tuples. The way
>>>> currently to get around this is to use data tuples and have a separate
>>> port
>>>> for such tuples that sends tuples to all partitions of the downstream
>>>> operators, which is not exactly developer friendly.
>>>>
>>>> We have already seen a number of use cases that can use this feature:
>>>>
>>>> 1) Batch support: We need to tell all operators of the physical DAG
>> when
>>> a
>>>> batch starts and ends, so the operators can do whatever that is needed
>>> upon
>>>> the start or the end of a batch.
>>>>
>>>> 2) Watermark: To support the concepts of event time windowing, the
>>>> watermark control tuple is needed to tell which windows should be
>>>> considered late.
>>>>
>>>> 3) Changing operator properties: We do have the support of changing
>>>> operator properties on the fly, but with a custom control tuple, the
>>>> command to change operator properties can be window aligned for all
>>>> partitions and also across the DAG.
>>>>
>>>> 4) Recording tuples: Like changing operator properties, we do have this
>>>> support now but only at the individual physical operator level, and
>>> without
>>>> control of which window to record tuples for. With a custom control
>>> tuple,
>>>> because a control tuple must belong to a window, all operators in the
>> DAG
>>>> can start (and stop) recording for the same windows.
>>>>
>>>> I can think of two options to achieve this:
>>>>
>>>> 1) new custom control tuple type that takes user's serializable object.
>>>>
>>>> 2) piggy back the current BEGIN_WINDOW and END_WINDOW control tuples.
>>>>
>>>> Please provide your feedback. Thank you.
>>>>
>>>> David
>>>
>>
