Hi Kostas,
The late elements are immediately getting triggered with the code I have
sent,
I have tested it with a test case as follows : (I am doing the outer-join
operation by doing the union of stream1 and stream2)
1. Push 5 records to Kafka Topic 1 -----> sourceStream1
2. Wait for few minutes -- (Now by this time sourceStream1 elements are
triggered as non-matched elements)
3. Push the matching 5 records to Kafka Topic 2 ------> sourceStream2 (here
the watermark has already reached ahead)
Now according the custom trigger whenever *onElement is called it
immediately fires the window and purge it*
I have kept the onEventTime code same as 1.0.3, where it will fire and purge
Am I doing something wrong ? (I have kept allowedLateness to Long.MAX_VALUE)
Regards,
Vinay Patil
On Tue, Oct 4, 2016 at 9:58 AM, Kostas Kloudas [via Apache Flink User
Mailing List archive.] <ml-node+s2336050n9325...@n4.nabble.com> wrote:
> Hi Vinay,
>
> From what I understand from your code, the only difference of your trigger
> compared to the
> one shipping with Flink is that for the late elements, instead of firing
> and keeping the element,
> you fire and purge, i.e. clean the window state.
>
> This does not solve the problem of dropping the super late elements if
> their window has expired
> (currentWatermark >= window.maxTimestamp + allowedLateness).
>
> Cheers,
> Kostas
>
> On Oct 4, 2016, at 3:25 PM, vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9325&i=0>> wrote:
>
> Hi Kostas,
>
> Thank you for your reply, yes that will be a good functionality to have,
> but for now the Custom Trigger as close to 1.0.3 works for me.
> public TriggerResult onElement(Object element, long timestamp, TimeWindow
> window, TriggerContext ctx) throws Exception { if(window.maxTimestamp()
> <= ctx.getCurrentWatermark()) { *return TriggerResult.FIRE_AND_PURGE;* }
> else { ctx.registerEventTimeTimer(window.maxTimestamp()); return
> TriggerResult.CONTINUE; } } public TriggerResult onEventTime(long time,
> TimeWindow window, TriggerContext ctx) { return
> *TriggerResult.FIRE_AND_PURGE;* }
>
> This is the change I have done in my custom trigger which is similar to
> 1.0.3 , does this looks good to you ?
> (I am not using canMerge and onMerge methods since I am not aware of it)
>
>
> Regards,
> Vinay Patil
>
> On Tue, Oct 4, 2016 at 3:58 AM, Kostas Kloudas [via Apache Flink User
> Mailing List archive.] <<a href="x-msg://9/user/
> SendEmail.jtp?type=node&node=9323&i=0" target="_top"
> rel="nofollow" link="external" class="">[hidden email]> wrote:
>
>> Hello LF and Vinay,
>>
>> With the introduction of “allowed lateness” elements and windows are
>> kept around until the watermark
>> passes the window.maxTimestamp + allowed_lateness and then they are
>> cleaned up (garbage collected)
>>
>> Every element that comes in and belongs to a window that is garbage
>> collected is dropped as super-late.
>> Elements that are late, but no more than the allowed lateness, they are
>> kept the window fires as before.
>>
>> If you know what the maximum latency is, then the best way is to set the
>> allowed lateness to that value.
>>
>> Currently Flink drops super-late elements and does not provide any
>> mechanism to manually handle these
>> elements, BUT there are discussions about adding such a functionality
>> that will allow you to (probably) have
>> a separate stream with only these elements.
>>
>> Thanks,
>> Kostas
>>
>> On Oct 3, 2016, at 5:28 PM, [hidden email]
>> <http://user/SendEmail.jtp?type=node&node=9307&i=0> wrote:
>>
>> Not yet.
>> I'm hoping a Flink export on this mailing list will reply.
>>
>>
>> - LF
>>
>>
>>
>> ------------------------------
>> *From:* vinay patil <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=9307&i=1>>
>> *To:* [hidden email] <http://user/SendEmail.jtp?type=node&node=9307&i=2>
>> *Sent:* Monday, October 3, 2016 8:09 AM
>> *Subject:* Re: Regarding Late Elements
>>
>> Hi LF,
>>
>> So did you manage to get the workaround for it ?
>>
>> I am using a Custom Trigger which is similar to 1.0.3 with few changes
>>
>> Regards,
>> Vinay Patil
>>
>> On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing
>> List archive.] <[hidden email]> wrote:
>>
>> We have the same requirement - we cannot discard any data even if it
>> arrives late.
>>
>>
>> - LF
>>
>>
>>
>>
>> ------------------------------
>> *From:* Vinay Patil <[hidden email]>
>> *To:* [hidden email]
>>
>> *Sent:* Sunday, October 2, 2016 8:21 PM
>> *Subject:* Regarding Late Elements
>>
>> Hi Guys,
>>
>> Just wanted to get an idea on Why Flink decided to completely discard
>> late elements in the latest version ?, this was not the case in 1.0.3
>>
>>
>> P.S In our case the data is critical so we cannot discard a single record
>> even if it is late, I have written a custom trigger (as suggested by
>> Aljoscha) to even accept late elements.
>>
>>
>> Regards,
>> Vinay Patil
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user- mailing-list-archive.2336050.
>> n4.nabble.com/Regarding-Late- Elements-tp9284p9292.html
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9292.html>
>> To start a new topic under Apache Flink User Mailing List archive., email
>> [hidden
>> email]
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Regarding Late Elements
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9294.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com <http://nabble.com/>.
>>
>>
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Regarding-Late-Elements-tp9284p9307.html
>> To start a new topic under Apache Flink User Mailing List archive., email
>> <a href="x-msg://9/user/SendEmail.jtp?type=node&node=9323&i=1"
>> target="_top" rel="nofollow" link="external" class="">[hidden email]
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Regarding Late Elements
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9323.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com <http://nabble.com>.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/Regarding-Late-Elements-tp9284p9325.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1...@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9328.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
