RE: Custom time window in Flink

[Radu Tudoran]

Hi,

Easiest way is to just start from the code of an existing one

https://github.com/apache/flink/tree/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/evictors

If you take the example of Timeevictor you would just need to use the same code 
and modify the public int evict class.

Same story with the triggers

https://github.com/apache/flink/tree/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers

Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R Division

[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München

E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone: +49 891588344173

HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, 
www.huawei.com<http://www.huawei.com/>
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN
This e-mail and its attachments contain confidential information from HUAWEI, 
which is intended only for the person or entity whose address is listed above. 
Any use of the information contained herein in any way (including, but not 
limited to, total or partial disclosure, reproduction, or dissemination) by 
persons other than the intended recipient(s) is prohibited. If you receive this 
e-mail in error, please notify the sender by phone or email immediately and 
delete it!

From: Piyush Shrivastava [mailto:piyush...@yahoo.co.in]
Sent: Wednesday, April 20, 2016 11:24 AM
To: user@flink.apache.org
Subject: Re: Custom time window in Flink

Hello,
Thanks a lot for your reply. Can you share a sample code or example which I can 
refer while creating a custom evictor and trigger?

Thanks and Regards,
Piyush Shrivastava<mailto:piy...@webograffiti.com>
[WeboGraffiti]
http://webograffiti.com<http://webograffiti.com/>

On Wednesday, 20 April 2016 2:50 PM, Radu Tudoran 
<radu.tudo...@huawei.com<mailto:radu.tudo...@huawei.com>> wrote:

Hi,

Tha way to do this is to create your own evictor. In the evictor you can than 
decide when the events are removed. I would suggest creating a symmetric 
trigger as well because I would assume that you also need to fire the 
computation first after 1 hour and then at each 5 minutes.
The logic would be that you have a field that marks whether a window was 
created or not (e.g., a Boolean field in the evictor class), which once a 
window is created you can set it to false and from there one you operate on 5 
minutes windows


Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R Division

[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München

E-mail: radu.tudo...@huawei.com<mailto:radu.tudo...@huawei.com>
Mobile: +49 15209084330
Telephone: +49 891588344173

HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, 
www.huawei.com<http://www.huawei.com/>
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN
This e-mail and its attachments contain confidential information from HUAWEI, 
which is intended only for the person or entity whose address is listed above. 
Any use of the information contained herein in any way (including, but not 
limited to, total or partial disclosure, reproduction, or dissemination) by 
persons other than the intended recipient(s) is prohibited. If you receive this 
e-mail in error, please notify the sender by phone or email immediately and 
delete it!

From: Piyush Shrivastava [mailto:piyush...@yahoo.co.in]
Sent: Wednesday, April 20, 2016 9:59 AM
To: user@flink.apache.org<mailto:user@flink.apache.org>
Subject: Custom time window in Flink

Hello,
I wanted to enquire how a job I am trying to do with Flink can be done.
I have also posted a question on StackOverflow. PFB the link:
http://stackoverflow.com/questions/36720192/custom-windows-charging-in-flink

I am using Flink's TimeWindow functionality to perform some computations. I am 
creating a 5 minute Window. However I want to create a one hour Window for only 
the first time. The next Windows I need are of 5 minutes.
Such that for the first hour, data is collected and my operation is performed 
on it. Once this is done, every five minutes the same operation is performed.

Can you kindly help me with this? How can such a functionality be implemented?


Thanks and Regards,
Piyush Shrivastava<mailto:piy...@webograffiti.com>
[WeboGraffiti]
http://webograffiti.com<http://webograffiti.com/>

Re: Custom time window in Flink

[Piyush Shrivastava]

Hello,Thanks a lot for your reply. Can you share a sample code or example which 
I can refer while creating a custom evictor and trigger? Thanks and 
Regards,Piyush Shrivastava
http://webograffiti.com
 

On Wednesday, 20 April 2016 2:50 PM, Radu Tudoran  
wrote:
 

 #yiv9940457293 #yiv9940457293 -- _filtered #yiv9940457293 
{font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv9940457293 
{font-family:SimSun;panose-1:2 1 6 0 3 1 1 1 1 1;} _filtered #yiv9940457293 
{panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv9940457293 
{font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv9940457293 
{font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;} _filtered #yiv9940457293 
{panose-1:2 1 6 0 3 1 1 1 1 1;}#yiv9940457293 #yiv9940457293 
p.yiv9940457293MsoNormal, #yiv9940457293 li.yiv9940457293MsoNormal, 
#yiv9940457293 div.yiv9940457293MsoNormal 
{margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv9940457293 a:link, 
#yiv9940457293 span.yiv9940457293MsoHyperlink 
{color:blue;text-decoration:underline;}#yiv9940457293 a:visited, #yiv9940457293 
span.yiv9940457293MsoHyperlinkFollowed 
{color:purple;text-decoration:underline;}#yiv9940457293 code {}#yiv9940457293 
p.yiv9940457293MsoAcetate, #yiv9940457293 li.yiv9940457293MsoAcetate, 
#yiv9940457293 div.yiv9940457293MsoAcetate 
{margin:0in;margin-bottom:.0001pt;font-size:8.0pt;}#yiv9940457293 
span.yiv9940457293EmailStyle18 {color:#1F497D;}#yiv9940457293 
span.yiv9940457293BalloonTextChar {}#yiv9940457293 .yiv9940457293MsoChpDefault 
{font-size:10.0pt;} _filtered #yiv9940457293 {margin:1.0in 1.0in 1.0in 
1.0in;}#yiv9940457293 div.yiv9940457293WordSection1 {}#yiv9940457293 Hi,    Tha 
way to do this is to create your own evictor. In the evictor you can than 
decide when the events are removed. I would suggest creating a symmetric 
trigger as well because I would assume that you also need to fire the 
computation first after 1 hour and then at each 5 minutes. The logic would be 
that you have a field that marks whether a window was created or not (e.g., a 
Boolean field in the evictor class), which once a window is created you can set 
it to false and from there one you operate on 5 minutes windows       Dr. Radu 
Tudoran Research Engineer - Big Data Expert IT R Division     HUAWEI 
TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 
München    E-mail:radu.tudo...@huawei.com Mobile: +49 15209084330 Telephone: 
+49 891588344173    HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN This e-mail and its 
attachments contain confidential information from HUAWEI, which is intended 
only for the person or entity whose address is listed above. Any use of the 
information contained herein in any way (including, but not limited to, total 
or partial disclosure, reproduction, or dissemination) by persons other than 
the intended recipient(s) is prohibited. If you receive this e-mail in error, 
please notify the sender by phone or email immediately and delete it!    From: 
Piyush Shrivastava [mailto:piyush...@yahoo.co.in]
Sent: Wednesday, April 20, 2016 9:59 AM
To: user@flink.apache.org
Subject: Custom time window in Flink    Hello, I wanted to enquire how a job I 
am trying to do with Flink can be done. I have also posted a question on 
StackOverflow. PFB the link: 
http://stackoverflow.com/questions/36720192/custom-windows-charging-in-flink    
I am using Flink'sTimeWindow functionality to perform some computations. I am 
creating a 5 minute Window. However I want to create a one hourWindow for only 
the first time. The next Windows I need are of 5 minutes.  Such that for the 
first hour, data is collected and my operation is performed on it. Once this is 
done, every five minutes the same operation is performed.     Can you kindly 
help me with this? How can such a functionality be implemented?

      Thanks and Regards, Piyush Shrivastava  http://webograffiti.com 

  

RE: Custom time window in Flink

[Radu Tudoran]

Hi,

Tha way to do this is to create your own evictor. In the evictor you can than 
decide when the events are removed. I would suggest creating a symmetric 
trigger as well because I would assume that you also need to fire the 
computation first after 1 hour and then at each 5 minutes.
The logic would be that you have a field that marks whether a window was 
created or not (e.g., a Boolean field in the evictor class), which once a 
window is created you can set it to false and from there one you operate on 5 
minutes windows


Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R Division

[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München

E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone: +49 891588344173

HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, 
www.huawei.com
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN
This e-mail and its attachments contain confidential information from HUAWEI, 
which is intended only for the person or entity whose address is listed above. 
Any use of the information contained herein in any way (including, but not 
limited to, total or partial disclosure, reproduction, or dissemination) by 
persons other than the intended recipient(s) is prohibited. If you receive this 
e-mail in error, please notify the sender by phone or email immediately and 
delete it!

From: Piyush Shrivastava [mailto:piyush...@yahoo.co.in]
Sent: Wednesday, April 20, 2016 9:59 AM
To: user@flink.apache.org
Subject: Custom time window in Flink

Hello,
I wanted to enquire how a job I am trying to do with Flink can be done.
I have also posted a question on StackOverflow. PFB the link:
http://stackoverflow.com/questions/36720192/custom-windows-charging-in-flink

I am using Flink's TimeWindow functionality to perform some computations. I am 
creating a 5 minute Window. However I want to create a one hour Window for only 
the first time. The next Windows I need are of 5 minutes.
Such that for the first hour, data is collected and my operation is performed 
on it. Once this is done, every five minutes the same operation is performed.

Can you kindly help me with this? How can such a functionality be implemented?



Thanks and Regards,
Piyush Shrivastava
[WeboGraffiti]
http://webograffiti.com