Re: Using FlinkML algorithms in Streaming
Hi Márton, I want to train and get the residuals. Thanks and Regards,Piyush Shrivastava http://webograffiti.com On Wednesday, 11 May 2016 7:19 PM, Márton Balassi <balassi.mar...@gmail.com> wrote: Hey Piyush, Would you like to train or predict on the streaming data? Best, Marton On Wed, May 11, 2016 at 3:44 PM, Piyush Shrivastava <piyush...@yahoo.co.in> wrote: Hello all, I want to perform linear regression using FlinkML's MultipleLinearRegression() function on streaming data. This function takes a DataSet as an input and I cannot create a DataSet inside the MapFunction of a DataStream. How can I use this function on my DataStream? Thanks and Regards,Piyush Shrivastava http://webograffiti.com
Re: Sorted output
This is now solved, thank you. :-) Thanks and Regards,Piyush Shrivastava http://webograffiti.com On Monday, 9 May 2016 3:47 PM, Piyush Shrivastava <piyush...@yahoo.co.in> wrote: Hello all, I have a time series based logic written with Flink. Due to the parallelism, I am not getting the output in a proper series.For example, 3> (12:00:00, "value") appears before 1> (11:59:00, "value") while the timestamp of the latter is smaller than the former. I am using TimeWindow and apply (WindowFunction) functionalities. How can I sort the output? Thanks and Regards,Piyush Shrivastava http://webograffiti.com
Sorted output
Hello all, I have a time series based logic written with Flink. Due to the parallelism, I am not getting the output in a proper series.For example, 3> (12:00:00, "value") appears before 1> (11:59:00, "value") while the timestamp of the latter is smaller than the former. I am using TimeWindow and apply (WindowFunction) functionalities. How can I sort the output? Thanks and Regards,Piyush Shrivastava http://webograffiti.com
Re: Accessing elements from DataStream
Hi Robert,
Can you share an example where flatmap is used to access elements?
Thanks and Regards,Piyush Shrivastava
http://webograffiti.com
On Thursday, 5 May 2016 4:45 PM, Robert Metzger <rmetz...@apache.org> wrote:
Hi,
you can just use a flatMap() on a DataStream to access individual elements from
a stream.
On Thu, May 5, 2016 at 1:00 PM, Piyush Shrivastava <piyush...@yahoo.co.in>
wrote:
Hi all,
Can we access individual elements from a DataStream through an iterator like we
can in a WindowedStream with the apply function?
I am able to access the elements of a WindowedStream using the apply function
and using the Iterable and Collector interfaces:
val ds = ws.apply((K, W, input: Iterable[T], output: Collector[R]) => {
val it = input.toIterator
while(it.hasNext){ ... }})
the output of this function is a DataStream. I want to perform a keyBy on this
DataStream and then again iterate through it's individual elements. We cannot
use the apply function on DataStreams like in WindowedStreams. Is there any
other way? Thanks and Regards,Piyush Shrivastava
http://webograffiti.com
Accessing elements from DataStream
Hi all,
Can we access individual elements from a DataStream through an iterator like we
can in a WindowedStream with the apply function?
I am able to access the elements of a WindowedStream using the apply function
and using the Iterable and Collector interfaces:
val ds = ws.apply((K, W, input: Iterable[T], output: Collector[R]) => {
val it = input.toIterator
while(it.hasNext){ ... }})
the output of this function is a DataStream. I want to perform a keyBy on this
DataStream and then again iterate through it's individual elements. We cannot
use the apply function on DataStreams like in WindowedStreams. Is there any
other way? Thanks and Regards,Piyush Shrivastava
http://webograffiti.com
Re: Understanding Sliding Windows
Hello Dominik, Thanks for the information. Since my window is getting triggered every 10 seconds, the results I am getting before 5 minutes would be irrelevant as I need to consider data coming in every 5 minutes. Is there a way I can skip the results that are output before the first 5 minutes? Thanks and Regards,Piyush Shrivastava http://webograffiti.com On Tuesday, 26 April 2016 8:54 PM, Dominik Choma <dominik.ch...@gmail.com> wrote: Piyush, You created sliding window witch is triggered every 10 seconds Flink fires up this window every 10 seconds, without waiting at 5 min buffer to be filled up It seems to me that first argument is rather "maximum data buffer retention" than " the initial threshold" Dominik Dominik 2016-04-26 12:16 GMT+02:00 Piyush Shrivastava <piyush...@yahoo.co.in>: Hi all,I wanted to know how exactly sliding windows produce results in Flink.Suppose I create a sliding window of 5 minutes which is refreshed in every 10 seconds: .timeWindow(Time.minutes(5), Time.seconds(10)) So in every 10 seconds we are looking at data from the past 5 minutes. But what happens before the initial 5 minutes have passed?Suppose we start the computation at 10:00. At 10:05 we will get the result for 10:00-10:05. But what are the results which we get in between this? i.e. at 10:00:10, 10:00:20 and so on.Basically why do Flink start producing results before the initial threshold has passed? What do these results signify? Thanks and Regards,Piyush Shrivastava http://webograffiti.com
Understanding Sliding Windows
Hi all,I wanted to know how exactly sliding windows produce results in Flink.Suppose I create a sliding window of 5 minutes which is refreshed in every 10 seconds: .timeWindow(Time.minutes(5), Time.seconds(10)) So in every 10 seconds we are looking at data from the past 5 minutes. But what happens before the initial 5 minutes have passed?Suppose we start the computation at 10:00. At 10:05 we will get the result for 10:00-10:05. But what are the results which we get in between this? i.e. at 10:00:10, 10:00:20 and so on.Basically why do Flink start producing results before the initial threshold has passed? What do these results signify? Thanks and Regards,Piyush Shrivastava http://webograffiti.com
Re: Custom Trigger Implementation
Thanks a lot Kostas. This solved my problem. Thanks and Regards,Piyush
Shrivastava
http://webograffiti.com
On Monday, 25 April 2016 3:27 PM, Kostas Kloudas
<k.klou...@data-artisans.com> wrote:
Hi,
Let me also add that you should also override the clear() method in order to
clear you state.and delete the pending timers.
Kostas
On Apr 25, 2016, at 11:52 AM, Kostas Kloudas <k.klou...@data-artisans.com>
wrote:
Hi Piyush,
In the onElement function, you register a timer every time you receive an
element.
When the next watermark arrives, in the flag==false case, this will lead to
every element adding a timer for its timestamp+6ms. The same for flag==true
case, with 2ms interval.
What you can try is to set only once, at the first element the initial trigger
for 60 sec, and then just set all the rest in the on the onEventTime with 20
sec.
To have a look at a custom trigger you can look at
here:https://github.com/kl0u/flink-examples/blob/master/src/main/java/com/dataartisans/flinksolo/beam_comparison/customTriggers/EventTimeTriggerWithEarlyAndLateFiring.java
I hope this helped.Let me know if you need any help.
Kostas
On Apr 25, 2016, at 11:22 AM, Piyush Shrivastava <piyush...@yahoo.co.in> wrote:
Hi all,I want to implement a custom Trigger which fired a GlobalWindow in 1
minute for the first time and every 20 seconds after that.I believe I cannot
get this logic right in the implementation of my custom Trigger. Please help me
with this.
Here is the code of my custom Trigger:
public class TradeTrigger extends Trigger<Object, W> {
/**
*
*/
private static final long serialVersionUID = 1L;
private TradeTrigger() {
}
@Override
public TriggerResult onElement(
Object element,
long timestamp,
W window,
org.apache.flink.streaming.api.windowing.triggers.Trigger.TriggerContext ctx)
throws Exception {
ctx.registerEventTimeTimer(timestamp);
return TriggerResult.CONTINUE;
}
@Override
public TriggerResult onEventTime(
long timestamp,
W window,
org.apache.flink.streaming.api.windowing.triggers.Trigger.TriggerContext ctx)
throws Exception {
ValueState state = ctx.getPartitionedState(new
ValueStateDescriptor("flag", Boolean.TYPE, false));
if(state.value()==false){
ctx.registerEventTimeTimer(timestamp+6);
state.update(true);
return TriggerResult.FIRE;
}else{
System.out.println(""+state.value());
ctx.registerEventTimeTimer(timestamp+2);
return TriggerResult.FIRE;
}
}
@Override
public TriggerResult onProcessingTime(
long arg0,
W arg1,
org.apache.flink.streaming.api.windowing.triggers.Trigger.TriggerContext arg2)
throws Exception {
// TODO Auto-generated method stub
return TriggerResult.CONTINUE;
}
public static TradeTrigger of() {
return new TradeTrigger<>();
}
}
Thanks and Regards,Piyush Shrivastava
http://webograffiti.com
Operation of Windows and Triggers
I wanted to know how Windows and Triggers work in Flink. I am creating a time window of 20 seconds and a count trigger of 100. stream.keyBy(0) .timeWindow(Time.seconds(20)) .trigger(CountTrigger.of(100)) In this case, when will my window get triggered? When 20 seconds has passed, 100 messages are passed? Thanks and Regards,Piyush Shrivastava http://webograffiti.com
Re: Custom time window in Flink
Hello,Thanks a lot for your reply. Can you share a sample code or example which
I can refer while creating a custom evictor and trigger? Thanks and
Regards,Piyush Shrivastava
http://webograffiti.com
On Wednesday, 20 April 2016 2:50 PM, Radu Tudoran <radu.tudo...@huawei.com>
wrote:
#yiv9940457293 #yiv9940457293 -- _filtered #yiv9940457293
{font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv9940457293
{font-family:SimSun;panose-1:2 1 6 0 3 1 1 1 1 1;} _filtered #yiv9940457293
{panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv9940457293
{font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv9940457293
{font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;} _filtered #yiv9940457293
{panose-1:2 1 6 0 3 1 1 1 1 1;}#yiv9940457293 #yiv9940457293
p.yiv9940457293MsoNormal, #yiv9940457293 li.yiv9940457293MsoNormal,
#yiv9940457293 div.yiv9940457293MsoNormal
{margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv9940457293 a:link,
#yiv9940457293 span.yiv9940457293MsoHyperlink
{color:blue;text-decoration:underline;}#yiv9940457293 a:visited, #yiv9940457293
span.yiv9940457293MsoHyperlinkFollowed
{color:purple;text-decoration:underline;}#yiv9940457293 code {}#yiv9940457293
p.yiv9940457293MsoAcetate, #yiv9940457293 li.yiv9940457293MsoAcetate,
#yiv9940457293 div.yiv9940457293MsoAcetate
{margin:0in;margin-bottom:.0001pt;font-size:8.0pt;}#yiv9940457293
span.yiv9940457293EmailStyle18 {color:#1F497D;}#yiv9940457293
span.yiv9940457293BalloonTextChar {}#yiv9940457293 .yiv9940457293MsoChpDefault
{font-size:10.0pt;} _filtered #yiv9940457293 {margin:1.0in 1.0in 1.0in
1.0in;}#yiv9940457293 div.yiv9940457293WordSection1 {}#yiv9940457293 Hi, Tha
way to do this is to create your own evictor. In the evictor you can than
decide when the events are removed. I would suggest creating a symmetric
trigger as well because I would assume that you also need to fire the
computation first after 1 hour and then at each 5 minutes. The logic would be
that you have a field that marks whether a window was created or not (e.g., a
Boolean field in the evictor class), which once a window is created you can set
it to false and from there one you operate on 5 minutes windows Dr. Radu
Tudoran Research Engineer - Big Data Expert IT R Division HUAWEI
TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992
München E-mail:radu.tudo...@huawei.com Mobile: +49 15209084330 Telephone:
+49 891588344173 HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN This e-mail and its
attachments contain confidential information from HUAWEI, which is intended
only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total
or partial disclosure, reproduction, or dissemination) by persons other than
the intended recipient(s) is prohibited. If you receive this e-mail in error,
please notify the sender by phone or email immediately and delete it! From:
Piyush Shrivastava [mailto:piyush...@yahoo.co.in]
Sent: Wednesday, April 20, 2016 9:59 AM
To: user@flink.apache.org
Subject: Custom time window in Flink Hello, I wanted to enquire how a job I
am trying to do with Flink can be done. I have also posted a question on
StackOverflow. PFB the link:
http://stackoverflow.com/questions/36720192/custom-windows-charging-in-flink
I am using Flink'sTimeWindow functionality to perform some computations. I am
creating a 5 minute Window. However I want to create a one hourWindow for only
the first time. The next Windows I need are of 5 minutes. Such that for the
first hour, data is collected and my operation is performed on it. Once this is
done, every five minutes the same operation is performed. Can you kindly
help me with this? How can such a functionality be implemented?
Thanks and Regards, Piyush Shrivastava http://webograffiti.com
Custom time window in Flink
Hello,I wanted to enquire how a job I am trying to do with Flink can be done.I have also posted a question on StackOverflow. PFB the link:http://stackoverflow.com/questions/36720192/custom-windows-charging-in-flink I am using Flink's TimeWindow functionality to perform some computations. I am creating a 5 minute Window. However I want to create a one hour Window for only the first time. The next Windows I need are of 5 minutes. Such that for the first hour, data is collected and my operation is performed on it. Once this is done, every five minutes the same operation is performed. Can you kindly help me with this? How can such a functionality be implemented? Thanks and Regards,Piyush Shrivastava http://webograffiti.com
