Re: Tiny topology shows '0' for all stats.

[Nick Dimiduk]

For my own understanding, are you suggesting the FLINK-2944 (or a subtask)
is the appropriate place to implement exposure of metrics such as bytes,
records in, out of Streaming sources and sinks?

On Tue, Dec 15, 2015 at 5:24 AM, Niels Basjes  wrote:

> Hi,
>
> @Ufuk: I added the env.disableOperatorChaining() and indeed now I see two
> things on the screen and there are numbers counting what has happened.
> @Stephan: Yes, I understand these numbers now.
>
> I found that this is already a jira ticket to add what I was looking for:
> https://issues.apache.org/jira/browse/FLINK-2944
> https://issues.apache.org/jira/browse/FLINK-3130
>
> Niels
>
>
>
> On Mon, Dec 14, 2015 at 5:03 PM, Ufuk Celebi  wrote:
>
>>
>> > On 14 Dec 2015, at 16:25, Niels Basjes  wrote:
>> >
>> > Hi,
>> >
>> > I have a very small topology here.
>> > In fact this is a thing that generates synthetic data and puts it into
>> Kafka.
>> > When looking at the web UI I see that all counters (i.e. Bytes
>> received, Records received, Bytes sent,  Records sent) all remain 0.
>> > I verified and I'm seeing thousands of records arriving into Kafka.
>> >
>> > Is this a bug in Flink or am I misinterpreting the meaning of these
>> numbers?
>>
>> Sources and sinks do not show the number of received or sent records,
>> because of the internals of how these numbers are collected. I agree that
>> this is confusing. Big +1 to improve this.
>>
>> You actually don’t see any numbers, because the operators are chained and
>> hence you only have one task, which acts as both source and sink.
>>
>> You should see some sent and received numbers if you break up the chain
>> (env.disableOperatorChaining()). Can you confirm this?
>>
>> – Ufuk
>>
>>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

Re: Published test artifacts for flink streaming

[lofifnc]

Hi,

If you wan't to play with it you can find the source and basic documentation
here: https://github.com/ottogroup/flink-spector.
The framework is for now feature complete. At the moment I'm working on
exposing some more functionality to the user, making the dsl more intuitive
and scalatest support. As soon as I've finished the first two points and
made the core compatible with scala I will release maven artifacts.

Best Alex!



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Published-test-artifacts-for-flink-streaming-tp3379p4055.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

RE: Behaviour of CountWindowAll

[Radu Tudoran]

Hi,



I believe this question might have been asked before - so sorry for repeating 
it (I just did not find the discussion on the mailing list).



Is it possible somehow to create a new DataStream from the elements that are 
evicted from a window?



A simple use case for this is:

We have data coming from a sensor every second. We want to compute all the 
times the average on the last 5 seconds and on the interval from 5 seconds ago 
until 10 seconds ago.

I would be interested in how the data evicted from the main window that keeps 
the fresh data could be feed into a new stream on which I could apply again a 
window of 5 seconds.

(Having a 10 seconds window and selecting only the most ancient 5 second data 
is not a viable option)



Regards,

Re: flink streaming documentation

[Matthias J. Sax]

Thanks for reporting!

Would you like to fix this and open a PR?


-Matthias

On 12/15/2015 04:43 AM, Radu Tudoran wrote:
> Hi,
> 
>  
> 
> I believe i found 2 small inconsistencies in the documentation for the
> description of Window Apply
> 
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#windows-on-unkeyed-data-streams
> 
>  
> 
> 1) in the example provided I believe it should be corrected to remove
> the extra > and add ")":
> 
>  
> 
> |(new WindowFunction,Integer, Tuple, Window>() {
> *...* });|
> 
>  
> 
> instead of
> 
>  
> 
> |(new WindowFunction,Integer>, Tuple, Window>() {
> *...* };|
> 
>  
> 
>  
> 
> 2) for AllWindowedStream it seems there is a need for a
> AllWindowFunction not a WindowFunction
> 
> I would propose to duplicate the existing example to cover also this
> case, particularly that it has a slightly different interface
> 
>  
> 
> |allWindowedStream.apply (new
> AllWindowFunction,Integer, TimeWindow>() { public
> void apply (TimeWindow window, Iterable> values,
> Collector out) throws Exception { int sum = 0; for (value t:
> values) { sum += t.f1; } out.collect (new Integer(sum)); } });|
> 
>  
> 
> Regards,
> 
>  
> 
> Radu
> 
>  
> 



signature.asc
Description: OpenPGP digital signature