Re: Writing an Integration test for flink-metrics

[Colin Williams]

Thanks for the help. I ended up creating a custom metric reporter and
accessing it's fields in an integration test. However I do think that the
test that Martin checked in is another good way to test. I opened
https://issues.apache.org/jira/browse/FLINK-7907 regarding the missing
Scala examples in the metrics documentation. I can take a stab at that on
the weekend if somebody doesn't beat me to it.

On Fri, Oct 13, 2017 at 6:09 AM, Martin Eden 
wrote:

> Hi,
> Not merged in yet but this is an example pr that is mocking metrics and
> checking they are properly updated:
> https://github.com/apache/flink/pull/4725
>
>
> On Fri, Oct 13, 2017 at 1:49 PM, Aljoscha Krettek 
> wrote:
>
>> I think we could add this functionality to the (operator) test harnesses.
>> I.e. add a mock MetricGroup thingy in there that you can query to check the
>> state of metrics.
>>
>>
>> On 13. Oct 2017, at 13:50, Chesnay Schepler  wrote:
>>
>> I meant that you could unit-test the behavior of the function in
>> isolation. You could create a dummy metric group that
>> verifies that the correct counters are being registered (based on names i
>> guess), as well as provide access to them.
>> Mock some input and observe whether the counter value is being modified.
>>
>> Whether this is a viable option depends a bit on the complexity of the
>> function of course, that is how much how mocking
>> you would have to do.
>>
>> On 13.10.2017 11:18, Piotr Nowojski wrote:
>>
>> For testing Link applications in general you can read
>> https://ci.apache.org/projects/flink/flink-docs-release
>> -1.4/dev/stream/testing.html
>>
>> However as we said before, testing metrics would require using custom or
>> a imx reporter.
>>
>> Yes, please report this bug in Jira.
>>
>> Thanks, Piotrek
>>
>> On 13 Oct 2017, at 04:31, Colin Williams > com> wrote:
>>
>> Team wants an integration test, I'm not sure what unit test you had in
>> mind. Actually feel that I've been trying to avoid the reporter method but
>> that would be more end to end.
>>
>> The documentation for metrics and Scala are missing with the exception of
>> Gauge: https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>> monitoring/metrics.html . Should I file a issue against that?
>>
>> Then it leaves you guessing a little bit how to implement Counters. One
>> approach tried was using objects
>>
>> object PointFilter extends RichMapFunction[...
>>
>>   @transient lazy val someCounter = 
>> getRuntimeContext.getMetricGroup.counter(...)
>>
>>
>> This allowed access to the counter before and after execution . However
>> between the unit tests the Counter kept its value also and that's a no for
>> the test. Think that might be an issue with ScalaTest.
>>
>> I've tried to get at the counter from some other directions like trying
>> to find a way to inject a reporter to get it's state. But don't see a way
>> to do it. So probably the best thing to do is fire up something to collect
>> the metrics from the reporter.
>>
>> On Thu, Oct 12, 2017 at 5:29 AM, Chesnay Schepler 
>> wrote:
>>
>>> Well damn, i should've read the second part of the initial mail.
>>>
>>> I'm wondering though, could you not unit-test this behavior?
>>>
>>>
>>> On 12.10.2017 14:25, Chesnay Schepler wrote:
>>>
 You could also write a custom reporter that opens a socket or similar
 for communication purposes.

 You can then either query it for the metrics, or even just trigger the
 verification in the reporter,
 and fail with an error if the reporter returns an error.

 On 12.10.2017 14:02, Piotr Nowojski wrote:

> Hi,
>
> Doing as you proposed using JMXReporter (or custom reporter) should
> work. I think there is no easier way to do this at the moment.
>
> Piotrek
>
> On 12 Oct 2017, at 04:58, Colin Williams <
>> colin.williams.seat...@gmail.com> wrote:
>>
>> I have a RichMapFunction and I'd like to ensure Meter fields are
>> properly incremented. I've been trying to think of the best way to do 
>> this.
>> Currently I think that I'd need to either implement my own reporter (or 
>> use
>> JMX) and write to a socket, create a listener and wait for the reporter 
>> to
>> send the message.
>>
>> Is this a good approach for writing the test, or should I be
>> considering something else?
>>
>
>


>>>
>>
>>
>>
>>
>

Re: Writing an Integration test for flink-metrics

[Martin Eden]

Hi,
Not merged in yet but this is an example pr that is mocking metrics and
checking they are properly updated:
https://github.com/apache/flink/pull/4725


On Fri, Oct 13, 2017 at 1:49 PM, Aljoscha Krettek 
wrote:

> I think we could add this functionality to the (operator) test harnesses.
> I.e. add a mock MetricGroup thingy in there that you can query to check the
> state of metrics.
>
>
> On 13. Oct 2017, at 13:50, Chesnay Schepler  wrote:
>
> I meant that you could unit-test the behavior of the function in
> isolation. You could create a dummy metric group that
> verifies that the correct counters are being registered (based on names i
> guess), as well as provide access to them.
> Mock some input and observe whether the counter value is being modified.
>
> Whether this is a viable option depends a bit on the complexity of the
> function of course, that is how much how mocking
> you would have to do.
>
> On 13.10.2017 11:18, Piotr Nowojski wrote:
>
> For testing Link applications in general you can read
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/
> testing.html
>
> However as we said before, testing metrics would require using custom or a
> imx reporter.
>
> Yes, please report this bug in Jira.
>
> Thanks, Piotrek
>
> On 13 Oct 2017, at 04:31, Colin Williams 
> wrote:
>
> Team wants an integration test, I'm not sure what unit test you had in
> mind. Actually feel that I've been trying to avoid the reporter method but
> that would be more end to end.
>
> The documentation for metrics and Scala are missing with the exception of
> Gauge: https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/monitoring/metrics.html . Should I file a issue against that?
>
> Then it leaves you guessing a little bit how to implement Counters. One
> approach tried was using objects
>
> object PointFilter extends RichMapFunction[...
>
>   @transient lazy val someCounter = 
> getRuntimeContext.getMetricGroup.counter(...)
>
>
> This allowed access to the counter before and after execution . However
> between the unit tests the Counter kept its value also and that's a no for
> the test. Think that might be an issue with ScalaTest.
>
> I've tried to get at the counter from some other directions like trying to
> find a way to inject a reporter to get it's state. But don't see a way to
> do it. So probably the best thing to do is fire up something to collect the
> metrics from the reporter.
>
> On Thu, Oct 12, 2017 at 5:29 AM, Chesnay Schepler 
> wrote:
>
>> Well damn, i should've read the second part of the initial mail.
>>
>> I'm wondering though, could you not unit-test this behavior?
>>
>>
>> On 12.10.2017 14:25, Chesnay Schepler wrote:
>>
>>> You could also write a custom reporter that opens a socket or similar
>>> for communication purposes.
>>>
>>> You can then either query it for the metrics, or even just trigger the
>>> verification in the reporter,
>>> and fail with an error if the reporter returns an error.
>>>
>>> On 12.10.2017 14:02, Piotr Nowojski wrote:
>>>
 Hi,

 Doing as you proposed using JMXReporter (or custom reporter) should
 work. I think there is no easier way to do this at the moment.

 Piotrek

 On 12 Oct 2017, at 04:58, Colin Williams  com> wrote:
>
> I have a RichMapFunction and I'd like to ensure Meter fields are
> properly incremented. I've been trying to think of the best way to do 
> this.
> Currently I think that I'd need to either implement my own reporter (or 
> use
> JMX) and write to a socket, create a listener and wait for the reporter to
> send the message.
>
> Is this a good approach for writing the test, or should I be
> considering something else?
>


>>>
>>>
>>
>
>
>
>

Re: Writing an Integration test for flink-metrics

[Aljoscha Krettek]

I think we could add this functionality to the (operator) test harnesses. I.e. 
add a mock MetricGroup thingy in there that you can query to check the state of 
metrics. 

> On 13. Oct 2017, at 13:50, Chesnay Schepler  wrote:
> 
> I meant that you could unit-test the behavior of the function in isolation. 
> You could create a dummy metric group that
> verifies that the correct counters are being registered (based on names i 
> guess), as well as provide access to them.
> Mock some input and observe whether the counter value is being modified.
> 
> Whether this is a viable option depends a bit on the complexity of the 
> function of course, that is how much how mocking
> you would have to do.
> 
> On 13.10.2017 11:18, Piotr Nowojski wrote:
>> For testing Link applications in general you can read 
>> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html
>>  
>> 
>> 
>> However as we said before, testing metrics would require using custom or a 
>> imx reporter.
>> 
>> Yes, please report this bug in Jira. 
>> 
>> Thanks, Piotrek
>> 
>>> On 13 Oct 2017, at 04:31, Colin Williams >> > wrote:
>>> 
>>> Team wants an integration test, I'm not sure what unit test you had in 
>>> mind. Actually feel that I've been trying to avoid the reporter method but 
>>> that would be more end to end.
>>> 
>>> The documentation for metrics and Scala are missing with the exception of 
>>> Gauge: 
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html
>>>  
>>> 
>>>  . Should I file a issue against that?
>>> 
>>> Then it leaves you guessing a little bit how to implement Counters. One 
>>> approach tried was using objects
>>> 
>>> object PointFilter extends RichMapFunction[...
>>>   @transient lazy val someCounter = 
>>> getRuntimeContext.getMetricGroup.counter(...)
>>> 
>>> This allowed access to the counter before and after execution . However 
>>> between the unit tests the Counter kept its value also and that's a no for 
>>> the test. Think that might be an issue with ScalaTest. 
>>> 
>>> I've tried to get at the counter from some other directions like trying to 
>>> find a way to inject a reporter to get it's state. But don't see a way to 
>>> do it. So probably the best thing to do is fire up something to collect the 
>>> metrics from the reporter.
>>> 
>>> On Thu, Oct 12, 2017 at 5:29 AM, Chesnay Schepler >> > wrote:
>>> Well damn, i should've read the second part of the initial mail.
>>> 
>>> I'm wondering though, could you not unit-test this behavior?
>>> 
>>> 
>>> On 12.10.2017 14:25, Chesnay Schepler wrote:
>>> You could also write a custom reporter that opens a socket or similar for 
>>> communication purposes.
>>> 
>>> You can then either query it for the metrics, or even just trigger the 
>>> verification in the reporter,
>>> and fail with an error if the reporter returns an error.
>>> 
>>> On 12.10.2017 14:02, Piotr Nowojski wrote:
>>> Hi,
>>> 
>>> Doing as you proposed using JMXReporter (or custom reporter) should work. I 
>>> think there is no easier way to do this at the moment.
>>> 
>>> Piotrek
>>> 
>>> On 12 Oct 2017, at 04:58, Colin Williams >> > wrote:
>>> 
>>> I have a RichMapFunction and I'd like to ensure Meter fields are properly 
>>> incremented. I've been trying to think of the best way to do this. 
>>> Currently I think that I'd need to either implement my own reporter (or use 
>>> JMX) and write to a socket, create a listener and wait for the reporter to 
>>> send the message.
>>> 
>>> Is this a good approach for writing the test, or should I be considering 
>>> something else?
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
> 

Re: Writing an Integration test for flink-metrics

[Chesnay Schepler]

I meant that you could unit-test the behavior of the function in 
isolation. You could create a dummy metric group that
verifies that the correct counters are being registered (based on names 
i guess), as well as provide access to them.

Mock some input and observe whether the counter value is being modified.

Whether this is a viable option depends a bit on the complexity of the 
function of course, that is how much how mocking

you would have to do.

On 13.10.2017 11:18, Piotr Nowojski wrote:
For testing Link applications in general you can read 
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html


However as we said before, testing metrics would require using custom 
or a imx reporter.


Yes, please report this bug in Jira.

Thanks, Piotrek

On 13 Oct 2017, at 04:31, Colin Williams 
> wrote:


Team wants an integration test, I'm not sure what unit test you had 
in mind. Actually feel that I've been trying to avoid the reporter 
method but that would be more end to end.


The documentation for metrics and Scala are missing with the 
exception of Gauge: 
https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html 
. Should I file a issue against that?


Then it leaves you guessing a little bit how to implement Counters. 
One approach tried was using objects


object PointFilterextends RichMapFunction[...
   @transient lazy val someCounter = 
getRuntimeContext.getMetricGroup.counter(...)

This allowed access to the counter before and after execution . 
However between the unit tests the Counter kept its value also and 
that's a no for the test. Think that might be an issue with ScalaTest.


I've tried to get at the counter from some other directions like 
trying to find a way to inject a reporter to get it's state. But 
don't see a way to do it. So probably the best thing to do is fire up 
something to collect the metrics from the reporter.


On Thu, Oct 12, 2017 at 5:29 AM, Chesnay Schepler > wrote:


Well damn, i should've read the second part of the initial mail.

I'm wondering though, could you not unit-test this behavior?


On 12.10.2017 14:25, Chesnay Schepler wrote:

You could also write a custom reporter that opens a socket or
similar for communication purposes.

You can then either query it for the metrics, or even just
trigger the verification in the reporter,
and fail with an error if the reporter returns an error.

On 12.10.2017 14:02, Piotr Nowojski wrote:

Hi,

Doing as you proposed using JMXReporter (or custom
reporter) should work. I think there is no easier way to
do this at the moment.

Piotrek

On 12 Oct 2017, at 04:58, Colin Williams
mailto:colin.williams.seat...@gmail.com>> wrote:

I have a RichMapFunction and I'd like to ensure Meter
fields are properly incremented. I've been trying to
think of the best way to do this. Currently I think
that I'd need to either implement my own reporter (or
use JMX) and write to a socket, create a listener and
wait for the reporter to send the message.

Is this a good approach for writing the test, or
should I be considering something else?

Re: Writing an Integration test for flink-metrics

[Piotr Nowojski]

For testing Link applications in general you can read 
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html
 


However as we said before, testing metrics would require using custom or a imx 
reporter.

Yes, please report this bug in Jira. 

Thanks, Piotrek

> On 13 Oct 2017, at 04:31, Colin Williams  
> wrote:
> 
> Team wants an integration test, I'm not sure what unit test you had in mind. 
> Actually feel that I've been trying to avoid the reporter method but that 
> would be more end to end.
> 
> The documentation for metrics and Scala are missing with the exception of 
> Gauge: 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html
>  
> 
>  . Should I file a issue against that?
> 
> Then it leaves you guessing a little bit how to implement Counters. One 
> approach tried was using objects
> 
> object PointFilter extends RichMapFunction[...
>   @transient lazy val someCounter = 
> getRuntimeContext.getMetricGroup.counter(...)
> 
> This allowed access to the counter before and after execution . However 
> between the unit tests the Counter kept its value also and that's a no for 
> the test. Think that might be an issue with ScalaTest. 
> 
> I've tried to get at the counter from some other directions like trying to 
> find a way to inject a reporter to get it's state. But don't see a way to do 
> it. So probably the best thing to do is fire up something to collect the 
> metrics from the reporter.
> 
> On Thu, Oct 12, 2017 at 5:29 AM, Chesnay Schepler  > wrote:
> Well damn, i should've read the second part of the initial mail.
> 
> I'm wondering though, could you not unit-test this behavior?
> 
> 
> On 12.10.2017 14:25, Chesnay Schepler wrote:
> You could also write a custom reporter that opens a socket or similar for 
> communication purposes.
> 
> You can then either query it for the metrics, or even just trigger the 
> verification in the reporter,
> and fail with an error if the reporter returns an error.
> 
> On 12.10.2017 14:02, Piotr Nowojski wrote:
> Hi,
> 
> Doing as you proposed using JMXReporter (or custom reporter) should work. I 
> think there is no easier way to do this at the moment.
> 
> Piotrek
> 
> On 12 Oct 2017, at 04:58, Colin Williams  > wrote:
> 
> I have a RichMapFunction and I'd like to ensure Meter fields are properly 
> incremented. I've been trying to think of the best way to do this. Currently 
> I think that I'd need to either implement my own reporter (or use JMX) and 
> write to a socket, create a listener and wait for the reporter to send the 
> message.
> 
> Is this a good approach for writing the test, or should I be considering 
> something else?
> 
> 
> 
> 
> 

Re: Writing an Integration test for flink-metrics

[Colin Williams]

Team wants an integration test, I'm not sure what unit test you had in
mind. Actually feel that I've been trying to avoid the reporter method but
that would be more end to end.

The documentation for metrics and Scala are missing with the exception of
Gauge:
https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html
. Should I file a issue against that?

Then it leaves you guessing a little bit how to implement Counters. One
approach tried was using objects

object PointFilter extends RichMapFunction[...

  @transient lazy val someCounter =
getRuntimeContext.getMetricGroup.counter(...)


This allowed access to the counter before and after execution . However
between the unit tests the Counter kept its value also and that's a no for
the test. Think that might be an issue with ScalaTest.

I've tried to get at the counter from some other directions like trying to
find a way to inject a reporter to get it's state. But don't see a way to
do it. So probably the best thing to do is fire up something to collect the
metrics from the reporter.

On Thu, Oct 12, 2017 at 5:29 AM, Chesnay Schepler 
wrote:

> Well damn, i should've read the second part of the initial mail.
>
> I'm wondering though, could you not unit-test this behavior?
>
>
> On 12.10.2017 14:25, Chesnay Schepler wrote:
>
>> You could also write a custom reporter that opens a socket or similar for
>> communication purposes.
>>
>> You can then either query it for the metrics, or even just trigger the
>> verification in the reporter,
>> and fail with an error if the reporter returns an error.
>>
>> On 12.10.2017 14:02, Piotr Nowojski wrote:
>>
>>> Hi,
>>>
>>> Doing as you proposed using JMXReporter (or custom reporter) should
>>> work. I think there is no easier way to do this at the moment.
>>>
>>> Piotrek
>>>
>>> On 12 Oct 2017, at 04:58, Colin Williams >>> com> wrote:

 I have a RichMapFunction and I'd like to ensure Meter fields are
 properly incremented. I've been trying to think of the best way to do this.
 Currently I think that I'd need to either implement my own reporter (or use
 JMX) and write to a socket, create a listener and wait for the reporter to
 send the message.

 Is this a good approach for writing the test, or should I be
 considering something else?

>>>
>>>
>>
>>
>

Re: Writing an Integration test for flink-metrics

[Chesnay Schepler]

Well damn, i should've read the second part of the initial mail.

I'm wondering though, could you not unit-test this behavior?

On 12.10.2017 14:25, Chesnay Schepler wrote:
You could also write a custom reporter that opens a socket or similar 
for communication purposes.


You can then either query it for the metrics, or even just trigger the 
verification in the reporter,

and fail with an error if the reporter returns an error.

On 12.10.2017 14:02, Piotr Nowojski wrote:

Hi,

Doing as you proposed using JMXReporter (or custom reporter) should 
work. I think there is no easier way to do this at the moment.


Piotrek

On 12 Oct 2017, at 04:58, Colin Williams 
 wrote:


I have a RichMapFunction and I'd like to ensure Meter fields are 
properly incremented. I've been trying to think of the best way to 
do this. Currently I think that I'd need to either implement my own 
reporter (or use JMX) and write to a socket, create a listener and 
wait for the reporter to send the message.


Is this a good approach for writing the test, or should I be 
considering something else?

Re: Writing an Integration test for flink-metrics

[Chesnay Schepler]

You could also write a custom reporter that opens a socket or similar 
for communication purposes.


You can then either query it for the metrics, or even just trigger the 
verification in the reporter,

and fail with an error if the reporter returns an error.

On 12.10.2017 14:02, Piotr Nowojski wrote:

Hi,

Doing as you proposed using JMXReporter (or custom reporter) should work. I 
think there is no easier way to do this at the moment.

Piotrek


On 12 Oct 2017, at 04:58, Colin Williams  
wrote:

I have a RichMapFunction and I'd like to ensure Meter fields are properly 
incremented. I've been trying to think of the best way to do this. Currently I 
think that I'd need to either implement my own reporter (or use JMX) and write 
to a socket, create a listener and wait for the reporter to send the message.

Is this a good approach for writing the test, or should I be considering 
something else?

Re: Writing an Integration test for flink-metrics

[Piotr Nowojski]

Hi,

Doing as you proposed using JMXReporter (or custom reporter) should work. I 
think there is no easier way to do this at the moment.

Piotrek

> On 12 Oct 2017, at 04:58, Colin Williams  
> wrote:
> 
> I have a RichMapFunction and I'd like to ensure Meter fields are properly 
> incremented. I've been trying to think of the best way to do this. Currently 
> I think that I'd need to either implement my own reporter (or use JMX) and 
> write to a socket, create a listener and wait for the reporter to send the 
> message.
> 
> Is this a good approach for writing the test, or should I be considering 
> something else?