Re: [Feature Proposal] Add metrics dropwizards like gauges, meters, histogram etc to Apex Platform

[Deepak Narkhede]

Thanks all for the suggestions. I'll incorporate all the
suggestions/questions by each one of you and come up with concrete design
with multiple phases/staging.
   - Deepak

On Thu, May 10, 2018 at 8:22 AM, Ananth G  wrote:

> +1 for the feature.
>
> +1 for an abstracted way of metrics library integration.
>
> Some additional thoughts on this:
>
> - We might have implications of adding a set of dependencies from drop
> wizard into the engine core ( guava versions etc).
> - Flink as pointed out by Thomas seems to have taken an interesting
> approach to metrics ( apart from supporting multiple reporters ) - Any
> reporter is only enabled if the relevant jar is put in the classpath.
> - There seems to be two worlds for generating the metric names. 1. Dot
> notation separated ( or some separator thereof. Ex: dropwizard style) and
> 2. high dimensional metric names using key value pair tags.  ( prometheus
> style). We might have to "transform" the dot hierarchical notation to a key
> value pair based notation based on the reporter that is chosen by the end
> user.
> - The scope of the implementation is too big as I understand it and perhaps
> this feature needs to be done in multiple JIRAs. Also malhar is still at
> java-7 while engine is at java-8.
>
> Some questions:
>
> - Is there a plan to expose the metric via a jetty end point on each JVM ?
> ( if configured )
> - How do we plan to handle dynamic partitioning of operators for metrics ?
> ( i.e possibly short-lived JVMs )
> - This feature implies that we will no longer support equivalent of
> ComplexType of Autometrics ?
> - This feature also implies that we will no longer support autometric
> aggregators ? ( and leave this to the metric tools functionality)
>
>
> Regards,
> Ananth
>
> On Thu, May 10, 2018 at 12:37 PM, Chinmay Kolhatkar <
> chinmaykolhatka...@gmail.com> wrote:
>
> > +1 for approach 1. As Thomas mentioned, I think there metrics layer
> should
> > be abstracted out from its implementation. This way one can plugin
> > different metrics systems in apex.
> >
> > Also keep in mind that there is a lot of code which uses AutoMetrics
> > annotations. We should help a smooth transition from that. As next
> release
> > will be a major version release, this is a good opportunity for getting
> rid
> > of old AutoMetrics and counters functionality.
> >
> > Regards,
> > Chinmay.
> >
> >
> > On Thu, 10 May 2018, 4:47 am Vlad Rozov,  wrote:
> >
> > > +1 for the #1 proposal.
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > > On 5/9/18 07:14, Thomas Weise wrote:
> > > > +1 for the initiative
> > > >
> > > > Some thoughts:
> > > >
> > > > - it is probably good to retain a level of abstraction, to avoid
> direct
> > > > dependency on dropwizard
> > > > - support programmatic metric creation (not just annotations)
> > > > - remove deprecated counter and auto-metric code and migrate
> operators
> > to
> > > > use new API
> > > > - which metric reporting systems will be supported out of the box
> > > >
> > > > You can also take a look at how this was structured in Flink:
> > > >
> > > https://ci.apache.org/projects/flink/flink-docs-
> release-1.4/monitoring/
> > metrics.html
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > >
> > > > On Tue, May 8, 2018 at 8:56 AM, Deepak Narkhede <
> > mailtodeep...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hi Community,
> > > >>
> > > >> I want to propose addition of metrics like gauges, meters, counters
> > and
> > > >> historgram for the following components.
> > > >> 1) Addition of metrics for Container Stats.
> > > >> 2) Addition of metrics for Operator Stats.
> > > >> 3) Addition of  metrics for  Stram Application Master stats.
> > > >> 4) Addition of metrics for JVM related stats for all containers.
> > > >>
> > > >> To implement them would be using metrics dropwizard api's. (
> > > >> http://metrics.dropwizard.io/)
> > > >> Use cases:
> > > >> 1) Can be directly pushed to external visualisation system like
> > > Graphite.
> > > >> 2) Can be viewed in visualVM tools through JMX.
> > > >> 3) Can be outputted to console.
> > > >> 4) It is also possible to push the metrics to custom sink.
> > > >>
> > > >> We will also need to write sinks and reporter, if required for
> custom
> > > >> sinks.
> > > >>
> > > >> Design/Implementation approach:
> > > >> Way #1:
> > > >> 1) Create new annotations like @MetricTypeGauge, @MetricTypeMeter,
> > > >> @MetricTypeCounter, @MetricTypeHistogram. They can be both fields
> and
> > > >> methods.
> > > >> 2) Add them to respective methods or fields like StreamingContainer,
> > > >> StreamingAppMasterService for extraction of relevant metrics.
> > > >> 3) While Node creation ( InputNode/GeneticNode/OiONode), we create
> > and
> > > >> initialise  the metrics registry depending on components.
> > > >> 4) While collectMetrics() part of operator runner thread (
> > InputNode.run
> > > >> /GenericNode.run), we actually invoke the annotations 

Re: [Feature Proposal] Add metrics dropwizards like gauges, meters, histogram etc to Apex Platform

[Ananth G]

+1 for the feature.

+1 for an abstracted way of metrics library integration.

Some additional thoughts on this:

- We might have implications of adding a set of dependencies from drop
wizard into the engine core ( guava versions etc).
- Flink as pointed out by Thomas seems to have taken an interesting
approach to metrics ( apart from supporting multiple reporters ) - Any
reporter is only enabled if the relevant jar is put in the classpath.
- There seems to be two worlds for generating the metric names. 1. Dot
notation separated ( or some separator thereof. Ex: dropwizard style) and
2. high dimensional metric names using key value pair tags.  ( prometheus
style). We might have to "transform" the dot hierarchical notation to a key
value pair based notation based on the reporter that is chosen by the end
user.
- The scope of the implementation is too big as I understand it and perhaps
this feature needs to be done in multiple JIRAs. Also malhar is still at
java-7 while engine is at java-8.

Some questions:

- Is there a plan to expose the metric via a jetty end point on each JVM ?
( if configured )
- How do we plan to handle dynamic partitioning of operators for metrics ?
( i.e possibly short-lived JVMs )
- This feature implies that we will no longer support equivalent of
ComplexType of Autometrics ?
- This feature also implies that we will no longer support autometric
aggregators ? ( and leave this to the metric tools functionality)


Regards,
Ananth

On Thu, May 10, 2018 at 12:37 PM, Chinmay Kolhatkar <
chinmaykolhatka...@gmail.com> wrote:

> +1 for approach 1. As Thomas mentioned, I think there metrics layer should
> be abstracted out from its implementation. This way one can plugin
> different metrics systems in apex.
>
> Also keep in mind that there is a lot of code which uses AutoMetrics
> annotations. We should help a smooth transition from that. As next release
> will be a major version release, this is a good opportunity for getting rid
> of old AutoMetrics and counters functionality.
>
> Regards,
> Chinmay.
>
>
> On Thu, 10 May 2018, 4:47 am Vlad Rozov,  wrote:
>
> > +1 for the #1 proposal.
> >
> > Thank you,
> >
> > Vlad
> >
> > On 5/9/18 07:14, Thomas Weise wrote:
> > > +1 for the initiative
> > >
> > > Some thoughts:
> > >
> > > - it is probably good to retain a level of abstraction, to avoid direct
> > > dependency on dropwizard
> > > - support programmatic metric creation (not just annotations)
> > > - remove deprecated counter and auto-metric code and migrate operators
> to
> > > use new API
> > > - which metric reporting systems will be supported out of the box
> > >
> > > You can also take a look at how this was structured in Flink:
> > >
> > https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/
> metrics.html
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > > On Tue, May 8, 2018 at 8:56 AM, Deepak Narkhede <
> mailtodeep...@gmail.com
> > >
> > > wrote:
> > >
> > >> Hi Community,
> > >>
> > >> I want to propose addition of metrics like gauges, meters, counters
> and
> > >> historgram for the following components.
> > >> 1) Addition of metrics for Container Stats.
> > >> 2) Addition of metrics for Operator Stats.
> > >> 3) Addition of  metrics for  Stram Application Master stats.
> > >> 4) Addition of metrics for JVM related stats for all containers.
> > >>
> > >> To implement them would be using metrics dropwizard api's. (
> > >> http://metrics.dropwizard.io/)
> > >> Use cases:
> > >> 1) Can be directly pushed to external visualisation system like
> > Graphite.
> > >> 2) Can be viewed in visualVM tools through JMX.
> > >> 3) Can be outputted to console.
> > >> 4) It is also possible to push the metrics to custom sink.
> > >>
> > >> We will also need to write sinks and reporter, if required for custom
> > >> sinks.
> > >>
> > >> Design/Implementation approach:
> > >> Way #1:
> > >> 1) Create new annotations like @MetricTypeGauge, @MetricTypeMeter,
> > >> @MetricTypeCounter, @MetricTypeHistogram. They can be both fields and
> > >> methods.
> > >> 2) Add them to respective methods or fields like StreamingContainer,
> > >> StreamingAppMasterService for extraction of relevant metrics.
> > >> 3) While Node creation ( InputNode/GeneticNode/OiONode), we create
> and
> > >> initialise  the metrics registry depending on components.
> > >> 4) While collectMetrics() part of operator runner thread (
> InputNode.run
> > >> /GenericNode.run), we actually invoke the annotations methods and
> > collect
> > >> different types of metrics.
> > >> 5) We can have a sink which pushes the metrics to reporter like
> Console,
> > >> JMX etc.
> > >>
> > >> Way #2:
> > >> Use existing AutoMetrics annotations, convert some metrics to
> different
> > >> types like gauge, counter etc..But this cannot be done generically as
> we
> > >> don't know the types. Still more investigation is going on this
> > approach.
> > >>
> > >> I would prefer first way.
> > >>
> > >> Note: There are 

Re: [Feature Proposal] Add metrics dropwizards like gauges, meters, histogram etc to Apex Platform

[Chinmay Kolhatkar]

+1 for approach 1. As Thomas mentioned, I think there metrics layer should
be abstracted out from its implementation. This way one can plugin
different metrics systems in apex.

Also keep in mind that there is a lot of code which uses AutoMetrics
annotations. We should help a smooth transition from that. As next release
will be a major version release, this is a good opportunity for getting rid
of old AutoMetrics and counters functionality.

Regards,
Chinmay.


On Thu, 10 May 2018, 4:47 am Vlad Rozov,  wrote:

> +1 for the #1 proposal.
>
> Thank you,
>
> Vlad
>
> On 5/9/18 07:14, Thomas Weise wrote:
> > +1 for the initiative
> >
> > Some thoughts:
> >
> > - it is probably good to retain a level of abstraction, to avoid direct
> > dependency on dropwizard
> > - support programmatic metric creation (not just annotations)
> > - remove deprecated counter and auto-metric code and migrate operators to
> > use new API
> > - which metric reporting systems will be supported out of the box
> >
> > You can also take a look at how this was structured in Flink:
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html
> >
> > Thanks,
> > Thomas
> >
> >
> > On Tue, May 8, 2018 at 8:56 AM, Deepak Narkhede  >
> > wrote:
> >
> >> Hi Community,
> >>
> >> I want to propose addition of metrics like gauges, meters, counters and
> >> historgram for the following components.
> >> 1) Addition of metrics for Container Stats.
> >> 2) Addition of metrics for Operator Stats.
> >> 3) Addition of  metrics for  Stram Application Master stats.
> >> 4) Addition of metrics for JVM related stats for all containers.
> >>
> >> To implement them would be using metrics dropwizard api's. (
> >> http://metrics.dropwizard.io/)
> >> Use cases:
> >> 1) Can be directly pushed to external visualisation system like
> Graphite.
> >> 2) Can be viewed in visualVM tools through JMX.
> >> 3) Can be outputted to console.
> >> 4) It is also possible to push the metrics to custom sink.
> >>
> >> We will also need to write sinks and reporter, if required for custom
> >> sinks.
> >>
> >> Design/Implementation approach:
> >> Way #1:
> >> 1) Create new annotations like @MetricTypeGauge, @MetricTypeMeter,
> >> @MetricTypeCounter, @MetricTypeHistogram. They can be both fields and
> >> methods.
> >> 2) Add them to respective methods or fields like StreamingContainer,
> >> StreamingAppMasterService for extraction of relevant metrics.
> >> 3) While Node creation ( InputNode/GeneticNode/OiONode), we create and
> >> initialise  the metrics registry depending on components.
> >> 4) While collectMetrics() part of operator runner thread ( InputNode.run
> >> /GenericNode.run), we actually invoke the annotations methods and
> collect
> >> different types of metrics.
> >> 5) We can have a sink which pushes the metrics to reporter like Console,
> >> JMX etc.
> >>
> >> Way #2:
> >> Use existing AutoMetrics annotations, convert some metrics to different
> >> types like gauge, counter etc..But this cannot be done generically as we
> >> don't know the types. Still more investigation is going on this
> approach.
> >>
> >> I would prefer first way.
> >>
> >> Note: There are some complications, if two operators are deployed on
> same
> >> jvm conatiner. But  I think it can be resolved by creating two different
> >> metrics registry with unique id from JVM.
> >>
> >> Let me know your thoughts on this.
> >>
> >> Thanks,
> >> Deepak
> >>
>
>

Re: [Feature Proposal] Add metrics dropwizards like gauges, meters, histogram etc to Apex Platform

[Vlad Rozov]

+1 for the #1 proposal.

Thank you,

Vlad

On 5/9/18 07:14, Thomas Weise wrote:

+1 for the initiative

Some thoughts:

- it is probably good to retain a level of abstraction, to avoid direct
dependency on dropwizard
- support programmatic metric creation (not just annotations)
- remove deprecated counter and auto-metric code and migrate operators to
use new API
- which metric reporting systems will be supported out of the box

You can also take a look at how this was structured in Flink:
https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html

Thanks,
Thomas


On Tue, May 8, 2018 at 8:56 AM, Deepak Narkhede 
wrote:


Hi Community,

I want to propose addition of metrics like gauges, meters, counters and
historgram for the following components.
1) Addition of metrics for Container Stats.
2) Addition of metrics for Operator Stats.
3) Addition of  metrics for  Stram Application Master stats.
4) Addition of metrics for JVM related stats for all containers.

To implement them would be using metrics dropwizard api's. (
http://metrics.dropwizard.io/)
Use cases:
1) Can be directly pushed to external visualisation system like Graphite.
2) Can be viewed in visualVM tools through JMX.
3) Can be outputted to console.
4) It is also possible to push the metrics to custom sink.

We will also need to write sinks and reporter, if required for custom
sinks.

Design/Implementation approach:
Way #1:
1) Create new annotations like @MetricTypeGauge, @MetricTypeMeter,
@MetricTypeCounter, @MetricTypeHistogram. They can be both fields and
methods.
2) Add them to respective methods or fields like StreamingContainer,
StreamingAppMasterService for extraction of relevant metrics.
3) While Node creation ( InputNode/GeneticNode/OiONode), we create and
initialise  the metrics registry depending on components.
4) While collectMetrics() part of operator runner thread ( InputNode.run
/GenericNode.run), we actually invoke the annotations methods and collect
different types of metrics.
5) We can have a sink which pushes the metrics to reporter like Console,
JMX etc.

Way #2:
Use existing AutoMetrics annotations, convert some metrics to different
types like gauge, counter etc..But this cannot be done generically as we
don't know the types. Still more investigation is going on this approach.

I would prefer first way.

Note: There are some complications, if two operators are deployed on same
jvm conatiner. But  I think it can be resolved by creating two different
metrics registry with unique id from JVM.

Let me know your thoughts on this.

Thanks,
Deepak

Re: [Feature Proposal] Add metrics dropwizards like gauges, meters, histogram etc to Apex Platform

[Thomas Weise]

+1 for the initiative

Some thoughts:

- it is probably good to retain a level of abstraction, to avoid direct
dependency on dropwizard
- support programmatic metric creation (not just annotations)
- remove deprecated counter and auto-metric code and migrate operators to
use new API
- which metric reporting systems will be supported out of the box

You can also take a look at how this was structured in Flink:
https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html

Thanks,
Thomas


On Tue, May 8, 2018 at 8:56 AM, Deepak Narkhede 
wrote:

> Hi Community,
>
> I want to propose addition of metrics like gauges, meters, counters and
> historgram for the following components.
> 1) Addition of metrics for Container Stats.
> 2) Addition of metrics for Operator Stats.
> 3) Addition of  metrics for  Stram Application Master stats.
> 4) Addition of metrics for JVM related stats for all containers.
>
> To implement them would be using metrics dropwizard api's. (
> http://metrics.dropwizard.io/)
> Use cases:
> 1) Can be directly pushed to external visualisation system like Graphite.
> 2) Can be viewed in visualVM tools through JMX.
> 3) Can be outputted to console.
> 4) It is also possible to push the metrics to custom sink.
>
> We will also need to write sinks and reporter, if required for custom
> sinks.
>
> Design/Implementation approach:
> Way #1:
> 1) Create new annotations like @MetricTypeGauge, @MetricTypeMeter,
> @MetricTypeCounter, @MetricTypeHistogram. They can be both fields and
> methods.
> 2) Add them to respective methods or fields like StreamingContainer,
> StreamingAppMasterService for extraction of relevant metrics.
> 3) While Node creation ( InputNode/GeneticNode/OiONode), we create and
> initialise  the metrics registry depending on components.
> 4) While collectMetrics() part of operator runner thread ( InputNode.run
> /GenericNode.run), we actually invoke the annotations methods and collect
> different types of metrics.
> 5) We can have a sink which pushes the metrics to reporter like Console,
> JMX etc.
>
> Way #2:
> Use existing AutoMetrics annotations, convert some metrics to different
> types like gauge, counter etc..But this cannot be done generically as we
> don't know the types. Still more investigation is going on this approach.
>
> I would prefer first way.
>
> Note: There are some complications, if two operators are deployed on same
> jvm conatiner. But  I think it can be resolved by creating two different
> metrics registry with unique id from JVM.
>
> Let me know your thoughts on this.
>
> Thanks,
> Deepak
>