[
https://issues.apache.org/jira/browse/NIP-45?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Secules updated NIP-45:
----------------------------
Description:
h1. Add Optional Counter / Gauge Labels to NiFi API
h2. Motivation:
NiFi allows developers to write their own ComponentMetricReporter which can be
used to report existing NiFi counters and gauges to an external service.
Oftentimes the NiFi counters and gauges will end up in prometheus or some other
metrics storage which supports adding key/value attributes to each metric which
allow a users of the metrics storage and reporting layer (Grafana for example)
to create dashboards that aggregate based on these attributes.
Currently there is a usability gap if a Processor developer wants to record a
counter or gauge with one more more attributes related to the domain of the
processor. They would need to encode them in the counter or gauge name and then
write a companion ComponentMetricReporter which understands how to parse the
base metrics name and attributes out.
Concretely, the ConsumeKafka processor has several counters it records per
topic and best practices for reporting metics would dictate that the metric
name would be something like `consumekafka_records_acknowledged` with an
attribute for `topic` but instead the counter's name according to the UI is
`Records Acknowledged for <topic>`. Without framework support for metric
attributes it is impossible to develop a generic ComponentMetricReporter for a
metrics sink that accurately reports all metric attributes from all processors.
h2. Scope
Limited to adding new implementations of `recordGauge` and `adjustCounter` and
adding the attribute map to `GaugeRecord` and `CounterRecord` records
h2. Compatibility
Fully backwards compatible.
h2. Description
We will add 4-argument variants of `recordGauge` and `adjustCounter` which take
a `Map<String, String>`
`recordGauge` and `adjustCounter` with 3 arguments will simply call their 4
argument counterpart with an empty map.
The attribute map will feed into the `GaugeRecord` and `CounterRecord` records
where ComponentMetricReporters can use it.
Each metric series in NiFi will need to be shown in the NiFi counters UI
including both the base name and the key/value pairs of attributes.
h2. Verification
Add test coverage in `ComponentMetricReporterIT` for updating metrics that have
attributes. The test should validate that different attribute values are
correctly recorded.
h2. Alternatives
A workaround exists if a developer is willing to write a bespoke
`ComponentMetricsReporter` which parses the attribute key/value pairs out of
the metric names.
was:
h1. Add Optional Counter / Gauge Labels to NiFi API
h2. Motivation:
NiFi allows developers to write their own ComponentMetricReporter which can be
used to report existing NiFi counters and gauges to an external service.
Oftentimes the NiFi counters and gauges will end up in prometheus or some other
metrics storage which supports adding key/value attributes to each metric which
allow a users of the metrics storage and reporting layer (Grafana for example)
to create dashboards that aggregate based on these attributes.
Currently there is a usability gap if a Processor developer wants to record a
counter or gauge with one more more attributes related to the domain of the
processor. They would need to encode them in the counter or gauge name and then
write a companion ComponentMetricReporter which understands how to parse the
base metrics name and attributes out.
Concretely, the ConsumeKafka processor has several counters it records per
topic and best practices for reporting metics would dictate that the metric
name would be something like `consumekafka_records_acknowledged` with an
attribute for `topic` but instead the counter's name according to the UI is
`Records Acknowledged for <topic>`. Without framework support for metric
attributes it is impossible to develop a generic ComponentMetricReporter for a
metrics sink that accurately reports all metric attributes from all processors.
h2. Scope
Limited to adding new implementations of `recordGauge` and `adjustCounter` and
adding the attribute map to `GaugeRecord` and `CounterRecord` records
h2. Compatibility
Fully backwards compatible.
h2. Description
We will add 4-argument variants of `recordGauge` and `adjustCounter` which take
a `Map<String, String>`
`recordGauge` and `adjustCounter` with 3 arguments will simply call their 4
argument counterpart with an empty map.
The attribute map will feed into the `GaugeRecord` and `CounterRecord` records
where ComponentMetricReporters can use it.
h2. Verification
Add test coverage in `ComponentMetricReporterIT` for updating metrics that have
attributes. The test should validate that different attribute values are
correctly recorded.
h2. Alternatives
A workaround exists if a developer is willing to write a bespoke
`ComponentMetricsReporter` which parses the attribute key/value pairs out of
the metric names.
> Add Optional Counter / Gauge Labels to NiFi API
> -----------------------------------------------
>
> Key: NIP-45
> URL: https://issues.apache.org/jira/browse/NIP-45
> Project: NiFi Improvement Proposal
> Issue Type: Improvement
> Reporter: Eric Secules
> Priority: Major
>
> h1. Add Optional Counter / Gauge Labels to NiFi API
> h2. Motivation:
> NiFi allows developers to write their own ComponentMetricReporter which can
> be used to report existing NiFi counters and gauges to an external service.
> Oftentimes the NiFi counters and gauges will end up in prometheus or some
> other metrics storage which supports adding key/value attributes to each
> metric which allow a users of the metrics storage and reporting layer
> (Grafana for example) to create dashboards that aggregate based on these
> attributes.
> Currently there is a usability gap if a Processor developer wants to record a
> counter or gauge with one more more attributes related to the domain of the
> processor. They would need to encode them in the counter or gauge name and
> then write a companion ComponentMetricReporter which understands how to parse
> the base metrics name and attributes out.
> Concretely, the ConsumeKafka processor has several counters it records per
> topic and best practices for reporting metics would dictate that the metric
> name would be something like `consumekafka_records_acknowledged` with an
> attribute for `topic` but instead the counter's name according to the UI is
> `Records Acknowledged for <topic>`. Without framework support for metric
> attributes it is impossible to develop a generic ComponentMetricReporter for
> a metrics sink that accurately reports all metric attributes from all
> processors.
> h2. Scope
> Limited to adding new implementations of `recordGauge` and `adjustCounter`
> and adding the attribute map to `GaugeRecord` and `CounterRecord` records
> h2. Compatibility
> Fully backwards compatible.
> h2. Description
> We will add 4-argument variants of `recordGauge` and `adjustCounter` which
> take a `Map<String, String>`
> `recordGauge` and `adjustCounter` with 3 arguments will simply call their 4
> argument counterpart with an empty map.
> The attribute map will feed into the `GaugeRecord` and `CounterRecord`
> records where ComponentMetricReporters can use it.
> Each metric series in NiFi will need to be shown in the NiFi counters UI
> including both the base name and the key/value pairs of attributes.
> h2. Verification
> Add test coverage in `ComponentMetricReporterIT` for updating metrics that
> have attributes. The test should validate that different attribute values are
> correctly recorded.
> h2. Alternatives
> A workaround exists if a developer is willing to write a bespoke
> `ComponentMetricsReporter` which parses the attribute key/value pairs out of
> the metric names.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
