[
https://issues.apache.org/jira/browse/NIFI-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adela Jaworowska-Nowak updated NIFI-16608:
------------------------------------------
Description:
h3. Background
UpdateCounter records values through ProcessSession.adjustCounter(). Those
values are stored in CounterRepository and are queryable via GET
/nifi-api/counters (reset via PUT /nifi-api/counters and PUT
/nifi-api/counters/\{id\}).
UpdateGauge records values through ProcessSession.recordGauge(). Those values
are forwarded only to ComponentMetricReporter. If no reporter is configured,
DefaultComponentMetricReporter drops the gauge. There is no framework store and
no REST API to read the last recorded value.
Counters and gauges are therefore inconsistent for local inspection, debugging,
and automated tests. Flows that use UpdateCounter can assert counts through GET
/nifi-api/counters. Flows that use UpdateGauge (for example last-ingestion time
or latency) cannot assert the last recorded value the same way: there is no GET
/nifi-api/gauges, so tests can only infer that a gauge processor ran
(provenance / FlowFile counts), not that it stored the expected value.
h3. Proposal
Persist the latest gauge value in a GaugeRepository when recordGauge() is
applied, using the same commit-timing rules as counters (NOW vs
SESSION_COMMITTED). Continue to call ComponentMetricReporter.recordGauge() so
OpenTelemetry (or any other reporter) remains the export path. The store is for
local query, not a replacement for reporting.
h3. Identity
* Counters remain keyed by context + name. Metric attributes/labels are
reporter-only.
* Gauges are keyed by name + attributes/labels, plus the same two contexts as
counters:
{code}
{componentName} ({componentId})
All {componentType}'s
{code}
* Distinct label sets (for example per-table last-ingestion time) must not
overwrite each other.
h3. REST and authorization
Analogous to counters:
{code}
GET /nifi-api/gauges
PUT /nifi-api/gauges
PUT /nifi-api/gauges/{id}
{code}
* PUT with no id resets all values to 0
* PUT with an id resets that gauge to 0
* Authorization resource /gauges (ResourceType.Gauges)
* Toolkit GaugesClient
No Angular listing page is required. The policy type should appear so /gauges
can be granted like /counters.
h3. Cluster
Do not sum gauges. When merging node snapshots, the value with the latest
recorded timestamp wins.
h3. Storage and configuration (must be designed, not copied from Counters)
A Gauge store is not a drop-in CounterRepository. Counters are a small
in-memory map keyed by context + name, lost on restart, with no
nifi.properties. Gauges are last-value series whose identity includes labels,
so cardinality can grow with distinct label combinations (not just processor
count).
Before implementing, decide explicitly:
* Memory vs disk (in-memory last value vs a persisted repository).
* Bound: max entries and/or TTL so dropped label sets (e.g. a table no longer
in the flow) do not live forever.
* Whether that needs new nifi.properties (max size, expiration, persist
directory).
* Cluster: per-node maps plus GET merge is not enough without a defined merge
rule, clock skew, and nodes that never saw a given series.
The existing ComponentMetricReporter / OTel path already has cardinality limits
and expiration. A NiFi GaugeRepository must not ignore those concerns. If the
only consumer is tests, prefer asserting OTel output instead of adding a
framework store.
h3. Out of scope
* Changing processor recordGauge APIs
* Requiring a ComponentMetricReporter implementation in order to read gauges
from a running node
* A UI page equivalent to Counters
was:
h3. Background
UpdateCounter records values through ProcessSession.adjustCounter(). Those
values are stored in CounterRepository and are queryable via GET
/nifi-api/counters (reset via PUT /nifi-api/counters and PUT
/nifi-api/counters/\{id\}).
UpdateGauge records values through ProcessSession.recordGauge(). Those values
are forwarded only to ComponentMetricReporter. If no reporter is configured,
DefaultComponentMetricReporter drops the gauge. There is no framework store and
no REST API to read the last recorded value.
Counters and gauges are therefore inconsistent for local inspection, debugging,
and automated tests. Flows that use UpdateCounter can assert counts through GET
/nifi-api/counters. Flows that use UpdateGauge (for example last-ingestion time
or latency) cannot assert the last recorded value the same way: there is no GET
/nifi-api/gauges, so tests can only infer that a gauge processor ran
(provenance / FlowFile counts), not that it stored the expected value.
h3. Proposal
Persist the latest gauge value in a GaugeRepository when recordGauge() is
applied, using the same commit-timing rules as counters (NOW vs
SESSION_COMMITTED). Continue to call ComponentMetricReporter.recordGauge() so
OpenTelemetry (or any other reporter) remains the export path. The store is for
local query, not a replacement for reporting.
h3. Identity
* Counters remain keyed by context + name. Metric attributes/labels are
reporter-only.
* Gauges are keyed by name + attributes/labels, plus the same two contexts as
counters:
{code}
{componentName} ({componentId})
All {componentType}'s
{code}
* Distinct label sets (for example per-table last-ingestion time) must not
overwrite each other.
h3. REST and authorization
Analogous to counters:
{code}
GET /nifi-api/gauges
PUT /nifi-api/gauges
PUT /nifi-api/gauges/{id}
{code}
* PUT with no id resets all values to 0
* PUT with an id resets that gauge to 0
* Authorization resource /gauges (ResourceType.Gauges)
* Toolkit GaugesClient
No Angular listing page is required. The policy type should appear so /gauges
can be granted like /counters.
h3. Cluster
Do not sum gauges. When merging node snapshots, the value with the latest
recorded timestamp wins.
h3. Out of scope
* Changing processor recordGauge APIs
* Requiring a ComponentMetricReporter implementation in order to read gauges
from a running node
* A UI page equivalent to Counters
> Add GaugeRepository and REST API analogous to Counters
> ------------------------------------------------------
>
> Key: NIFI-16608
> URL: https://issues.apache.org/jira/browse/NIFI-16608
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Adela Jaworowska-Nowak
> Assignee: Adela Jaworowska-Nowak
> Priority: Major
>
> h3. Background
> UpdateCounter records values through ProcessSession.adjustCounter(). Those
> values are stored in CounterRepository and are queryable via GET
> /nifi-api/counters (reset via PUT /nifi-api/counters and PUT
> /nifi-api/counters/\{id\}).
> UpdateGauge records values through ProcessSession.recordGauge(). Those values
> are forwarded only to ComponentMetricReporter. If no reporter is configured,
> DefaultComponentMetricReporter drops the gauge. There is no framework store
> and no REST API to read the last recorded value.
> Counters and gauges are therefore inconsistent for local inspection,
> debugging, and automated tests. Flows that use UpdateCounter can assert
> counts through GET /nifi-api/counters. Flows that use UpdateGauge (for
> example last-ingestion time or latency) cannot assert the last recorded value
> the same way: there is no GET /nifi-api/gauges, so tests can only infer that
> a gauge processor ran (provenance / FlowFile counts), not that it stored the
> expected value.
> h3. Proposal
> Persist the latest gauge value in a GaugeRepository when recordGauge() is
> applied, using the same commit-timing rules as counters (NOW vs
> SESSION_COMMITTED). Continue to call ComponentMetricReporter.recordGauge() so
> OpenTelemetry (or any other reporter) remains the export path. The store is
> for local query, not a replacement for reporting.
> h3. Identity
> * Counters remain keyed by context + name. Metric attributes/labels are
> reporter-only.
> * Gauges are keyed by name + attributes/labels, plus the same two contexts as
> counters:
> {code}
> {componentName} ({componentId})
> All {componentType}'s
> {code}
> * Distinct label sets (for example per-table last-ingestion time) must not
> overwrite each other.
> h3. REST and authorization
> Analogous to counters:
> {code}
> GET /nifi-api/gauges
> PUT /nifi-api/gauges
> PUT /nifi-api/gauges/{id}
> {code}
> * PUT with no id resets all values to 0
> * PUT with an id resets that gauge to 0
> * Authorization resource /gauges (ResourceType.Gauges)
> * Toolkit GaugesClient
> No Angular listing page is required. The policy type should appear so /gauges
> can be granted like /counters.
> h3. Cluster
> Do not sum gauges. When merging node snapshots, the value with the latest
> recorded timestamp wins.
> h3. Storage and configuration (must be designed, not copied from Counters)
> A Gauge store is not a drop-in CounterRepository. Counters are a small
> in-memory map keyed by context + name, lost on restart, with no
> nifi.properties. Gauges are last-value series whose identity includes labels,
> so cardinality can grow with distinct label combinations (not just processor
> count).
> Before implementing, decide explicitly:
> * Memory vs disk (in-memory last value vs a persisted repository).
> * Bound: max entries and/or TTL so dropped label sets (e.g. a table no longer
> in the flow) do not live forever.
> * Whether that needs new nifi.properties (max size, expiration, persist
> directory).
> * Cluster: per-node maps plus GET merge is not enough without a defined merge
> rule, clock skew, and nodes that never saw a given series.
> The existing ComponentMetricReporter / OTel path already has cardinality
> limits and expiration. A NiFi GaugeRepository must not ignore those concerns.
> If the only consumer is tests, prefer asserting OTel output instead of adding
> a framework store.
> h3. Out of scope
> * Changing processor recordGauge APIs
> * Requiring a ComponentMetricReporter implementation in order to read gauges
> from a running node
> * A UI page equivalent to Counters
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)