David Handermann created NIFI-16380:
---------------------------------------
Summary: Add Lag and Records Metrics to ConsumeKinesis
Key: NIFI-16380
URL: https://issues.apache.org/jira/browse/NIFI-16380
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: David Handermann
Assignee: David Handermann
Following the pattern of {{ConsumeKafka}}, the {{ConsumeKinesis}} Processor
should track lag and record metrics for each invocation. The Map of attributes
for Counters and Gauges should include the Kinesis Stream and Shard information.
The metrics should include the following:
* kinesis.bytes.consumed
** Type: Counter
* kinesis.records.consumed
** Type: Counter
* kinesis.consumer.behind.ms
** Type: Gauge
* kinesis.records.parsed.errors
** Type: Counter
Kinesis supports the concept of
[MillisBehindLatest|https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html#API_GetRecords_ResponseSyntax]
to indicate the number milliseconds behind the consumer is from the latest
record in the stream. This is a loose indicator of lag because it does not
provide the number of records and interpreting the number requires an
understanding of expected periodicity for records added to the Kinesis Stream.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
