Theo Diefenthal created NIFI-9273:
-------------------------------------
Summary: Metrics: Allow and provice processor custom metrics |
refactor metric system
Key: NIFI-9273
URL: https://issues.apache.org/jira/browse/NIFI-9273
Project: Apache NiFi
Issue Type: Wish
Components: Core Framework
Reporter: Theo Diefenthal
In my current (beginner) understanding of NiFi, each processor reports the very
same metrics like processing time, received bytes/files, sent bytes/files. No
processor specific metrics exist.
I argue that at least most of the source processors should have their own
custom metrics which would drastically improve the user experience when
operating NiFi on different agents.
An example: Let's say we have a ConsumeKafka Source connected to a PutKafka
Sink mirroring data from a remote kafka to a (nifi-)local kafka cluster. Via
NiFi monitoring, we could find backpressure within NiFi, but we wouldn't see if
the consumer doesn't catch up reading the messages. We'd thus need additionally
tooling to monitor the remote kafka cluster. Same goes e.g. for the GetFile
processor where it would be substantial to know the current number of files in
the input directory in order to know whether some back pressure builds up. Of
course, that doesn't sound like a realistic scenario (Reading from disk would
be slower than writing new files) but image reading from a network drive where
log files are written local but received via the network.
As a processor developer, I'd also like to include my own custom metrics. From
my understanding, I currently can only add deltas to counter types. Gauges thus
can't be maintained.
In many applications, thanks to frameworks like Dropwizard or MicroMeter,
adding and updating metrics is as easy as writing logging statements.
Applications like Spark or Flink don't utilize those libraries but do their
best to provide similar features which let you easily define your own metrics
anywhere you write user-code and which include metrics from all source
components, e.g. when writing to kafka, report all possible
kafka-producer-metrics from the kafkaProducer java library. In NiFi, I'm
missing those metrics.
In NiFi, a ReportingTask gets some "status" classes about each
processor/connection and so on. From that, it builds itself the metrics like
Counters and Gauges it wants to report to an external system. In the systems I
mentioned earlier, the user can write its own reporter similar to NiFi. The
difference is: The reporter doesn't get specific classes like
{{{color:#000000}ProcessorStatus}} but instead gets a metric registry with
counters, gauges and so on which can then be easily exported to an external
system. This way, each processor could add any custom metrics without the need
to implement logic for those new metrics in each ReportingTask.
{color}
{color:#000000}Maybe, the Metric handling in NiFi could be inspired by Apache
Flink: In Flink, each user code class can receive a "MetricGroup". Those metric
groups are kind of pre-configured and properly setup for the task. Let's say in
NiFi, we would provide a MetricGroup to some processor. That group would make
sure that all metrics added to the group are already labeled with the
processor-id and processor-name. And NiFi could as well add all default metrics
for this group beforehand, i.e. for each processor something like timeSpent,
flowfiles received and so on. {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
