Koji Kawamura created NIFI-1890:
-----------------------------------
Summary: Add JSON log format support for
ControllerStatusReportingTask
Key: NIFI-1890
URL: https://issues.apache.org/jira/browse/NIFI-1890
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Koji Kawamura
ControllerStatusReportingTask can report the 5-minute status for Processors and
Connections to log files, those are informative from monitoring and alerting
system perspective. However, current log format spreads across multiple lines,
and separates columns with '|', it can be more user-friendly to integrate with
3rd party monitoring tools.
For example, if a user would like to setup monitoring & alerting mechanism on
"FlowFiles Queued" column in this log format, they has to first grep the target
log message lines, then using complex regular expression to extract target
values.
If ControllerStatusReportingTask has a option to write these statistics in JSON
format to a single log message, it would be much easier to parse log message,
extract required fields and pass those into other 3rd party monitoring tools,
such as [Logstash|https://www.elastic.co/products/logstash] and
[Kibana|https://www.elastic.co/products/kibana].
Below is the current log format for reference:
{code}
2016-05-17 09:08:23,254 INFO [Timer-Driven Process Thread-6]
o.a.n.c.C.Processors Processor Statuses:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Processor Name | Processor ID |
Processor Type | Run Status | Flow
Files In | Flow Files Out |
Bytes Read | Bytes Written | Tasks |
Proc Time |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| GenerateFlowFile | cff43e4a-2b0c-4221-bdab-fbeb1400b94b |
GenerateFlowFile | Running | 0 / 0 bytes (+0/+0
bytes) | 100 / 100 bytes (+100/+100 bytes) | 0 bytes (+0
bytes) | 100 bytes (+100 bytes) | 100 (+100) | 00:00:00.131
(+00:00:00.131) |
| LogAttribute | bbf54a49-a948-4a17-a5fd-de95a2898bd5 |
LogAttribute | Running | 0 / 0 bytes (+0/+0
bytes) | 0 / 0 bytes (+0/+0 bytes) | 0 bytes (+0
bytes) | 0 bytes (+0 bytes) | 0 (+0) | 00:00:00.000
(+00:00:00.000) |
| UpdateAttribute | e960a719-bcf9-4bdf-9122-6438548becc6 |
UpdateAttribute | Stopped | 0 / 0 bytes (+0/+0
bytes) | 0 / 0 bytes (+0/+0 bytes) | 0 bytes (+0
bytes) | 0 bytes (+0 bytes) | 0 (+0) | 00:00:00.000
(+00:00:00.000) |
| LogAttribute | 4677d849-3042-48c7-8872-96b75eb81664 |
LogAttribute | Running | 0 / 0 bytes (+0/+0
bytes) | 0 / 0 bytes (+0/+0 bytes) | 0 bytes (+0
bytes) | 0 bytes (+0 bytes) | 0 (+0) | 00:00:00.000
(+00:00:00.000) |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2016-05-17 09:08:23,255 INFO [Timer-Driven Process Thread-6]
o.a.n.c.C.Connections Connection Statuses:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Connection ID | Source |
Connection Name | Destination |
Flow Files In | Flow Files
Out | FlowFiles Queued |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| 3bf440da-a2aa-4581-87f8-984efe709631 | GenerateFlowFile |
success | UpdateAttribute |
100 / 100 bytes (+100/+100 bytes) | 0 / 0 bytes (+0/+0
bytes) | 10748 / 10.5 KB (+10748/+10.5 KB) |
| 7318685f-bf3e-4ec8-8507-6aacbaf9ef52 | UpdateAttribute |
success | output-1 |
0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0
bytes) | 9 / 9 bytes (+9/+9 bytes) |
| ee097acf-1688-4e08-a638-1a144c5df465 | output-1 |
| LogAttribute |
0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
0 / 0 bytes (+0/+0 bytes) |
| a98e015e-73b6-4ddb-9983-3c9a99e68f94 | input-1 |
| LogAttribute |
0 / 0 bytes (+0/+0 bytes) | 0 / 0 bytes (+0/+0 bytes) |
0 / 0 bytes (+0/+0 bytes) |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
