Hello Everyone, Few Months ago, I was looking for a comprehensive solution that can help me monitor our Apache NiFi clusters.
Some of the expectations I had in Mind from this solution were: - It should be customisable, So I can choose which processors , queues, etc. to monitor - I should be able to set thresholds on certain parameters and get email/slack/teams alerts whenever these thresholds are crossed. - If a node is down. I should be notified. - I should be able to see historical data about the state of my cluster, for example how is the disk space filling up, are certain connections under backpressure during specific times etc. - It should be reliable i.e. even if the Cluster goes down, I should get an alert. So something that runs externally and not within NiFi. - It Should support on-prem or air gapped NiFi Clusters. I didn't find anything open source that would help on this and was easy to use. Thanks to the community, I came across Apache NiFi Rest API https://nifi.apache.org/docs/nifi-docs/rest-api/index.html <https://nifi.apache.org/docs/nifi-docs/rest-api/index.html>. Using Data from this Rest API, Over the past few months I have developed & Tested *MonitoFi*. It is supposed to work as a health and performance monitor for Apache NiFi. How does it work? 1. It runs externally in a docker container and collects data periodically from any of the GET endpoints (You can choose which ones) in Apache NiFi Rest API. 2. This Data is Stored in InfluxDB that is running in another container along with Grafana. https://github.com/tushardhadiwal/docker-influxdb-grafana <https://github.com/tushardhadiwal/docker-influxdb-grafana> 3. Grafana is used to showcase Graphs and Manage Alerts etc. It is also possible to store data in Azure if needed for backups etc. If someone wants to try it, I have included a deployment script <http://install.monitofi.com/> that handles end2end deployment of containers, Grafana and InfluxDB Configuration and Imports Grafana Dashboards which include about 10 sample queries tracking various parameters for a 3 node NiFi Cluster. Feel free to open up an issue on github if you have any trouble getting this to run. I am happy to share it with the community and gather some feedback on whether this is something that people find to be useful. I have few more improvements in mind for the near future. *More info is available at http://www.monitofi.com/ <http://www.monitofi.com/>* Let me know what you think?! Cheers, Tushar Dhadiwal <https://www.linkedin.com/in/tushar-dhadiwal/> [image: MonitoFiDashboard.png]
