Hi Eric, For monitoring, you have following tools: 1. Monitoring in web UI ( https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#monitoring , https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#routing-and-mediation ) 2. Using reporting tasks ( https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks )
As for processing the data: NIFI is not ETL tool. it's a data FLOW management tool. Yes, you have a lot of processors that can extract data and process, make decisions, fork flows based on different attributes or content, etc. But usually your NIFI clusters are tiny, and ETL tools a sitting on much bigger and powerful clusters. But, you obviously can do ETL too, Writing processors explained number of time (e.g. http://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/, https://community.hortonworks.com/articles/4318/build-custom-nifi-processor.html ). Regards, Ed. On Thu, Jul 19, 2018 at 8:59 AM Eric Desjardins <[email protected]> wrote: > Hi, > > I am evaluating nifi to replace a local ETL system. I went through the > video and the doc and it looks great. > > We have a large number of data feeds that ends up in different folder on a > sftp server. I could use a SFTP processor to fetch data from the server. > Than, I could write my own processor to process the data. For this, I would > need to know information about the feed to fetch a configuration. How would > I do that? > > Than, I am curious about monitoring all those feeds. I need to know what`s > going on but also if data arrive on schedule. > > How would I setup my flow? > > Kind Regards, > Eric > >
