Hi Kirk, Without knowing more of the details of your situation, my suggestions would be as follows:
* Abstract the details to follow specific conventions (i.e. always read from a standard directory path when loading data files, etc.) * If you feel you need an instance of NiFi on every remote system (the source servers), investigate MiNiFi [1][2]. You can build a single flow in NiFi and export it as a class of flows to MiNiFi. It will run as a “well-behaved” guest agent rather than NiFi, which can sometimes be resource-heavy and more often is the main tenant on a system. * If you don’t think you need an instance of NiFi to do processing and transmission on each endpoint system, use standard tools like rsync, FTP, UDP, etc. to transmit the data from each of the collection systems to a common NiFi endpoint, where you can listen for each protocol/origin and aggregate the data into the form you wish to process and operate on in a single location. I know that was vague but hopefully it will help you identify the core functionality you need and will lead to steps toward a reusable and versatile solution. If you have further specifics, people may be able to provide additional advice. [1] https://nifi.apache.org/minifi/ <https://nifi.apache.org/minifi/> [2] https://cwiki.apache.org/confluence/display/MINIFI/Design <https://cwiki.apache.org/confluence/display/MINIFI/Design> Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 9, 2016, at 2:38 PM, Tarou, Kirk <[email protected]> > wrote: > > I'm trying to transition my data collection from a variety of configuration > files and scripts running in cron to NiFi. > > Given the hundreds of servers I have to collect from, creating a flow for > each server and data type would be very time consuming. Even utilizing > templates, creating everything in the GUI would take too long. > > I've considered writing a script that uses the API to generate the flows, but > I'm concerned that it will be difficult to make changes without deleting > whole sections of my flow and regenerating them. > > I've written a few custom processors, so I'm considering building what I need > on my own and leveraging the existing processors when I can. > > If anyone else has done something similar and would be willing to share their > strategy, I'd really appreciate it. > > -Kirk
signature.asc
Description: Message signed with OpenPGP using GPGMail
