I am reviewing NiFi as a possible replacement for an internally developed data transfer tool.
NiFi appears to be of the "the data is guaranteed to be delivered at least once" variety. Where my needs are "the data is guaranteed to be delivered once"; to the point that I'm willing to manually review and resolve failures that occur "beyond the point of no return" to ensure 1x delivery and no data loss. Some of my data transfers are of the sequential transactional type, where they must be transferred and processed, in sequence. Take for instance GetFTP, I see holes in the commit model (from the perspective of what I'm trying to accomplish). Looking at GetFTP (via FetchFileTransfer.java), session.commit() occurs before deleting or renaming the source file. So, if that step fails the file will be retrieved and processed again and again. PutSQL appears to have similar issues as it relates to updating a database more than once should the transfer die before the db commit is recognized by NiFi, so the the FlowFile(s) get rolled back. Are my needs outside of Nifi's objectives?
