Joe - Permissions is not an issue since both users are using the same credentials.
I do not remove the files after pickup as the original directory needs to mirror the destination. However, your description fits with what I have been thinking which is as follows: ListFile generates a list of the files and sends to FetchFile. If FetchFile does not forward a file to PutSFTP then there would be no error in PutSFTP and the ListFile processor will not re-submit the file since it already did it once. I think what makes the most sense is some way to create a confirmation loop. In other words, once PutSFTP has sent a file then update the file in some way that would allow the system to know that the file is sent. Maybe something like this: 1. Use putfile to move the file from one directory to another 2. Execute PutSFTP from that directory 3. If successful move the file back to original directory with a flag that indicates to putfile not to pick it up. Maybe set the timestamp in the past? FTP --> Directory A --> Putfile, with deletion --> Directory B --> PutSFTP --> PutFile with deletion --> Directory A (set timestamp to past) If anything fails along the way it would be obvious. A bit complicated but more definite. Jim K. On Wed, May 3, 2017 at 7:30 AM Joe Skora <[email protected]> wrote: > Jim, > > There are a couple tickets related to the ListFile processors, I'm not > sure if they apply to your situation but you might want to check them out > to see if they help clarify anything. > > - NIFI-1671 <https://issues.apache.org/jira/browse/NIFI-1671> relates > to tests failing possible due to Windows account configurations, > - NIFI-2383 <https://issues.apache.org/jira/browse/NIFI-2383> mentions > files not being picked up, and > - NIFI-3332 <https://issues.apache.org/jira/browse/NIFI-3332> relates > to a race condition in the ListXXX processor's high watermark > implementation. > > I agree with James Wing that your problem could be permissions, especially > if your users send similar volumes of files. But if the missed files are > sent in larger batches and/or sent over slower connections it might be the > timestamp issue. > > Does your configuration move or delete files from the source directory > after they are processed? If you remove processed files, you should be > able to force the processor to pickup everything in the directory by > clearing the processor state. If you do not remove processed files but can > identify which ones weren't processed, you could update the file > timestamps, with "touch" for example, to cause the processor to pick them > up. > > Regards, > Joe S > > On Tue, May 2, 2017 at 9:00 AM, James Keeney <[email protected]> wrote: > >> I have setup a processor chain with ListFile, FetchFile and then PutSFTP. >> >> Users FTP to the server that has NIFI and then NIFI is used to transfer >> the files to our production servers using PutSFTP. >> >> When one user performs the transfer everything works, but when another >> does it some of the files are not transferred. >> >> I have checked and all of the files made it onto the server with the >> initial FTP transfer. I setup the PutSFTP so a reject or failure would >> route the file to a directory using putfile. No files end up in that >> directory so PutSFTP appears to be working correctly. So, It appears that >> the breakdown occurs in the ListFile, FetchFile process. How would I go >> about debugging this? No errors are thrown that I can see. >> >> Jim K. >> -- >> Jim Keeney >> President, FitterWeb >> E: [email protected] >> M: 703-568-5887 <(703)%20568-5887> >> >> *FitterWeb Consulting* >> *Are you lean and agile enough? * >> > > -- Jim Keeney President, FitterWeb E: [email protected] M: 703-568-5887 *FitterWeb Consulting* *Are you lean and agile enough? *
