Do your files get passed out of ListFile? If so, that eliminates the NIFI-2383 and NIFI-3332 issues. Specifically, NIFI-3332 occurs when a block of files is being written with the same timestamp and the processor runs in the middle of those writes, so it picks up the timestamp as it's highwater mark.
If your files go ListFile -> FetchFile but then not FetchFile -> PutSFTP that's a different problem. Depending on your flow and systems, you may be able to use move or rename instead of PutFile/Delete, on a single file system that can be very efficient and not require copying content. On Wed, May 3, 2017 at 8:44 AM, James Keeney <[email protected]> wrote: > Joe - > > Permissions is not an issue since both users are using the same > credentials. > > I do not remove the files after pickup as the original directory needs to > mirror the destination. > > However, your description fits with what I have been thinking which is as > follows: > > ListFile generates a list of the files and sends to FetchFile. If > FetchFile does not forward a file to PutSFTP then there would be no error > in PutSFTP and the ListFile processor will not re-submit the file since it > already did it once. > > I think what makes the most sense is some way to create a confirmation > loop. In other words, once PutSFTP has sent a file then update the file in > some way that would allow the system to know that the file is sent. > > Maybe something like this: > > > 1. Use putfile to move the file from one directory to another > 2. Execute PutSFTP from that directory > 3. If successful move the file back to original directory with a flag > that indicates to putfile not to pick it up. Maybe set the timestamp in the > past? > > > FTP --> Directory A --> Putfile, with deletion --> Directory B --> PutSFTP > --> PutFile with deletion --> Directory A (set timestamp to past) > > If anything fails along the way it would be obvious. > > A bit complicated but more definite. > > Jim K. > > On Wed, May 3, 2017 at 7:30 AM Joe Skora <[email protected]> wrote: > >> Jim, >> >> There are a couple tickets related to the ListFile processors, I'm not >> sure if they apply to your situation but you might want to check them out >> to see if they help clarify anything. >> >> - NIFI-1671 <https://issues.apache.org/jira/browse/NIFI-1671> relates >> to tests failing possible due to Windows account configurations, >> - NIFI-2383 <https://issues.apache.org/jira/browse/NIFI-2383> >> mentions files not being picked up, and >> - NIFI-3332 <https://issues.apache.org/jira/browse/NIFI-3332> relates >> to a race condition in the ListXXX processor's high watermark >> implementation. >> >> I agree with James Wing that your problem could be permissions, >> especially if your users send similar volumes of files. But if the missed >> files are sent in larger batches and/or sent over slower connections it >> might be the timestamp issue. >> >> Does your configuration move or delete files from the source directory >> after they are processed? If you remove processed files, you should be >> able to force the processor to pickup everything in the directory by >> clearing the processor state. If you do not remove processed files but can >> identify which ones weren't processed, you could update the file >> timestamps, with "touch" for example, to cause the processor to pick them >> up. >> >> Regards, >> Joe S >> >> On Tue, May 2, 2017 at 9:00 AM, James Keeney <[email protected]> >> wrote: >> >>> I have setup a processor chain with ListFile, FetchFile and then >>> PutSFTP. >>> >>> Users FTP to the server that has NIFI and then NIFI is used to transfer >>> the files to our production servers using PutSFTP. >>> >>> When one user performs the transfer everything works, but when another >>> does it some of the files are not transferred. >>> >>> I have checked and all of the files made it onto the server with the >>> initial FTP transfer. I setup the PutSFTP so a reject or failure would >>> route the file to a directory using putfile. No files end up in that >>> directory so PutSFTP appears to be working correctly. So, It appears that >>> the breakdown occurs in the ListFile, FetchFile process. How would I go >>> about debugging this? No errors are thrown that I can see. >>> >>> Jim K. >>> -- >>> Jim Keeney >>> President, FitterWeb >>> E: [email protected] >>> M: 703-568-5887 <(703)%20568-5887> >>> >>> *FitterWeb Consulting* >>> *Are you lean and agile enough? * >>> >> >> -- > Jim Keeney > President, FitterWeb > E: [email protected] > M: 703-568-5887 <(703)%20568-5887> > > *FitterWeb Consulting* > *Are you lean and agile enough? * >
