Thanks for your comment. However, I can't queue everything as the total size of the data is around 560GB. Right now, I am using a GetFile processor and it has been running for a few days. If I look at my end point, it looks like it should be done pretty soon but data is still streaming in at the same rate so I was wondering if the processor remembers every single file it has already processed or if it is simply going through all the files alphabetically or in whatever order it decides.
Thanks Jean-Sébastien Vachon Co-Founder & Architect Brizo Data, Inc. www.brizodata.com<https://outlook.office365.com/mail/options/mail/messageContent/www.brizodata.com> ________________________________ From: Arpad Boda <[email protected]> Sent: Thursday, February 18, 2021 8:29 AM To: [email protected] <[email protected]> Subject: Re: Questions about the GetFile processor You can use the combination of listfile and fetchfile. In the queue between the two you are going to see the number of (flow)files left to be processed. On Thu, Feb 18, 2021 at 2:14 PM Jean-Sebastien Vachon <[email protected]<mailto:[email protected]>> wrote: Hi all, If I configure a GetFile processor to list all JSON files under a given folder, will it stops sending flows once it has processed all files? My folder contains thousands of files and the processor reads them by small batch (10) every 30s. Is there a way to know how many files are left to processed? Thanks Jean-Sébastien Vachon Co-Founder & Architect Brizo Data, Inc. www.brizodata.com<https://outlook.office365.com/mail/options/mail/messageContent/www.brizodata.com>
