OK, it seems like NiFi was actually trying to find files in the directory, although there were way too many and it was running out of memory... I guess I need to run something to archive the old logs and start with only the new ones.
Thanks for your help Mark! On Tue, 28 May 2019 at 21:33, Stephen Greszczyszyn <sgres...@gmail.com> wrote: > OK, thanks, this us useful information, I did also test the filenames > with a regex parser, but didn't know NiFi was different. I really don't > need an expression anyway as I want to get all the files. I'm wondering if > the problem is that I have too many files in the directory, so I'll just > set up a test directory and no regex and see what happens. > > > On Tue, 28 May 2019 at 20:21, Mark Payne <marka...@hotmail.com> wrote: > >> Stephen, >> >> Egrep evaluates regex'es very differently than NiFi does. With egrep, any >> line that contains text matching the regex >> will be output. With NiFi, however, the entire line of text must match >> the regex exactly in order to be selected. So '^c.r' will >> not match but '^c.r.*' will. >> >> Thanks >> -Mark >> >> On May 28, 2019, at 3:15 PM, Stephen Greszczyszyn <sgres...@gmail.com> >> wrote: >> >> I don't think that is the problem as I have checked the regex and can >> find the files (this is from inside the container so I also can read the >> directory): >> >> nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r' >> cdr_AP2-HKG-CCP-UCM_12_201905281539_35251 >> cdr_AP2-HKG-CCP-UCM_06_201905281539_297892 >> cmr_AP2-HKG-CCP-UCM_06_201905281539_297892 >> cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435 >> cdr_AP2-HKG-CCP-UCM_11_201905281539_98483 >> >> On Tue, 28 May 2019 at 17:02, Mark Payne <marka...@hotmail.com> wrote: >> >>> Stephen, >>> >>> The regex that you have there in interpreted by NiFi as "the filename >>> must consist of 3 characters, beginning with a c and ending with an r". >>> What you want >>> is "the filename must begin with a c, followed by any character, >>> followed by an r, followed by anything else." So you'd want "c.r.*" I do >>> believe. >>> >>> Thanks >>> -Mark >>> >>> >>> > On May 28, 2019, at 11:44 AM, Stephen Greszczyszyn <sgres...@gmail.com> >>> wrote: >>> > >>> > Hi there, >>> > >>> > I'm running NiFi in docker. I'm trying to enable flow debugs and also >>> to determine if NiFi can actually see in my data directory path. I'm >>> trying to use a simple recursive list file processor to find the files >>> based on a regular expression "^c.r". >>> > >>> > I can docker exec into the container and can see that the nifi >>> container can "see" into my filepath OK: >>> > >>> > nifi@4f614c6c67c4:/data/sftp/cdr$ ls -1t | head -5 | egrep '^c.r' >>> > cdr_AP2-HKG-CCP-UCM_12_201905281539_35251 >>> > cdr_AP2-HKG-CCP-UCM_06_201905281539_297892 >>> > cmr_AP2-HKG-CCP-UCM_06_201905281539_297892 >>> > cdr_AP9-SYD-UCCE-UCM_05_201905281539_200435 >>> > cdr_AP2-HKG-CCP-UCM_11_201905281539_98483 >>> > >>> > However, I can't get the ListFile processor to work and find those >>> files. Any ideas on debugging strategies? >>> > >>> > Thanks. >>> >>> >>
