Imagine a filename construct where you wanted to pick up any file that begins with the phrase 'start' but does NOT end in the phrase 'part'.
The name is of a form 'begin.middle.end'. This filename start.middle.ok would get picked up. This filename start.middle.part would not. The pattern for that example would be start\..+\.(?!part) The key part of that is the negative lookahead for ensuring it does not end in part. Thanks On Wed, Dec 6, 2017 at 2:29 AM, Ravi Papisetti (rpapiset) <[email protected]> wrote: > Yeah..that is good idea, but we are already using this option to copy file > with certain prefix. Not sure how I can use this field to meet both exclusion > and inclusion criterion. > > Any thoughts. > > Thanks, > Ravi Papisetti > > On 06/12/17, 1:26 AM, "Joe Witt" <[email protected]> wrote: > > Ravi > > Please use the 'File Filter' property of ListFile to control ignoring > filenames until they no longer end in 'part'. > > Thanks > > On Wed, Dec 6, 2017 at 2:14 AM, Ravi Papisetti (rpapiset) > <[email protected]> wrote: > > Hi, > > > > > > > > We are using Apache NiFi 1.3.0 > > > > > > > > We have a process flow to copy files from NFS to HDFS (with processors > > ListFile, FetchFile and PutHDFS) > > > > > > > > In the NiFi process flow, ListFile is configured to listen to a > directory on > > NFS. When a file (ex: x.csv) is being copied from a windows machine to > NFS > > (while transfer is in the middle), a part file(x.csv.part) is created > at NFS > > until transfer is complete. > > > > > > > > ListFile has picked up this x.csv.part file and fetchFile picked up > this to > > transfer to HDFS, didn’t update the file name back to x.csv in HDFS when > > transfer is complete. > > > > > > > > But, in case a file from linux file system, while file copy to NFS is in > > progress it created (.x.csv) and when transfer is complete, at both NFS > and > > HDFS, filename is updated to x.csv (from .x.csv). > > > > > > > > Any thought how we can configure ListFile not to pickup these part > files or > > any configurations in NiFi that fixes file names for these windows part > > files? > > > > > > > > Appreciate your help. > > > > > > > > Thanks, > > > > Ravi Papisetti > >
