I am 100% in favor of keeping the relative path (I brought up out of band the value if the Lister and Fetcher were different machines with different mount points). I think is just a matter of what attribute to fill with what value.
On Wed, Nov 25, 2015 at 2:09 PM, Joe Skora <[email protected]> wrote: > Mark, > > What you described is the behavior of ListFile (in spite of confusing doc > info). > > JoeW, > > Consistency with ListHDFS makes sense, and if that is the desired behavior > it's easy to change ListFile. But CoreAttributres state "The flowfile's > path indicates the relative directory" and if that's not true, does > CoreAttributes need revision too? > > Thanks, > Joe > > On Wed, Nov 25, 2015 at 2:00 PM, Mark Payne <[email protected]> wrote: > > > Tony, > > > > I would recommend that ListFile add both 'path' and 'absolute.path'. The > > 'path' would be relative to the base directory being listed. > > For example, if ListFile is configured to list files from /data/nifi/in > > and recurse subdirectories, and it finds a file named: > > /data/nifi/in/123/myfile.txt > > then i would expect the following attributes: > > > > absolute.path = /data/nifi/in/123 > > path = ./123 > > filename = myfile.txt > > > > Thanks > > -Mark > > > > > > > On Nov 25, 2015, at 1:56 PM, Tony Kurc <[email protected]> wrote: > > > > > > All, > > > Joe and I commented on NIFI-631 that it didn't "just work" when wiring > > the > > > processors together. ListFile was populating the attributes as > > > described in CoreAttributes.java > > > [1] (path being relative to the input directory, and absolute being the > > > full path). FetchFile was using ${path}/${filename} as the default, > which > > > wouldn't grab the directory. I'm puzzled as to what the correct > behavior > > > should be. The description of path said it is relative ... relative to > > > what? ListHDFS appears to state path is absolute [2] [3], and I expect > we > > > should have consistent behavior between ListHDFS and ListFile. > > > > > > So, I guess I'm not sure what guidance to give on a review of NIFI-631. > > > Should the default of FetchFile be changed to > > ${absolute.path}/${filename} > > > (which may be inconsistent with other List/Fetch processor combos), or > > > should ListFile be changed to have path be absolute? > > > > > > [1] > > > > > > https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/flowfile/attributes/CoreAttributes.java > > > [2] > > > > > > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L79 > > > [3] > > > > > > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L442 > > > > >
