[
https://issues.apache.org/jira/browse/NIFI-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984540#comment-14984540
]
Joe Skora commented on NIFI-631:
--------------------------------
[~mpetronic]
The changes are almost done. I modeled the properties on GetFile, so no
functionality would be lost. Details are below.
I'm working out issues with file attribute differences on Linux vs Windows and
making sure the tests work on both platforms. I hope to have that done
tonight, but if I don't do you want a crack at the Linux source?
Joe
----
The properties now consist of
* Input Directory - root directory for file searches,
* Recurse Subdirectories - directory recursion,
* File Filter - regex file name pattern,
* Path Filter - regex path pattern, not including file name, if recurse is true,
* Minimum File Age - Time Unit delta for oldest file to process (i.e. 2 days, 1
hour, etc.),
* Maximum File Age - Time Unit delta for newest file to process,
* Minimum FIle Size - file size minimum to process,
* Maximum File Size - file size maximum to process, and
* Ignore Hidden Files - allows suppression of hidden files.
> Create ListFile and FetchFile processors
> ----------------------------------------
>
> Key: NIFI-631
> URL: https://issues.apache.org/jira/browse/NIFI-631
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Mark Payne
> Assignee: Joe Skora
> Attachments:
> 0001-NIFI-631-Initial-implementation-of-FetchFile-process.patch
>
>
> This pair of Processors will provide several benefits over the existing
> GetFile processor:
> 1. Currently, GetFile will continually pull the same files if the "Keep
> Source File" property is set to true. There is no way to pull the file and
> leave it in the directory without continually pulling the same file. We could
> implement state here, but it would either be a huge amount of state to
> remember everything pulled or it would have to always pull the oldest file
> first so that we can maintain just the Last Modified Date of the last file
> pulled plus all files with the same Last Modified Date that have already been
> pulled.
> 2. If pulling from a network attached storage such as NFS, this would allow a
> single processor to run ListFiles and then distribute those FlowFiles to the
> cluster so that the cluster can share the work of pulling the data.
> 3. There are use cases when we may want to pull a specific file (for example,
> in conjunction with ProcessHttpRequest/ProcessHttpResponse) rather than just
> pull all files in a directory. GetFile does not support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
