[
https://issues.apache.org/jira/browse/NIFI-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15026099#comment-15026099
]
Tony Kurc commented on NIFI-631:
--------------------------------
[~jskora], [~markap14], I put together both of your patches and rebased them on
master. I also put InputRequirements on them and had them reference each other.
I started testing them together and things seem to work. I think there may be
some minor tweaks needed
[~jskora] - I don't understand this code. Won't the if condition never hit?
{code}
String relativePathString = relativePath.toString() + "/";
if (relativePathString.isEmpty()) {
relativePathString = "./";
}
{code}
This description misled me to misconfigure my flow. I believe that path is set
to a relative path, not absolute. Also, the attribute absolute.path is
populated but not described. When I wired ListFile and FetchFile together with
defaults, they did not work, which I wasn't expecting. I'll upload a flow I
made which does work (I used absolute.path in the EL for the file).
{code}
@WritesAttribute(attribute="path", description="The path is set to the
absolute path of the file's directory " +
"on filesystem. For example, if the Directory property is set
to /tmp, then files picked up from " +
"/tmp will have the path attribute set to \"./\". If the
Recurse Subdirectories property is set to " +
"true and a file is picked up from /tmp/abc/1/2/3, then the
path attribute will be set to " +
"\"/tmp/abc/1/2/3\"."),
{code}
Why I believe it is a relative path:
{code}
attributes.put(CoreAttributes.PATH.key(), relativePathString);
{code}
Anyhow, they both totally seem to work. Please review my rebasing and changes
to unify them together and add InputRequirements.
> Create ListFile and FetchFile processors
> ----------------------------------------
>
> Key: NIFI-631
> URL: https://issues.apache.org/jira/browse/NIFI-631
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Mark Payne
> Assignee: Joe Skora
> Attachments:
> 0001-NIFI-631-Initial-implementation-of-FetchFile-process.patch,
> NIFI-631.000.patch
>
>
> This pair of Processors will provide several benefits over the existing
> GetFile processor:
> 1. Currently, GetFile will continually pull the same files if the "Keep
> Source File" property is set to true. There is no way to pull the file and
> leave it in the directory without continually pulling the same file. We could
> implement state here, but it would either be a huge amount of state to
> remember everything pulled or it would have to always pull the oldest file
> first so that we can maintain just the Last Modified Date of the last file
> pulled plus all files with the same Last Modified Date that have already been
> pulled.
> 2. If pulling from a network attached storage such as NFS, this would allow a
> single processor to run ListFiles and then distribute those FlowFiles to the
> cluster so that the cluster can share the work of pulling the data.
> 3. There are use cases when we may want to pull a specific file (for example,
> in conjunction with ProcessHttpRequest/ProcessHttpResponse) rather than just
> pull all files in a directory. GetFile does not support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
