Nicolas Dupont created NIFI-2946:
------------------------------------
Summary: Nifi can skip data in List processors and never ingest it
afterwards
Key: NIFI-2946
URL: https://issues.apache.org/jira/browse/NIFI-2946
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 1.0.0
Environment: All
Reporter: Nicolas Dupont
With current design for Listing processors, it seems like Nifi may skip some
folder's contents which will never be ingested later.
Example in FTPTransfer.java :
if (recurse && file.isDirectory())
{
try
{
listing.addAll(getListing(newFullForwardPath, depth + 1, maxResults - count));
}
catch (final IOException e)
{
logger.error("Unable to get listing from " + newFullForwardPath + "; skipping
this subdirectory");
}
}
With this design, you can encounter a situation in which you can't list one or
more subfolder's contents (for any reason like a sudden time-out) BUT still
update the component state timestamps : at next runs previous "forgotten" files
will never be pushed down because of their timestamps + they can't be
reinjected through an error relationship.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
