[
https://issues.apache.org/jira/browse/NIFI-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Dupont updated NIFI-2946:
---------------------------------
Description:
With current design for Listing processors, it seems like Nifi may skip some
folder's contents which will never be ingested later.
{code:title=Example from FTPTransfer.java|borderStyle=solid}
if (recurse && file.isDirectory())
{
try
{
listing.addAll(getListing(newFullForwardPath, depth + 1, maxResults - count));
}
catch (final IOException e)
{
logger.error("Unable to get listing from " + newFullForwardPath + "; skipping
this subdirectory");
}
}
{code}
With this design, you can encounter a situation in which you can't list one or
more subfolder's contents (for any reason like a sudden time-out) BUT still
update the component state timestamps : at next runs previous "forgotten" files
will never be pushed down because of their timestamps + they can't be
reinjected through an error relationship.
was:
With current design for Listing processors, it seems like Nifi may skip some
folder's contents which will never be ingested later.
Example in FTPTransfer.java :
if (recurse && file.isDirectory())
{
try
{
listing.addAll(getListing(newFullForwardPath, depth + 1, maxResults - count));
}
catch (final IOException e)
{
logger.error("Unable to get listing from " + newFullForwardPath + "; skipping
this subdirectory");
}
}
With this design, you can encounter a situation in which you can't list one or
more subfolder's contents (for any reason like a sudden time-out) BUT still
update the component state timestamps : at next runs previous "forgotten" files
will never be pushed down because of their timestamps + they can't be
reinjected through an error relationship.
> Nifi can skip data in List processors and never ingest it afterwards
> --------------------------------------------------------------------
>
> Key: NIFI-2946
> URL: https://issues.apache.org/jira/browse/NIFI-2946
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 1.0.0
> Environment: All
> Reporter: Nicolas Dupont
> Labels: errorhandling, listFiles, nifi, skip
>
> With current design for Listing processors, it seems like Nifi may skip some
> folder's contents which will never be ingested later.
> {code:title=Example from FTPTransfer.java|borderStyle=solid}
> if (recurse && file.isDirectory())
> {
> try
> {
> listing.addAll(getListing(newFullForwardPath, depth + 1, maxResults - count));
> }
> catch (final IOException e)
> {
> logger.error("Unable to get listing from " + newFullForwardPath + "; skipping
> this subdirectory");
> }
> }
> {code}
> With this design, you can encounter a situation in which you can't list one
> or more subfolder's contents (for any reason like a sudden time-out) BUT
> still update the component state timestamps : at next runs previous
> "forgotten" files will never be pushed down because of their timestamps +
> they can't be reinjected through an error relationship.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
