I opened a ticket:
https://issues.apache.org/jira/browse/NIFI-4715
*
Root cause is: *
When the file gets uploaded to S3 simultaneously when List S3 is in
progress.
onTrigger--> maxTimestamp is initiated as 0L.
This is clearing keys as per the code below
When lastModifiedTime on S3 object is same as currentTimestamp for the
listed key it should be skipped. As the key is cleared, it is loading the
same file again.
I think fix should be to initiate the maxTimestamp with currentTimestamp not
0L.
{code}
long maxTimestamp = currentTimestamp;
{code}
Following block is clearing keys.
{code:title=org.apache.nifi.processors.aws.s3.ListS3.java|borderStyle=solid}
if (lastModified > maxTimestamp) {
maxTimestamp = lastModified;
currentKeys.clear();
getLogger().debug("clearing keys");
}
{code}
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
