Is there a way to troubleshoot this further? I did an upgrade to 1.7.1. Changed the 'Minimum Object Age' to 1 minute, but I _think_ NiFi is somehow still not picking up all objects from the S3 bucket.
On 20-Sep-18 19:55, Mark Rachelski wrote: > The S3Fetch processor is a stateful processor using NiFi state storage > to track the time of the most recent object name list extracted from > the bucket. On subsequent runs, it will only pull objects that have an > update time newer than the internally stored time stored in state (the > time of the last S3Fetch run). > > Is this possibly contributing to your missing files? > > mark. > > On Fri, Sep 21, 2018 at 6:21 AM Vets, Laurens <[email protected] > <mailto:[email protected]>> wrote: > > Hello, > > I'm using NiFi to read an S3 bucket containing all our AWS CloudTrail > logs. While debugging an issue, I noticed that not all objects are > fetched or listed. Basically, some events which I can find manually by > grepping the S3 files, I can't find in our Kibana dashboard. Is it > therefor possible that there might be an issue with the S3 processors > whereby it doesn't pick up all S3 objects? > > I'm using NiFi 1.3.0. While reading the release notes for the > newer NiFi > versions, I found https://issues.apache.org/jira/browse/NIFI-4876 and > wondering whether this might be related? > > Can anyone shed some light on this? >
