hello i think we would need to see directory listing from s3, state info and config from nifi, and logs.
you can step through in a debug session to see what the proc sees as well. thanks On Fri, Sep 21, 2018, 6:55 PM Vets, Laurens <[email protected]> wrote: > Is there a way to troubleshoot this further? I did an upgrade to 1.7.1. > Changed the 'Minimum Object Age' to 1 minute, but I _think_ NiFi is > somehow still not picking up all objects from the S3 bucket. > > On 20-Sep-18 19:55, Mark Rachelski wrote: > > The S3Fetch processor is a stateful processor using NiFi state storage > > to track the time of the most recent object name list extracted from > > the bucket. On subsequent runs, it will only pull objects that have an > > update time newer than the internally stored time stored in state (the > > time of the last S3Fetch run). > > > > Is this possibly contributing to your missing files? > > > > mark. > > > > On Fri, Sep 21, 2018 at 6:21 AM Vets, Laurens <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hello, > > > > I'm using NiFi to read an S3 bucket containing all our AWS CloudTrail > > logs. While debugging an issue, I noticed that not all objects are > > fetched or listed. Basically, some events which I can find manually > by > > grepping the S3 files, I can't find in our Kibana dashboard. Is it > > therefor possible that there might be an issue with the S3 processors > > whereby it doesn't pick up all S3 objects? > > > > I'm using NiFi 1.3.0. While reading the release notes for the > > newer NiFi > > versions, I found https://issues.apache.org/jira/browse/NIFI-4876 > and > > wondering whether this might be related? > > > > Can anyone shed some light on this? > > >
