The S3 User Guide Working with Folders
explains how S3 provides a conceptual directory hierarchy using key name
prefixes but that buckets really just hold a flat collection of objects.
ListS3 will query S3 for the list of objects and then uses the object
timestamp as James pointed out to determine what's new to be processed.
(Though, it uses last modified timestamp not the last read timestamp.)
You can populate the "Prefix" property of the processor so that to S3 can
filter the object list (as if for a directory tree) before sending the list
back to NiFi to make things more efficient when dealing with subsets of the
On Tue, Aug 8, 2017 at 11:22 AM, Laurens Vets <laur...@daemon.be> wrote:
> Hi list,
> Does the ListS3 processor keep state of multiple directories in a bucket?
> For instance, suppose I have a directory "logs" with subdirectories
> "host1", "host2" & "host3". Each directory contains logfiles which are
> added dailty.
> Will ListS3 keep state correctly here for all 3 subdirectories?
> Thanks in advance.