Yes, that's my understanding too.

On Tue, Aug 8, 2017 at 1:14 PM, Laurens Vets <laur...@daemon.be> wrote:

> Thank you for this information. There's no internal notion of directories
> in S3, only objects, so I suspect I'm good if I only set the bucketname?
>
> On 2017-08-08 09:55, Joe Skora wrote:
>
> Laurens,
>
> The S3 User Guide Working with Folders
> <http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html> page
> explains how S3 provides a conceptual directory hierarchy using key name
> prefixes but that buckets really just hold a flat collection of objects.
>
> ListS3 will query S3 for the list of objects and then uses the object
> timestamp as James pointed out to determine what's new to be processed.
>  (Though, it uses last modified timestamp not the last read timestamp.)
>  You can populate the "Prefix" property of the processor so that to S3 can
> filter the object list (as if for a directory tree) before sending the list
> back to NiFi to make things more efficient when dealing with subsets of the
> bucket contents.
>
> Regards,
> Joe S
>
>
> On Tue, Aug 8, 2017 at 11:22 AM, Laurens Vets <laur...@daemon.be> wrote:
>
>> Hi list,
>>
>> Does the ListS3 processor keep state of multiple directories in a bucket?
>>
>> For instance, suppose I have a directory "logs" with subdirectories
>> "host1", "host2" & "host3". Each directory contains logfiles which are
>> added dailty.
>>
>> Will ListS3 keep state correctly here for all 3 subdirectories?
>>
>> Thanks in advance.
>>
>> Laurens
>>
>
>

Reply via email to