Kyle, I think you can do what you want right now without ListS3 by using S3 event notifications. You can configure an event notification to publish to SQS and then use GetSQS to retrieve the events and FetchS3Object to get the JSON file and the rest of the flow could be written as you have in mind.
Depending on your scale, this might be preferable because it's slow/expensive to do listings on S3 prefixes that have a lot of file matches. -joey > On Jan 30, 2016, at 11:40 AM, Joe Skora <[email protected]> wrote: > > Kyle, > > Processors exist to Put, Fetch, and Delete S3Objects, but ListS3 is in the > backlog on ticket NIFI-840 at the moment. It should fit the List/Fetch > metaphor like the List/Fetch processors pairs for xFile, xHDFS, xSFTP, etc. > > Regards, > Joe Skora > >> On Sat, Jan 30, 2016 at 10:14 AM, Kyle Burke <[email protected]> >> wrote: >> All, >> I'm trying to get Nifi set up to a move data around S3. My first attempt >> is to just monitor a S3 folder where json files are placed and then copy the >> file, convert it to Avro, and the drop it in a different S3 folder. The >> documentation is pretty slim for working with S3. I can't seem to get it >> working and was wondering if anyone had any S3 examples for monitoring an S3 >> folder (i.e.. something like a ListS3 processer similar to what is available >> on a local file system?) >> >> Respectfully, >> >> Kyle Burke | Data Science Engineer >> IgnitionOne - Marketing Technology. Simplified. >> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 >> Direct: 404.961.3918 >
