Joe/Joe, Thanks for the response. It makes sense to use SNS and SQS to respond to S3 file changes. I’m going see if my company will give me access to those Amazon services. I found an article that explains how to setup on this functionality in the Amazon console. Once that’s setup it seems pretty straight forward to use GetSQS/DeleteSQS. I suspect many will want this functionality but I’m not sure what’s the best method (i.e. Template or user doc) that explains how to solve this in nifi. I’ll be happy to submit something if you let me know the right method.
http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html Respectfully, Kyle Burke | Data Science Engineer IgnitionOne - Marketing Technology. Simplified. Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 From: Joe Witt Reply-To: "[email protected]<mailto:[email protected]>" Date: Saturday, January 30, 2016 at 2:06 PM To: "[email protected]<mailto:[email protected]>" Subject: Re: ListS3 processor? Kyle Let us know if that doesn't get you what you need. We have a decent set of templates but I didn't see one that demonstrates interaction with amazon services. Thanks Joe On Jan 30, 2016 12:56 PM, "Joey Frazee" <[email protected]<mailto:[email protected]>> wrote: Kyle, I think you can do what you want right now without ListS3 by using S3 event notifications. You can configure an event notification to publish to SQS and then use GetSQS to retrieve the events and FetchS3Object to get the JSON file and the rest of the flow could be written as you have in mind. Depending on your scale, this might be preferable because it's slow/expensive to do listings on S3 prefixes that have a lot of file matches. -joey On Jan 30, 2016, at 11:40 AM, Joe Skora <[email protected]<mailto:[email protected]>> wrote: Kyle, Processors exist to Put, Fetch, and Delete S3Objects, but ListS3 is in the backlog on ticket NIFI-840<https://issues.apache.org/jira/browse/NIFI-840> at the moment. It should fit the List/Fetch metaphor like the List/Fetch processors pairs for xFile, xHDFS, xSFTP, etc. Regards, Joe Skora On Sat, Jan 30, 2016 at 10:14 AM, Kyle Burke <[email protected]<mailto:[email protected]>> wrote: All, I'm trying to get Nifi set up to a move data around S3. My first attempt is to just monitor a S3 folder where json files are placed and then copy the file, convert it to Avro, and the drop it in a different S3 folder. The documentation is pretty slim for working with S3. I can't seem to get it working and was wondering if anyone had any S3 examples for monitoring an S3 folder (i.e.. something like a ListS3 processer similar to what is available on a local file system?) Respectfully, Kyle Burke | Data Science Engineer IgnitionOne - Marketing Technology. Simplified. Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 Direct: 404.961.3918<tel:404.961.3918>
