Kyle, The ideal case for communicating how to do this would be both a template and an associated doc. Great for a blog or wiki page or something. We can of course give you perms to write to a wiki page on the nifi wiki if interested. The template itself can also be annotated with comments that show up right in the flow itself. That may be a fine option too.
Thanks Joe On Sat, Jan 30, 2016 at 2:52 PM, Kyle Burke <[email protected]> wrote: > Joe/Joe, > Thanks for the response. It makes sense to use SNS and SQS to respond to > S3 file changes. I’m going see if my company will give me access to those > Amazon services. I found an article that explains how to setup on this > functionality in the Amazon console. Once that’s setup it seems pretty > straight forward to use GetSQS/DeleteSQS. I suspect many will want this > functionality but I’m not sure what’s the best method (i.e. Template or user > doc) that explains how to solve this in nifi. I’ll be happy to submit > something if you let me know the right method. > > http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html > > Respectfully, > > Kyle Burke | Data Science Engineer > IgnitionOne - Marketing Technology. Simplified. > Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 > > > From: Joe Witt > Reply-To: "[email protected]" > Date: Saturday, January 30, 2016 at 2:06 PM > To: "[email protected]" > Subject: Re: ListS3 processor? > > Kyle > > Let us know if that doesn't get you what you need. We have a decent set of > templates but I didn't see one that demonstrates interaction with amazon > services. > > Thanks > Joe > > On Jan 30, 2016 12:56 PM, "Joey Frazee" <[email protected]> wrote: >> >> Kyle, >> >> I think you can do what you want right now without ListS3 by using S3 >> event notifications. You can configure an event notification to publish to >> SQS and then use GetSQS to retrieve the events and FetchS3Object to get the >> JSON file and the rest of the flow could be written as you have in mind. >> >> Depending on your scale, this might be preferable because it's >> slow/expensive to do listings on S3 prefixes that have a lot of file >> matches. >> >> >> -joey >> >> On Jan 30, 2016, at 11:40 AM, Joe Skora <[email protected]> wrote: >> >> Kyle, >> >> Processors exist to Put, Fetch, and Delete S3Objects, but ListS3 is in the >> backlog on ticket NIFI-840 at the moment. It should fit the List/Fetch >> metaphor like the List/Fetch processors pairs for xFile, xHDFS, xSFTP, etc. >> >> Regards, >> Joe Skora >> >> On Sat, Jan 30, 2016 at 10:14 AM, Kyle Burke <[email protected]> >> wrote: >>> >>> All, >>> I'm trying to get Nifi set up to a move data around S3. My first >>> attempt is to just monitor a S3 folder where json files are placed and then >>> copy the file, convert it to Avro, and the drop it in a different S3 folder. >>> The documentation is pretty slim for working with S3. I can't seem to get it >>> working and was wondering if anyone had any S3 examples for monitoring an S3 >>> folder (i.e.. something like a ListS3 processer similar to what is available >>> on a local file system?) >>> >>> Respectfully, >>> >>> Kyle Burke | Data Science Engineer >>> IgnitionOne - Marketing Technology. Simplified. >>> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 >>> Direct: 404.961.3918 >>> >> >
