Thanks for the info about the HDFSPutProcessor. I just grabbed the aws 0.2.0 version from here: http://central.maven.org/maven2/org/apache/nifi/nifi-aws-nar/
That also doesn't have a DELETE.. I dug into the source a little.. https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;a=blob;f=nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/FetchS3Object.java;h=2406b67f1281e29073500464fc19f082e0118444;hb=HEAD Def no hidden capability. It just gets the object. Presuming the file heads toward a successful relationship, I could probably just call the AmazonS3 deleteObject(..) method http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#deleteObject%28java.lang.String,%20java.lang.String%29 Recommendation: Add a Property "Auto-Delete After Download: true/false" If true, and it was successfully transferred call a delete on the object. Ryan On Fri, Jul 17, 2015 at 4:04 PM, Ryan Blue <[email protected]> wrote: > Ryan, > > There are a couple of AWS-related bundles in the build, thought I've not > looked into them very closely and don't know if they have the S3 support > you're looking for. > > You should, however, be able to get the HDFSPutProcessor to do what you > want. The Hadoop FileSystem API can be used to interact with S3 using s3n: > and s3a: path URIs. The HDFS processors have jets3t in the dependencies, > though I don't see hadoop-aws so I'm not sure whether you'd have to drop > extra dependencies into the lib directory to use the Hadoop S3 URIs. > > Once you've fixed the library issue (or confirmed that the libraries are > already there), you can also use the Kite processor with S3 to write to > partitioned datasets in S3 stored as either Parquet or Avro. > > I hope that helps! > > rb > > > On 07/16/2015 06:04 PM, Ryan Hendrickson wrote: > >> Hi all, >> I thought there was an Amazon S3 Processor.. I dug around the Jira >> Issues.. Looks like one was discussed: >> https://issues.apache.org/jira/browse/NIFI-25 >> >> Although I don't see it in 0.2.0, or 0.1.0... Am I missing something >> simple here, like an extra nar for extensions? >> >> Thanks, >> Ryan >> >> > > -- > Ryan Blue > Software Engineer > Cloudera, Inc. >
