Russell, The general contract is for ListXXX to identify files to process without moving content and FetchXXX to pull the content into the data flow. So, FetchS3 is ideally meant to work in conjunction with a ListS3 instance, but your solution of using ExecuteStreamCommand with the S3 CLI tools should allow you to approximate that behavior until ListS3 is released.
Joe On Tue, Jan 12, 2016 at 10:08 PM, Russell Whitaker < russell.whita...@gmail.com> wrote: > On Tue, Jan 12, 2016 at 6:39 PM, Corey Flowers <cflow...@onyxpoint.com> > wrote: > > Hello Russell, > > > > Sorry if that seemed short, I was running in to pick my son up > from > > my practice. > > Oh, I didn't mean to give the impression I'd taken it that way! > > > What I meant to say was that you are correct. Although I > > haven't worked on those processors, I do believe it is expecting the > listS3 > > processor to function and that is why you are getting that error. I would > > love to know if there is a work around because I would also love to work > > with these processors within the next week or so. > > > > I can think of a workaround that might involve wrapping s3cmd in an > ExecuteStreamCommand > processor, will let you know... > > Russell > > > On Tue, Jan 12, 2016 at 9:16 PM, Russell Whitaker > > <russell.whita...@gmail.com> wrote: > >> > >> On Tue, Jan 12, 2016 at 6:11 PM, Corey Flowers <cflow...@onyxpoint.com> > >> wrote: > >> > Ha ha! Well that would do it! :) > >> > > >> > >> I don't know what that would "do" other than confirm that the > >> FetchS3Object processor shipped > >> with v0.4.1 needs its doc to reflect the fact it's not yet useable > >> untl a ListS3* processor is implemented > >> and included in the narfile for the distribution. > >> > >> Russell > >> > >> > Sent from my iPhone > >> > > >> >> On Jan 12, 2016, at 9:10 PM, Russell Whitaker > >> >> <russell.whita...@gmail.com> wrote: > >> >> > >> >>> On Tue, Jan 12, 2016 at 6:02 PM, Corey Flowers > >> >>> <cflow...@onyxpoint.com> wrote: > >> >>> I haven't worked with this processor but I believe it is looking for > >> >>> the S3 list processor to generate the list of objects to fetch. Did > >> >>> you try that yet? > >> >> > >> >> I mentioned this: "There's no "ListS3Object" processor type which > >> >> might hypothetically populate > >> >> attributes for FetchS3Object to act upon." I should have made this > >> >> doubly explicit that I checked > >> >> in the processor creation dialogue. > >> >> > >> >> Also, this: > >> >> > >> >> > https://mail-archives.apache.org/mod_mbox/nifi-users/201510.mbox/%3cd23c06e8.ca0%25chakrader.dewaraga...@lifelock.com%3E > >> >> > >> >> "There is already a ticket > >> >> (NIFI-840<https://issues.apache.org/jira/browse/NIFI-840>) > >> >> in the hopper to create a ListS3Objects processor that can track > >> >> bucket contents and trigger > >> >> FetchS3Object." > >> >> > >> >> Oh god, it does appear that v0.4.1 ships with an implemented > >> >> FetchS3Object processor but no > >> >> List processor to feed it: > >> >> > >> >> https://issues.apache.org/jira/browse/NIFI-840 > >> >> > >> >> Status: unresolved > >> >> > >> >> Description: "A processor is needed that can provide an S3 listing to > >> >> use in conjunction with FetchS3Object. This is to provide a similar > >> >> user experience as with the HDFS processors that perform List/Get." > >> >> > >> >> I think this means I'm horked. And the Relationships section of the > >> >> FetchS3Object doc is still wrong. > >> >> > >> >> Russell > >> >> > >> >> > >> >>> Sent from my iPhone > >> >>> > >> >>>> On Jan 12, 2016, at 8:38 PM, Russell Whitaker > >> >>>> <russell.whita...@gmail.com> wrote: > >> >>>> > >> >>>> I'm running v0.4.1 Nifi, and seeing this (taken from nifi-app.log, > >> >>>> also seeing on mouseover of the "!" icon on the processor on the > >> >>>> canvas): > >> >>>> > >> >>>> 2016-01-12 17:08:50,357 ERROR [NiFi Web Server-18] > >> >>>> o.a.nifi.groups.StandardProcessGroup Unable to start > >> >>>> FetchS3Object[id=f4253204-a2e2-4ce6-ba09-9415e8024dca] due to {} > >> >>>> java.lang.IllegalStateException: Processor FetchS3Object is not in > a > >> >>>> valid state due to ['Upstream Connections' is invalid because > >> >>>> Processor requires an upstream connection but currently has none] > >> >>>> > >> >>>> Per: > >> >>>> > >> >>>> > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.aws.s3.FetchS3Object/index.html > , > >> >>>> FetchS3Object "Retrieves the contents of an S3 Object and writes it > >> >>>> to > >> >>>> the content of a FlowFile," which would seem to indicate this is an > >> >>>> "edge" processor that doesn't expect a flowfile from an upstream > >> >>>> processor. > >> >>>> > >> >>>> The "Tags" on the doc are: "Amazon, S3, AWS, Get, Fetch" > >> >>>> > >> >>>> The processor configuration settings themselves strongly indicate > it > >> >>>> expects to connect to S3 using the supplied > >> >>>> credentials/bucket/objectkey settings, with no upstream processor. > >> >>>> > >> >>>> But I get this error. What am I missing? There's no GetS3Object > >> >>>> anymore; surely this is the edge processor for directly downloading > >> >>>> S3 > >> >>>> objects, yes? There's no "ListS3Object" processor type which might > >> >>>> hypothetically populate attributes for FetchS3Object to act upon. > >> >>>> > >> >>>> Also, there are these obviously incorrect copy/paste lines in the > >> >>>> Relationships area of the API doc referenced above: > >> >>>> > >> >>>> "success - FlowFiles are routed to success after being successfully > >> >>>> copied to Amazon S3" > >> >>>> "failure - FlowFiles are routed to failure if unable to be copied > to > >> >>>> Amazon S3" > >> >>>> > >> >>>> No, that's obviously lifted from the PutS3Object doc page, where > it's > >> >>>> actually correct: > >> >>>> > >> >>>> > >> >>>> > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.aws.s3.PutS3Object/index.html > >> >>>> > >> >>>> Anyone have any insight into this? Thanks in advance. > >> >>>> > >> >>>> Russell > >> >> > >> >> -- > >> >> Russell Whitaker > >> >> http://twitter.com/OrthoNormalRuss > >> >> http://www.linkedin.com/pub/russell-whitaker/0/b86/329 > >> > >> > >> > >> -- > >> Russell Whitaker > >> http://twitter.com/OrthoNormalRuss > >> http://www.linkedin.com/pub/russell-whitaker/0/b86/329 > > > > > > > > > > -- > > Corey Flowers > > Vice President, Onyx Point, Inc > > (410) 541-6699 > > cflow...@onyxpoint.com > > > > -- This account not approved for unencrypted proprietary information -- > > > > -- > Russell Whitaker > http://twitter.com/OrthoNormalRuss > http://www.linkedin.com/pub/russell-whitaker/0/b86/329 >
