On Tue, Jan 12, 2016 at 6:39 PM, Corey Flowers <cflow...@onyxpoint.com> wrote: > Hello Russell, > > Sorry if that seemed short, I was running in to pick my son up from > my practice.
Oh, I didn't mean to give the impression I'd taken it that way! > What I meant to say was that you are correct. Although I > haven't worked on those processors, I do believe it is expecting the listS3 > processor to function and that is why you are getting that error. I would > love to know if there is a work around because I would also love to work > with these processors within the next week or so. > I can think of a workaround that might involve wrapping s3cmd in an ExecuteStreamCommand processor, will let you know... Russell > On Tue, Jan 12, 2016 at 9:16 PM, Russell Whitaker > <russell.whita...@gmail.com> wrote: >> >> On Tue, Jan 12, 2016 at 6:11 PM, Corey Flowers <cflow...@onyxpoint.com> >> wrote: >> > Ha ha! Well that would do it! :) >> > >> >> I don't know what that would "do" other than confirm that the >> FetchS3Object processor shipped >> with v0.4.1 needs its doc to reflect the fact it's not yet useable >> untl a ListS3* processor is implemented >> and included in the narfile for the distribution. >> >> Russell >> >> > Sent from my iPhone >> > >> >> On Jan 12, 2016, at 9:10 PM, Russell Whitaker >> >> <russell.whita...@gmail.com> wrote: >> >> >> >>> On Tue, Jan 12, 2016 at 6:02 PM, Corey Flowers >> >>> <cflow...@onyxpoint.com> wrote: >> >>> I haven't worked with this processor but I believe it is looking for >> >>> the S3 list processor to generate the list of objects to fetch. Did >> >>> you try that yet? >> >> >> >> I mentioned this: "There's no "ListS3Object" processor type which >> >> might hypothetically populate >> >> attributes for FetchS3Object to act upon." I should have made this >> >> doubly explicit that I checked >> >> in the processor creation dialogue. >> >> >> >> Also, this: >> >> >> >> https://mail-archives.apache.org/mod_mbox/nifi-users/201510.mbox/%3cd23c06e8.ca0%25chakrader.dewaraga...@lifelock.com%3E >> >> >> >> "There is already a ticket >> >> (NIFI-840<https://issues.apache.org/jira/browse/NIFI-840>) >> >> in the hopper to create a ListS3Objects processor that can track >> >> bucket contents and trigger >> >> FetchS3Object." >> >> >> >> Oh god, it does appear that v0.4.1 ships with an implemented >> >> FetchS3Object processor but no >> >> List processor to feed it: >> >> >> >> https://issues.apache.org/jira/browse/NIFI-840 >> >> >> >> Status: unresolved >> >> >> >> Description: "A processor is needed that can provide an S3 listing to >> >> use in conjunction with FetchS3Object. This is to provide a similar >> >> user experience as with the HDFS processors that perform List/Get." >> >> >> >> I think this means I'm horked. And the Relationships section of the >> >> FetchS3Object doc is still wrong. >> >> >> >> Russell >> >> >> >> >> >>> Sent from my iPhone >> >>> >> >>>> On Jan 12, 2016, at 8:38 PM, Russell Whitaker >> >>>> <russell.whita...@gmail.com> wrote: >> >>>> >> >>>> I'm running v0.4.1 Nifi, and seeing this (taken from nifi-app.log, >> >>>> also seeing on mouseover of the "!" icon on the processor on the >> >>>> canvas): >> >>>> >> >>>> 2016-01-12 17:08:50,357 ERROR [NiFi Web Server-18] >> >>>> o.a.nifi.groups.StandardProcessGroup Unable to start >> >>>> FetchS3Object[id=f4253204-a2e2-4ce6-ba09-9415e8024dca] due to {} >> >>>> java.lang.IllegalStateException: Processor FetchS3Object is not in a >> >>>> valid state due to ['Upstream Connections' is invalid because >> >>>> Processor requires an upstream connection but currently has none] >> >>>> >> >>>> Per: >> >>>> >> >>>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.aws.s3.FetchS3Object/index.html, >> >>>> FetchS3Object "Retrieves the contents of an S3 Object and writes it >> >>>> to >> >>>> the content of a FlowFile," which would seem to indicate this is an >> >>>> "edge" processor that doesn't expect a flowfile from an upstream >> >>>> processor. >> >>>> >> >>>> The "Tags" on the doc are: "Amazon, S3, AWS, Get, Fetch" >> >>>> >> >>>> The processor configuration settings themselves strongly indicate it >> >>>> expects to connect to S3 using the supplied >> >>>> credentials/bucket/objectkey settings, with no upstream processor. >> >>>> >> >>>> But I get this error. What am I missing? There's no GetS3Object >> >>>> anymore; surely this is the edge processor for directly downloading >> >>>> S3 >> >>>> objects, yes? There's no "ListS3Object" processor type which might >> >>>> hypothetically populate attributes for FetchS3Object to act upon. >> >>>> >> >>>> Also, there are these obviously incorrect copy/paste lines in the >> >>>> Relationships area of the API doc referenced above: >> >>>> >> >>>> "success - FlowFiles are routed to success after being successfully >> >>>> copied to Amazon S3" >> >>>> "failure - FlowFiles are routed to failure if unable to be copied to >> >>>> Amazon S3" >> >>>> >> >>>> No, that's obviously lifted from the PutS3Object doc page, where it's >> >>>> actually correct: >> >>>> >> >>>> >> >>>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.aws.s3.PutS3Object/index.html >> >>>> >> >>>> Anyone have any insight into this? Thanks in advance. >> >>>> >> >>>> Russell >> >> >> >> -- >> >> Russell Whitaker >> >> http://twitter.com/OrthoNormalRuss >> >> http://www.linkedin.com/pub/russell-whitaker/0/b86/329 >> >> >> >> -- >> Russell Whitaker >> http://twitter.com/OrthoNormalRuss >> http://www.linkedin.com/pub/russell-whitaker/0/b86/329 > > > > > -- > Corey Flowers > Vice President, Onyx Point, Inc > (410) 541-6699 > cflow...@onyxpoint.com > > -- This account not approved for unencrypted proprietary information -- -- Russell Whitaker http://twitter.com/OrthoNormalRuss http://www.linkedin.com/pub/russell-whitaker/0/b86/329
