Adam, A PenalizeFlowFile processor could be pretty useful, please feel free to file a New Feature Jira for this if you like.
In the meantime you could use ExecuteScript (with Groovy for this example) and the following: def flowFile = session.get() if(!flowFile) return flowFile = session.penalize(flowFile) session.transfer(flowFile, REL_SUCCESS) In this case the "success" relationship is awkward, it means you successfully penalized the flow file. But then you can route it back/forward to the appropriate processor. If you create a template from this single processor, then dragging the template onto the canvas is somewhat equivalent to dragging a "PenalizeFlowFile" processor onto the canvas (meaning I suggest the template is named PenalizeFlowFile). Regards, Matt On Wed, Aug 31, 2016 at 10:46 PM, Adam Taft <[email protected]> wrote: > In the wild west of HTTP response codes, a 500 Server Error could mean > practically anything. In my experience, you can't infer any semantic > meaning for what a 500 status code could mean, unless you're very familiar > with the server application. > > I'd even go so far as to suggest, if a modification is made to PostHTTP, > that all non-200 response codes should be penalized. The dataflow manager > can always adjust the penalization timeout towards zero if a processing > delay is not warranted. > > Unrelated, but this also reminds me, we really need a PenalizeFlowFile > processor, which would allow a dataflow manager to penalize a flowfile > anywhere that is deemed necessary, even if other processors haven't done so > (have routed to success). > > > On Wed, Aug 31, 2016 at 1:54 PM, Andrew Grande <[email protected]> wrote: > >> Wasn't HTTP 400 Bad Request meant for that? 500 only means the server >> failed, not necessarily due to user input. >> >> Andrew >> >> On Wed, Aug 31, 2016, 10:16 AM Mark Payne <[email protected]> wrote: >> >> > Hey Chris, >> > >> > I think it is reasonable to penalize when we receive a 500 response. 500 >> > means Internal Server Error, and it is >> > very reasonable to believe that the Internal Server Error occurred due to >> > the specific input (i.e., that it may not >> > always occur with different input). So penalizing the FlowFile so that it >> > can be retried after a little bit is reasonable >> > IMO. >> > >> > When using the prioritizers, any FlowFile that is penalized will not hold >> > up other FlowFiles. They are always at the >> > bottom of the queue until the penalization expires. >> > >> > Thanks >> > -Mark >> > >> > >> > > On Aug 31, 2016, at 10:06 AM, McDermott, Chris Kevin (MSDU - >> > STaTS/StorefrontRemote) <[email protected]> wrote: >> > > >> > > I wanted to ask if it would be at all sane to have the PostHTTP >> > processor penalize a flowfile on 5xx response. 5xx indicates that the >> > request may be good but it cannot be handle by the server Currently it >> > seems the processor routes files eliciting this response to the failure >> > output but does not penalize them. What do we think of adding such >> > penalization? >> > > >> > > On a related note. If a file penalized file is routed to a funnel that >> > is connect to a processor via a connection with the OldestFlowFileFirst >> > prioritizer will the consumption of files from that connection be blocked >> > until penalization period is over? >> > > >> > > What I am trying to accomplish is this: I am using PostHTTP to send >> > files to web service that is throttling incoming data by returning a 500 >> > response. When that happens I want to slow down files being to that that >> > service. >> > > >> > > Thanks, >> > > >> > > Chris McDermott. >> > > >> > > Remote Business Analytics >> > > STaTS/StoreFront Remote >> > > HPE Storage >> > > Hewlett Packard Enterprise >> > > Mobile: +1 978-697-5315 >> > > >> > > >> > >> > >>
