Is there any means to tell how full my StandardHttpContextMap is getting as Http requests are processed and/or fail processing? Keep alive messages of 0 bytes show up as flowFiles via Http Request. I filter those out but do not Http Respond to them as the original message may still be in process and so still requires the connection (to respond to the requestor). I suspect these keep alives are leaving entries in the StandardHttpContextMap, and are not being purged by Request Expiration. Is there any means to flush results from StandardHttpContextMap? Mine fills up over a few days flow and then blocks the acceptance of any other Http requests. I do set Request Expiration on it. I had originally set to 10 min, but have dropped that to 2 min. Thank you.
On Wed, Feb 22, 2017 at 11:08 AM, James McMahon <[email protected]> wrote: > Mark, based on what you've said don't "failure orphans" populate on the > Context Map in many other cases too, not just failure at the Response? I > have Failure from ExecuteScriptf eeding back into my ExecuteScript > processor, for example. I gather that is not advisable, and each and every > possible failure from any processor should flow to a HandleHttpResponse > processor. Is that correct? That would still not help us overcome the > problem of a Failure at the Response - but JIRA 3517 should solve that > problem when it is implemented. > > On Wed, Feb 22, 2017 at 10:36 AM, Mark Payne <[email protected]> wrote: > >> Jim, >> >> re: two Success paths - Yes, you should send only one of them to the >> HandleHttpResponse. I'm curious though - why use >> a disabled processor and queue data up instead of using the Data >> Provenance feature? >> >> Yes, StandardHttpContextMap should be removing any entries on its own >> that exceed the timeout. How many requests per >> second are you seeing? I am assuming that you are receiving a pretty high >> rate if it is at the point of containing over 10K entries >> with a 10 minute timeout. If you are not seeing that many requests, then >> there may be something else going on there. >> >> Thanks >> -Mark >> >> >> >> On Feb 22, 2017, at 10:31 AM, James McMahon <[email protected]> wrote: >> >> Additional questions about this. Immediately following my >> HandleHttpRequest processor, I have an ExecuteScript processor that then >> sends flowfile copies out two Success paths. one path eventually culminates >> in a HandleHttpResponse that has the aforementioned auto-termintaion of >> Success and Failure results. The second path is to a MonitorActivity >> processor that is disabled, to permit me to queue up and review incoming >> flowfile results after ExecuteScript during dev and test. Does that second >> path also have to send a response? Isn't it enough that the ContextMap is >> cleared by the response from the first path? >> >> Second question: how does this ever happen? Doesn't the Request >> Expiration I set on the StandradHttpContextMap force the obliteration of >> all entries that age beyond that point? >> >> Jim >> >> On Wed, Feb 22, 2017 at 10:13 AM, James McMahon <[email protected]> >> wrote: >> >>> I may well have that Mark. I have a number of paths where I have >>> HandleHttpResponse that auto terminates Failures. That would cause such a >>> problem, wouldn't it? >>> >>> How do people handle this situation: app does a POST, and so we handle >>> the request. App closes or timesout for whatever the reason may be. The >>> HanldeHttpResponse is unable to reply. Should those not be auto terminated? >>> >>> In a situation like this then Mark, are these the steps to recover? >>> 1. HanldeHttpResponse at end of all paths >>> 2. do not autoterminate failure conditions >>> 3. DELETE the StandardHttpContextMap (to clear the log jam) >>> 4. Recreate it fresh, which I presume creates it empty (I hope) >>> >>> What else must I do to recover? And how do I properly handle those >>> "broken connection" situations? >>> >>> On Wed, Feb 22, 2017 at 10:06 AM, Mark Payne <[email protected]> >>> wrote: >>> >>>> Jim, >>>> >>>> You likely have a path through your flow where you are receiving an >>>> HTTP Request via HandleHttpRequest >>>> but you never respond via a HandleHttpResponse. When using these >>>> processors, it's important that every >>>> incoming FlowFile go to a HandleHttpResponse processor. Do you have >>>> some path in your flow where you >>>> are not responding to the request? >>>> >>>> Thanks >>>> -Mark >>>> >>>> >>>> > On Feb 22, 2017, at 9:58 AM, James McMahon <[email protected]> >>>> wrote: >>>> > >>>> > I am getting the following errors when my users attempt to use curl >>>> or python to post to my HandleHttpRequest processor (cannot export actual >>>> messages, must select pieces and retype here): >>>> > WARNING >>>> > Received request from [IP address is here] but could not process it >>>> because too many requests are already outstaning; responding with >>>> SERVICE_UNAVAILABLE >>>> > ERROR >>>> > ...claim=StandardContentClaim.... >>>> > transfer relationship not specified >>>> > >>>> > None of my apps can post to NiFi. >>>> > >>>> > I have a StandradSSLContextService and a standradHttpContextMap, both >>>> of which are enabled. I suspect I may have inadvertently caused this >>>> problem by setting my ContextMap parameters badly. Here are those params: >>>> > Maximum Outstanding Requests: 10000 >>>> > Request Expiration 10 min >>>> > >>>> > I've looked across my workflow and no flowfiles are queued up. So my >>>> expectation is that there should be ample space in my ContextMap. But these >>>> errors indicate otherwise. How do I fix this? >>>> > Thanks very much in advance for your help. >>>> > Jim >>>> >>>> >>> >> >> >
