Joe, FWIW, I would personally avoid adding a Regex property. I think we use regexes already too much. I would just add the HTTP Headers or not... or you could even add a property that specifies whether or not to add the headers as attributes.
This is very much a preference here, though. I think regexes get messy too quickly though and are very difficult to write and maintain. -Mark > On Nov 9, 2015, at 3:35 PM, Joe Percivall <[email protected]> wrote: > > Mans, > > I did a bit of testing using my refactor and I understand what the issue is. > I was able to successfully complete the first use-case where "follow > redirect" is true and within the same ontrigger follow the redirect. > > The second use-case where the "follow redirect" is false didn't work as > intended. It ends up routing to "No Retry" and emitting no response flowfile. > The problem arises that the location header is not added to the request > flowfile so there is no way to know where it was supposed to redirect to > later. > > This actually brings up a bigger issue that is at the core of this issue. > That is when the processor reaches out to a website and routes to any of the > not 2xx relationships, the headers (other than status code and message) that > are returned by the response aren't stored anywhere. I'm thinking we have a > property that is a regular expression that allows users to match specific > headers to add as attributes to the request flowfile. Thoughts? > > Not trying to discourage you from contributing but I'm going to knock this > out as part of 1086. There are plenty of other tickets that could use > contributors! > > Thanks for bringing this up, > Joe > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: [email protected] > > > > > On Monday, November 9, 2015 2:05 PM, Joe Percivall <[email protected]> > wrote: > Hey Mans, > > Thanks for creating the ticket. First, have you tried the patch I supplied in > 1086 [1]? I changed the underlying implementation of InvokeHttp so it very > well could already be fixed. > > The functionality should work where if you have "follow redirects" to true > then it should follow the 3xx response to hit the other server automatically > within the same ontrigger call. If "follow redirects" is false then it would > not follow the redirect to the location set in the header and instead route > the request to the 3xx relationship. > > > [1] https://issues.apache.org/jira/browse/NIFI-1086 > Thanks, > Joe > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: [email protected] > > > > > > On Monday, November 9, 2015 1:54 PM, M Singh <[email protected]> wrote: > > > > Hi: > > I could not find Jira ticket for InvokeHTTP processor not saving location > header, so I've created a Jira ticket for this - [NIFI-1133] InvokeHTTP > Processor does not save Location header for 3xx responses - ASF JIRA. > > My understanding is that once the http status is saved and a relationship for > 3xx is available then we can route the flow file with appropriate attribute > (based on location header) to the invoke http processor so it can make a new > request. > > > Please let me know if it ok and if I can start working on it. > > Thanks > > > [NIFI-1133] InvokeHTTP Processor does not save Location header for 3xx > responses - ASF JIRA > InvokeHTTP processor does not save the Location header for 3xx responses. So, > we can cannot hook up the processor to route based on redirect location > header values. > View on issues.apache.org Preview by Yahoo > > > > > > On Monday, November 9, 2015 6:51 AM, Joe Witt <[email protected]> wrote: > > > > Mans, > > Being new to NiFi or even contributing to open source at all is not a > problem. We're here to help. Ask questions as needed and we can help > you contribute. > > Thanks > Joe > > > On Mon, Nov 9, 2015 at 9:38 AM, M Singh <[email protected]> wrote: >> Hi Joe: >> >> I was looking at the InvokeHttp code and I can work on enhancing the 3xx >> issue based on the pattern for handling other statuses. However, I would >> like to add a newbie caveat here. >> >> Let me know if that would help. >> >> Thanks >> >> Mans >> >> >> >> On Sunday, November 8, 2015 8:44 PM, M Singh <[email protected]> wrote: >> >> >> Hi Joe: >> >> You are right - setting follow-redirects did not work and I did mix retry >> with redirect. >> >> I will wait for your enhancements. >> >> Thanks again for your help. >> >> >> >> On Sunday, November 8, 2015 8:31 PM, Joe Percivall <[email protected]> >> wrote: >> >> >> Hello, >> >> >> Firstly, I think you're mixing up "retry" and "redirect". The 3xx status >> code is for redirecting to another url and 5xx is to try again. The property >> we have is "Follow Redirects". Retrying doesn't involve a location header >> but the redirect does. That being said, I did a bit of digging I don't think >> InvokeHttp was handling redirects properly. All we were doing was setting >> the "setInstanceFollowRedirects" to true, which according to this site [1] >> doesn't fully handle it. >> >> I am going to attach a patch to ticket 1086[2] tonight for InvokeHttp's >> refactor to use OkHttp instead of HttpUrlConnection. If you'd like to test >> that out and see if that it solves the redirect and location header problem >> that would be awesome. >> >> [1] >> http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/ >> [2] https://issues.apache.org/jira/browse/NIFI-1086 >> >> Joe >> - - - - - - >> Joseph Percivall >> linkedin.com/in/Percivall >> e: [email protected] >> >> >> >> >> On Sunday, November 8, 2015 9:53 PM, Joe Witt <[email protected]> wrote: >> >> >> >> Joe P, >> >> Do you have any other recommendations light of the work you're doing >> now to Invoke HTTP? >> >> Thanks >> Joe >> >> >> On Sun, Nov 8, 2015 at 9:49 PM, M Singh <[email protected]> wrote: >>> Thanks Joe. >>> >>> When I look at the provenance of the flow file, it shows the status as 301 >>> as shown below but no Location attribute. >>> >>> If I curl the url and check the headers it does show the Location >>> attribute. >>> >>> invokehttp.status.code 301 >>> >>> >>> >>> >>> On Sunday, November 8, 2015 4:58 PM, Joe Witt <[email protected]> wrote: >>> >>> >>> Hello >>> >>> You can use provenance or a LogAttributes processor to see what the >>> headers are of the flow file after InvokeHTTP executes. You may find >>> the location header present as one of the attributes. If so then you >>> should be able to use that attribute to establish the URL it will hit >>> next time. Does that help? >>> >>> Thanks >>> Joe >>> >>> On Sun, Nov 8, 2015 at 7:51 PM, M Singh <[email protected]> wrote: >>>> Hi: >>>> >>>> I am trying to use InvokeHTTP and have set the follow-retry to true, and >>>> have associated a self referencing relation for the InvokeHttp for >>>> no-retry >>>> (1xx, 2xx, and 3xx) and retry (5xx) relations. But it looks like it >>>> retires >>>> only status 500 codes requests and for 3xx does not pick up the Location >>>> header for retrying. >>>> >>>> However, since I am new to nifi - I might be missing something or using >>>> the >>>> wrong settings or wrong processor. >>>> >>>> If anyone has any suggestion, please let me know. >>>> >>>> Thanks >>>> >>> >>> >> >> >> >>
