Mans, I did a bit of testing using my refactor and I understand what the issue is. I was able to successfully complete the first use-case where "follow redirect" is true and within the same ontrigger follow the redirect.
The second use-case where the "follow redirect" is false didn't work as intended. It ends up routing to "No Retry" and emitting no response flowfile. The problem arises that the location header is not added to the request flowfile so there is no way to know where it was supposed to redirect to later. This actually brings up a bigger issue that is at the core of this issue. That is when the processor reaches out to a website and routes to any of the not 2xx relationships, the headers (other than status code and message) that are returned by the response aren't stored anywhere. I'm thinking we have a property that is a regular expression that allows users to match specific headers to add as attributes to the request flowfile. Thoughts? Not trying to discourage you from contributing but I'm going to knock this out as part of 1086. There are plenty of other tickets that could use contributors! Thanks for bringing this up, Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Monday, November 9, 2015 2:05 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: Hey Mans, Thanks for creating the ticket. First, have you tried the patch I supplied in 1086 [1]? I changed the underlying implementation of InvokeHttp so it very well could already be fixed. The functionality should work where if you have "follow redirects" to true then it should follow the 3xx response to hit the other server automatically within the same ontrigger call. If "follow redirects" is false then it would not follow the redirect to the location set in the header and instead route the request to the 3xx relationship. [1] https://issues.apache.org/jira/browse/NIFI-1086 Thanks, Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Monday, November 9, 2015 1:54 PM, M Singh <mans2si...@yahoo.com> wrote: Hi: I could not find Jira ticket for InvokeHTTP processor not saving location header, so I've created a Jira ticket for this - [NIFI-1133] InvokeHTTP Processor does not save Location header for 3xx responses - ASF JIRA. My understanding is that once the http status is saved and a relationship for 3xx is available then we can route the flow file with appropriate attribute (based on location header) to the invoke http processor so it can make a new request. Please let me know if it ok and if I can start working on it. Thanks [NIFI-1133] InvokeHTTP Processor does not save Location header for 3xx responses - ASF JIRA InvokeHTTP processor does not save the Location header for 3xx responses. So, we can cannot hook up the processor to route based on redirect location header values. View on issues.apache.org Preview by Yahoo On Monday, November 9, 2015 6:51 AM, Joe Witt <joe.w...@gmail.com> wrote: Mans, Being new to NiFi or even contributing to open source at all is not a problem. We're here to help. Ask questions as needed and we can help you contribute. Thanks Joe On Mon, Nov 9, 2015 at 9:38 AM, M Singh <mans2si...@yahoo.com> wrote: > Hi Joe: > > I was looking at the InvokeHttp code and I can work on enhancing the 3xx > issue based on the pattern for handling other statuses. However, I would > like to add a newbie caveat here. > > Let me know if that would help. > > Thanks > > Mans > > > > On Sunday, November 8, 2015 8:44 PM, M Singh <mans2si...@yahoo.com> wrote: > > > Hi Joe: > > You are right - setting follow-redirects did not work and I did mix retry > with redirect. > > I will wait for your enhancements. > > Thanks again for your help. > > > > On Sunday, November 8, 2015 8:31 PM, Joe Percivall <joeperciv...@yahoo.com> > wrote: > > > Hello, > > > Firstly, I think you're mixing up "retry" and "redirect". The 3xx status > code is for redirecting to another url and 5xx is to try again. The property > we have is "Follow Redirects". Retrying doesn't involve a location header > but the redirect does. That being said, I did a bit of digging I don't think > InvokeHttp was handling redirects properly. All we were doing was setting > the "setInstanceFollowRedirects" to true, which according to this site [1] > doesn't fully handle it. > > I am going to attach a patch to ticket 1086[2] tonight for InvokeHttp's > refactor to use OkHttp instead of HttpUrlConnection. If you'd like to test > that out and see if that it solves the redirect and location header problem > that would be awesome. > > [1] > http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/ > [2] https://issues.apache.org/jira/browse/NIFI-1086 > > Joe > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: joeperciv...@yahoo.com > > > > > On Sunday, November 8, 2015 9:53 PM, Joe Witt <joe.w...@gmail.com> wrote: > > > > Joe P, > > Do you have any other recommendations light of the work you're doing > now to Invoke HTTP? > > Thanks > Joe > > > On Sun, Nov 8, 2015 at 9:49 PM, M Singh <mans2si...@yahoo.com> wrote: >> Thanks Joe. >> >> When I look at the provenance of the flow file, it shows the status as 301 >> as shown below but no Location attribute. >> >> If I curl the url and check the headers it does show the Location >> attribute. >> >> invokehttp.status.code 301 >> >> >> >> >> On Sunday, November 8, 2015 4:58 PM, Joe Witt <joe.w...@gmail.com> wrote: >> >> >> Hello >> >> You can use provenance or a LogAttributes processor to see what the >> headers are of the flow file after InvokeHTTP executes. You may find >> the location header present as one of the attributes. If so then you >> should be able to use that attribute to establish the URL it will hit >> next time. Does that help? >> >> Thanks >> Joe >> >> On Sun, Nov 8, 2015 at 7:51 PM, M Singh <mans2si...@yahoo.com> wrote: >>> Hi: >>> >>> I am trying to use InvokeHTTP and have set the follow-retry to true, and >>> have associated a self referencing relation for the InvokeHttp for >>> no-retry >>> (1xx, 2xx, and 3xx) and retry (5xx) relations. But it looks like it >>> retires >>> only status 500 codes requests and for 3xx does not pick up the Location >>> header for retrying. >>> >>> However, since I am new to nifi - I might be missing something or using >>> the >>> wrong settings or wrong processor. >>> >>> If anyone has any suggestion, please let me know. >>> >>> Thanks >>> >> >> > > > >
