Joe, Just as a quick observation, this statement isn't completely accurate:
> "... and can stream the contents instead of loading into memory" The original InvokeHTTP code (pre okhttp) explicitly set the content-length header, because it was known (the flowfile payload content length is always known). This does not, however, imply that the entire contents were loaded into memory. The previous InvokeHTTP used the #setFixedLengthStreamingMode(long), which is described as: "This method is used to enable streaming of a HTTP request body without internal buffering, when the content length is known in advance." [1] HttpURLConnection doesn't need to buffer if the length is known in advance. It's only when it doesn't know the length that it either needs to buffer to determine it or use chunked encoding. I think it's important to be able to support non-chunked encoded POST requests. There are many "legacy" (or even "broken") web services that don't work with chunked encoding, obviously like in this case. Unfortunately, I don't recall that okhttp has similar direct support for "fixed length streaming". It's probable that a custom implementation of okhttp.RequestBody would need to be created to support this. [2] [1] https://docs.oracle.com/javase/8/docs/api/java/net/HttpURLConnection.html#setFixedLengthStreamingMode-long- [2] http://square.github.io/okhttp/3.x/okhttp/okhttp3/RequestBody.html On Thu, Jan 14, 2016 at 10:29 PM, Joe Percivall < joeperciv...@yahoo.com.invalid> wrote: > Hello Evan, > > Glad to hear you're enjoying NiFi! > > I was able to replicate your results so I dug in a bit and noticed in > Wireshark that the "Transfer-Encoding" header for InvokeHttp was set to > "chunked". When I tried using the same flag for curl it failed so I'm > relatively confident that is the problem. Currently InvokeHttp requires > using the chunk encoding for POST (primarily because you don't need to know > the content-length and can stream the contents instead of loading into > memory). > > PostHttp does have a "Use Chunked Encoding" option which would solve your > problem except that it doesn't work properly. PostHttp is using the > "EntityTemplate" which streams the content so the content length will never > be implemented and thus it will alway use the chunked encoding. I created a > ticket for it [1]. > > > Also as a note, when creating a template you have to either explicitly > select the connections or not select anything and create a template for the > whole canvas (your template didn't have any connections). > > [1] https://issues.apache.org/jira/browse/NIFI-1396 > > Cheers, > Joe > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: joeperciv...@yahoo.com > > > > On Thursday, January 14, 2016 8:07 PM, "yuchen....@thomsonreuters.com" < > yuchen....@thomsonreuters.com> wrote: > > > > > Hi Guys, > > Not sure if it is the correct way to raise issue by sending this email, if > not, let me know where the post the issue, thanks. > > We are using NiFi InvokeHttp processor to do POST to an webpage. > URL: > http://www.hkexnews.hk/listedco/listconews/advancedsearch/search_active_main.aspx > Request header: Content-Type: application/x-www-form-urlencoded > POST Data: > txt_stock_code=24984&sel_DateOfReleaseFrom_y=2016&sel_DateOfReleaseFrom_m=01&sel_DateOfReleaseFrom_d=04&sel_DateOfReleaseTo_y=2016&sel_DateOfReleaseTo_m=01&sel_DateOfReleaseTo_d=11&IsFromNewList=False&sel_tier_1=-2&sel_tier_2_group=-2&sel_tier_2=-2 > > To make sure the request header and request body are correct, we use > Fiddler to compose the post request. > And the response show the request header and post data are correct. > > > Attached file is the template we are using, it is working fine on version > 0.3.0 > > But not on the latest version 0.4.1 > > So we suppose it is potential defect of the InvokeHttp processor in this > version. > We checked the source code and try to locate the issue, and found it is > using com.squareup.okhttp.Request; to do the request, so we are not go any > further to dig the issue… > Currently we are using Curl to do the POST as a workaround. > > Let me know your comments, thanks. > > Finally, NiFi is a great tool!!! You guys are awesome!!! > > Best Regards, > Evan from Thomson Reuters >
