On Wed, Sep 15, 2021 at 12:36 AM Christopher Faulet <[email protected]> wrote:
> Le 9/14/21 à 3:14 AM, Ryan Burn a écrit : > > > > > > On Thu, Sep 9, 2021 at 12:22 AM Christopher Faulet <[email protected] > > <mailto:[email protected]>> wrote: > > > > Le 8/11/21 à 2:53 AM, Ryan Burn a écrit : > > > I'm working on integrating HAProxy with traceable.ai > > <http://traceable.ai> <http://traceable.ai <http://traceable.ai>>'s > > > security product. > > > > > > As part of the integration, we'd like to capture the contents of > any http > > > responses processed by HAProxy and send them to a service either > via SPOA > > or an > > > RPC call from Lua. The response contents are used by the product > to help > > > identify possible security threats. > > > > > > I've tried a few things, but haven't found a reliable way to > capture the > > > contents of response bodies. Is this possible with HAProxy? > > > > > > Here are the approaches I've explored so far: > > > > > > 1. I used the "res.body" fetch but that only provides the contents > > sometimes (I > > > presume if it's available in a buffer): > > > > > > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/extcap.conf#L19 > > < > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/extcap.conf#L19 > > > > > > > > > < > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/extcap.conf#L19 > > < > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/extcap.conf#L19 > >> > > > > > > 2. I also tried accessing the contents of the response channel > from a Lua > > > action, but that fails with "Cannot manipulate HAProxy channels > in HTTP mode" > > > > > > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/response.lua#L5 > > < > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/response.lua#L5 > > > > > > > > > < > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/response.lua#L5 > > < > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/response.lua#L5 > >> > > > > > > About the sample fetches, on HAProxy 2.3 and lower, there is no way > to get the > > response payload because it is not possible to wait for it. There is > no > > equivalent to the "http-buffer-request" option on the response side. > On > > HAProxy-2.4, it is possible by using "wait-for-body" HTTP rule, > available on > > the > > request and the response side. However, it is still limited by the > buffer size. > > > > > > Thanks Christopher! Do you know how to access the response body from a > SPOA if > > you add the "wait-for-body"? > > > > I added the wait-for-proxy rules to my example project, but the > "res.body" > > argument still doesn't consistently provide the full body. > > > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/haproxy.cfg#L15-L16 > > < > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/haproxy.cfg#L15-L16 > > > > I've checked your configuration and your SPOE message is sent on the > 'on-http-response' event. This event is triggered before 'http-response' > ruleset > evaluation. Thus the 'wait-for-body' action is not performed yet at this > stage. > Here, you should use a SPOE group and send it using 'send-spoe-group' > action. > The same should be done on the request side. > > -- > Christopher Faulet > Thanks Christopher! This is working for us after making this change. On the size limit, ideally we'd like to capture up to the first 128k of the body. But after doing some tests, it looks like we can only get up the first 15k, even if we specify "http-response wait-for-body time 10s at-least 128k". When you say the capture is limited by the buffer size, do you mean this size https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#3.2-tune.bufsize ?
