Hello,
Sorry for top posting, but has there been any progress in getting the
ability to rewrite response body with Lua in HAproxy (easy way)? I would
assume AppletHTTP could be used for this, but I see that http-response
doesn't support use-service.
Regards,
Nenad
On 10/26/2015 12:00 PM, Thierry FOURNIER wrote:
> On Sun, 25 Oct 2015 02:09:15 +0100
> PiBa-NL <piba.nl....@gmail.com> wrote:
>
>> Hi Thierry, haproxy-list,
>>
>> Op 19-10-2015 om 11:24 schreef thierry.fourn...@arpalert.org:
>>> On Mon, 19 Oct 2015 01:31:42 +0200
>>> PiBa-NL <piba.nl....@gmail.com> wrote:
>>>
>>>> Hi Thierry,
>>>>
>>>> Op 18-10-2015 om 21:37 schreef thierry.fourn...@arpalert.org:
>>>>> On Sun, 18 Oct 2015 00:07:13 +0200
>>>>> PiBa-NL <piba.nl....@gmail.com> wrote:
>>>>>
>>>>>> Hi haproxy list,
>>>>>>
>>>>>> For testing purposes i am trying to 'modify' a response of a webserver
>>>>>> but only having limited success. Is this supposed to work?
>>>>>> As a more usefull goal than the current LAL to TST replacement i imagine
>>>>>> rewriting absolute links on a webpage could be possible which is
>>>>>> sometimes problematic with 'dumb' webapplications..
>>>>>>
>>>>>> Or is it outside of the current scope of implemented functionality? If
>>>>>> so, it on the 'lua todo list' ?
>>>>>>
>>>>>> I tried for example a configuration like below. And get several
>>>>>> different results in the browser.
>>>>>> -Sometimes i get 4 times TSTA
>>>>>> -Sometimes i see after the 8th TSTA- Connection: keep-alive << this
>>>>>> happens most of the time..
>>>>>> -Sometimes i get 9 times TSTA + STOP << this would be the desired
>>>>>> outcome (only seen very few times..)
>>>>>>
>>>>>> Probably due to the response-buffer being filled differently due to
>>>>>> 'timing'..
>>>>>>
>>>>>> The "connection: keep-alive" text is probably from the actual server
>>>>>> reply which is 'appended' behind the response generated by my lua
>>>>>> script.?. However shouldn't the .done() prevent that from being send to
>>>>>> the client?
>>>>>>
>>>>>> Ive tried putting a loop into the lua script to call res:get() multiple
>>>>>> times but that didnt seem to work..
>>>>>>
>>>>>> Also to properly modify a page i would need to know all changes before
>>>>>> sending the headers with changed content-length back to the client..
>>>>>>
>>>>>> Can someone confirm this is or isn't (reliably) possible? Or how this
>>>>>> can be scripted in lua differently?
>>>>> Hello,
>>>>>
>>>>> Your script replace 3 bytes by 3 bytes, this must run with HTTP, but if
>>>>> your replacement change the length of the response, you can have some
>>>>> difficulties with clients, or with keepalive.
>>>> Yes i started with replacing with the same number of bytes to avoid some
>>>> of the possible troubles caused by changing the length.. And as seen in
>>>> the haproxy.cfg it is configured with 'mode http'.
>>>>> The res:get(), returns the current content of the response buffer.
>>>>> Maybe it not contains the full response. You must execute a loop with
>>>>> regular "core.yield()" to get back the hand to HAProxy and wait for new
>>>> Calling yield does allow to 'wait' for more data to come in.. No
>>>> guarantee that it only takes 1 yield for data to 'grow'..
>>>>
>>>> [info] 278/055943 (77431) : luahttpresponse Content-Length XYZ: 14115
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 2477
>>>> [info] 278/055943 (77431) : luahttpresponse LOOP
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 6221
>>>> [info] 278/055943 (77431) : luahttpresponse LOOP
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 7469
>>>> [info] 278/055943 (77431) : luahttpresponse LOOP
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 7469
>>>> [info] 278/055943 (77431) : luahttpresponse LOOP
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 7469
>>>> [info] 278/055943 (77431) : luahttpresponse LOOP
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 7469
>>>> [info] 278/055943 (77431) : luahttpresponse LOOP
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 7469
>>>> [info] 278/055943 (77431) : luahttpresponse LOOP
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 7469
>>>> [info] 278/055943 (77431) : luahttpresponse LOOP
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 7469
>>>> [info] 278/055943 (77431) : luahttpresponse LOOP
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 8717
>>>> [info] 278/055943 (77431) : luahttpresponse LOOP
>>>> [info] 278/055943 (77431) : luahttpresponse SIZE: 14337
>>>> [info] 278/055943 (77431) : luahttpresponse DONE?: 14337
>>>>
>>>>> data. When all the data are read, res:get() returns an error.
>>>> Not sure when/how this error would happen.? The result of res:get only
>>>> seems to get bigger while the webserver is sending the response..
>>>>> The res:send() is dangerous because it send data directly to the client
>>>>> without the end of haproxy analysis. Maybe it is the cause o your
>>>>> problem.
>>>>>
>>>>> Try to use res:set().
>>>> Ok tried that, new try with function below.
>>>>> The difficulty is that another "res:get()" returns the same data that
>>>>> these you put.
>>>>>
>>>>> I don't known if you can modify an http response greater than one
>>>>> buffer.
>>>> Would be nice if that was somehow possible. But my current lua script
>>>> cannot..
>>>>> The function res:close() closes the connection even if HAProxy want to
>>>>> keep the connection alive. I suggest that you don't use this function.
>>>> It seems txn.res:close() does not exist? txn:done()
>>>>> I reproduce the error message using curl. By default curl tries
>>>>> to transfer data with keepalive, and it is not happy if the all the
>>>>> announced data are not transfered.
>>>>>
>>>>> Connection: keep-alive curl: (18) transfer closed with outstanding
>>>>> read data remaining
>>>>>
>>>>> It seems that i reproduce a bug. I'm looking for.
>>>> Ok if you can create a patch, let me know. Happy to test if it solves
>>>> some of the issues i see.
>>>
>>> Hi,
>>>
>>> I catch the error. While we execute http actions, the content of the
>>> buffer must contains the http headers, otherwise, the request is
>>> invalid.
>>>
>>> It is because the http action are applied on the content and not on
>>> the stream. So this is th behaviour:
>>>
>>> -> The function res:get() remove the data from the haproxy input buffer
>>>
>>> -> The function res:send() send the modified data directly to the
>>> client (throught the output buffer), this data is not yet avalaible
>>> in the input buffer.
>>>
>>> -> Now HAProxy is locked in the processing, because it wait for a
>>> valid reponse header for continuing the Lua processing.
>>>
>>> This behaviour is not really a bug, it is the process who garantee that
>>> a valid http response is present before continuing the processing.
>>>
>>> Maybe, a patch will send an error in the case of we enter in an action
>>> with a valid request/response and we out from the processing without a
>>> valid request/response.
>>>
>>> So, regarding the HAProxy global behaviour, the action are designed for
>>> manipulating http header, and not the body.
>>>
>>> If you want to manipulates the body, you can use a tcp action.
>> Ok the script below works, even for larger responses and sets the new
>> content-length.. That is.. for my single test page i tried it with.
>>
>> It could be troublesome that the lua script must take into account every
>> variation that might happen like chunked/compressed bodies, and parsing
>> the headers. Was kinda hoping that haproxy in http mode would be able to
>> take care of at least some of these kind of things. All advantages of
>> running a haproxy frontent in 'mode http' are now nolonger available so
>> thats a downside in my opinion..
>
>
> You're absolutely right. The http action allow only the header
> manipulation and not the body manipulation. So a new Lua binding is
> currently discussed. This will be pluged between the receive buffer and
> the send buffer. It will be dedicated for protocol transformation.
>
> This enhancement requires heavy modification in HAProxy. Luckyly, these
> modification will be the common with http/2.
>
>
>> Also the script below buffers the whole response (which must have a
>> content-length header) before forwarding it to the client. This could
>> significantly increase the memoryusage of a single connection through
>> haproxy..
>
>
> I agree again. It increase also the latency :(.
>
>
>> But hey, it does open up a world of new 'rewriting'
>> possibilities for those backends that really really really need it..
>
>
> I have no doubts about this.
>
>
>> Anyway i wanted to share this for those people that might be in need of
>> doing something similar..
>>
>> And if anyone knows a way to simplify any part of the lua script please
>> let me know :)
>> Yes searching the start and end of the content-length header twice isnt
>> needed, and the txn:Info() statements should be removed.. but im asking
>> more for other 'major' improvements.
>
>
> Thank you for sharing.
>
> Thierry
>
>
>> Best regards,
>> PiBa-NL
>>
>> listen proxytcpresponse
>> bind :10009
>> mode tcp
>> tcp-response content lua.luatcpresponse
>> server x 192.168.0.40:302
>>
>> function luatcpresponse(txn)
>> txn:Info("#### TCP RESPONSE ENTERING LUA FUNCTION ###\n")
>> local responsecomplete = ""
>> local headersparsed = false
>> local headerlength = 0
>> local contentsize = 0
>> local response2 = txn.res:get()
>> local headers = ""
>> while (response2) and (string.len(response2) > 0) do
>> txn:Info("lua tcp response LOOP")
>> txn:Info("lua tcp response SIZE: " .. string.len(response2))
>> response2 = string.gsub(response2,"LAL","TST")
>> responsecomplete = responsecomplete .. response2
>>
>> if (not headersparsed) and
>> (string.find(responsecomplete,"\r\n\r\n") > 0) then
>> txn:Info("headers complete")
>>
>> headersparsed = true
>> _ , headerlength = string.find(responsecomplete,"\r\n\r\n")
>> headers = string.sub(responsecomplete,0,headerlength)
>> local cl,cle = string.find(headers, "Content%-Length: ")
>> local sizeend = string.find(responsecomplete,"\r\n", cle)
>> contentsize = tonumber(string.sub(headers, cle+1,
>> sizeend-1))
>>
>> txn:Info("Headersize: " .. headerlength)
>> txn:Info("Content-Length: " .. contentsize)
>> responsecomplete = string.sub(responsecomplete,
>> headerlength+1)
>> end
>> txn:Info("RES size: " .. string.len(responsecomplete) .. "
>> H:" .. headerlength .. " S:" .. contentsize)
>> if (string.len(responsecomplete) == contentsize) then
>> txn:Info("RESPONSE COMPLETE")
>> local i = 0
>>
>> -- remove the scheme and host part from response href
>> links.. (nothing checked..)
>> responsecomplete =
>> string.gsub(responsecomplete,"href=\"http://localhost:302/","href=\"/";)
>>
>> local ressize = string.len(responsecomplete)
>> local cl,cle = string.find(headers, "Content%-Length: ")
>> local sizeend = string.find(headers,"\r\n", cle)
>> local firstheaders = string.sub(headers, 0, cl-1)
>> local lastheaders = string.sub(headers, sizeend)
>> headers = firstheaders .. "Content-Length: " .. ressize
>> .. lastheaders
>> txn.res:send(headers .. responsecomplete)
>> responsecomplete = ""
>> headersparsed = false
>> end
>>
>> response2 = txn.res:get()
>> end
>> txn:Info("TCP RESPONSE LUA FUNCTION, EXIT\n")
>> end
>>
>> core.register_action("luatcpresponse" , { "tcp-res" }, luatcpresponse);
>>
>>
>>
>>>
>>> You can use also an http action, but the buffer must contains a valid
>>> http request each time when HAProxy give back the hand. In this case,
>>> you cannot modify requests greater than an haproxy buffer.
>>>
>>> res:get() -> get all the data and remove it from the input buffer
>>> res:dup() -> just duplicates data
>>> res:set() -> set data in the input buffer, HAProxy will continue the
>>> analysis.
>>> res:send()-> send data in the output buffer, HAProy cannot analyse
>>> these data.
>>>
>>> Thierry
>>>
>>>>> Thierry
>>>>>
>>>> This function seems to work for responses up to +-15KB.
>>>> Sometimes the number of loops it runs is different, and it seems kinda
>>>> in-efficient to just run loops until the response is 'complete', another
>>>> strange observation is that the res:set inside the loop is required,
>>>> even though it doesn't set a modified response, eventually the complete
>>>> response is modified in the browser result. Second request over a
>>>> keep-alive connection also fails. Adding http-server-close also closes
>>>> the client connection, but does avoid the problem with the second
>>>> request.. In the lua script I dont account for headers size yet when
>>>> checking if the response is completely read, but i dont think thats
>>>> affected the test..
>>>>
>>>> Is there anything i can do to improve this function? (Besides removing
>>>> the txn:Info() lines.)
>>>>
>>>> function luahttpresponse(txn)
>>>> local resheaders = txn.http:res_get_headers()
>>>> local contentlength = tonumber(resheaders["content-length"][0])
>>>> local response2 = txn.res:get()
>>>> txn:Info("luahttpresponse Content-Length XYZ: " .. contentlength)
>>>> txn:Info("luahttpresponse SIZE: " .. string.len(response2))
>>>> while string.len(response2) < contentlength do
>>>> txn:Info("luahttpresponse LOOP")
>>>> txn.res:set(response2)
>>>> core.yield()
>>>> response2 = txn.res:get()
>>>> txn:Info("luahttpresponse SIZE: " .. string.len(response2))
>>>> end
>>>> response2 = string.gsub(response2,"LAL","TST")
>>>> txn.res:set(response2)
>>>> txn:Info("luahttpresponse DONE?: " .. string.len(response2))
>>>> end
>>>>
>>>> Regards,
>>>> PiBa-NL
>>>>>> Thanks in advance,
>>>>>> PiBa-NL
>>>>>>
>>>>>> ## haproxy.cfg
>>>>>> listen proxyresponse
>>>>>> bind :10006
>>>>>> mode http
>>>>>> http-response lua.luahttpresponse
>>>>>> server x 192.168.0.40:302
>>>>>>
>>>>>> ## script.lua
>>>>>> function luahttpresponse(txn)
>>>>>> local response2 = txn.res:get()
>>>>>> response2 = string.gsub(response2,"LAL","TST")
>>>>>> txn.res:send(response2)
>>>>>> txn.done()
>>>>>> end
>>>>>> core.register_action("luahttpresponse" , { "http-res" },
>>>>>> luahttpresponse);
>>>>>>
>>>>>> ## webpage.aspx , with 2.7KB output.
>>>>>> ------------------
>>>>>> <%@ Page Language="C#" %>
>>>>>> <html><body>START
>>>>>> <% var x = new String('x', 250);
>>>>>> x = "<input type=\"hidden\" value=\""+x+"\" />"; %>
>>>>>> 1 -LALA- <% Response.Write(x); %>
>>>>>> 2 -LALA- <% Response.Write(x); %>
>>>>>> 3 -LALA- <% Response.Write(x); %>
>>>>>> 4 -LALA- <% Response.Write(x); %>
>>>>>> 5 -LALA- <% Response.Write(x); %>
>>>>>> 6 -LALA- <% Response.Write(x); %>
>>>>>> 7 -LALA- <% Response.Write(x); %>
>>>>>> 8 -LALA- <% Response.Write(x); %>
>>>>>> 9 -LALA- <% Response.Write(x); %>
>>>>>> STOP
>>>>>> --------------------
>>>>>>
>>>>>>
>>>>
>>
>>
>
