On Mon, 19 Oct 2015 01:31:42 +0200 PiBa-NL <[email protected]> wrote:
> Hi Thierry, > > Op 18-10-2015 om 21:37 schreef [email protected]: > > On Sun, 18 Oct 2015 00:07:13 +0200 > > PiBa-NL <[email protected]> wrote: > > > >> Hi haproxy list, > >> > >> For testing purposes i am trying to 'modify' a response of a webserver > >> but only having limited success. Is this supposed to work? > >> As a more usefull goal than the current LAL to TST replacement i imagine > >> rewriting absolute links on a webpage could be possible which is > >> sometimes problematic with 'dumb' webapplications.. > >> > >> Or is it outside of the current scope of implemented functionality? If > >> so, it on the 'lua todo list' ? > >> > >> I tried for example a configuration like below. And get several > >> different results in the browser. > >> -Sometimes i get 4 times TSTA > >> -Sometimes i see after the 8th TSTA- Connection: keep-alive << this > >> happens most of the time.. > >> -Sometimes i get 9 times TSTA + STOP << this would be the desired > >> outcome (only seen very few times..) > >> > >> Probably due to the response-buffer being filled differently due to > >> 'timing'.. > >> > >> The "connection: keep-alive" text is probably from the actual server > >> reply which is 'appended' behind the response generated by my lua > >> script.?. However shouldn't the .done() prevent that from being send to > >> the client? > >> > >> Ive tried putting a loop into the lua script to call res:get() multiple > >> times but that didnt seem to work.. > >> > >> Also to properly modify a page i would need to know all changes before > >> sending the headers with changed content-length back to the client.. > >> > >> Can someone confirm this is or isn't (reliably) possible? Or how this > >> can be scripted in lua differently? > > > > Hello, > > > > Your script replace 3 bytes by 3 bytes, this must run with HTTP, but if > > your replacement change the length of the response, you can have some > > difficulties with clients, or with keepalive. > Yes i started with replacing with the same number of bytes to avoid some > of the possible troubles caused by changing the length.. And as seen in > the haproxy.cfg it is configured with 'mode http'. > > > > The res:get(), returns the current content of the response buffer. > > Maybe it not contains the full response. You must execute a loop with > > regular "core.yield()" to get back the hand to HAProxy and wait for new > Calling yield does allow to 'wait' for more data to come in.. No > guarantee that it only takes 1 yield for data to 'grow'.. > > [info] 278/055943 (77431) : luahttpresponse Content-Length XYZ: 14115 > [info] 278/055943 (77431) : luahttpresponse SIZE: 2477 > [info] 278/055943 (77431) : luahttpresponse LOOP > [info] 278/055943 (77431) : luahttpresponse SIZE: 6221 > [info] 278/055943 (77431) : luahttpresponse LOOP > [info] 278/055943 (77431) : luahttpresponse SIZE: 7469 > [info] 278/055943 (77431) : luahttpresponse LOOP > [info] 278/055943 (77431) : luahttpresponse SIZE: 7469 > [info] 278/055943 (77431) : luahttpresponse LOOP > [info] 278/055943 (77431) : luahttpresponse SIZE: 7469 > [info] 278/055943 (77431) : luahttpresponse LOOP > [info] 278/055943 (77431) : luahttpresponse SIZE: 7469 > [info] 278/055943 (77431) : luahttpresponse LOOP > [info] 278/055943 (77431) : luahttpresponse SIZE: 7469 > [info] 278/055943 (77431) : luahttpresponse LOOP > [info] 278/055943 (77431) : luahttpresponse SIZE: 7469 > [info] 278/055943 (77431) : luahttpresponse LOOP > [info] 278/055943 (77431) : luahttpresponse SIZE: 7469 > [info] 278/055943 (77431) : luahttpresponse LOOP > [info] 278/055943 (77431) : luahttpresponse SIZE: 8717 > [info] 278/055943 (77431) : luahttpresponse LOOP > [info] 278/055943 (77431) : luahttpresponse SIZE: 14337 > [info] 278/055943 (77431) : luahttpresponse DONE?: 14337 > > > data. When all the data are read, res:get() returns an error. > Not sure when/how this error would happen.? The result of res:get only > seems to get bigger while the webserver is sending the response.. > > > > The res:send() is dangerous because it send data directly to the client > > without the end of haproxy analysis. Maybe it is the cause o your > > problem. > > > > Try to use res:set(). > Ok tried that, new try with function below. > > > > The difficulty is that another "res:get()" returns the same data that > > these you put. > > > > I don't known if you can modify an http response greater than one > > buffer. > Would be nice if that was somehow possible. But my current lua script > cannot.. > > > > The function res:close() closes the connection even if HAProxy want to > > keep the connection alive. I suggest that you don't use this function. > It seems txn.res:close() does not exist? txn:done() > > > > I reproduce the error message using curl. By default curl tries > > to transfer data with keepalive, and it is not happy if the all the > > announced data are not transfered. > > > > Connection: keep-alive curl: (18) transfer closed with outstanding > > read data remaining > > > > It seems that i reproduce a bug. I'm looking for. > Ok if you can create a patch, let me know. Happy to test if it solves > some of the issues i see. Hi, I catch the error. While we execute http actions, the content of the buffer must contains the http headers, otherwise, the request is invalid. It is because the http action are applied on the content and not on the stream. So this is th behaviour: -> The function res:get() remove the data from the haproxy input buffer -> The function res:send() send the modified data directly to the client (throught the output buffer), this data is not yet avalaible in the input buffer. -> Now HAProxy is locked in the processing, because it wait for a valid reponse header for continuing the Lua processing. This behaviour is not really a bug, it is the process who garantee that a valid http response is present before continuing the processing. Maybe, a patch will send an error in the case of we enter in an action with a valid request/response and we out from the processing without a valid request/response. So, regarding the HAProxy global behaviour, the action are designed for manipulating http header, and not the body. If you want to manipulates the body, you can use a tcp action. You can use also an http action, but the buffer must contains a valid http request each time when HAProxy give back the hand. In this case, you cannot modify requests greater than an haproxy buffer. res:get() -> get all the data and remove it from the input buffer res:dup() -> just duplicates data res:set() -> set data in the input buffer, HAProxy will continue the analysis. res:send()-> send data in the output buffer, HAProy cannot analyse these data. Thierry > > > > Thierry > > > This function seems to work for responses up to +-15KB. > Sometimes the number of loops it runs is different, and it seems kinda > in-efficient to just run loops until the response is 'complete', another > strange observation is that the res:set inside the loop is required, > even though it doesn't set a modified response, eventually the complete > response is modified in the browser result. Second request over a > keep-alive connection also fails. Adding http-server-close also closes > the client connection, but does avoid the problem with the second > request.. In the lua script I dont account for headers size yet when > checking if the response is completely read, but i dont think thats > affected the test.. > > Is there anything i can do to improve this function? (Besides removing > the txn:Info() lines.) > > function luahttpresponse(txn) > local resheaders = txn.http:res_get_headers() > local contentlength = tonumber(resheaders["content-length"][0]) > local response2 = txn.res:get() > txn:Info("luahttpresponse Content-Length XYZ: " .. contentlength) > txn:Info("luahttpresponse SIZE: " .. string.len(response2)) > while string.len(response2) < contentlength do > txn:Info("luahttpresponse LOOP") > txn.res:set(response2) > core.yield() > response2 = txn.res:get() > txn:Info("luahttpresponse SIZE: " .. string.len(response2)) > end > response2 = string.gsub(response2,"LAL","TST") > txn.res:set(response2) > txn:Info("luahttpresponse DONE?: " .. string.len(response2)) > end > > Regards, > PiBa-NL > > > >> Thanks in advance, > >> PiBa-NL > >> > >> ## haproxy.cfg > >> listen proxyresponse > >> bind :10006 > >> mode http > >> http-response lua.luahttpresponse > >> server x 192.168.0.40:302 > >> > >> ## script.lua > >> function luahttpresponse(txn) > >> local response2 = txn.res:get() > >> response2 = string.gsub(response2,"LAL","TST") > >> txn.res:send(response2) > >> txn.done() > >> end > >> core.register_action("luahttpresponse" , { "http-res" }, luahttpresponse); > >> > >> ## webpage.aspx , with 2.7KB output. > >> ------------------ > >> <%@ Page Language="C#" %> > >> <html><body>START > >> <% var x = new String('x', 250); > >> x = "<input type=\"hidden\" value=\""+x+"\" />"; %> > >> 1 -LALA- <% Response.Write(x); %> > >> 2 -LALA- <% Response.Write(x); %> > >> 3 -LALA- <% Response.Write(x); %> > >> 4 -LALA- <% Response.Write(x); %> > >> 5 -LALA- <% Response.Write(x); %> > >> 6 -LALA- <% Response.Write(x); %> > >> 7 -LALA- <% Response.Write(x); %> > >> 8 -LALA- <% Response.Write(x); %> > >> 9 -LALA- <% Response.Write(x); %> > >> STOP > >> -------------------- > >> > >> > >
