Re: LUA, 'retry' failed requests

[PiBa-NL]

Hi Thierry,
Op 5-11-2015 om 8:08 schreef Thierry FOURNIER:

Hi,

Now, because of you I have my own freebsd installed :). I can
reproduced the segfault. I suppose that the OS other than Freebsd are
impacted, but luckyly it not visible.
OK thanks for installing and trying on FreeBSD :) i suppose some OS's 
are more likely to catch/evade a problem then others.


I encounter an easy compilation error. So can you test the two attached
patches. The first one fix the compilation issue, and the second one
fix he segfault.
First patch removes the warning below though it always build fine as far 
as i could tell. Anyway less warnings is better.

include/common/mini-clist.h:114:9: warning: 'LIST_PREV' macro redefined
#define LIST_PREV(lh, pt, el) (LIST_ELEM((lh)->p, pt, el))

Second patch i confirm fixes the core dump.

Thanks as always!

Regards,
PiBa-NL


Thierry


On Mon, 2 Nov 2015 20:50:01 +0100
PiBa-NL  wrote:


Op 2-11-2015 om 10:03 schreef Thierry FOURNIER:

On Sat, 31 Oct 2015 21:22:14 +0100
PiBa-NL  wrote:


Hi Thierry, haproxy-list,

Hi Pieter,

Hi Thierry,



I've created another possibly interesting lua script, and it works :)
(mostly). (on my test machine..)

When i visit the 192.168.0.120:9003 website i always see the 'Hello
World' page. So in that regard this is usable, it is left to the browser
to send the request again, not sure how safe this is in regard to
mutations being send twice. It should probably check for POST requests
and then just return the error without replacing it with a redirect..
Not sure if that would catch all problem cases..

Ive created a lua service that counts how many requests are made, and
returns a error status every 5th request.
Second there is a lua response script that checks the status, and
replaces it by a redirect if it sees the faulty status 500.
This does currently result in the connection being closed and reopened
probably due to the txn.res:send().?.

Though i am still struggling with what is and isn't supposed to be possible.
For example the scripts below are running in 'mode http' and mostly just
changing 'headers'.
I expected to be able to simply read the status by calling
txn.f:status() but this always seems to result in 'null'.
Manually parsing the response buffer duplicate works but seems ugly..

txn.f:status()  <  it doesnt result in the actual status.

This is a bug wich I reproduce. Can you try the attached patches ?

With the patches it works without my 'workaround', thanks.

txn.res:set()  < if used in place of send() causes 30 second delay

This function put data in the input part of the response buffer. This
new data follows the HAProxy stream when the Lua script is finished.
It is your case.

I can't reproduce this behaviour, I suppose that its because I work
locally, and I'm not impacted by the network latency.

Even when i bind everything to 0.0.0.0 and use 127.0.0.1 to query the
9003 port it still waits for the timeout to strike..
I'm not sure why it doesn't happen in your setup.. Of course i'm running
on FreeBSD, but i don't expect that to affect this..



txn.done()  < dumps core. (im not sure when ever to call it? the script
below seems to match the description that this function has.?.)

I can't reproduce too, for the same reasons, I guess.

Please note that both set() and done() need to be uncommented for the
dump to happen, with the 5th request.

Not sure if it helps, but backtrace of the dump below (would 'bt full'
be more usefull?):
(gdb) bt
#0  0x000801a76bb5 in memmove () from /lib/libc.so.7
#1  0x00417523 in buffer_insert_line2 (b=0x8024a,
  pos=0x8024a0035 "\r\n\ncontent-type: text/plain\r\ncontent-length:
394\r\n\r\nError 5\r\nversion\t\n[HTTP/1.1]\t\nf\t\n   0\t\n
[userdata: 0x802683a68]\t\nsc\t\n   0\t\n   [userdata:
0x802683be8]\t\nc\t\n 0\t\n   [userdata: 0x802683b68]\t\nheader"...,
  str=0x58c695 "Connection: keep-alive", len=22) at src/buffer.c:126
#2  0x0047b3a5 in http_header_add_tail2 (msg=0x8024bb290,
hdr_idx=0x8024bb280, text=0x58c695 "Connection: keep-alive", len=22)
  at src/proto_http.c:595
#3  0x0047f943 in http_change_connection_header
(txn=0x8024bb280, msg=0x8024bb290, wanted=8388608) at src/proto_http.c:2079
#4  0x004900fd in http_process_res_common (s=0x802485600,
rep=0x802485650, an_bit=262144, px=0x8024de000) at src/proto_http.c:6882
#5  0x004d6c90 in process_stream (t=0x8024ab710) at
src/stream.c:1918
#6  0x00420588 in process_runnable_tasks () at src/task.c:238
#7  0x0040ce0e in run_poll_loop () at src/haproxy.c:1559
#8  0x0040dcb2 in main (argc=4, argv=0x7fffeb00) at
src/haproxy.c:1912


Am i trying to do it wrong?

p.s. Is 'health checking' using lua possible? The redis example looks
like a health 'ping'.. It could possibly be much much more flexible then
the tcp-check send  / tcp-check expect routines..

It is not possible. You can write a task which do something (like an

Re: LUA, 'retry' failed requests

[Willy Tarreau]

Hi Pieter,

On Thu, Nov 05, 2015 at 08:21:26PM +0100, PiBa-NL wrote:
> >I encounter an easy compilation error. So can you test the two attached
> >patches. The first one fix the compilation issue, and the second one
> >fix he segfault.
> First patch removes the warning below though it always build fine as far 
> as i could tell. Anyway less warnings is better.

Yes definitely. No warning is the goal unless we really have no other
option. I don't have a single warning on my builds and that helps me
a lot to spot issues I introduce during backports.

> include/common/mini-clist.h:114:9: warning: 'LIST_PREV' macro redefined
> #define LIST_PREV(lh, pt, el) (LIST_ELEM((lh)->p, pt, el))
> 
> Second patch i confirm fixes the core dump.

Great, thanks for this useful report. I've thus merged the patches
into 1.7 and 1.6.

Cheers,
Willy

Re: LUA, 'retry' failed requests

[Thierry FOURNIER]

Hi,

Now, because of you I have my own freebsd installed :). I can
reproduced the segfault. I suppose that the OS other than Freebsd are
impacted, but luckyly it not visible.

I encounter an easy compilation error. So can you test the two attached
patches. The first one fix the compilation issue, and the second one
fix he segfault.

Thierry


On Mon, 2 Nov 2015 20:50:01 +0100
PiBa-NL  wrote:

> Op 2-11-2015 om 10:03 schreef Thierry FOURNIER:
> > On Sat, 31 Oct 2015 21:22:14 +0100
> > PiBa-NL  wrote:
> >
> >> Hi Thierry, haproxy-list,
> >
> > Hi Pieter,
> Hi Thierry,
> >
> >
> >> I've created another possibly interesting lua script, and it works :)
> >> (mostly). (on my test machine..)
> >>
> >> When i visit the 192.168.0.120:9003 website i always see the 'Hello
> >> World' page. So in that regard this is usable, it is left to the browser
> >> to send the request again, not sure how safe this is in regard to
> >> mutations being send twice. It should probably check for POST requests
> >> and then just return the error without replacing it with a redirect..
> >> Not sure if that would catch all problem cases..
> >>
> >> Ive created a lua service that counts how many requests are made, and
> >> returns a error status every 5th request.
> >> Second there is a lua response script that checks the status, and
> >> replaces it by a redirect if it sees the faulty status 500.
> >> This does currently result in the connection being closed and reopened
> >> probably due to the txn.res:send().?.
> >>
> >> Though i am still struggling with what is and isn't supposed to be 
> >> possible.
> >> For example the scripts below are running in 'mode http' and mostly just
> >> changing 'headers'.
> >> I expected to be able to simply read the status by calling
> >> txn.f:status() but this always seems to result in 'null'.
> >> Manually parsing the response buffer duplicate works but seems ugly..
> >>
> >> txn.f:status()  <  it doesnt result in the actual status.
> >
> > This is a bug wich I reproduce. Can you try the attached patches ?
> With the patches it works without my 'workaround', thanks.
> >
> >> txn.res:set()  < if used in place of send() causes 30 second delay
> >
> > This function put data in the input part of the response buffer. This
> > new data follows the HAProxy stream when the Lua script is finished.
> > It is your case.
> >
> > I can't reproduce this behaviour, I suppose that its because I work
> > locally, and I'm not impacted by the network latency.
> Even when i bind everything to 0.0.0.0 and use 127.0.0.1 to query the 
> 9003 port it still waits for the timeout to strike..
> I'm not sure why it doesn't happen in your setup.. Of course i'm running 
> on FreeBSD, but i don't expect that to affect this..
> >
> >
> >> txn.done()  < dumps core. (im not sure when ever to call it? the script
> >> below seems to match the description that this function has.?.)
> >
> > I can't reproduce too, for the same reasons, I guess.
> Please note that both set() and done() need to be uncommented for the 
> dump to happen, with the 5th request.
> 
> Not sure if it helps, but backtrace of the dump below (would 'bt full' 
> be more usefull?):
> (gdb) bt
> #0  0x000801a76bb5 in memmove () from /lib/libc.so.7
> #1  0x00417523 in buffer_insert_line2 (b=0x8024a,
>  pos=0x8024a0035 "\r\n\ncontent-type: text/plain\r\ncontent-length: 
> 394\r\n\r\nError 5\r\nversion\t\n[HTTP/1.1]\t\nf\t\n   0\t\n   
> [userdata: 0x802683a68]\t\nsc\t\n   0\t\n   [userdata: 
> 0x802683be8]\t\nc\t\n 0\t\n   [userdata: 0x802683b68]\t\nheader"...,
>  str=0x58c695 "Connection: keep-alive", len=22) at src/buffer.c:126
> #2  0x0047b3a5 in http_header_add_tail2 (msg=0x8024bb290, 
> hdr_idx=0x8024bb280, text=0x58c695 "Connection: keep-alive", len=22)
>  at src/proto_http.c:595
> #3  0x0047f943 in http_change_connection_header 
> (txn=0x8024bb280, msg=0x8024bb290, wanted=8388608) at src/proto_http.c:2079
> #4  0x004900fd in http_process_res_common (s=0x802485600, 
> rep=0x802485650, an_bit=262144, px=0x8024de000) at src/proto_http.c:6882
> #5  0x004d6c90 in process_stream (t=0x8024ab710) at 
> src/stream.c:1918
> #6  0x00420588 in process_runnable_tasks () at src/task.c:238
> #7  0x0040ce0e in run_poll_loop () at src/haproxy.c:1559
> #8  0x0040dcb2 in main (argc=4, argv=0x7fffeb00) at 
> src/haproxy.c:1912
> 
> >
> >> Am i trying to do it wrong?
> >>
> >> p.s. Is 'health checking' using lua possible? The redis example looks
> >> like a health 'ping'.. It could possibly be much much more flexible then
> >> the tcp-check send  / tcp-check expect routines..
> >
> > It is not possible. You can write a task which do something (like an
> > http request) and reuse the result in the request processing Lua code,
> > but this task cannot set the status of the server.
> The doc does say for core.register_task(func) "For example this type of 

Re: LUA, 'retry' failed requests

[PiBa-NL]

Op 2-11-2015 om 10:03 schreef Thierry FOURNIER:

On Sat, 31 Oct 2015 21:22:14 +0100
PiBa-NL  wrote:


Hi Thierry, haproxy-list,


Hi Pieter,

Hi Thierry,




I've created another possibly interesting lua script, and it works :)
(mostly). (on my test machine..)

When i visit the 192.168.0.120:9003 website i always see the 'Hello
World' page. So in that regard this is usable, it is left to the browser
to send the request again, not sure how safe this is in regard to
mutations being send twice. It should probably check for POST requests
and then just return the error without replacing it with a redirect..
Not sure if that would catch all problem cases..

Ive created a lua service that counts how many requests are made, and
returns a error status every 5th request.
Second there is a lua response script that checks the status, and
replaces it by a redirect if it sees the faulty status 500.
This does currently result in the connection being closed and reopened
probably due to the txn.res:send().?.

Though i am still struggling with what is and isn't supposed to be possible.
For example the scripts below are running in 'mode http' and mostly just
changing 'headers'.
I expected to be able to simply read the status by calling
txn.f:status() but this always seems to result in 'null'.
Manually parsing the response buffer duplicate works but seems ugly..

txn.f:status()  <  it doesnt result in the actual status.


This is a bug wich I reproduce. Can you try the attached patches ?

With the patches it works without my 'workaround', thanks.



txn.res:set()  < if used in place of send() causes 30 second delay


This function put data in the input part of the response buffer. This
new data follows the HAProxy stream when the Lua script is finished.
It is your case.

I can't reproduce this behaviour, I suppose that its because I work
locally, and I'm not impacted by the network latency.
Even when i bind everything to 0.0.0.0 and use 127.0.0.1 to query the 
9003 port it still waits for the timeout to strike..
I'm not sure why it doesn't happen in your setup.. Of course i'm running 
on FreeBSD, but i don't expect that to affect this..




txn.done()  < dumps core. (im not sure when ever to call it? the script
below seems to match the description that this function has.?.)


I can't reproduce too, for the same reasons, I guess.
Please note that both set() and done() need to be uncommented for the 
dump to happen, with the 5th request.


Not sure if it helps, but backtrace of the dump below (would 'bt full' 
be more usefull?):

(gdb) bt
#0  0x000801a76bb5 in memmove () from /lib/libc.so.7
#1  0x00417523 in buffer_insert_line2 (b=0x8024a,
pos=0x8024a0035 "\r\n\ncontent-type: text/plain\r\ncontent-length: 
394\r\n\r\nError 5\r\nversion\t\n[HTTP/1.1]\t\nf\t\n   0\t\n   
[userdata: 0x802683a68]\t\nsc\t\n   0\t\n   [userdata: 
0x802683be8]\t\nc\t\n 0\t\n   [userdata: 0x802683b68]\t\nheader"...,

str=0x58c695 "Connection: keep-alive", len=22) at src/buffer.c:126
#2  0x0047b3a5 in http_header_add_tail2 (msg=0x8024bb290, 
hdr_idx=0x8024bb280, text=0x58c695 "Connection: keep-alive", len=22)

at src/proto_http.c:595
#3  0x0047f943 in http_change_connection_header 
(txn=0x8024bb280, msg=0x8024bb290, wanted=8388608) at src/proto_http.c:2079
#4  0x004900fd in http_process_res_common (s=0x802485600, 
rep=0x802485650, an_bit=262144, px=0x8024de000) at src/proto_http.c:6882
#5  0x004d6c90 in process_stream (t=0x8024ab710) at 
src/stream.c:1918

#6  0x00420588 in process_runnable_tasks () at src/task.c:238
#7  0x0040ce0e in run_poll_loop () at src/haproxy.c:1559
#8  0x0040dcb2 in main (argc=4, argv=0x7fffeb00) at 
src/haproxy.c:1912





Am i trying to do it wrong?

p.s. Is 'health checking' using lua possible? The redis example looks
like a health 'ping'.. It could possibly be much much more flexible then
the tcp-check send  / tcp-check expect routines..


It is not possible. You can write a task which do something (like an
http request) and reuse the result in the request processing Lua code,
but this task cannot set the status of the server.
The doc does say for core.register_task(func) "For example this type of 
tasks can be executed to perform complex health checks." but if i 
understand correctly it is so that we can perform healthchecks with it, 
but the results of such a check cannot be used to change a servers 
health.? Probably the text near this register_task could use some 
tweaking then?.



Thierry



I'm currently testing with HA-Proxy version 1.7-dev0-e4c4b7d and the
following configuration files:

 haproxy.cfg ###
global
  maxconn6000
  lua-load/var/etc/haproxy/luascript_lua-count5error
defaults
  timeout connect3
  timeout server3
  timeout client3
  modehttp
  logglobal
frontend TEST-lua-count
  

Re: LUA, 'retry' failed requests

[Thierry FOURNIER]

On Sat, 31 Oct 2015 21:22:14 +0100
PiBa-NL  wrote:

> Hi Thierry, haproxy-list,


Hi Pieter,


> I've created another possibly interesting lua script, and it works :) 
> (mostly). (on my test machine..)
> 
> When i visit the 192.168.0.120:9003 website i always see the 'Hello 
> World' page. So in that regard this is usable, it is left to the browser 
> to send the request again, not sure how safe this is in regard to 
> mutations being send twice. It should probably check for POST requests 
> and then just return the error without replacing it with a redirect.. 
> Not sure if that would catch all problem cases..
> 
> Ive created a lua service that counts how many requests are made, and 
> returns a error status every 5th request.
> Second there is a lua response script that checks the status, and 
> replaces it by a redirect if it sees the faulty status 500.
> This does currently result in the connection being closed and reopened 
> probably due to the txn.res:send().?.
> 
> Though i am still struggling with what is and isn't supposed to be possible.
> For example the scripts below are running in 'mode http' and mostly just 
> changing 'headers'.
> I expected to be able to simply read the status by calling 
> txn.f:status() but this always seems to result in 'null'.
> Manually parsing the response buffer duplicate works but seems ugly..
> 
> txn.f:status()  <  it doesnt result in the actual status.


This is a bug wich I reproduce. Can you try the attached patches ?


> txn.res:set()  < if used in place of send() causes 30 second delay


This function put data in the input part of the response buffer. This
new data follows the HAProxy stream when the Lua script is finished.
It is your case.

I can't reproduce this behaviour, I suppose that its because I work
locally, and I'm not impacted by the network latency.


> txn.done()  < dumps core. (im not sure when ever to call it? the script 
> below seems to match the description that this function has.?.)


I can't reproduce too, for the same reasons, I guess.


> Am i trying to do it wrong?
> 
> p.s. Is 'health checking' using lua possible? The redis example looks 
> like a health 'ping'.. It could possibly be much much more flexible then 
> the tcp-check send  / tcp-check expect routines..


It is not possible. You can write a task which do something (like an
http request) and reuse the result in the request processing Lua code,
but this task cannot set the status of the server.

Thierry


> I'm currently testing with HA-Proxy version 1.7-dev0-e4c4b7d and the 
> following configuration files:
> 
>  haproxy.cfg ###
> global
>  maxconn6000
>  lua-load/var/etc/haproxy/luascript_lua-count5error
> defaults
>  timeout connect3
>  timeout server3
>  timeout client3
>  modehttp
>  logglobal
> frontend TEST-lua-count
>  bind192.168.0.120:9002
>  optionhttp-keep-alive
>  http-request use-service lua.lua-count
> 
> frontend TEST-lua-retry-serverror
>  bind192.168.0.120:9003
>  optionhttp-keep-alive
>  http-request lua.retrystorerequest
>  http-response lua.retryerrors
>  default_backend hap_9002_http_ipvANY
> 
> backend hap_9002_http_ipvANY
>  modehttp
>  retries3
>  server192.168.0.120_9002 192.168.0.120:9002
> 
> ### luascript_lua-count5error ###
> core.register_action("retryerrors" , { "http-res" }, function(txn)
>  local clientip = txn.f:src()
>  txn:Info("  LUA client " .. clientip)
> 
>  local s = txn.f:status() -- doesnt work?
>  if s == null then
>  core.Info("LUA txn.s:status RETURNED: NULL, fallback needed ??")
>  local req = txn.res:dup()
>  local statusstr = string.sub(req, 10, 13)
>  s = tonumber(statusstr)
>  end
>  core.Info("LUA status " .. s)
> 
>  if s ~= 200 then
>  txn:Info("LUA REDIRECT IT ! " .. s)
> 
>  local url = txn:get_priv()
>  local response = ""
>  response = response .. "HTTP/1.1 302 Moved\r\n"
>  response = response .. "Location: " .. url .."\r\n"
>  response = response .. "\r\n"
> 
>  txn.res:send(response)
>  --txn.res:set(response) -- causes 30 second delay..
>  --txn:done() --dumps core..
>  end
> end);
> 
> core.register_action("retrystorerequest" , { "http-req" }, function(txn)
>  local url = txn.f:url()
>  txn:set_priv(url);
> end);
> 
> core.register_service("lua-count", "http", function(applet)
> if test == null then
>test = 0
> end
> test = test + 1
> local response = ""
> if test % 5 == 0 then
>applet:set_status(500)
>response = "Error " .. test
> else
>applet:set_status(200)
>response = "Hello World !" .. test
> end
> applet:add_header("content-length", 

LUA, 'retry' failed requests

[PiBa-NL]

Hi Thierry, haproxy-list,

I've created another possibly interesting lua script, and it works :) 
(mostly). (on my test machine..)


When i visit the 192.168.0.120:9003 website i always see the 'Hello 
World' page. So in that regard this is usable, it is left to the browser 
to send the request again, not sure how safe this is in regard to 
mutations being send twice. It should probably check for POST requests 
and then just return the error without replacing it with a redirect.. 
Not sure if that would catch all problem cases..


Ive created a lua service that counts how many requests are made, and 
returns a error status every 5th request.
Second there is a lua response script that checks the status, and 
replaces it by a redirect if it sees the faulty status 500.
This does currently result in the connection being closed and reopened 
probably due to the txn.res:send().?.


Though i am still struggling with what is and isn't supposed to be possible.
For example the scripts below are running in 'mode http' and mostly just 
changing 'headers'.
I expected to be able to simply read the status by calling 
txn.f:status() but this always seems to result in 'null'.

Manually parsing the response buffer duplicate works but seems ugly..

txn.f:status()  <  it doesnt result in the actual status.
txn.res:set()  < if used in place of send() causes 30 second delay
txn.done()  < dumps core. (im not sure when ever to call it? the script 
below seems to match the description that this function has.?.)


Am i trying to do it wrong?

p.s. Is 'health checking' using lua possible? The redis example looks 
like a health 'ping'.. It could possibly be much much more flexible then 
the tcp-check send  / tcp-check expect routines..


I'm currently testing with HA-Proxy version 1.7-dev0-e4c4b7d and the 
following configuration files:


 haproxy.cfg ###
global
maxconn6000
lua-load/var/etc/haproxy/luascript_lua-count5error
defaults
timeout connect3
timeout server3
timeout client3
modehttp
logglobal
frontend TEST-lua-count
bind192.168.0.120:9002
optionhttp-keep-alive
http-request use-service lua.lua-count

frontend TEST-lua-retry-serverror
bind192.168.0.120:9003
optionhttp-keep-alive
http-request lua.retrystorerequest
http-response lua.retryerrors
default_backend hap_9002_http_ipvANY

backend hap_9002_http_ipvANY
modehttp
retries3
server192.168.0.120_9002 192.168.0.120:9002

### luascript_lua-count5error ###
core.register_action("retryerrors" , { "http-res" }, function(txn)
local clientip = txn.f:src()
txn:Info("  LUA client " .. clientip)

local s = txn.f:status() -- doesnt work?
if s == null then
core.Info("LUA txn.s:status RETURNED: NULL, fallback needed ??")
local req = txn.res:dup()
local statusstr = string.sub(req, 10, 13)
s = tonumber(statusstr)
end
core.Info("LUA status " .. s)

if s ~= 200 then
txn:Info("LUA REDIRECT IT ! " .. s)

local url = txn:get_priv()
local response = ""
response = response .. "HTTP/1.1 302 Moved\r\n"
response = response .. "Location: " .. url .."\r\n"
response = response .. "\r\n"

txn.res:send(response)
--txn.res:set(response) -- causes 30 second delay..
--txn:done() --dumps core..
end
end);

core.register_action("retrystorerequest" , { "http-req" }, function(txn)
local url = txn.f:url()
txn:set_priv(url);
end);

core.register_service("lua-count", "http", function(applet)
   if test == null then
  test = 0
   end
   test = test + 1
   local response = ""
   if test % 5 == 0 then
  applet:set_status(500)
  response = "Error " .. test
   else
  applet:set_status(200)
  response = "Hello World !" .. test
   end
   applet:add_header("content-length", string.len(response))
   applet:add_header("content-type", "text/plain")
   applet:start_response()
   applet:send(response)
end)