Re: Lua patchset merged
On Tue, Mar 10, 2015 at 12:40:03AM +0100, Thierry FOURNIER wrote: On Mon, 09 Mar 2015 22:11:56 +0100 Cyril Bonté cyril.bo...@free.fr wrote: I've also seen this commit : MEDIUM: lua: use the Lua-5.3 version of the library [1] This one may be annoying in the short term for some users, because Lua 5.3 is not available in all distributions (for example debian, ubuntu, ...). Currently for such distributions, it requires to recompile Lua 5.3 with a small patch to generate the .so dynamic library. For those who want to make some tests, you can have a look at [2] and [3] and the patches they both provide. Arg, you're right :(. The Lua 5.3 is required for the session timeout. In the same way, it is necessary for the forced-yield. This system forces a yield and permits to haproxy to get the hand for processing other things than Lua. With the version 5.2, this is not possible without frequently Lua error. Maybe I can do cohabitation for 5.2 and 5.3 version. The forced yield will be deactivated with compilation 5.2. The execution timeout remain active. BUt, I'm not sure that is a good idea. Given that there's no way to break an infinite loop before 5.2 and that Lua will be used by people who experiment with scripts, I'd rather stick to 5.3 unless someone finds an acceptable workaround working with 5.2. I'd rather avoid seeing people report unresponsive haproxy just because of bugs in their Lua scripts :-/ Willy
Re: Lua patchset merged
On Tue, Mar 10, 2015 at 02:08:40AM +0100, Thierry FOURNIER wrote: On Tue, 10 Mar 2015 01:05:50 +0100 Cyril Bonté cyril.bo...@free.fr wrote: Hi again, Le 10/03/2015 00:40, Thierry FOURNIER a écrit : On Mon, 09 Mar 2015 22:11:56 +0100 Cyril Bonté cyril.bo...@free.fr wrote: I've seen new commits that have been merged on the git repository. The bad news are that the previous test that I reported (sending a response larger than the buffer) doesn't work anymore :-/ Resulting in : [ALERT] 067/220744 (27176) : Lua function 'hello_world': execution timeout. Hi cyril, This is due to the implementation of the Lua execution timeout. This is a system used to prevent loops in scripts. The Timeout is set by default to 4s. You can see tune.lua.session-timeout, tune.lua.task-timeout and tune.lua.forced-yield http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#tune.lua.session-timeout Of course, but it shouldn't take 4 seconds, the answer is immediate in my test case. Actually, I could find that it was reproducible beginning with a response greater or equal to 16392 bytes (I've not read the code yet). Thank you Cyril, the bug is partially reproduced and fixed (the buffer is not sent, but the error timeout is after 4 seconds as expected). I attach the patch. I think that Willy must check this patch, because it is possible than the comparison which I modify, did make sense. Cool, will merge all this, thanks guys. Willy
Re: Lua patchset merged
All patches merged, thanks Thierry. Willy
Re: Lua patchset merged
Hi Thierry, Le 04/03/2015 21:20, Cyril Bonté a écrit : Hi Thierry, Le 04/03/2015 11:51, Thierry FOURNIER a écrit : Thank you Cyril for the bug repport. I join the patch that fix this issue. Great ! I didn't have time to investigate further. I can confirm that the patch works well ;-) I've seen new commits that have been merged on the git repository. The bad news are that the previous test that I reported (sending a response larger than the buffer) doesn't work anymore :-/ Resulting in : [ALERT] 067/220744 (27176) : Lua function 'hello_world': execution timeout. I've also seen this commit : MEDIUM: lua: use the Lua-5.3 version of the library [1] This one may be annoying in the short term for some users, because Lua 5.3 is not available in all distributions (for example debian, ubuntu, ...). Currently for such distributions, it requires to recompile Lua 5.3 with a small patch to generate the .so dynamic library. For those who want to make some tests, you can have a look at [2] and [3] and the patches they both provide. Also, we'll have to : - modify the haproxy Makefile in order to effectively detect Lua 5.3 and not 5.2 anymore (only the comment was modified) - modify the version check in src/hlua.c to trigger an error if the version number is lesser than 503. [1] http://www.haproxy.org/git?p=haproxy.git;a=commit;h=f90838b71a3c7f84e1d8b4ff85760a35d60c6910 [2] http://www.linuxfromscratch.org/blfs/view/svn/general/lua.html [3] https://aur.archlinux.org/packages/lua53/ -- Cyril Bonté
Re: Lua patchset merged
On Mon, 09 Mar 2015 22:11:56 +0100 Cyril Bonté cyril.bo...@free.fr wrote: Hi Thierry, Le 04/03/2015 21:20, Cyril Bonté a écrit : Hi Thierry, Le 04/03/2015 11:51, Thierry FOURNIER a écrit : Thank you Cyril for the bug repport. I join the patch that fix this issue. Great ! I didn't have time to investigate further. I can confirm that the patch works well ;-) I've seen new commits that have been merged on the git repository. The bad news are that the previous test that I reported (sending a response larger than the buffer) doesn't work anymore :-/ Resulting in : [ALERT] 067/220744 (27176) : Lua function 'hello_world': execution timeout. Hi cyril, This is due to the implementation of the Lua execution timeout. This is a system used to prevent loops in scripts. The Timeout is set by default to 4s. You can see tune.lua.session-timeout, tune.lua.task-timeout and tune.lua.forced-yield http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#tune.lua.session-timeout I've also seen this commit : MEDIUM: lua: use the Lua-5.3 version of the library [1] This one may be annoying in the short term for some users, because Lua 5.3 is not available in all distributions (for example debian, ubuntu, ...). Currently for such distributions, it requires to recompile Lua 5.3 with a small patch to generate the .so dynamic library. For those who want to make some tests, you can have a look at [2] and [3] and the patches they both provide. Arg, you're right :(. The Lua 5.3 is required for the session timeout. In the same way, it is necessary for the forced-yield. This system forces a yield and permits to haproxy to get the hand for processing other things than Lua. With the version 5.2, this is not possible without frequently Lua error. Maybe I can do cohabitation for 5.2 and 5.3 version. The forced yield will be deactivated with compilation 5.2. The execution timeout remain active. BUt, I'm not sure that is a good idea. Also, we'll have to : - modify the haproxy Makefile in order to effectively detect Lua 5.3 and not 5.2 anymore (only the comment was modified) - modify the version check in src/hlua.c to trigger an error if the version number is lesser than 503. You're right. The patch is writed and attached. Thank you Thierry [1] http://www.haproxy.org/git?p=haproxy.git;a=commit;h=f90838b71a3c7f84e1d8b4ff85760a35d60c6910 [2] http://www.linuxfromscratch.org/blfs/view/svn/general/lua.html [3] https://aur.archlinux.org/packages/lua53/ -- Cyril Bonté From 0d2bfc50892b1b09e3eb78ebaced9c0ed4334063 Mon Sep 17 00:00:00 2001 From: Thierry FOURNIER tfourn...@exceliance.fr Date: Tue, 10 Mar 2015 00:35:36 +0100 Subject: [PATCH] BUG/BUILD: lua: The strict Lua 5.3 version check is not done. This patch fix the Lua library check. Only the version 5.3 or later is allowed. This bug is added by the patch MEDIUM: lua: use the Lua-5.3 version of the library with commit id f90838b71a3c7f84e1d8b4ff85760a35d60c6910 --- Makefile |4 ++-- src/hlua.c |4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 1a31792..a12e4c9 100644 --- a/Makefile +++ b/Makefile @@ -569,9 +569,9 @@ OPTIONS_CFLAGS += -DUSE_LUA $(if $(LUA_INC),-I$(LUA_INC)) LUA_LD_FLAGS := $(if $(LUA_LIB),-L$(LUA_LIB)) ifeq ($(LUA_LIB_NAME),) # Try to automatically detect the Lua library -LUA_LIB_NAME := $(firstword $(foreach lib,lua5.2 lua52 lua,$(call check_lua_lib,$(lib),$(LUA_LD_FLAGS +LUA_LIB_NAME := $(firstword $(foreach lib,lua5.3 lua53 lua,$(call check_lua_lib,$(lib),$(LUA_LD_FLAGS ifeq ($(LUA_LIB_NAME),) -$(error unable to automatically detect the Lua library name, you can enforce its name with LUA_LIB_NAME=name (where name can be lua5.2, lua52, lua, ...)) +$(error unable to automatically detect the Lua library name, you can enforce its name with LUA_LIB_NAME=name (where name can be lua5.3, lua53, lua, ...)) endif endif diff --git a/src/hlua.c b/src/hlua.c index 564f5ba..fb39dc1 100644 --- a/src/hlua.c +++ b/src/hlua.c @@ -4,8 +4,8 @@ #include lua.h #include lualib.h -#if !defined(LUA_VERSION_NUM) || LUA_VERSION_NUM 502 -#error Requires Lua 5.2 or later. +#if !defined(LUA_VERSION_NUM) || LUA_VERSION_NUM 503 +#error Requires Lua 5.3 or later. #endif #include ebpttree.h -- 1.7.10.4
Re: Lua patchset merged
Hi again, Le 10/03/2015 00:40, Thierry FOURNIER a écrit : On Mon, 09 Mar 2015 22:11:56 +0100 Cyril Bonté cyril.bo...@free.fr wrote: I've seen new commits that have been merged on the git repository. The bad news are that the previous test that I reported (sending a response larger than the buffer) doesn't work anymore :-/ Resulting in : [ALERT] 067/220744 (27176) : Lua function 'hello_world': execution timeout. Hi cyril, This is due to the implementation of the Lua execution timeout. This is a system used to prevent loops in scripts. The Timeout is set by default to 4s. You can see tune.lua.session-timeout, tune.lua.task-timeout and tune.lua.forced-yield http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#tune.lua.session-timeout Of course, but it shouldn't take 4 seconds, the answer is immediate in my test case. Actually, I could find that it was reproducible beginning with a response greater or equal to 16392 bytes (I've not read the code yet). haproxy.cfg: global lua-load hello_world.lua listen proxy bind 127.0.0.1:10001 tcp-request content lua hello_world hello_world.lua: function hello_world(txn) local res = txn:res_channel() s = for i = 1,16392 do s = s .. x end res:send(s) end -- Cyril Bonté
Re: Lua patchset merged
On Tue, 10 Mar 2015 01:05:50 +0100 Cyril Bonté cyril.bo...@free.fr wrote: Hi again, Le 10/03/2015 00:40, Thierry FOURNIER a écrit : On Mon, 09 Mar 2015 22:11:56 +0100 Cyril Bonté cyril.bo...@free.fr wrote: I've seen new commits that have been merged on the git repository. The bad news are that the previous test that I reported (sending a response larger than the buffer) doesn't work anymore :-/ Resulting in : [ALERT] 067/220744 (27176) : Lua function 'hello_world': execution timeout. Hi cyril, This is due to the implementation of the Lua execution timeout. This is a system used to prevent loops in scripts. The Timeout is set by default to 4s. You can see tune.lua.session-timeout, tune.lua.task-timeout and tune.lua.forced-yield http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#tune.lua.session-timeout Of course, but it shouldn't take 4 seconds, the answer is immediate in my test case. Actually, I could find that it was reproducible beginning with a response greater or equal to 16392 bytes (I've not read the code yet). Thank you Cyril, the bug is partially reproduced and fixed (the buffer is not sent, but the error timeout is after 4 seconds as expected). I attach the patch. I think that Willy must check this patch, because it is possible than the comparison which I modify, did make sense. Thierry haproxy.cfg: global lua-load hello_world.lua listen proxy bind 127.0.0.1:10001 tcp-request content lua hello_world hello_world.lua: function hello_world(txn) local res = txn:res_channel() s = for i = 1,16392 do s = s .. x end res:send(s) end -- Cyril Bonté From f0185cae264f0309e386b500be115051a30e120b Mon Sep 17 00:00:00 2001 From: Thierry FOURNIER tfourn...@exceliance.fr Date: Tue, 10 Mar 2015 01:55:01 +0100 Subject: [PATCH] BUG/MEDIUM: buffer: one byte miss in buffer free space check Space is not avalaible only if the end of the data inserted is strictly greater than the end of buffer. If these two value are equal, the space is avamaible. --- src/buffer.c |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/buffer.c b/src/buffer.c index e156991..3c7f6cc 100644 --- a/src/buffer.c +++ b/src/buffer.c @@ -75,7 +75,7 @@ int buffer_replace2(struct buffer *b, char *pos, char *end, const char *str, int delta = len - (end - pos); - if (bi_end(b) + delta = b-data + b-size) + if (bi_end(b) + delta b-data + b-size) return 0; /* no space left */ if (buffer_not_empty(b) -- 1.7.10.4
Re: Lua patchset merged
Hi Steven, On Thu, Mar 05, 2015 at 11:16:14PM +0100, Steven Le Roux wrote: Hi, why not just a github so that people can fork, play and submit their own Lua scripts? Well, anyone can just do that by himself. I meant something more discussion-oriented than development-oriented. Probably the type of place were you'd find the links to the most common github projects for example. It was suggested that Reddit could be suited for this, we'll check. Willy
Re: Lua patchset merged
Hi Thierry, Le 04/03/2015 11:51, Thierry FOURNIER a écrit : Thank you Cyril for the bug repport. I join the patch that fix this issue. Great ! I didn't have time to investigate further. I can confirm that the patch works well ;-) -- Cyril Bonté
Re: Lua patchset merged
On Tue, 03 Mar 2015 19:35:34 +0100 Cyril Bonté cyril.bo...@free.fr wrote: Hi Thierry, Le 01/03/2015 13:47, Thierry FOURNIER a écrit : HAProxy Lua Hello world HAProxy configuration file (hello_world.conf): global lua-load hello_world.lua listen proxy bind 127.0.0.1:10001 tcp-request content lua hello_world HAProxy Lua file (hello_world.lua): function hello_world(txn) local res = txn:res_channel() res:send(hello world\n) end I was wondering what would happen if a response was greater than the buffer. It appears that sometimes it works, sometime not. The function I used : function hello_world(txn) local res = txn:res_channel() s = for i = 1,32768 do s = s .. x end res:send(s) end Then : $ while true; do s=$(curl -s http://localhost:10001/|wc -c); echo $s; if [[ $s != 32768 ]]; then break; fi; done 32768 32768 32768 32768 32768 32768 32768 32768 32768 32768 32768 32768 539952 = Randomly, it will send extra data after the 16384 first bytes (with a buffer of 16384 bytes), containing other parts of the memory, which may leak other requests data. Thank you Cyril for the bug repport. I join the patch that fix this issue. Thierry From 32fa948331282ae22728c6895d12bdb20125 Mon Sep 17 00:00:00 2001 From: Thierry FOURNIER tfourn...@exceliance.fr Date: Wed, 4 Mar 2015 11:44:47 +0100 Subject: [PATCH] BUG/MAJOR: lua: send function fails and return bad bytes In some cases the Lua send function fails. This is caused by the return of buffer_replace2() is not tested. I checked avalaible space in the buffer and I supposed than buffer_replace2() took all the data. In some cases, buffer_replace2() cannot take the incoming data, it returns the amount of data copied. This patch check the amount of data really copied by buffer_replace2() and advance the buffer with taking this value in account. --- src/hlua.c |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/hlua.c b/src/hlua.c index 1df18b6..d3f302f 100644 --- a/src/hlua.c +++ b/src/hlua.c @@ -2091,7 +2091,7 @@ __LJMP static int _hlua_channel_send(lua_State *L) if (max len - l) max = len - l; - buffer_replace2(chn-chn-buf, chn-chn-buf-p, chn-chn-buf-p, str+l, max); + max = buffer_replace2(chn-chn-buf, chn-chn-buf-p, chn-chn-buf-p, str+l, max); /* buffer replace considers that the input part is filled. * so, I must forward these new data in the output part. */ -- 1.7.10.4
Re: Lua patchset merged
Hi Thierry, Le 01/03/2015 13:47, Thierry FOURNIER a écrit : HAProxy Lua Hello world HAProxy configuration file (hello_world.conf): global lua-load hello_world.lua listen proxy bind 127.0.0.1:10001 tcp-request content lua hello_world HAProxy Lua file (hello_world.lua): function hello_world(txn) local res = txn:res_channel() res:send(hello world\n) end I was wondering what would happen if a response was greater than the buffer. It appears that sometimes it works, sometime not. The function I used : function hello_world(txn) local res = txn:res_channel() s = for i = 1,32768 do s = s .. x end res:send(s) end Then : $ while true; do s=$(curl -s http://localhost:10001/|wc -c); echo $s; if [[ $s != 32768 ]]; then break; fi; done 32768 32768 32768 32768 32768 32768 32768 32768 32768 32768 32768 32768 539952 = Randomly, it will send extra data after the 16384 first bytes (with a buffer of 16384 bytes), containing other parts of the memory, which may leak other requests data. -- Cyril Bonté
Re: Lua patchset merged
I love it ! Just wrote, as a proof of concept, a forward proxy... That said, it seems my lua script is blocking... I mean, if the remote server is slow to deliver the response, then HAProxy doesn't process any other request or response. Baptiste
Re: Lua patchset merged
On Sun, 01 Mar 2015 17:34:36 +0100 Aleksandar Lazic al-hapr...@none.at wrote: Dear Thierry. Cool work ;-) Am 01-03-2015 13:47, schrieb Thierry FOURNIER: Hi everyone, Thanks Willy for the introduction. I join the first part of the document. All the function are not yet available. The doc uses a markdown format like that uses GitHub. I join the original format, and the HTML conversion. The next steps are: - collecting users ideas and bug reports, - add missing functions like native header manipulations, - check the usage of existing libraries, - change the 'sleep' engine, - add execution timeout to prevent infinite loops, - some other tasks ... [snipp] I assume you know the lua module for nginx which is also fully non blocking https://github.com/openresty/lua-nginx-module What I like in this module is the deep integration into nginx. Maybe it would be possible to integrate lua-scripting into the check agent framework? Hi, thank you. It is certainly possible. Have you some concrete usage case ? crazy-idea inject lua code over contorl socket ;-) /crazy-idea It is technicaly possible. I'm not sure that it is useful for other thing than development. For now, the tag crazy-idea is welcome :) Thierry enjoy Thierry br Aleks On Sat, 28 Feb 2015 23:33:56 +0100 Willy Tarreau w...@1wt.eu wrote: Hi folks, this is just a quick mail to let you know that I've just merged the recent work from my coworker Thierry Fournier (in Cc) about supporting Lua scripting in haproxy. I won't go into lengths explaining how that's supposed to be used because I don't really know myself, but I'll let him provide some explanations and working examples. The real purpose would be to get some feedback, suggestions, criticisms to get further into this. At the moment it is possible to declare some Lua-based sample fetch functions, some converters as well and even to create new tcp-request and http-req/resp actions. These last ones (the actions) support yielding so that it is possible to make blocking calls from the Lua code and benefit from haproxy's native session management and scheduling to switch to something else while things are making progress. Thus the whole system is totally non-blocking and event-driven, even if in Lua you appear to be blocking. The long term goal is to be able to quickly provide some features that are not yet possible natively, and eventually to implement some of them natively if they're often requested or have a performance impact. Some of the most complex parts such as writing authentication systems relying on side-traffic would possibly remain written in Lua only, except if we manage to see a big use of certain things and we see a way to implement them cleanly. Thierry is busy these days, and that's why I asked him to push his code prior to switching to other tasks despite the doc not being there yet. So it's possible we'll have to wait a bit for some doc, but we'll really value testing, bug reports and feedback. Nothing is set in stone yet, he does have some ideas on how to improve things. I do still have some concerns about long-term stability and resource usage, as you can expect. So use it as a toy first, play with it and tell us if you like it or not, and if not, why. Thanks! Willy
Re: Lua patchset merged
Am 02-03-2015 14:51, schrieb Thierry FOURNIER: On Sun, 01 Mar 2015 17:34:36 +0100 Aleksandar Lazic al-hapr...@none.at wrote: [snipp] Maybe it would be possible to integrate lua-scripting into the check agent framework? Hi, thank you. It is certainly possible. Have you some concrete usage case ? I'm sorry not yet. crazy-idea inject lua code over contorl socket ;-) /crazy-idea It is technicaly possible. I'm not sure that it is useful for other thing than development. Well you know things change. For now, the tag crazy-idea is welcome :) Thanks. Thierry enjoy Thierry br Aleks On Sat, 28 Feb 2015 23:33:56 +0100 Willy Tarreau w...@1wt.eu wrote: Hi folks, this is just a quick mail to let you know that I've just merged the recent work from my coworker Thierry Fournier (in Cc) about supporting Lua scripting in haproxy. I won't go into lengths explaining how that's supposed to be used because I don't really know myself, but I'll let him provide some explanations and working examples. The real purpose would be to get some feedback, suggestions, criticisms to get further into this. At the moment it is possible to declare some Lua-based sample fetch functions, some converters as well and even to create new tcp-request and http-req/resp actions. These last ones (the actions) support yielding so that it is possible to make blocking calls from the Lua code and benefit from haproxy's native session management and scheduling to switch to something else while things are making progress. Thus the whole system is totally non-blocking and event-driven, even if in Lua you appear to be blocking. The long term goal is to be able to quickly provide some features that are not yet possible natively, and eventually to implement some of them natively if they're often requested or have a performance impact. Some of the most complex parts such as writing authentication systems relying on side-traffic would possibly remain written in Lua only, except if we manage to see a big use of certain things and we see a way to implement them cleanly. Thierry is busy these days, and that's why I asked him to push his code prior to switching to other tasks despite the doc not being there yet. So it's possible we'll have to wait a bit for some doc, but we'll really value testing, bug reports and feedback. Nothing is set in stone yet, he does have some ideas on how to improve things. I do still have some concerns about long-term stability and resource usage, as you can expect. So use it as a toy first, play with it and tell us if you like it or not, and if not, why. Thanks! Willy
Re: Lua patchset merged
how do you pass arguments to a lua function? Imagine I want to call the following lua function: function download (host, file) Baptiste
Re: Lua patchset merged
Hi again Thierry, Le 01/03/2015 18:22, Cyril Bonté a écrit : I think I'll try some LUA scripts before the end of the week-end (which is approaching too quickly) ;-) OK, I could play with some Lua scripts. I could declare some custom sample fetches, from which I could call internal fetches ;-) And that's where I also could raise some segfaults, when I use sample fetches that don't require arguments (for example req_ver). A simple Lua script which reproduces the segfault : core.register_fetches(hello, function(txn, ...) return txn.req_ver(txn) end With this haproxy configuration : global lua-load hello_world.lua listen proxy mode http bind 127.0.0.1:10001 http-request redirect location /%[lua.hello] The time for me to commit and I'll send the fix. -- Cyril Bonté
Re: Lua patchset merged
Hi everyone, Thanks Willy for the introduction. I join the first part of the document. All the function are not yet available. The doc uses a markdown format like that uses GitHub. I join the original format, and the HTML conversion. The next steps are: - collecting users ideas and bug reports, - add missing functions like native header manipulations, - check the usage of existing libraries, - change the 'sleep' engine, - add execution timeout to prevent infinite loops, - some other tasks ... Below, I add the HAProxy configuration documentation: lua-load file scope: global This function declares and load the Lua file. A syntax parsing is executed and this directive fails if the syntax contain errors. The main Lua code is executed. This code can be used to initialize the Lua bindings in HAProxy. tcp-request content lua function scope: frontend / backend / listen This keyword registers and Lua function executed when a tcp connection receive content. tcp-response content lua function scope: frontend / backend / listen This keyword registers and Lua function executed when a tcp connection receive content from the response channel. http-request lua function scope: frontend / backend / listen This keyword registers and Lua function executed when an http request is sent. http-response lua function scope: frontend / backend / listen This keyword registers and Lua function executed when an http response is received. enjoy Thierry On Sat, 28 Feb 2015 23:33:56 +0100 Willy Tarreau w...@1wt.eu wrote: Hi folks, this is just a quick mail to let you know that I've just merged the recent work from my coworker Thierry Fournier (in Cc) about supporting Lua scripting in haproxy. I won't go into lengths explaining how that's supposed to be used because I don't really know myself, but I'll let him provide some explanations and working examples. The real purpose would be to get some feedback, suggestions, criticisms to get further into this. At the moment it is possible to declare some Lua-based sample fetch functions, some converters as well and even to create new tcp-request and http-req/resp actions. These last ones (the actions) support yielding so that it is possible to make blocking calls from the Lua code and benefit from haproxy's native session management and scheduling to switch to something else while things are making progress. Thus the whole system is totally non-blocking and event-driven, even if in Lua you appear to be blocking. The long term goal is to be able to quickly provide some features that are not yet possible natively, and eventually to implement some of them natively if they're often requested or have a performance impact. Some of the most complex parts such as writing authentication systems relying on side-traffic would possibly remain written in Lua only, except if we manage to see a big use of certain things and we see a way to implement them cleanly. Thierry is busy these days, and that's why I asked him to push his code prior to switching to other tasks despite the doc not being there yet. So it's possible we'll have to wait a bit for some doc, but we'll really value testing, bug reports and feedback. Nothing is set in stone yet, he does have some ideas on how to improve things. I do still have some concerns about long-term stability and resource usage, as you can expect. So use it as a toy first, play with it and tell us if you like it or not, and if not, why. Thanks! Willy channel.fig Description: application/xfig How Lua runs in HAProxy HAProxy Lua running contexts The Lua code executed in HAProxy can be processed in 2 main modes. The first one is the initialisation mode, and the second is the runtime mode. In the initialisation mode, we can perform DNS solves, but we cannot perform socket I/O. In this initialisation mode, HAProxy still blocked during the execution of the Lua program. In the runtime mode, we cannot perform DNS solves, but we can use sockets. The execution of the Lua code is multiplexed with the requests processing, so the Lua code seems to be run in blocking, but it is not the case. The Lua code is loaded in one or more files. These files contains main code and functions. Lua have 6 execution context. The Lua file body context. It is executed during the load of the Lua file in the HAProxy [global] section with the directive lua-load. It is executed in initialisation mode. This section is use for configuring Lua bindings in HAProxy. The Lua init context. It is an Lua function executed just after the HAProxy configuration parsing. The execution is in initialisation mode. In this context the HAProxy environment are already initialized. It is useful to check configuration, or initializing socket connections or tasks. These functions are declared in the body context with the Lua function core.register_init(). The prototype of the function is a simple
Re: Lua patchset merged
Dear Thierry. Cool work ;-) Am 01-03-2015 13:47, schrieb Thierry FOURNIER: Hi everyone, Thanks Willy for the introduction. I join the first part of the document. All the function are not yet available. The doc uses a markdown format like that uses GitHub. I join the original format, and the HTML conversion. The next steps are: - collecting users ideas and bug reports, - add missing functions like native header manipulations, - check the usage of existing libraries, - change the 'sleep' engine, - add execution timeout to prevent infinite loops, - some other tasks ... [snipp] I assume you know the lua module for nginx which is also fully non blocking https://github.com/openresty/lua-nginx-module What I like in this module is the deep integration into nginx. Maybe it would be possible to integrate lua-scripting into the check agent framework? crazy-idea inject lua code over contorl socket ;-) /crazy-idea enjoy Thierry br Aleks On Sat, 28 Feb 2015 23:33:56 +0100 Willy Tarreau w...@1wt.eu wrote: Hi folks, this is just a quick mail to let you know that I've just merged the recent work from my coworker Thierry Fournier (in Cc) about supporting Lua scripting in haproxy. I won't go into lengths explaining how that's supposed to be used because I don't really know myself, but I'll let him provide some explanations and working examples. The real purpose would be to get some feedback, suggestions, criticisms to get further into this. At the moment it is possible to declare some Lua-based sample fetch functions, some converters as well and even to create new tcp-request and http-req/resp actions. These last ones (the actions) support yielding so that it is possible to make blocking calls from the Lua code and benefit from haproxy's native session management and scheduling to switch to something else while things are making progress. Thus the whole system is totally non-blocking and event-driven, even if in Lua you appear to be blocking. The long term goal is to be able to quickly provide some features that are not yet possible natively, and eventually to implement some of them natively if they're often requested or have a performance impact. Some of the most complex parts such as writing authentication systems relying on side-traffic would possibly remain written in Lua only, except if we manage to see a big use of certain things and we see a way to implement them cleanly. Thierry is busy these days, and that's why I asked him to push his code prior to switching to other tasks despite the doc not being there yet. So it's possible we'll have to wait a bit for some doc, but we'll really value testing, bug reports and feedback. Nothing is set in stone yet, he does have some ideas on how to improve things. I do still have some concerns about long-term stability and resource usage, as you can expect. So use it as a toy first, play with it and tell us if you like it or not, and if not, why. Thanks! Willy
Re: Lua patchset merged
Great news, congratulation Thierry! Baptiste
Re: Lua patchset merged
Hi Tierry, Huge work ! I've not played with it yet, but I've already compiled it successfully ;-) Some early feedbacks : - It appears that the code requires at least LUA 5.2. - Maybe we'll have to work on the Makefile to ease the compilation. For example, on debian, I have to add -llua5.2 instead of -llua - There's a small typo in an error message (hlua.c:423) : Malformad argument mask instead of Malformed argument mask for the hlua_lua2arg_check() function. I'll send a patch later for that. - Talking about hlua_lua2arg_check(), There are 2 other points : 1. The function comments has some typos. While trying to fix them, I'm realizing I don't understand the comment and I'm not sure to rewrite it correctly. Can you have a look at it ? 2. I think we can have a buffer overflow with the following test : if (idx = ARGM_NBARGS argp[idx].type != ARGT_STOP) The calling function (hlua_run_sample_fetch) already allows a same buffer overflow : struct arg args[ARGM_NBARGS]; and args[i].type = ARGT_STOP; where `i' can be equal to ARGM_NBARGS. - As it is done for other libraries, maybe we can add the compiled version of LUA when haproxy -vv is called. I'll also send a patch for that. - Still about the version : maybe we can add a #error when LUA_VERSION_NUM is not defined or less than 502 : # LUA 5.0.x : not devined # LUA 5.1.x : equal to 501 # LUA 5.2.x : equal to 502 # LUA 5.3.x : equal to 503 I think I'll try some LUA scripts before the end of the week-end (which is approaching too quickly) ;-) -- Cyril Bonté
Re: Lua patchset merged
A few ifdef missing when SSL is not compiled in HAProxy:
diff --git a/src/hlua.c b/src/hlua.c
index a0e4d91..3d69c5d 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -1542,6 +1542,7 @@ __LJMP static int hlua_socket_connect(struct lua_State *L)
return 0;
}
+#ifdef USE_OPENSSL
__LJMP static int hlua_socket_connect_ssl(struct lua_State *L)
{
struct hlua_socket *socket;
@@ -1551,6 +1552,7 @@ __LJMP static int hlua_socket_connect_ssl(struct
lua_State *L)
socket-s-target = socket_ssl.obj_type;
return MAY_LJMP(hlua_socket_connect(L));
}
+#endif
__LJMP static int hlua_socket_setoption(struct lua_State *L)
{
@@ -3563,7 +3565,9 @@ void hlua_init(void)
lua_pushstring(gL.T, __index);
lua_newtable(gL.T);
+#ifdef USE_OPENSSL
hlua_class_function(gL.T, connect_ssl, hlua_socket_connect_ssl);
+#endif
hlua_class_function(gL.T, connect, hlua_socket_connect);
hlua_class_function(gL.T, send,hlua_socket_send);
hlua_class_function(gL.T, receive, hlua_socket_receive);
Baptiste
On Sun, Mar 1, 2015 at 6:22 PM, Cyril Bonté cyril.bo...@free.fr wrote:
Hi Tierry,
Huge work ! I've not played with it yet, but I've already compiled it
successfully ;-)
Some early feedbacks :
- It appears that the code requires at least LUA 5.2.
- Maybe we'll have to work on the Makefile to ease the compilation. For
example, on debian, I have to add -llua5.2 instead of -llua
- There's a small typo in an error message (hlua.c:423) : Malformad
argument mask instead of Malformed argument mask for the
hlua_lua2arg_check() function.
I'll send a patch later for that.
- Talking about hlua_lua2arg_check(), There are 2 other points :
1. The function comments has some typos. While trying to fix them, I'm
realizing I don't understand the comment and I'm not sure to rewrite it
correctly. Can you have a look at it ?
2. I think we can have a buffer overflow with the following test :
if (idx = ARGM_NBARGS argp[idx].type != ARGT_STOP)
The calling function (hlua_run_sample_fetch) already allows a same buffer
overflow :
struct arg args[ARGM_NBARGS];
and
args[i].type = ARGT_STOP;
where `i' can be equal to ARGM_NBARGS.
- As it is done for other libraries, maybe we can add the compiled version
of LUA when haproxy -vv is called.
I'll also send a patch for that.
- Still about the version : maybe we can add a #error when LUA_VERSION_NUM
is not defined or less than 502 :
# LUA 5.0.x : not devined
# LUA 5.1.x : equal to 501
# LUA 5.2.x : equal to 502
# LUA 5.3.x : equal to 503
I think I'll try some LUA scripts before the end of the week-end (which is
approaching too quickly) ;-)
--
Cyril Bonté
