Re: Lua patchset merged

2015-03-10 Thread Willy Tarreau
On Tue, Mar 10, 2015 at 12:40:03AM +0100, Thierry FOURNIER wrote:
 On Mon, 09 Mar 2015 22:11:56 +0100
 Cyril Bonté cyril.bo...@free.fr wrote:
  I've also seen this commit : MEDIUM: lua: use the Lua-5.3 version of 
  the library [1]
  This one may be annoying in the short term for some users, because Lua 
  5.3 is not available in all distributions (for example debian, ubuntu, ...).
  Currently for such distributions, it requires to recompile Lua 5.3 with 
  a small patch to generate the .so dynamic library. For those who want 
  to make some tests, you can have a look at [2] and [3] and the patches 
  they both provide.
 
 
 Arg, you're right :(. The Lua 5.3 is required for the session timeout.
 In the same way, it is necessary for the forced-yield. This system
 forces a yield and permits to haproxy to get the hand for processing
 other things than Lua.
 
 With the version 5.2, this is not possible without frequently Lua error.
 
 Maybe I can do cohabitation for 5.2 and 5.3 version. The forced yield
 will be deactivated with compilation 5.2. The execution timeout remain
 active. BUt, I'm not sure that is a good idea.

Given that there's no way to break an infinite loop before 5.2 and that
Lua will be used by people who experiment with scripts, I'd rather stick
to 5.3 unless someone finds an acceptable workaround working with 5.2.
I'd rather avoid seeing people report unresponsive haproxy just because
of bugs in their Lua scripts :-/

Willy




Re: Lua patchset merged

2015-03-10 Thread Willy Tarreau
On Tue, Mar 10, 2015 at 02:08:40AM +0100, Thierry FOURNIER wrote:
 On Tue, 10 Mar 2015 01:05:50 +0100
 Cyril Bonté cyril.bo...@free.fr wrote:
 
  Hi again,
  
  Le 10/03/2015 00:40, Thierry FOURNIER a écrit :
   On Mon, 09 Mar 2015 22:11:56 +0100
   Cyril Bonté cyril.bo...@free.fr wrote:
   I've seen new commits that have been merged on the git repository.
   The bad news are that the previous test that I reported (sending a
   response larger than the buffer) doesn't work anymore :-/
   Resulting in :
  
   [ALERT] 067/220744 (27176) : Lua function 'hello_world': execution 
   timeout.
  
  
   Hi cyril,
  
   This is due to the implementation of the Lua execution timeout. This is
   a system used to prevent loops in scripts. The Timeout is set by
   default to 4s. You can see tune.lua.session-timeout,
   tune.lua.task-timeout and tune.lua.forced-yield
  
   
   http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#tune.lua.session-timeout
  
  Of course, but it shouldn't take 4 seconds, the answer is immediate in 
  my test case.
  Actually, I could find that it was reproducible beginning with a 
  response greater or equal to 16392 bytes (I've not read the code yet).
 
 
 Thank you Cyril, the bug is partially reproduced and fixed (the buffer
 is not sent, but the error timeout is after 4 seconds as expected). I
 attach the patch. I think that Willy must check this patch, because
 it is possible than the comparison which I modify, did make sense.

Cool, will merge all this, thanks guys.

Willy




Re: Lua patchset merged

2015-03-10 Thread Willy Tarreau
All patches merged, thanks Thierry.

Willy




Re: Lua patchset merged

2015-03-09 Thread Cyril Bonté

Hi Thierry,

Le 04/03/2015 21:20, Cyril Bonté a écrit :

Hi Thierry,

Le 04/03/2015 11:51, Thierry FOURNIER a écrit :

Thank you Cyril for the bug repport. I join the patch that fix this
issue.


Great ! I didn't have time to investigate further.
I can confirm that the patch works well ;-)


I've seen new commits that have been merged on the git repository.
The bad news are that the previous test that I reported (sending a 
response larger than the buffer) doesn't work anymore :-/

Resulting in :

[ALERT] 067/220744 (27176) : Lua function 'hello_world': execution timeout.

I've also seen this commit : MEDIUM: lua: use the Lua-5.3 version of 
the library [1]
This one may be annoying in the short term for some users, because Lua 
5.3 is not available in all distributions (for example debian, ubuntu, ...).
Currently for such distributions, it requires to recompile Lua 5.3 with 
a small patch to generate the .so dynamic library. For those who want 
to make some tests, you can have a look at [2] and [3] and the patches 
they both provide.

Also, we'll have to :
- modify the haproxy Makefile in order to effectively detect Lua 5.3 and 
not 5.2 anymore (only the comment was modified)
- modify the version check in src/hlua.c to trigger an error if the 
version number is lesser than 503.


[1] 
http://www.haproxy.org/git?p=haproxy.git;a=commit;h=f90838b71a3c7f84e1d8b4ff85760a35d60c6910

[2] http://www.linuxfromscratch.org/blfs/view/svn/general/lua.html
[3] https://aur.archlinux.org/packages/lua53/


--
Cyril Bonté



Re: Lua patchset merged

2015-03-09 Thread Thierry FOURNIER
On Mon, 09 Mar 2015 22:11:56 +0100
Cyril Bonté cyril.bo...@free.fr wrote:

 Hi Thierry,
 
 Le 04/03/2015 21:20, Cyril Bonté a écrit :
  Hi Thierry,
 
  Le 04/03/2015 11:51, Thierry FOURNIER a écrit :
  Thank you Cyril for the bug repport. I join the patch that fix this
  issue.
 
  Great ! I didn't have time to investigate further.
  I can confirm that the patch works well ;-)
 
 I've seen new commits that have been merged on the git repository.
 The bad news are that the previous test that I reported (sending a 
 response larger than the buffer) doesn't work anymore :-/
 Resulting in :
 
 [ALERT] 067/220744 (27176) : Lua function 'hello_world': execution timeout.


Hi cyril,

This is due to the implementation of the Lua execution timeout. This is
a system used to prevent loops in scripts. The Timeout is set by
default to 4s. You can see tune.lua.session-timeout,
tune.lua.task-timeout and tune.lua.forced-yield

   
http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#tune.lua.session-timeout


 I've also seen this commit : MEDIUM: lua: use the Lua-5.3 version of 
 the library [1]
 This one may be annoying in the short term for some users, because Lua 
 5.3 is not available in all distributions (for example debian, ubuntu, ...).
 Currently for such distributions, it requires to recompile Lua 5.3 with 
 a small patch to generate the .so dynamic library. For those who want 
 to make some tests, you can have a look at [2] and [3] and the patches 
 they both provide.


Arg, you're right :(. The Lua 5.3 is required for the session timeout.
In the same way, it is necessary for the forced-yield. This system
forces a yield and permits to haproxy to get the hand for processing
other things than Lua.

With the version 5.2, this is not possible without frequently Lua error.

Maybe I can do cohabitation for 5.2 and 5.3 version. The forced yield
will be deactivated with compilation 5.2. The execution timeout remain
active. BUt, I'm not sure that is a good idea.


 Also, we'll have to :
 - modify the haproxy Makefile in order to effectively detect Lua 5.3 and 
 not 5.2 anymore (only the comment was modified)
 - modify the version check in src/hlua.c to trigger an error if the 
 version number is lesser than 503.


You're right. The patch is writed and attached.

Thank you
Thierry


 
 [1] 
 http://www.haproxy.org/git?p=haproxy.git;a=commit;h=f90838b71a3c7f84e1d8b4ff85760a35d60c6910
 [2] http://www.linuxfromscratch.org/blfs/view/svn/general/lua.html
 [3] https://aur.archlinux.org/packages/lua53/
 
 
 -- 
 Cyril Bonté
 
From 0d2bfc50892b1b09e3eb78ebaced9c0ed4334063 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER tfourn...@exceliance.fr
Date: Tue, 10 Mar 2015 00:35:36 +0100
Subject: [PATCH] BUG/BUILD: lua: The strict Lua 5.3 version check is not
 done.

This patch fix the Lua library check. Only the version
5.3 or later is allowed.

This bug is added by the patch MEDIUM: lua: use the
Lua-5.3 version of the library with commit id

   f90838b71a3c7f84e1d8b4ff85760a35d60c6910
---
 Makefile   |4 ++--
 src/hlua.c |4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index 1a31792..a12e4c9 100644
--- a/Makefile
+++ b/Makefile
@@ -569,9 +569,9 @@ OPTIONS_CFLAGS  += -DUSE_LUA $(if $(LUA_INC),-I$(LUA_INC))
 LUA_LD_FLAGS := $(if $(LUA_LIB),-L$(LUA_LIB))
 ifeq ($(LUA_LIB_NAME),)
 # Try to automatically detect the Lua library
-LUA_LIB_NAME := $(firstword $(foreach lib,lua5.2 lua52 lua,$(call check_lua_lib,$(lib),$(LUA_LD_FLAGS
+LUA_LIB_NAME := $(firstword $(foreach lib,lua5.3 lua53 lua,$(call check_lua_lib,$(lib),$(LUA_LD_FLAGS
 ifeq ($(LUA_LIB_NAME),)
-$(error unable to automatically detect the Lua library name, you can enforce its name with LUA_LIB_NAME=name (where name can be lua5.2, lua52, lua, ...))
+$(error unable to automatically detect the Lua library name, you can enforce its name with LUA_LIB_NAME=name (where name can be lua5.3, lua53, lua, ...))
 endif
 endif
 
diff --git a/src/hlua.c b/src/hlua.c
index 564f5ba..fb39dc1 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -4,8 +4,8 @@
 #include lua.h
 #include lualib.h
 
-#if !defined(LUA_VERSION_NUM) || LUA_VERSION_NUM  502
-#error Requires Lua 5.2 or later.
+#if !defined(LUA_VERSION_NUM) || LUA_VERSION_NUM  503
+#error Requires Lua 5.3 or later.
 #endif
 
 #include ebpttree.h
-- 
1.7.10.4



Re: Lua patchset merged

2015-03-09 Thread Cyril Bonté

Hi again,

Le 10/03/2015 00:40, Thierry FOURNIER a écrit :

On Mon, 09 Mar 2015 22:11:56 +0100
Cyril Bonté cyril.bo...@free.fr wrote:

I've seen new commits that have been merged on the git repository.
The bad news are that the previous test that I reported (sending a
response larger than the buffer) doesn't work anymore :-/
Resulting in :

[ALERT] 067/220744 (27176) : Lua function 'hello_world': execution timeout.



Hi cyril,

This is due to the implementation of the Lua execution timeout. This is
a system used to prevent loops in scripts. The Timeout is set by
default to 4s. You can see tune.lua.session-timeout,
tune.lua.task-timeout and tune.lua.forced-yield


http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#tune.lua.session-timeout


Of course, but it shouldn't take 4 seconds, the answer is immediate in 
my test case.
Actually, I could find that it was reproducible beginning with a 
response greater or equal to 16392 bytes (I've not read the code yet).


haproxy.cfg:
global
   lua-load hello_world.lua

listen proxy
   bind 127.0.0.1:10001
   tcp-request content lua hello_world


hello_world.lua:
function hello_world(txn)
   local res = txn:res_channel()

   s = 
   for i = 1,16392 do
   s = s .. x
   end
   res:send(s)
end

--
Cyril Bonté



Re: Lua patchset merged

2015-03-09 Thread Thierry FOURNIER
On Tue, 10 Mar 2015 01:05:50 +0100
Cyril Bonté cyril.bo...@free.fr wrote:

 Hi again,
 
 Le 10/03/2015 00:40, Thierry FOURNIER a écrit :
  On Mon, 09 Mar 2015 22:11:56 +0100
  Cyril Bonté cyril.bo...@free.fr wrote:
  I've seen new commits that have been merged on the git repository.
  The bad news are that the previous test that I reported (sending a
  response larger than the buffer) doesn't work anymore :-/
  Resulting in :
 
  [ALERT] 067/220744 (27176) : Lua function 'hello_world': execution timeout.
 
 
  Hi cyril,
 
  This is due to the implementation of the Lua execution timeout. This is
  a system used to prevent loops in scripts. The Timeout is set by
  default to 4s. You can see tune.lua.session-timeout,
  tune.lua.task-timeout and tune.lua.forced-yield
 
  
  http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#tune.lua.session-timeout
 
 Of course, but it shouldn't take 4 seconds, the answer is immediate in 
 my test case.
 Actually, I could find that it was reproducible beginning with a 
 response greater or equal to 16392 bytes (I've not read the code yet).


Thank you Cyril, the bug is partially reproduced and fixed (the buffer
is not sent, but the error timeout is after 4 seconds as expected). I
attach the patch. I think that Willy must check this patch, because
it is possible than the comparison which I modify, did make sense.

Thierry


 haproxy.cfg:
 global
 lua-load hello_world.lua
 
 listen proxy
 bind 127.0.0.1:10001
 tcp-request content lua hello_world
 
 
 hello_world.lua:
 function hello_world(txn)
 local res = txn:res_channel()
 
 s = 
 for i = 1,16392 do
  s = s .. x
 end
 res:send(s)
 end
 
 -- 
 Cyril Bonté
 
From f0185cae264f0309e386b500be115051a30e120b Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER tfourn...@exceliance.fr
Date: Tue, 10 Mar 2015 01:55:01 +0100
Subject: [PATCH] BUG/MEDIUM: buffer: one byte miss in buffer free space check

Space is not avalaible only if the end of the data inserted
is strictly greater than the end of buffer. If these two value
are equal, the space is avamaible.
---
 src/buffer.c |2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/buffer.c b/src/buffer.c
index e156991..3c7f6cc 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -75,7 +75,7 @@ int buffer_replace2(struct buffer *b, char *pos, char *end, const char *str, int
 
 	delta = len - (end - pos);
 
-	if (bi_end(b) + delta = b-data + b-size)
+	if (bi_end(b) + delta  b-data + b-size)
 		return 0;  /* no space left */
 
 	if (buffer_not_empty(b) 
-- 
1.7.10.4



Re: Lua patchset merged

2015-03-05 Thread Steven Le Roux
Hi,

why not just a github so that people can fork, play and submit their
own Lua scripts?

On Thu, Mar 5, 2015 at 8:00 AM, Willy Tarreau w...@1wt.eu wrote:
 On Mon, Mar 02, 2015 at 11:33:39PM +0100, Baptiste wrote:
 I love it !

 Just wrote, as a proof of concept, a forward proxy...
 That said, it seems my lua script is blocking... I mean, if the
 remote server is slow to deliver the response, then HAProxy doesn't
 process any other request or response.

 As explained by Thierry, this is because you used the native sockets
 instead of haproxy's non-blocking ones. I think that after some time
 we'll start to collect some best practices regarding Lua and such
 issues will not happen anymore since (as usual), people will write
 configs by copy-pasting working ones.

 We're still thinking about setting up a forum dediated to Lua so that
 people can share their code easily. It's a bit early for now since
 we're mostly fixing the code and/or design, but once things settle
 down it should be helpful.

 Willy





-- 
Steven Le Roux
Jabber-ID : ste...@jabber.fr
0x39494CCB ste...@le-roux.info
2FF7 226B 552E 4709 03F0  6281 72D7 A010 3949 4CCB



Re: Lua patchset merged

2015-03-05 Thread Willy Tarreau
Hi Steven,

On Thu, Mar 05, 2015 at 11:16:14PM +0100, Steven Le Roux wrote:
 Hi,
 
 why not just a github so that people can fork, play and submit their
 own Lua scripts?

Well, anyone can just do that by himself. I meant something more
discussion-oriented than development-oriented. Probably the type
of place were you'd find the links to the most common github
projects for example. It was suggested that Reddit could be suited
for this, we'll check.

Willy




Re: Lua patchset merged

2015-03-04 Thread Cyril Bonté

Hi Thierry,

Le 04/03/2015 11:51, Thierry FOURNIER a écrit :

Thank you Cyril for the bug repport. I join the patch that fix this issue.


Great ! I didn't have time to investigate further.
I can confirm that the patch works well ;-)

--
Cyril Bonté



Re: Lua patchset merged

2015-03-04 Thread Thierry FOURNIER
On Tue, 03 Mar 2015 19:35:34 +0100
Cyril Bonté cyril.bo...@free.fr wrote:

 Hi Thierry,
 
 Le 01/03/2015 13:47, Thierry FOURNIER a écrit :
 
  HAProxy Lua Hello world
 
  HAProxy configuration file (hello_world.conf):
 
  global
  lua-load hello_world.lua
 
  listen proxy
  bind 127.0.0.1:10001
  tcp-request content lua hello_world
 
 
  HAProxy Lua file (hello_world.lua):
 
  function hello_world(txn)
  local res = txn:res_channel()
  res:send(hello world\n)
  end
 
 
 I was wondering what would happen if a response was greater than the buffer.
 It appears that sometimes it works, sometime not.
 
 The function I used :
 function hello_world(txn)
 local res = txn:res_channel()
 s = 
 for i = 1,32768 do
  s = s .. x
 end
 res:send(s)
 end
 
 Then :
 $ while true; do s=$(curl -s http://localhost:10001/|wc -c); echo $s; if 
 [[ $s != 32768 ]]; then break; fi; done
 32768
 32768
 32768
 32768
 32768
 32768
 32768
 32768
 32768
 32768
 32768
 32768
 539952
 = Randomly, it will send extra data after the 16384 first bytes (with a 
 buffer of 16384 bytes), containing other parts of the memory, which may 
 leak other requests data.
 

Thank you Cyril for the bug repport. I join the patch that fix this issue.

Thierry
From 32fa948331282ae22728c6895d12bdb20125 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER tfourn...@exceliance.fr
Date: Wed, 4 Mar 2015 11:44:47 +0100
Subject: [PATCH] BUG/MAJOR: lua: send function fails and return bad bytes

In some cases the Lua send function fails. This is caused by the
return of buffer_replace2() is not tested. I checked avalaible space
in the buffer and I supposed than buffer_replace2() took all the
data. In some cases, buffer_replace2() cannot take the incoming
data, it returns the amount of data copied.

This patch check the amount of data really copied by buffer_replace2()
and advance the buffer with taking this value in account.
---
 src/hlua.c |2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/hlua.c b/src/hlua.c
index 1df18b6..d3f302f 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -2091,7 +2091,7 @@ __LJMP static int _hlua_channel_send(lua_State *L)
 	if (max  len - l)
 		max = len - l;
 
-	buffer_replace2(chn-chn-buf, chn-chn-buf-p, chn-chn-buf-p, str+l, max);
+	max = buffer_replace2(chn-chn-buf, chn-chn-buf-p, chn-chn-buf-p, str+l, max);
 	/* buffer replace considers that the input part is filled.
 	 * so, I must forward these new data in the output part.
 	 */
-- 
1.7.10.4



Re: Lua patchset merged

2015-03-03 Thread Cyril Bonté

Hi Thierry,

Le 01/03/2015 13:47, Thierry FOURNIER a écrit :


HAProxy Lua Hello world

HAProxy configuration file (hello_world.conf):

global
lua-load hello_world.lua

listen proxy
bind 127.0.0.1:10001
tcp-request content lua hello_world


HAProxy Lua file (hello_world.lua):

function hello_world(txn)
local res = txn:res_channel()
res:send(hello world\n)
end



I was wondering what would happen if a response was greater than the buffer.
It appears that sometimes it works, sometime not.

The function I used :
function hello_world(txn)
   local res = txn:res_channel()
   s = 
   for i = 1,32768 do
   s = s .. x
   end
   res:send(s)
end

Then :
$ while true; do s=$(curl -s http://localhost:10001/|wc -c); echo $s; if 
[[ $s != 32768 ]]; then break; fi; done

32768
32768
32768
32768
32768
32768
32768
32768
32768
32768
32768
32768
539952
= Randomly, it will send extra data after the 16384 first bytes (with a 
buffer of 16384 bytes), containing other parts of the memory, which may 
leak other requests data.



--
Cyril Bonté



Re: Lua patchset merged

2015-03-02 Thread Baptiste
I love it !

Just wrote, as a proof of concept, a forward proxy...
That said, it seems my lua script is blocking... I mean, if the
remote server is slow to deliver the response, then HAProxy doesn't
process any other request or response.

Baptiste



Re: Lua patchset merged

2015-03-02 Thread Thierry FOURNIER
On Sun, 01 Mar 2015 17:34:36 +0100
Aleksandar Lazic al-hapr...@none.at wrote:

 Dear Thierry.
 
 Cool work ;-)
 
 Am 01-03-2015 13:47, schrieb Thierry FOURNIER:
  Hi everyone,
  
  Thanks Willy for the introduction.
  
  I join the first part of the document. All the function are not yet
  available. The doc uses a markdown format like that uses GitHub. I join
  the original format, and the HTML conversion.
  
  The next steps are:
   - collecting users ideas and bug reports,
   - add missing functions like native header manipulations,
   - check the usage of existing libraries,
   - change the 'sleep' engine,
   - add execution timeout to prevent infinite loops,
   - some other tasks ...
 
 [snipp]
 
 I assume you know the lua module for nginx which is also fully non 
 blocking
 
 https://github.com/openresty/lua-nginx-module
 
 What I like in this module is the deep integration into nginx.
 
 Maybe it would be possible to integrate lua-scripting into the check 
 agent framework?


Hi,

thank you.

It is certainly possible. Have you some concrete usage case ?


 crazy-idea
 inject lua code over contorl socket ;-)
 /crazy-idea


It is technicaly possible. I'm not sure that it is useful for other
thing than development. For now, the tag crazy-idea is welcome :)

Thierry


  enjoy
  Thierry
 
 br Aleks
 
 
  On Sat, 28 Feb 2015 23:33:56 +0100
  Willy Tarreau w...@1wt.eu wrote:
  
  Hi folks,
  
  this is just a quick mail to let you know that I've just merged the 
  recent
  work from my coworker Thierry Fournier (in Cc) about supporting Lua 
  scripting
  in haproxy. I won't go into lengths explaining how that's supposed to 
  be used
  because I don't really know myself, but I'll let him provide some 
  explanations
  and working examples. The real purpose would be to get some feedback,
  suggestions, criticisms to get further into this.
  
  At the moment it is possible to declare some Lua-based sample fetch 
  functions,
  some converters as well and even to create new tcp-request and 
  http-req/resp
  actions. These last ones (the actions) support yielding so that it is 
  possible
  to make blocking calls from the Lua code and benefit from haproxy's 
  native
  session management and scheduling to switch to something else while 
  things
  are making progress. Thus the whole system is totally non-blocking and
  event-driven, even if in Lua you appear to be blocking.
  
  The long term goal is to be able to quickly provide some features that 
  are
  not yet possible natively, and eventually to implement some of them 
  natively
  if they're often requested or have a performance impact. Some of the 
  most
  complex parts such as writing authentication systems relying on 
  side-traffic
  would possibly remain written in Lua only, except if we manage to see 
  a big
  use of certain things and we see a way to implement them cleanly.
  
  Thierry is busy these days, and that's why I asked him to push his 
  code
  prior to switching to other tasks despite the doc not being there yet. 
  So
  it's possible we'll have to wait a bit for some doc, but we'll really 
  value
  testing, bug reports and feedback. Nothing is set in stone yet, he 
  does have
  some ideas on how to improve things. I do still have some concerns 
  about
  long-term stability and resource usage, as you can expect. So use it 
  as a
  toy first, play with it and tell us if you like it or not, and if not, 
  why.
  
  Thanks!
  Willy
  
  
 



Re: Lua patchset merged

2015-03-02 Thread Thierry FOURNIER
On Sun, 01 Mar 2015 23:45:01 +0100
Cyril Bonté cyril.bo...@free.fr wrote:

 Hi again Thierry,
 
 Le 01/03/2015 18:22, Cyril Bonté a écrit :
  I think I'll try some LUA scripts before the end of the week-end (which
  is approaching too quickly) ;-)
 
 OK, I could play with some Lua scripts.
 I could declare some custom sample fetches, from which I could call 
 internal fetches ;-)
 And that's where I also could raise some segfaults, when I use sample 
 fetches that don't require arguments (for example req_ver).
 
 A simple Lua script which reproduces the segfault :
 core.register_fetches(hello, function(txn, ...)
return txn.req_ver(txn)
 end
 
 With this haproxy configuration :
 global
 lua-load hello_world.lua
 
 listen proxy
 mode http
 bind 127.0.0.1:10001
 http-request redirect location /%[lua.hello]
 
 The time for me to commit and I'll send the fix.


Hi cyril,

Thank you for the tests and for the associated patches.

Thierry


 
 -- 
 Cyril Bonté
 



Re: Lua patchset merged

2015-03-02 Thread Thierry FOURNIER
On Sun, 1 Mar 2015 23:20:14 +0100
Baptiste bed...@gmail.com wrote:

  how do you pass arguments to a lua function?
  Imagine I want to call the following lua function: function download
 (host, file)


If you ask about argument between HAProxy configuration file and Lua
function, I assume than my documentation is incomplete.

converters and sample-fetches can have a maximum of 5 arguments.
All the arguments are strings. If HAProxy, the declaration is like this:

   http-request redirect location %[lua.my_function(arg1, arg2, arg3)]

The Lua declaration is like this:

   function my_function(txn, arg1, arg2, arg3)
  -- do something
   end

The action functions can not have arguments.

If you ask about the convention call in Lua, the official
documentation will be more precise than me.

   http://www.lua.org/manual/5.2/manual.html#3.4.9

Thierry


 Baptiste
 



Re: Lua patchset merged

2015-03-02 Thread Aleksandar Lazic



Am 02-03-2015 14:51, schrieb Thierry FOURNIER:

On Sun, 01 Mar 2015 17:34:36 +0100
Aleksandar Lazic al-hapr...@none.at wrote:


[snipp]


Maybe it would be possible to integrate lua-scripting into the check
agent framework?



Hi,

thank you.

It is certainly possible. Have you some concrete usage case ?


I'm sorry not yet.


crazy-idea
inject lua code over contorl socket ;-)
/crazy-idea



It is technicaly possible. I'm not sure that it is useful for other
thing than development.


Well you know things change.


For now, the tag crazy-idea is welcome :)


Thanks.


Thierry



 enjoy
 Thierry

br Aleks


 On Sat, 28 Feb 2015 23:33:56 +0100
 Willy Tarreau w...@1wt.eu wrote:

 Hi folks,

 this is just a quick mail to let you know that I've just merged the
 recent
 work from my coworker Thierry Fournier (in Cc) about supporting Lua
 scripting
 in haproxy. I won't go into lengths explaining how that's supposed to
 be used
 because I don't really know myself, but I'll let him provide some
 explanations
 and working examples. The real purpose would be to get some feedback,
 suggestions, criticisms to get further into this.

 At the moment it is possible to declare some Lua-based sample fetch
 functions,
 some converters as well and even to create new tcp-request and
 http-req/resp
 actions. These last ones (the actions) support yielding so that it is
 possible
 to make blocking calls from the Lua code and benefit from haproxy's
 native
 session management and scheduling to switch to something else while
 things
 are making progress. Thus the whole system is totally non-blocking and
 event-driven, even if in Lua you appear to be blocking.

 The long term goal is to be able to quickly provide some features that
 are
 not yet possible natively, and eventually to implement some of them
 natively
 if they're often requested or have a performance impact. Some of the
 most
 complex parts such as writing authentication systems relying on
 side-traffic
 would possibly remain written in Lua only, except if we manage to see
 a big
 use of certain things and we see a way to implement them cleanly.

 Thierry is busy these days, and that's why I asked him to push his
 code
 prior to switching to other tasks despite the doc not being there yet.
 So
 it's possible we'll have to wait a bit for some doc, but we'll really
 value
 testing, bug reports and feedback. Nothing is set in stone yet, he
 does have
 some ideas on how to improve things. I do still have some concerns
 about
 long-term stability and resource usage, as you can expect. So use it
 as a
 toy first, play with it and tell us if you like it or not, and if not,
 why.

 Thanks!
 Willy







Re: Lua patchset merged

2015-03-01 Thread Baptiste
 how do you pass arguments to a lua function?
 Imagine I want to call the following lua function: function download
(host, file)

Baptiste



Re: Lua patchset merged

2015-03-01 Thread Cyril Bonté

Hi again Thierry,

Le 01/03/2015 18:22, Cyril Bonté a écrit :

I think I'll try some LUA scripts before the end of the week-end (which
is approaching too quickly) ;-)


OK, I could play with some Lua scripts.
I could declare some custom sample fetches, from which I could call 
internal fetches ;-)
And that's where I also could raise some segfaults, when I use sample 
fetches that don't require arguments (for example req_ver).


A simple Lua script which reproduces the segfault :
core.register_fetches(hello, function(txn, ...)
  return txn.req_ver(txn)
end

With this haproxy configuration :
global
   lua-load hello_world.lua

listen proxy
   mode http
   bind 127.0.0.1:10001
   http-request redirect location /%[lua.hello]

The time for me to commit and I'll send the fix.


--
Cyril Bonté



Re: Lua patchset merged

2015-03-01 Thread Thierry FOURNIER
Hi everyone,

Thanks Willy for the introduction.

I join the first part of the document. All the function are not yet
available. The doc uses a markdown format like that uses GitHub. I join
the original format, and the HTML conversion.

The next steps are:
 - collecting users ideas and bug reports,
 - add missing functions like native header manipulations,
 - check the usage of existing libraries,
 - change the 'sleep' engine,
 - add execution timeout to prevent infinite loops,
 - some other tasks ...

Below, I add the HAProxy configuration documentation:

lua-load file

   scope: global

   This function declares and load the Lua file. A syntax parsing is
   executed and this directive fails if the syntax contain errors. The
   main Lua code is executed. This code can be used to initialize the
   Lua bindings in HAProxy.

tcp-request content lua function

   scope: frontend / backend / listen

   This keyword registers and Lua function executed when a tcp
   connection receive content.

tcp-response content lua function

   scope: frontend / backend / listen

   This keyword registers and Lua function executed when a tcp
   connection receive content from the response channel.

http-request lua function

   scope: frontend / backend / listen

   This keyword registers and Lua function executed when an http
   request is sent.

http-response lua function

   scope: frontend / backend / listen

   This keyword registers and Lua function executed when an http
   response is received.

enjoy
Thierry


On Sat, 28 Feb 2015 23:33:56 +0100
Willy Tarreau w...@1wt.eu wrote:

 Hi folks,
 
 this is just a quick mail to let you know that I've just merged the recent
 work from my coworker Thierry Fournier (in Cc) about supporting Lua scripting
 in haproxy. I won't go into lengths explaining how that's supposed to be used
 because I don't really know myself, but I'll let him provide some explanations
 and working examples. The real purpose would be to get some feedback,
 suggestions, criticisms to get further into this.
 
 At the moment it is possible to declare some Lua-based sample fetch functions,
 some converters as well and even to create new tcp-request and http-req/resp
 actions. These last ones (the actions) support yielding so that it is possible
 to make blocking calls from the Lua code and benefit from haproxy's native
 session management and scheduling to switch to something else while things
 are making progress. Thus the whole system is totally non-blocking and
 event-driven, even if in Lua you appear to be blocking.
 
 The long term goal is to be able to quickly provide some features that are
 not yet possible natively, and eventually to implement some of them natively
 if they're often requested or have a performance impact. Some of the most
 complex parts such as writing authentication systems relying on side-traffic
 would possibly remain written in Lua only, except if we manage to see a big
 use of certain things and we see a way to implement them cleanly.
 
 Thierry is busy these days, and that's why I asked him to push his code
 prior to switching to other tasks despite the doc not being there yet. So
 it's possible we'll have to wait a bit for some doc, but we'll really value
 testing, bug reports and feedback. Nothing is set in stone yet, he does have
 some ideas on how to improve things. I do still have some concerns about
 long-term stability and resource usage, as you can expect. So use it as a
 toy first, play with it and tell us if you like it or not, and if not, why.
 
 Thanks!
 Willy
 
 


channel.fig
Description: application/xfig

How Lua runs in HAProxy

HAProxy Lua running contexts

The Lua code executed in HAProxy can be processed in 2 main modes. The first one
is the initialisation mode, and the second is the runtime mode.


In the initialisation mode, we can perform DNS solves, but we cannot
perform socket I/O. In this initialisation mode, HAProxy still blocked during
the execution of the Lua program.
In the runtime mode, we cannot perform DNS solves, but we can use sockets.
The execution of the Lua code is multiplexed with the requests processing, so
the Lua code seems to be run in blocking, but it is not the case.


The Lua code is loaded in one or more files. These files contains main code and
functions. Lua have 6 execution context.


The Lua file body context. It is executed during the load of the Lua file
in the HAProxy [global] section with the directive lua-load. It is
executed in initialisation mode. This section is use for configuring Lua
bindings in HAProxy.
The Lua init context. It is an Lua function executed just after the
HAProxy configuration parsing. The execution is in initialisation mode. In
this context the HAProxy environment are already initialized. It is useful to
check configuration, or initializing socket connections or tasks. These
functions are declared in the body context with the Lua function
core.register_init(). The prototype of the function is a simple 

Re: Lua patchset merged

2015-03-01 Thread Aleksandar Lazic

Dear Thierry.

Cool work ;-)

Am 01-03-2015 13:47, schrieb Thierry FOURNIER:

Hi everyone,

Thanks Willy for the introduction.

I join the first part of the document. All the function are not yet
available. The doc uses a markdown format like that uses GitHub. I join
the original format, and the HTML conversion.

The next steps are:
 - collecting users ideas and bug reports,
 - add missing functions like native header manipulations,
 - check the usage of existing libraries,
 - change the 'sleep' engine,
 - add execution timeout to prevent infinite loops,
 - some other tasks ...


[snipp]

I assume you know the lua module for nginx which is also fully non 
blocking


https://github.com/openresty/lua-nginx-module

What I like in this module is the deep integration into nginx.

Maybe it would be possible to integrate lua-scripting into the check 
agent framework?


crazy-idea
inject lua code over contorl socket ;-)
/crazy-idea


enjoy
Thierry


br Aleks



On Sat, 28 Feb 2015 23:33:56 +0100
Willy Tarreau w...@1wt.eu wrote:


Hi folks,

this is just a quick mail to let you know that I've just merged the 
recent
work from my coworker Thierry Fournier (in Cc) about supporting Lua 
scripting
in haproxy. I won't go into lengths explaining how that's supposed to 
be used
because I don't really know myself, but I'll let him provide some 
explanations

and working examples. The real purpose would be to get some feedback,
suggestions, criticisms to get further into this.

At the moment it is possible to declare some Lua-based sample fetch 
functions,
some converters as well and even to create new tcp-request and 
http-req/resp
actions. These last ones (the actions) support yielding so that it is 
possible
to make blocking calls from the Lua code and benefit from haproxy's 
native
session management and scheduling to switch to something else while 
things

are making progress. Thus the whole system is totally non-blocking and
event-driven, even if in Lua you appear to be blocking.

The long term goal is to be able to quickly provide some features that 
are
not yet possible natively, and eventually to implement some of them 
natively
if they're often requested or have a performance impact. Some of the 
most
complex parts such as writing authentication systems relying on 
side-traffic
would possibly remain written in Lua only, except if we manage to see 
a big

use of certain things and we see a way to implement them cleanly.

Thierry is busy these days, and that's why I asked him to push his 
code
prior to switching to other tasks despite the doc not being there yet. 
So
it's possible we'll have to wait a bit for some doc, but we'll really 
value
testing, bug reports and feedback. Nothing is set in stone yet, he 
does have
some ideas on how to improve things. I do still have some concerns 
about
long-term stability and resource usage, as you can expect. So use it 
as a
toy first, play with it and tell us if you like it or not, and if not, 
why.


Thanks!
Willy






Re: Lua patchset merged

2015-03-01 Thread Baptiste
Great news, congratulation Thierry!

Baptiste



Re: Lua patchset merged

2015-03-01 Thread Cyril Bonté

Hi Tierry,

Huge work ! I've not played with it yet, but I've already compiled it 
successfully ;-)


Some early feedbacks :
- It appears that the code requires at least LUA 5.2.

- Maybe we'll have to work on the Makefile to ease the compilation. For 
example, on debian, I have to add -llua5.2 instead of -llua


- There's a small typo in an error message (hlua.c:423) : Malformad 
argument mask instead of Malformed argument mask for the 
hlua_lua2arg_check() function.

I'll send a patch later for that.

- Talking about hlua_lua2arg_check(), There are 2 other points :
1. The function comments has some typos. While trying to fix them, I'm 
realizing I don't understand the comment and I'm not sure to rewrite it 
correctly. Can you have a look at it ?

2. I think we can have a buffer overflow with the following test :
  if (idx = ARGM_NBARGS  argp[idx].type != ARGT_STOP)

The calling function (hlua_run_sample_fetch) already allows a same 
buffer overflow :

  struct arg args[ARGM_NBARGS];
and
  args[i].type = ARGT_STOP;
where `i' can be equal to ARGM_NBARGS.

- As it is done for other libraries, maybe we can add the compiled 
version of LUA when haproxy -vv is called.

I'll also send a patch for that.

- Still about the version : maybe we can add a #error when 
LUA_VERSION_NUM is not defined or less than 502 :

# LUA 5.0.x : not devined
# LUA 5.1.x : equal to 501
# LUA 5.2.x : equal to 502
# LUA 5.3.x : equal to 503

I think I'll try some LUA scripts before the end of the week-end (which 
is approaching too quickly) ;-)


--
Cyril Bonté



Re: Lua patchset merged

2015-03-01 Thread Baptiste
A few ifdef missing when SSL is not compiled in HAProxy:

diff --git a/src/hlua.c b/src/hlua.c
index a0e4d91..3d69c5d 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -1542,6 +1542,7 @@ __LJMP static int hlua_socket_connect(struct lua_State *L)
return 0;
 }

+#ifdef USE_OPENSSL
 __LJMP static int hlua_socket_connect_ssl(struct lua_State *L)
 {
struct hlua_socket *socket;
@@ -1551,6 +1552,7 @@ __LJMP static int hlua_socket_connect_ssl(struct
lua_State *L)
socket-s-target = socket_ssl.obj_type;
return MAY_LJMP(hlua_socket_connect(L));
 }
+#endif

 __LJMP static int hlua_socket_setoption(struct lua_State *L)
 {
@@ -3563,7 +3565,9 @@ void hlua_init(void)
lua_pushstring(gL.T, __index);
lua_newtable(gL.T);

+#ifdef USE_OPENSSL
hlua_class_function(gL.T, connect_ssl, hlua_socket_connect_ssl);
+#endif
hlua_class_function(gL.T, connect, hlua_socket_connect);
hlua_class_function(gL.T, send,hlua_socket_send);
hlua_class_function(gL.T, receive, hlua_socket_receive);


Baptiste


On Sun, Mar 1, 2015 at 6:22 PM, Cyril Bonté cyril.bo...@free.fr wrote:
 Hi Tierry,

 Huge work ! I've not played with it yet, but I've already compiled it
 successfully ;-)

 Some early feedbacks :
 - It appears that the code requires at least LUA 5.2.

 - Maybe we'll have to work on the Makefile to ease the compilation. For
 example, on debian, I have to add -llua5.2 instead of -llua

 - There's a small typo in an error message (hlua.c:423) : Malformad
 argument mask instead of Malformed argument mask for the
 hlua_lua2arg_check() function.
 I'll send a patch later for that.

 - Talking about hlua_lua2arg_check(), There are 2 other points :
 1. The function comments has some typos. While trying to fix them, I'm
 realizing I don't understand the comment and I'm not sure to rewrite it
 correctly. Can you have a look at it ?
 2. I think we can have a buffer overflow with the following test :
   if (idx = ARGM_NBARGS  argp[idx].type != ARGT_STOP)

 The calling function (hlua_run_sample_fetch) already allows a same buffer
 overflow :
   struct arg args[ARGM_NBARGS];
 and
   args[i].type = ARGT_STOP;
 where `i' can be equal to ARGM_NBARGS.

 - As it is done for other libraries, maybe we can add the compiled version
 of LUA when haproxy -vv is called.
 I'll also send a patch for that.

 - Still about the version : maybe we can add a #error when LUA_VERSION_NUM
 is not defined or less than 502 :
 # LUA 5.0.x : not devined
 # LUA 5.1.x : equal to 501
 # LUA 5.2.x : equal to 502
 # LUA 5.3.x : equal to 503

 I think I'll try some LUA scripts before the end of the week-end (which is
 approaching too quickly) ;-)

 --
 Cyril Bonté