Re: [EXT] [EXT] Re: Regarding the new dark mode dashboard in 2.5

[Thierry Fournier]

> On 8 Mar 2022, at 14:52, Willy Tarreau  wrote:
> 
> Hi guys,
> 
> On Tue, Mar 08, 2022 at 01:46:50PM +, Marno Krahmer wrote:
>> Hey Tim,
>> 
>> I added a reference to the GitHub issue to the second line of the commit 
>> message.
> 
> Too late, I took the one you posted on the issue and that Ciprian
> confirmed. No big deal anyway.


Hi guys,

Thanks for the patch. Feel free to suggest colors, I have no
artistic taste :-) I just open the dark-mode way.

Thierry

Re: [PATCH] BUG/???: lua: Add missing call to RESET_SAFE_LJMP in hlua_filter_new()

[Thierry Fournier]

> On 12 Sep 2021, at 08:21, Willy Tarreau  wrote:
> 
> On Sat, Sep 11, 2021 at 11:17:25PM +0200, Tim Duesterhus wrote:
>> In one case before exiting leaving the function the panic handler was not
>> reset.
>> 
>> Introduced in 69c581a09271e91d306e7b9080502a36abdc415e, which is 2.5+.
>> No backport required.
> 
> Good catch, applied as medium since it seems none of us can clearly
> predict the effect :-)


Good question :-)

This system manage longjmp() calls used to catch Lua errors. The longjmp()
is set through the function lua_atpanic() before executing any Lua code.
The haproxy lua panic hanlder call a longjmp which return to the lua
execution caller and an error is processed.

When the Lua code execution is done (with or without error) reseting
this panic function restore the default Lua behavior which is an abort().

   see https://www.lua.org/manual/5.3/manual.html#4.6

So, when the longjmp is not reset, if some Lua code is executed ommiting
to set the longjmp (function lua_atpanic() crush the precedent registered
panic function) and the Lua code raises an exception a longjmp is processed
in a point of stack which contains random data.

In the best case, we have a segfault, in a worst case Haproxy continne
running doing crap.

The same case (Lua code is executed ommiting to set the longjmp and
the Lua code raises an exception) with the default panic handler throws
an abort().

A buffer overflow could be exploited only if haproxy try to execute
Lua code without setting the safe environment. I hope this is not the
case. I’m confident because if some lus code were executed without the
panic handler, abort() were already observed for a long time.

So, I guess this ommit is not a great bug, but the experieence learn
when we play with longjmp, MEDIUM is the right level for a patch.

Thierry

Undesirable space in a metric "Unstoppable Jobs"

[Thierry Fournier]

Hi list,

Listing metrics for supervision, I see an undesirable space in a
metric name 54 "Unstoppable Jobs".

44.CompressBpsRateLim.1:CLP:u32:0
47.Tasks.1:MGP:u32:455
48.Run_queue.1:MGP:u32:1
49.Idle_pct.1:MaP:u32:92
50.node.1:COS:str:ozon3
52.Stopping.1:MGP:u32:0
53.Jobs.1:MGP:u32:81
 -> 54.Unstoppable Jobs.1:MGP:u32:0
55.Listeners.1:MGP:u32:31
56.ActivePeers.1:MGP:u32:0
57.ConnectedPeers.1:MGP:u32:0
58.DroppedLogs.1:MGP:u32:0
59.BusyPolling.1:MGP:u32:0

This space is introduced two year ago. If anyone uses this variable
I suggest a rename with "_" in place of space.

Thierry

Lua error message patch

[Thierry Fournier]

Hi list,

A little patch to improve verbosity of some Lua errors.

A+
Thierry



0001-MINOR-hlua-Add-error-message-relative-to-the-Channel.patch
Description: Binary data

Re: [PATCH] MAJOR: contrib: porting spoa_server to support python3

[Thierry Fournier]

e mode 100644 haproxy/contrib/spoa_server/ps_python.h
>> 
> 
>> diff --git a/haproxy/contrib/spoa_server/Makefile 
>> b/haproxy/contrib/spoa_server/Makefile
>> index f075282..e7b20db 100644
>> --- a/haproxy/contrib/spoa_server/Makefile
>> +++ b/haproxy/contrib/spoa_server/Makefile
>> @@ -23,10 +23,47 @@ endif
>> 
>> ifneq ($(USE_PYTHON),)
>> OBJS + ps_python.o
>> +
>> +# "--embed" flag is supported (and required) only from python 3.8+
>> +check_python_config : $(shell if python3-config --embed; then echo 
>> "python3.8+"; \
>> +elif hash python3-config; then echo "python3"; \
>> +elif hash python-config; then echo "python2"; fi)
>> +
>> +ifeq ($(check_python_config), python3.8+)
>> +PYTHON_DEFAULT_INC : $(shell python3-config --includes)
>> +PYTHON_DEFAULT_LIB : $(shell python3-config --libs --embed)
>> +else ifeq ($(check_python_config), python3)
>> +PYTHON_DEFAULT_INC : $(shell python3-config --includes)
>> +PYTHON_DEFAULT_LIB : $(shell python3-config --libs)
>> +else ifeq ($(check_python_config), python2)
>> +PYTHON_DEFAULT_INC : $(shell python-config --includes)
>> +PYTHON_DEFAULT_LIB : $(shell python-config --libs)
>> +endif
>> +
>> +
>> +# Add default path
>> +ifneq ($(PYTHON_DEFAULT_INC),)
>> +CFLAGS + $(PYTHON_DEFAULT_INC)
>> +else
>> CFLAGS + -I/usr/include/python2.7
>> +endif
>> +ifneq ($(PYTHON_DEFAULT_LIB),)
>> +LDLIBS + $(PYTHON_DEFAULT_LIB)
>> +else
>> LDLIBS + -lpython2.7
>> endif
>> 
>> +# Add user additional paths if any
>> +ifneq ($(PYTHON_INC),)
>> +CFLAGS + -I$(PYTHON_INC)
>> +endif
>> +ifneq ($(PYTHON_LIB),)
>> +LDLIBS + -L$(PYTHON_LIB)
>> +endif
>> +
>> +LDLIBS +=-Wl,--export-dynamic
>> +endif
>> +
>> spoa: $(OBJS)
>>  $(LD) $(LDFLAGS) -o $@ $^ $(LDLIBS)
>> 
>> diff --git a/haproxy/contrib/spoa_server/README 
>> b/haproxy/contrib/spoa_server/README
>> index 341f5f9..f654a23 100644
>> --- a/haproxy/contrib/spoa_server/README
>> +++ b/haproxy/contrib/spoa_server/README
>> @@ -14,9 +14,10 @@ is done.
>> You have to install the development packages, either from the
>> distribution repositories or from the source.
>> 
>> -CentOS/RHEL: yum install python-devel
>> +CentOS/RHEL: sudo yum install python3-devel
>> 
>> -The current python version in use is 2.7.
>> +The current minimal python version compatible with this library is 2.7.
>> +It's recommended to use python version 3 where possible due to python 2 
>> deprecation.
>> 
>> 
>>   Compilation
>> @@ -28,6 +29,11 @@ USE_LUA=1 and/or USE_PYTHON=1.
>> You can add LUA_INC=.. LUA_LIB=.. to the make command to set the paths to
>> the lua header files and lua libraries.
>> 
>> +Similarly, you can add PYTHON_INC=.. PYTHON_LIB=.. to the make command to 
>> set the paths to
>> +the python header files and python libraries.
>> +By default, it will try to compile by detecting the default python 3 
>> parameters.
>> +It will fall back to python 2 if python 3 is not available.
>> +
>>   Start the service
>> -
>> 
>> diff --git a/haproxy/contrib/spoa_server/ps_python.c 
>> b/haproxy/contrib/spoa_server/ps_python.c
>> index 0a9fbff..019d125 100644
>> --- a/haproxy/contrib/spoa_server/ps_python.c
>> +++ b/haproxy/contrib/spoa_server/ps_python.c
>> @@ -1,13 +1,25 @@
>> /* spoa-server: processing Python
>>  *
>>  * Copyright 2018 OZON / Thierry Fournier 
>> + * Copyright (C) 2020  Gilchrist Dadaglo 
>>  *
>>  * This program is free software; you can redistribute it and/or
>>  * modify it under the terms of the GNU General Public License
>>  * as published by the Free Software Foundation; either version
>>  * 2 of the License, or (at your option) any later version.
>>  *
>> + * This program is provided in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + */
>> +
>> +/*
>> + *  Define PY_SSIZE_T_CLEAN before including Python.h
>> + *  as per 
>> https://antiphishing.ozon.io/4/0@_ixbUL4T4hw@E5GNuKwiNHkYpBKox487sQXNiWT2zi6XxBA4dwDPDRhTqnJB0fIQPU1gpsK7qk3xHzT5gIW2saBt2Fi7bNrF9TfMQPCkVEmSYqRZ6MLczUQ50fDoY-PN5w@fd97fcca979ec784c73757fc7af662f684c574a6
>>  and 
>> https://antiphishing.ozon.io/4/0@m-JgHdfit1E@MMo7vNjm_86dRyOfOBPvSY_YLSbh

Re: [PATCH v2 0/3] MINOR: lua: Add lua-prepend-path configuration option

[Thierry Fournier]

> On 14 Jan 2020, at 13:53, Willy Tarreau  wrote:
> 
> On Tue, Jan 14, 2020 at 01:46:55PM +0100, Thierry Fournier wrote:
>> Hi, It have two default path in the library path:
>> 
>> - path for lua libraries
>> - cpath for loading lua C libraries.
>> 
>> In some cases, haproxy requires C libraries like cjson which manipulates 
>> json.
>> 
>> I'm not sure, but it seems that the patch "lua-prepend-path" doesn't include 
>> the cpath.
> 
> Yes it does now with an optional "cpath" keyword on the line. Maybe it's
> not the most convenient way and you'd prefer separate keywords ?


Sorry, I don’t see the keyword "cpath". Maybe a lua-prepend-cpath
would be clearer than a final keyword ? I don’t known. Maybe the
cpath word would be specified at the start in place of the end ?
The most important it was the cpath was taken in account !

Just a little remark, I hope without consequences. Lua C libraries
are .so and not .lua:

   lua-prepend-path /etc/haproxy/lua-cmodules/?.lua cpath

becomes:

   lua-prepend-path /etc/haproxy/lua-cmodules/?.so cpath




Idea 1:

   lua-prepend-path path /etc/haproxy/lua-modules/?.lua
   lua-prepend-path cpath /etc/haproxy/lua-cmodules/?.lua

Idea 2:

   lua-prepend-path /etc/haproxy/lua-modules/?.lua
   lua-prepend-path cpath /etc/haproxy/lua-cmodules/?.lua

Idea 3:

   lua-prepend-path /etc/haproxy/lua-modules/?.lua
   lua-prepend-cpath /etc/haproxy/lua-cmodules/?.lua

Thierry

Re: [PATCH v2 0/3] MINOR: lua: Add lua-prepend-path configuration option

[Thierry Fournier]

> On 14 Jan 2020, at 07:08, Willy Tarreau  wrote:
> 
> On Sun, Jan 12, 2020 at 01:55:38PM +0100, Tim Duesterhus wrote:
>> Willy,
>> Thierry,
>> Vincent,
>> 
>> I've just prepared a new version of my proposal with the following 
>> differences:
>> 
>> 1. I adjusted the documentation as suggested by Willy.
>> 2. I added support for `package.cpath` as suggested by Thierry.
>> 3. I added a build time option to be used by distros as suggested by Vincent.
> (...)
> 
> Thanks Tim! Thierry, just let me know if you think it's OK for merging
> and I'll handle it, or if you need more time to review it or are still
> having any doubts.
> 


Hi, It have two default path in the library path:

 - path for lua libraries
 - cpath for loading lua C libraries.

In some cases, haproxy requires C libraries like cjson which manipulates json.

I’m not sure, but it seems that the patch “lua-prepend-path” doesn’t include 
the cpath.

Thierry

Re: Lua: forcing garbage collector after socket i/o

[Thierry Fournier]

> On 11 Jan 2020, at 07:48, Willy Tarreau  wrote:
> 
> On Fri, Jan 10, 2020 at 03:17:59PM -0800, Sadasiva Gujjarlapudi wrote:
>> After applying the patch and socket wasn't `close`d in Lua code
>> request/sec 7898.0976
>> $time haproxy
>> real 0m6.689s
>> user 0m1.043s
>> sys 0m1.059s
>> 
>> After applying the patch and socket was `close`d in Lua code
>> requests/sec: 9617.4609
>> $time haproxy
>> real 0m7.505s
>> user 0m0.738s
>> sys 0m0.930s
>> 
>> Same machine, no Lua code running
>> requests/sec: 14468.4468
>> $time haproxy
>> real 0m17.265s
>> user 0m0.243s
>> sys 0m0.670s
>> 
>> Also tested by causing the runtime error in Lua code before `close`ing
>> the socket.
>> $ulimit -n
>> 6000
>> Lua action/app created 20K connections during the test and everything
>> seems to be working as expected.
> 
> Excellent, this is very encouraging.
> 
>> My only concern with this patch is what if there is a persistent
>> connection and `conn_count` is always greater than zero and that
>> triggers GC.
> 
> It should not be a problem because the "lua" struct is per-Lua instance,
> hence per-stream, so we shouldn't have that many connections per stream
> and they'll be GC'd once the Lua instance or the stream is terminated.
> And if someone runs a self-contained Lua application creating that many
> sockets and never closing them, the current situation doesn't help
> either.


I confirm: GC cleanup only object where are no longer referenced. If the
connection  is global, the core.tcp object is referenced globaly and it
is not cleanned.


> What I propose if you agree is that I'll merge the patch into 2.2 next
> week. This will give us quite some time to observe if it requires more
> adjustments. And then you can pick it and apply it to your local tree
> in order to save some resources.


This is a good performance improvement. Easy and efficient !

Thierry

Re: [PATCH] MINOR: lua: Add lua-prepend-path configuration option

[Thierry Fournier]

> On 9 Jan 2020, at 22:02, Willy Tarreau  wrote:
> 
> On Thu, Jan 09, 2020 at 08:22:29PM +0100, Tim D�sterhus wrote:
>> Bob,
>> 
>> Am 09.01.20 um 19:44 schrieb Zakharychev, Bob:
>>> I'd say that since the issue can be solved naturally and completely within 
>>> the confines of Lua itself, new parameters should not be introduced in 
>>> HAProxy to give one ability to solve it differently while obscuring the 
>>> solution in configuration of the host program, however tempting this might 
>>> look.
>> 
>> To put my POV into a different perspective: HAProxy is the host program
>> of the Lua scripts and thus it is responsible for providing a working
>> environment to the script. It should not need to know how the file
>> system layout works.
>> Configuring this environment within HAProxy just feels natural even if
>> it might be possible to configure this setting from within a dedicated
>> "configuration script" as well.
> 
> The useful point I'm seeing here is that it allows users to use some
> external Lua programs with no modification and only specify the paths
> in the global section which centralizes everything specific to the
> deployment. Also it even allows to use environment variables there,
> which can be quite useful.
> 
> However, Bob, your concern confirms mine about the risks involved by
> adding the ability to execute some Lua code inlined directly from the
> haproxy config. So I think that if we want to stay reasonable, at least
> for a while, we should take Tim's patch as it is now to help configure
> the path, but not put Lua code in the config file.


Hi guys, I don’t have any opinion about this. But I extend the problem:

There are not only package.path, but also package.cpath which provides 
default location for .so libraries.

   https://www.lua.org/manual/5.3/manual.html#pdf-package.cpath

Thierry

Re: Lua: forcing garbage collector after socket i/o

[Thierry Fournier]

> On 10 Jan 2020, at 04:46, Willy Tarreau  wrote:
> 
> On Thu, Jan 09, 2020 at 05:30:32PM -0800, Sadasiva Gujjarlapudi wrote:
>> After applying Willy's patch
>> 
>> requests/sec 7181.9869
>> $time haproxy
>>  real 0m10.304s
>>  user 0m1.112s
>>  sys 0m1.184s
>> 
>> After applying Willy's patch and the following patch
>> 
>> @@ -955,10 +956,14 @@ void hlua_ctx_destroy(struct hlua *lua)
>> * the garbage collection.
>> */
>>if (lua->flags & HLUA_MUST_GC) {
>> -   if (!SET_SAFE_LJMP(gL.T))
>> -   return;
>> -   lua_gc(gL.T, LUA_GCCOLLECT, 0);
>> -   RESET_SAFE_LJMP(gL.T);
>> +   if (lua->gc_tune_count++ > 1000) {
>> +   lua->gc_tune_count = 0;
>> +   if (!SET_SAFE_LJMP(gL.T))
>> +   return;
>> +   lua_gc(gL.T, LUA_GCCOLLECT, 0);
>> +   RESET_SAFE_LJMP(gL.T);
>> +   }
>> +
>>}
>> 
>> requests/sec 8635.0402
>>  real 0m10.969s
>>  user 0m0.744s
>>  sys 0m1.116s
> 
> OK this is promising, thanks for the tests!
> 
>> Most of the time the CPU is the bottleneck and it may be ok to consume
>> little bit more memory on dedicated servers running haproxy.
>> Our use case/Lua application required to use as little CPU as possible.
> 
> Of course, but according to Thierry the problem is not so much about
> memory, but rather about connections/file descriptors.
> 
> Well, I'm thinking about something. Since this HLUA_MUST_GC flag is used
> only for connections, we could probably proceed differently. Instead of
> a flag we could have a connection counter, which gets incremented for
> each connection we create and decremented for each closed connection.
> Then we'd just run the GC in hlua_destroy_ctx() if the connection count
> indicates we did not kill all of them. This way, the GC will never run
> for normal situations where connections were cleanly closed, and it will
> only be run if absolutely needed.
> 
> I can propose you to give a try to the attached patch. I have no idea if
> it works or does something correct. It's only build-tested.


I like this way. If the Lua program works perfectly, the GC is not forced.
If some connections are pending the GC start and clean up.

Maybe it is must clever to rename the variable from “conn_count” to
“require_gc_count”. By this way if another resource have the same problem
than TCP connections, this var could be used ?

For testing this patch we need three cases:

- The Lua program work perfectly.

- The Lua program must fail after connexion (something like this:
  unexistant_array[‘void_key'] = 1). This test the unexpected abort. If the
  Lua fails, we must validate the socket is closed and HAProxy is not blocked
  because no connection available.

- The socket close must be commented out. The same kind of test than previsouly,
  but without the bug.

Thierry

Re: Lua: forcing garbage collector after socket i/o

[Thierry Fournier]

> On 9 Jan 2020, at 10:47, Willy Tarreau  wrote:
> 
> Hi Thierry,
> 
> On Thu, Jan 09, 2020 at 10:34:35AM +0100, Thierry Fournier wrote:
>> I'm remember: the execution of the GC were the only way to close TCP 
>> connexions
>> because in some cases the object provided by core.tcp() is not closed and the
>> connexion is not released. Without the GC execution, HAProxy reach the 
>> maximum
>> of available connexion, and the process were blocked. The flag forcing the 
>> GCC
>> is set only is we use a socket.
> 
> Ah OK then it makes sense, thanks for the explanation. However I'm seeing
> that the GC is forced for every yield/resume call, and not just on destroy.
> Maybe it could be acceptable not to call it on resume ?
> 
> Sada, would you be interested in checking if this solves most of the
> performance issue:
> 
> diff --git a/src/hlua.c b/src/hlua.c
> index 37f7866874..e5257efb54 100644
> --- a/src/hlua.c
> +++ b/src/hlua.c
> @@ -1195,7 +1195,7 @@ resume_execution:
>}
> 
>/* This GC permits to destroy some object when a Lua timeout strikes. 
> */
> -   if (lua->flags & HLUA_MUST_GC &&
> +   if (0 && lua->flags & HLUA_MUST_GC &&
>ret ! HLUA_E_AGAIN)
>lua_gc(lua->T, LUA_GCCOLLECT, 0);


This make sense, but I not remember if the GC at this point is important.

Just for information, there are the commit messages:

OPTIM/MEDIUM: lua: executes the garbage collector only when using cosocket

The garbage collector is a little bit heavy to run, and it was added
only for cosockets. This patch prevent useless executions when no
cosockets are used.




> 
>> Maybe we try to include new option "tune.lua.forced-gc".
> 
> Based on your description if we need an option, I'd rather have the
> opposite, like "tune.lua.disable-gc" so that it remains enabled by
> default, and with an explanation that one must not disable it.


I agree, but it make sense to run some tests:

 - First, testing your patch to decide if the GC line 1195 is relevant
 - Second, testing the behavior when GC is disabled and lua create connexion 
and lua fail without closing connexion.

Maybe the second case make the GC mandatory.

Thierry

Re: Lua: forcing garbage collector after socket i/o

[Thierry Fournier]

Hi,

I try this morning to reproduce the behaviour, but I do not have sufficient
time to finish the test. I’ml not surprised by the described behaviour because
we are talking about GC.

GC are program written to save the brain of developper implying a permanent
overconsumption of the CPU. So GC are complex algorithms using most CPU and
impacting performances.

I’m remember: the execution of the GC were the only way to close TCP connexions
because in some cases the object provided by core.tcp() is not closed and the
connexion is not released. Without the GC execution, HAProxy reach the maximum
of available connexion, and the process were blocked. The flag forcing the GCC
is set only is we use a socket.

Maybe we try to include new option "tune.lua.forced-gc”.

br,
Thierry

—
Thierry Fournier
Web Performance & Security Expert
m: +33 6 68 69 21 85  | e: thierry.fourn...@ozon.io
w: http://www.ozon.io/| b: http://blog.ozon.io/

> On 9 Jan 2020, at 06:25, Willy Tarreau  wrote:
> 
> Hi,
> 
> On Thu, Jan 02, 2020 at 04:36:58PM -0800, Sadasiva Gujjarlapudi wrote:
>> Hi,
>> We have observed (significant)request handling rate dropped if the TCP
>> sockets were used in `http-request` action.
>> Request handling rate recovered with slight increase in memory usage after
>> commenting the line in `hlua_socket_connect()`/hlua.c
>> `hlua->flags | HLUA_MUST_GC`
>> Doc: 
>> https://antiphishing.ozon.io/4/0@dIg8fWe3Pg0@uv152ypkX71_NyCG5u2ry1B654tGLFq2UKabFItkBLEp7NFIDiBuEUqAAogidiDiOGvMnTSTFK3ClXj9Ozc9-0FWyvq_z7qztGU6-WYKict5U8Wp-m3Gr7K1h-B9elAz@73c1dfb44abdfca30a93012148bd1e7ebfe3dee2
>> 
>> I want to add a param to `Socket:connect()` or add new method
>> `Socket:connect2()`.
>> to disable forced GC.
>> Any feedback is appreciated.
> 
> Interesting! CCing Thierry who's the one who knows this best. Thierry,
> please also have a look at Sada's follow up e-mail with his numbers and
> config. It looks like the GC is particularly aggressive here, and I don't
> know if this is really needed. Or maybe we could have a counter and only
> force it once every 100 times or so ?
> 
> thanks,
> Willy

Re: Lua + shared memory segment

[Thierry Fournier]

Hi,

You can’t use socket in sample-fetches for HAProxy internal reason.
If you want to take decision using socket+redis, you must set your
Redis request in “action”. according with the redis response, your
action set the result in shared context, and a sample fetche can
return the result. Something like that:


   http-request lua.do_my_redis_stuff
   use_backend bck_a if { lua.get_redis_result bck_a }
   use_backend bck_b if { lua.get_redis_result bck_b }

   core.register_action(“do_my_redis_stuff”, fucntion (txn)
  -- perform redis request
  network = core.tcp()
  ...
  -- get response
  response = network:read()
  -- store in context
  ctx = {}
  ctx[‘message’] = response
  txn:set_ctx(ctx)
   end)

   core.register_fetches("get_redis_result", function(txn)
  a = txn:get_ctx()
  return a[‘message']
   end)


Thierry


> On 7 Nov 2019, at 11:56, Kalantari  wrote:
> 
> 
> 
> 
> Hi Thierry,
> 
> I know this an old thread but I'm having a similar issue where HA Proxy 
> doesn't allow me to use Redis (not allowed to use socket in fetches). My 
> scenario is as below:
> 
> one frontend and multiple backends (proxies). (the frontend is just one IP 
> and always sends requests to the same url) 
> I use Lua script SmartRouting to:
> A. Look at the body of the body of the message and decide which back end to 
> use
> B. If the backend servers being used are too busy (active sessions > 
> threshold) then it will choose to send the requests to the offlineAgent
> 
> frontend http-in
> bind *:80 
> use_backend %[lua.SmartRouting(txn)]
> default_backend OnlineChargers
>  
> backend Onlineagent
> server testServer1 ${Server_1_IP}:${Server_1_Port} check
> backend ServersForRequestTypeX
> server testServer1 ${Server_2_IP}:${Server_2_Port} check
> backend offlineAgents
> server testServer1 ${Server_3_IP}:${Server_3_Port} check
> http-response lua.add_DegradedSession
> 
> Straight forward up to this point and no need for Redis, however if a message 
> is sent to offlineAgent proxy, then I want all the rest of the requests to be 
> sent to the offline agent. (each message can have a sessionID 
> inside the payload). I tried to add the sessionID for the messages inside my 
> SmartRouting to Redis as it was explained in your blogpost but HAProxy throws 
> an error and doesn't allow use of a socket in sample fetch. 
> below is my Lua script:
> 
> -- if number of active sessions goes higher than below then degraded mode is 
> detected
> DEGRADED_THRESHOLD = 10
> -- global object which will be used for routing of requests based on request 
> type xml tag
> -- if the request type is not found then HA proxy's default backend will be 
> used
> routingTable = {
> [""] = "Onlineagent",
> [""] = "ServersForRequestTypeX"
> }
> 
> local function SmartRouting(txn)
> --print_r(txn)
> local payload = txn.sf:req_body()
> -- extract request tag (final occurence of  local requestTag = string.match(payload, " local selectedBackend = routingTable[requestTag]
> -- check if there's enough capacity (degraded mode)
> -- this check is only neccessary if the
> if (selectedBackend == "Onlineagent") then
> local sessionCount = getTotalSessions()
> if (sessionCount > DEGRADED_THRESHOLD) then
> core.Debug("shit is hiting the fan! Sending requests to degraded 
> agent")
> --TODO:  Add the sessionID to Redis and check for future requests 
> if it's in Redis then redirect to Offline Agent
> return "offlineAgents"
> end
> end
> return selectedBackend
> end
> 
> function getTotalSessions()
> local total_sessions = 0
> for _, backend in pairs(core.backends) do
> if backend.name == "Onlineagent" then
> for _, server in pairs(backend.servers) do
> -- Get server's stats
> local stats = server:get_stats()
> 
> -- Get the backend's total number of current sessions
> if stats["status"] == "UP" then
> total_sessions = total_sessions + stats["scur"]
> end
> end
> end
> end
> return total_sessions
> end
> 
> -- register HAProxy "fetch"
> core.register_fetches("SmartRouting", dreSmartRouting)

Re: [PATCH 2/2] BUG/MINOR: lua: Make the arrays in the list of headers 1-indexed

[Thierry Fournier]

> On 30 Sep 2019, at 05:40, Willy Tarreau  wrote:
> 
> Hi guys,
> 
> On Thu, Sep 26, 2019 at 04:35:31PM +0200, Tim Düsterhus wrote:
>> Adis,
>> 
>> Am 26.09.19 um 16:01 schrieb Adis Nezirovic:
>>> While I agree that using zero based array indexing is a mistake (wearing
>>> my Lua hat), I don't think your proposal will make situation any better.
>>> Actually ot might hurt existing Lua code in unforeseen ways.
>> 
>> Yes, I understand that it breaks backwards compatibility. That's why I
>> added the cover letter with the rationale of duplicating the first value
>> instead of making the 0-index a nil.
> 
> To be honest I'm totally ignorant of this code area and it's even not
> very clear to me (even after reading the commit messages) what is the
> extent of this change nor what this headers array corresponds to.
> 
> However what I can say is that when implementing a transition towards
> a final solution by having an intermediate solution (like you're
> proposing by keeping index 0), it's important to have a plan to get rid
> of the intermediate solution at some point. Here, by duplicating entries
> I'm not seeing any way to properly cover this final transition, nor to
> safely avoid breakage of existing applications. Thus it will turn an
> existing problem into two other ones.
> 
> I could instead propose two solutions :
>  - either we knowingly break existing code for 2.1+ by changing the
>array numbering to match Lua standards ; those migrating from 2.0
>to 2.1 or 2.2 will have to modify their code. We can explain them
>how to make portable code by checking the haproxy version from
>within Lua if required;
> 
>  - or we decide to rename this "headers" field on the long term (if
>it's the only one, which is really the part I'm ignoring) and create
>a new one ("hdrs" ?) numbered from 1. We can thus keep the two in
>parallel till 2.2 included so that at least one LTS version covers
>both the old and new API, and in 2.3 we remove headers. If we can
>emit an API warning when "headers" is first met, it's even better.
>I just don't know how we can make one structure field alias another
>one, I can understand that it's not just a matter of setting +1 on
>a pointer, but I guess that we have hooks behind the curtains which
>intercept this "headers" thing.
> 
> There may also be other solutions, but definitely an API change needs
> to pass through something visible so that affected code at some point
> has to be fixed if we want to get rid of a past mistake.


Hi, hard question because everyone has right :-) I vote for a new function
which return an array of header starting at the index 1.

Thierry

Re: Creating a health check in Lua?

[Thierry Fournier]

Hi,

I not have change between 1.7 and 2.0dev for the functions you use.

Thierry

> On 2 May 2019, at 09:48, Gil Bahat  wrote:
> 
> Hi,
> 
> last I checked up on this, it was almost 2 years ago and it was haproxy 1.7.7 
> . has this changed perhaps for recent haproxy versions? there seems to be 
> some rekindled interest in this.
> 
> Gil
> 
> On Wed, Jul 5, 2017 at 5:55 PM Thierry FOURNIER  
> wrote:
> On Wed, 5 Jul 2017 11:29:10 +0300
> Gil Bahat  wrote:
> 
> > Hi,
> > 
> > I have some lua code I would like to use as a health check. Documentation
> > seems to hint this is possible somehow, but I have not been able to either
> > direct an applet:send to the health check (nor is it clear how it could
> > expect something in return) or formulate the strings on demand via lua,
> > which was the other idea I was looking at.
> 
> 
> Hi, the integration of Lua in the health check are in my todo list.
> I'm very busy by my professional activity and sadly, this development
> is not in my priorities.
> 
> 
> > as a not unrelated aside, the 'use-service' directive is missing from the
> > configuration manual.
> 
> 
> Ok, thanks I will check this.
> 
> 
> Thierry
> 
> 
> > Regards,
> > 
> > Gil

Re: [PATCH] new contrib proposal / exec Python & Lua scripts

[Thierry Fournier]

Hi Willy,

Great ! I thinked this patch for the trash. You can merge this work. The 
main goal is providing an environment to process spoe with lua or python.


Thierry

Le 11 mai 2019 11:43:14 Willy Tarreau  a écrit :


Hi Thierry,

I just stumbled upon the patch series below you sent a while ago. I see
that you didn't receive any feedback on it, but see no reason not to
merge it, as it must still be valid given that it's outside of the
core. Do you have any objection against it getting merged ? Or maybe
even a newer version ? This could be a nice bootstrap for people who
want to try to create new agents.

Thanks,
Willy

On Sun, Feb 25, 2018 at 10:00:01PM +0100, Thierry Fournier wrote:

Hi,

Some guy says that SPOE is a great method for some things. Actually it
is not really accessible because it requires C development.

I write a server which can bind SPOP messages on Python and/Lua
functions. The function process the necessary and return variables
using SPOP ack.

The patches are in attachment. Below, an example of python script.

Thierry


   import spoa
   import ipaddress

   def check_client_ip(args):

pprint(args)
# This display:
# [{'name': '', 'value': True},
#  {'name': '', 'value': 1234L},
#  {'name': '', 'value': IPv4Address(u'127.0.0.1')},
#  {'name': '', 'value': IPv6Address(u'::55')},
#  {'name': '', 'value': '127.0.0.1:10001'}]

spoa.set_var_null("null", spoa.scope_txn)
spoa.set_var_boolean("boolean", spoa.scope_txn, True)
spoa.set_var_int32("int32", spoa.scope_txn, 1234)
spoa.set_var_uint32("uint32", spoa.scope_txn, 1234)
spoa.set_var_int64("int64", spoa.scope_txn, 1234)
spoa.set_var_uint64("uint64", spoa.scope_txn, 1234)
spoa.set_var_ipv4("ipv4", spoa.scope_txn, 
ipaddress.IPv4Address(u"127.0.0.1"))
spoa.set_var_ipv6("ipv6", spoa.scope_txn, 
ipaddress.IPv6Address(u"1::f"))
spoa.set_var_str("str", spoa.scope_txn, "1::f")
spoa.set_var_bin("bin", spoa.scope_txn, "1:\x01:\x02f\x00\x00")
# HAProxy display:
# [debug converter] type: any <>
# [debug converter] type: bool <1>
# [debug converter] type: sint <1234>
# [debug converter] type: sint <1234>
# [debug converter] type: sint <1234>
# [debug converter] type: sint <1234>
# [debug converter] type: ipv4 <127.0.0.1>
# [debug converter] type: ipv6 <1::f>
    # [debug converter] type: str <1::f>
# [debug converter] type: bin <1:.:.f>

return





>From 0794044c73b7361560ebeb205d733f978bcd78af Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER 
Date: Fri, 23 Feb 2018 11:40:03 +0100
Subject: [PATCH 01/14] MINOR: spoa-server: Clone the v1.7 spoa-example project

This is a working base.
---
 contrib/spoa_server/Makefile |   24 +
 contrib/spoa_server/README   |   88 
 contrib/spoa_server/spoa.c   | 1152 ++
 3 files changed, 1264 insertions(+)
 create mode 100644 contrib/spoa_server/Makefile
 create mode 100644 contrib/spoa_server/README
 create mode 100644 contrib/spoa_server/spoa.c

diff --git a/contrib/spoa_server/Makefile b/contrib/spoa_server/Makefile
new file mode 100644
index 000..e6b7c53
--- /dev/null
+++ b/contrib/spoa_server/Makefile
@@ -0,0 +1,24 @@
+DESTDIR =
+PREFIX  = /usr/local
+BINDIR  = $(PREFIX)/bin
+
+CC = gcc
+LD = $(CC)
+
+CFLAGS  = -g -O2 -Wall -Werror -pthread
+LDFLAGS = -lpthread
+
+OBJS = spoa.o
+
+
+spoa: $(OBJS)
+   $(LD) $(LDFLAGS) -o $@ $^
+
+install: spoa
+   install spoa $(DESTDIR)$(BINDIR)
+
+clean:
+   rm -f spoa $(OBJS)
+
+%.o:   %.c
+   $(CC) $(CFLAGS) -c -o $@ $<
diff --git a/contrib/spoa_server/README b/contrib/spoa_server/README
new file mode 100644
index 000..7e376ee
--- /dev/null
+++ b/contrib/spoa_server/README
@@ -0,0 +1,88 @@
+A Random IP reputation service acting as a Stream Processing Offload Agent
+--
+
+This is a very simple service that implement a "random" ip reputation
+service. It will return random scores for all checked IP addresses. It only
+shows you how to implement a ip reputation service or such kind of services
+using the SPOE.
+
+
+  Start the service
+-
+
+After you have compiled it, to start the service, you just need to use "spoa"
+binary:
+
+$> ./spoa  -h
+Usage: ./spoa [-h] [-d] [-p ] [-n ]
+-h  Print this message
+-d  Enable the debug mode
+-pSpecify the port to listen on (default: 12345)
+-n Specify the number of workers (default: 5)
+
+Note: A worker is a thread.
+
+
+  Configure a SPOE to use 

Re: error in haproxy 1.9 using txn.req:send in lua

[Thierry Fournier]

Hi,

You can use something like that:

   --> receive request from client
   --> frontend a
   --> use-service lua.xxx
   --> [forward data using core.tcp(127.0.0.1:)
   --> frontend (bind 127.0.0.1:)
   --> [send request to your server]
   --> [receive response]
   --> [read response in lua.xxx]
   --> [modify reponse in lua.xxx]
   --> [forward response ]
   --> frontend a
   --> send reponse to client

It is a little bit ugly, and it eat manny memory and performances,
but it works !

BR,
Thierry


> On 13 Feb 2019, at 11:18, Laurent Penot  wrote:
> 
> Hi Christopher,
> 
> I'm so sad
> It was really working well in my use case with 1.8 versions.
> Thank's a lot for your answer
> 
> Best
> Laurent
> 
> 
> 
> On 13/02/2019 10:56, "Christopher Faulet"  wrote:
> 
>Le 13/02/2019 à 09:34, Laurent Penot a écrit :
>> Hi Thierry, guys,
>> 
>> When receiving a POST request on haproxy, I use lua to compute some 
>> values, and modify the body of the request before forwarding to the 
>> backend, so my backend can get these variables from the POST and use them.
>> 
>> Here is a sample cfg, and lua code to reproduce this.
>> 
>> # Conf (I removed all defauts, timeout and co ..) :
>> 
>> frontend front-nodes
>> 
>> bind :80
>> 
>> # option to wait for the body before processing (mandatory for POST 
>> requests)
>> 
>> option http-buffer-request
>> 
>> # default backend
>> 
>> default_backend be_test
>> 
>> http-request lua.manageRequests
>> 
>> # Lua :
>> 
>> *function */manageRequests/(txn)
>> 
>> -- create new postdata
>> 
>> *local *newPostData = /core/./concat/()
>> 
>> newPostData:add('POST /test.php HTTP/1.1\r\n')
>> 
>> newPostData:add('Host: test1\r\n')
>> 
>> newPostData:add('content-type: application/x-www-form-urlencoded\r\n')
>> 
>> *local *newBodyStr = 'var1=valueA=valueB'
>> 
>> *local *newBodyLen = string.len(newBodyStr)
>> 
>> newPostData:add('content-length: ' .. tostring(newBodyLen) .. '\r\n')
>> 
>> newPostData:add('\r\n')
>> 
>> newPostData:add(newBodyStr)
>> 
>> *local *newPostDataStr = tostring(newPostData:dump())
>> 
>> txn.req:send(newPostDataStr)
>> 
>> *end*
>> 
>> /core/./register_action/("manageRequests", { "http-req" }, /manageRequests/)
>> 
>> This is working well in haproxy 1.8.x (x : 14 to 18) but I get the 
>> following error with 1.9.4 (same error with 1.9.2, others 1.9.x versions 
>> not tested) :
>> 
>> Lua function 'manageRequests': runtime error: 0 from [C] method 'send', 
>> /etc/haproxy/lua/bench.lua:97 C function line 80.
>> 
>> Line 97 of my lua file is txn.req:send(newPostDataStr)
>> 
>> Maybe I’m missing something on 1.9.x but cant find what, or maybe it’s a 
>> bug, I can’t say.
>> 
>Hi Laurent,
> 
>It is not supported to modify an HTTP request/response calling Channel 
>functions. It means calling following functions within an HTTP proxy is 
>forbidden: Channel.get, Channel.dup, Channel.getline, Channel.set, 
>Channel.append, Channel.send, Channel.forward.
> 
>Since HAProxy 1.9, a runtime error is triggered (because there is no way 
>to do it during the configuration parsing, AFAIK). You may see this as a 
>regression, but in fact, it was never really supported. But because of a 
>lack check, no error was triggered. Because these functions totally 
>hijacked the HTTP parser, if used, the result is undefined. There are 
>many ways to crash HAProxy. Unfortunately, for now, there is no way to 
>rewrite the HTTP messages in Lua.
> 
>-- 
>Christopher Faulet
> 
> 

Re: Seamless reloads: file descriptors utilization in LUA

[Thierry Fournier]

Hi,

First, using:

 - fd for file system access
 - fd for tcp/udp connections throught Lua Socket

are a bad ideas because these actions block HAProxy. While the fd/socket is 
waiting for data, HAProxy does nothing, and it not process any other 
connections.

With other words: If you tcp connection reach a timeout of 30s, HAProxy will be 
blocked during 30s, and all the incomming or established connections wille be 
block during 30s.

 - For filesystem access, you can use a ramdisk FS (like /dev/shm on most 
distro), or preload files.
 - For tcp connections, you can user core.tcp()
 - For udp connection, I don’t have workaround.

1)

When you reload haproxy, the old processes are destroyed, and news process are 
created, so the FD are closed with the process. If the main process own FD, 
childrens inherit these socket, but, the main process can’t open more sockets, 
so you can’t reach limits. It is strange.

Do you have simple conf+lua code which reproduce the problem ?

2)

As I wrote, using socket with HAProxy + Lua is not a good idea.
I think that it is a bad idea to increase limits for preventing something 
linked to usage of a dangerous method.
The mindset of HAProxy is performance and resources saving first !

BR,
Thierry




> On 16 Jan 2019, at 08:59, Wert  wrote:
> 
>> CC'ing Thierry: as this has come on this discourse, can we have your
>> opinion about the FD's in LUA and howto best handle ulimit?
> 
> 
>> Apologies for the duplicate mail.
> 
> 
>> Thanks,
>> Lukas
> 
> 1. FD
> I don't know your architecture too much. From user-side I just see no reasons 
> to keep FD that created in LUA.
> For cases when I make redis-connection, open GEO-DB file or some socket to 
> send, there is no reason to keep such FDs for new instance.
> It might be option or default policy to completely stop transferring LUA-FDs 
> to new master.
> 
> If it is difficult, probably there could be some ways to make ability for 
> checking and cleaning it manually with CLI.
> 
> 2. Ulimit
> It is impossible to know - how many FDs would be used by LUA even after 
> fixing infinite grow.
> I use "ulimit-n 1000". Of cause it looks like dirty thing but I can't 
> imagine the case when it can make real harm, while low limit can a lot.
> If there is some harm, at least you may adjust current auto-calculated limit 
> with "+100","*2" or some similar modifier and it will cover many real cases.
> 
> ---
> Wert
> 

[THANKS] Re: Deadlock lua when calling register_task inside action function

[Thierry Fournier]

great ! 

Willy, could you apply the attached patch ?

thanks
Thierry


0001-BUG-MEDIUM-dead-lock-when-Lua-tasks-are-trigerred.patch
Description: Binary data

> On 6 Jan 2019, at 19:11, Thierry Fournier  
> wrote:
> 
> Hi,
> 
> Thanks for the bug report.
> Your "uneducated guess" was right.
> 
> Could you test the patch in attachment ?
> 
> Thanks
> Thierry
> <0001-BUG-MEDIUM-dead-lock-when-Lua-tasks-are-trigerred.patch>
> 
>> On 2 Jan 2019, at 01:40, Flakebi  wrote:
>> 
>> Hi,
>> 
>> I am currently trying to send an http request for some incoming requests to 
>> a logging server.
>> The incoming connection should be handled independent of the logging, so it 
>> should be possible
>> that the incoming connection is already closed, while the logging is still 
>> in progress.
>> 
>> I use lua to create a new http action. This action then registers a task to 
>> decouple the logging.
>> And the rest happens inside this task.
>> 
>> As far as I understand the lua api documentation, it should be possible to 
>> call `register_task` [1]
>> inside an action. However, HAProxy deadlocks when a request comes in.
>> My uneducated guess would be that HAProxy hangs in a spinlock because it 
>> tries to create a lua context
>> while still executing lua.
>> 
>> I tested version 1.8.14 and 1.9.0 and they show the same behaviour.
>> The lua version is 5.3.5.
>> 
>> Am I missing something? Should this be possible at all?
>> 
>> gdb tells the following stacktrace. HAProxy never leaves the hlua_ctx_init 
>> function:
>> 
>> #0  0x08eb8d04fd0f in hlua_ctx_init ()
>> #1  0x08eb8d0533f0 in ?? ()
>> #2  0x6de619c40d27 in ?? () from /usr/lib/liblua.so.5.3
>> #3  0x6de619c4db85 in ?? () from /usr/lib/liblua.so.5.3
>> #4  0x6de619c403b3 in ?? () from /usr/lib/liblua.so.5.3
>> #5  0x6de619c410a9 in lua_resume () from /usr/lib/liblua.so.5.3
>> #6  0x08eb8d04cf9a in ?? ()
>> #7  0x08eb8d05197e in ?? ()
>> #8  0x08eb8d05d037 in http_req_get_intercept_rule ()
>> #9  0x08eb8d063f96 in http_process_req_common ()
>> #10 0x08eb8d08f0f1 in process_stream ()
>> #11 0x08eb8d156b08 in process_runnable_tasks ()
>> #12 0x08eb8d0d443b in ?? ()
>> #13 0x08eb8d02b660 in main ()
>> 
>> 
>> haproxy.cfg:
>> global
>>   lua-load mytest.lua
>> 
>> listen server
>>   modehttp
>>   bind :7999
>>   http-request lua.test
>>   server s 127.0.0.1:8081
>> 
>> 
>> mytest.lua:
>> function test(txn)
>>   -- This works when commenting out the task creation
>>   core.register_task(function()
>>   core.Warning("Aha")
>>   end)
>> end
>> 
>> core.register_action("test", { "http-req" }, test, 0)
>> 
>> 
>> 
>> Cheers,
>> Flakebi
>> 
>> [1] 
>> https://www.arpalert.org/src/haproxy-lua-api/1.8/index.html#core.register_task
> 

[PATCH] Re: Question re lua and tcp content

[Thierry Fournier]

Hi,

Thanks fir the bug report. It is fixed by attached patch.

Willy, could you merge this patch ?

Thanks,
Thierry


0001-BUG-MINOR-bad-args-are-returned-for-Lua-actions.patch
Description: Binary data



> On 31 Dec 2018, at 18:53, FRANKS, Andy (SHREWSBURY AND TELFORD HOSPITAL NHS 
> TRUST)  wrote:
> 
> Hi all,
>   I’m an amateur with lua. I’ve been messing around with doing “something” 
> when either an http or tcp request happens as a way of teaching myself how to 
> do stuff.
> Many examples use the “hello world” lua script, and call it via either :
>  
> http-request lua.script
>  
> or 
>  
> tcp-request content lua.script
>  
> Which works fine; the script is called. My question though is around 
> arguments .. I understand the txn  gets passed over, so one can use a 
> script like this for example:
>  
> Function my_script(txn)
>   ip=txn.f:src()
>   core.Alert(ip)
> end
>  
> core.register_action(“my-script”, {“tcp-req”, “http-req”}, my_script)
>  
> .. and it works, but if we pass arguments, e.g.
>  
> http-request lua.script stuff
> or
> tcp-request content lua.script stuff
>  
> and in the script use 
>  
> Function my_script(txn, arg)
>   core.Alert(arg)
> end
> core.register_action(“my-script”, {“tcp-req”, “http-req”}, my_script, 1)
>  
> .. http works as expected, and sends the string “stuff”, but tcp just sends 
> the word “content”, not “stuff”
>  
> Am I missing something really obvious here!?
>  
> Thanks
> Andy
>  
>  
> 
> 
> 
> 
> This message may contain confidential information. If you are not the 
> intended recipient please inform the
> sender that you have received the message in error before deleting it.
> Please do not disclose, copy or distribute information in this e-mail or take 
> any action in relation to its contents. To do so is strictly prohibited and 
> may be unlawful. Thank you for your co-operation.
> 
> NHSmail is the secure email and directory service available for all NHS staff 
> in England and Scotland. NHSmail is approved for exchanging patient data and 
> other sensitive information with NHSmail and other accredited email services.
> 
> For more information and to find out how you can switch, 
> https://portal.nhs.net/help/joiningnhsmail

Re: Deadlock lua when calling register_task inside action function

[Thierry Fournier]

Hi,

Thanks for the bug report.
Your "uneducated guess" was right.

Could you test the patch in attachment ?

Thanks
Thierry


0001-BUG-MEDIUM-dead-lock-when-Lua-tasks-are-trigerred.patch
Description: Binary data


> On 2 Jan 2019, at 01:40, Flakebi  wrote:
> 
> Hi,
> 
> I am currently trying to send an http request for some incoming requests to a 
> logging server.
> The incoming connection should be handled independent of the logging, so it 
> should be possible
> that the incoming connection is already closed, while the logging is still in 
> progress.
> 
> I use lua to create a new http action. This action then registers a task to 
> decouple the logging.
> And the rest happens inside this task.
> 
> As far as I understand the lua api documentation, it should be possible to 
> call `register_task` [1]
> inside an action. However, HAProxy deadlocks when a request comes in.
> My uneducated guess would be that HAProxy hangs in a spinlock because it 
> tries to create a lua context
> while still executing lua.
> 
> I tested version 1.8.14 and 1.9.0 and they show the same behaviour.
> The lua version is 5.3.5.
> 
> Am I missing something? Should this be possible at all?
> 
> gdb tells the following stacktrace. HAProxy never leaves the hlua_ctx_init 
> function:
> 
> #0  0x08eb8d04fd0f in hlua_ctx_init ()
> #1  0x08eb8d0533f0 in ?? ()
> #2  0x6de619c40d27 in ?? () from /usr/lib/liblua.so.5.3
> #3  0x6de619c4db85 in ?? () from /usr/lib/liblua.so.5.3
> #4  0x6de619c403b3 in ?? () from /usr/lib/liblua.so.5.3
> #5  0x6de619c410a9 in lua_resume () from /usr/lib/liblua.so.5.3
> #6  0x08eb8d04cf9a in ?? ()
> #7  0x08eb8d05197e in ?? ()
> #8  0x08eb8d05d037 in http_req_get_intercept_rule ()
> #9  0x08eb8d063f96 in http_process_req_common ()
> #10 0x08eb8d08f0f1 in process_stream ()
> #11 0x08eb8d156b08 in process_runnable_tasks ()
> #12 0x08eb8d0d443b in ?? ()
> #13 0x08eb8d02b660 in main ()
> 
> 
> haproxy.cfg:
> global
>lua-load mytest.lua
> 
> listen server
>modehttp
>bind :7999
>http-request lua.test
>server s 127.0.0.1:8081
> 
> 
> mytest.lua:
> function test(txn)
>-- This works when commenting out the task creation
>core.register_task(function()
>core.Warning("Aha")
>end)
> end
> 
> core.register_action("test", { "http-req" }, test, 0)
> 
> 
> 
> Cheers,
> Flakebi
> 
> [1] 
> https://www.arpalert.org/src/haproxy-lua-api/1.8/index.html#core.register_task

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Thierry Fournier]

I Adis,

Sorry for the delay, I processed a quick review, and all seems to be ok for me!

BR,
Thierry


> On 27 Sep 2018, at 14:02, Adis Nezirovic  wrote:
> 
> On Mon, Sep 03, 2018 at 12:09:47PM +0200, Adis Nezirovic wrote:
>> Hi Thierry,
>> 
>> Have you had the time to review my patches?
> 
> Thierry,
> 
> Reviving this thread, do you have any objections about latest version of
> the patch (Lua stick table patch)
> 
> Best regards,
> Adis

Re: OpenSSL and per-context option problem

[Thierry Fournier]

—
Thierry Fournier
Web Performance & Security Expert
m: +33 6 68 69 21 85  | e: thierry.fourn...@ozon.io
w: http://www.ozon.io/| b: http://blog.ozon.io/

> On 17 Sep 2018, at 12:45, Emmanuel Hocdet  wrote:
> 
> 
> Hi Thierry,
> 
>> Le 15 sept. 2018 à 18:06, Thierry Fournier > <mailto:thierry.fourn...@ozon.io>> a écrit :
>> 
>> Hi,
>> 
>> I tried to use per-context options, in order to enable HTTP2 for a short
>> list of SNI. I just add lines like this:
>> 
>>   /certif1.pem [alpn h2,http/1.1] my-h2-host.com <http://my-h2-host.com/>
>>   /certif2.pem my-other-host.com <http://my-other-host.com/>
>> 
>> This configuration works fine on debian 8 with OpenSSL 1.0.2g, and doesn’t
>> work on Ubuntu 16.04 with OpenSSL 1.0.2l.
>> 
>> I compile the OpenSSL debian package 1.0.2g on Ubuntu, and the feature is
>> enabled.
>> 
>> My conclusion, is that some version of OpenSSL doesn’t support all 
>> per-context
>> options. 
>> 
>> Do you have an opinion ?
>> 
> 
> Are you sure it's not the opposite: doesn't work with 1.0.2g?
> 
> "Major changes between OpenSSL 1.0.2g and OpenSSL 1.0.2h [3 May 2016]
> 
> Modify behavior of ALPN to invoke callback after SNI/servername callback, 
> such that updates to the SSL_CTX affect ALPN. »
> 
Sorry, I mix versions. The following is the reality:

OpenSSL 1.0.2l  25 May 2017 => works
OpenSSL 1.0.2g  1 Mar 2016  => doesn’t work.

The change explain the observed behavior !

br,
Thierry

OpenSSL and per-context option problem

[Thierry Fournier]

Hi,

I tried to use per-context options, in order to enable HTTP2 for a short
list of SNI. I just add lines like this:

   /certif1.pem [alpn h2,http/1.1] my-h2-host.com
   /certif2.pem my-other-host.com

This configuration works fine on debian 8 with OpenSSL 1.0.2g, and doesn’t
work on Ubuntu 16.04 with OpenSSL 1.0.2l.

I compile the OpenSSL debian package 1.0.2g on Ubuntu, and the feature is
enabled.

My conclusion, is that some version of OpenSSL doesn’t support all per-context
options. 

Do you have an opinion ?

If my conclusion was right, maybe a warning during the SSL configuration
parser execution would be a good idea ?

BR,
Thierry

Re: [PATCH] BUG/MAJOR: thread: lua: Wrong SSL context initialization.

[Thierry Fournier]

Hi Pieter,

Your patch makes sense !
Good catch.
Willy, could you apply ?

Thierry

> On 29 Aug 2018, at 21:29, PiBa-NL  wrote:
> 
> Op 29-8-2018 om 14:29 schreef Olivier Houchard:
>> On Wed, Aug 29, 2018 at 02:11:45PM +0200, Frederic Lecaille wrote:
>>> This patch is in relation with one bug reproduced by the reg testing file
>>> sent by Pieter in this thread:
>>> https://www.mail-archive.com/haproxy@formilux.org/msg31079.html
>>> 
>>> Must be checked by Thierry.
>>> Must be backported to 1.8.
>>> 
>>> Note that Pieter reg testing files reg-tests/lua/b2.* come with this
>>> patch.
>>> 
>>> 
>>> Fred.
>>> From d6d38a354a89b55f91bb9962c5832a089d960b60 Mon Sep 17 00:00:00 2001
>>> From: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20L=C3=A9caille?= 
>>> Date: Wed, 29 Aug 2018 13:46:24 +0200
>>> Subject: [PATCH] BUG/MAJOR: thread: lua: Wrong SSL context initialization.
>>> 
>>> When calling ->prepare_srv() callback for SSL server which
>>> depends on global "nbthread" value, this latter was not already parsed,
>>> so equal to 1 default value. This lead to bad memory accesses.
>>> 
>>> Thank you to Pieter (PiBa-NL) for having reported this issue and
>>> for having provided a very helpful reg testing file to reproduce
>>> this issue (reg-test/lua/b2.*).
>>> 
>> That sounds good, nice catch !
>> 
>> And yes thanks Pieter, as usual :)
>> 
>> Olivier
> 
> As you've probably already verified, the issue is indeed fixed with this 
> patch applied on top of master.
> 
> Thanks Frederic & Olivier.
> 
> @Thierry, can you give the 'all okay' ? (or not okay, if it needs a different 
> fix..)
> 
> Regards,
> PiBa-NL (Pieter)
> 

Re: [PATCH] MEDIUM: reset lua transaction between http requests

[Thierry Fournier]

> On 24 Aug 2018, at 04:19, Willy Tarreau  wrote:
> 
> Hi Thierry,
> 
> On Thu, Aug 23, 2018 at 09:37:43AM +0200, Thierry Fournier wrote:
>> Hi,
>> 
>> Your patch make sense, that's the right appoach, but I have a doubt about
>> the place to use for doing the reinitialization. 
>> 
>> I add Willy in this thread in order to have http2 advisor.
>> 
>> Before the 1.8 the Lua context was reinitialized with the stream because
>> the stream was reinitialized between each http request in a keepalive
>> session.
>> 
>> With http2 I guess that this behavior change. So, Willy, do you have
>> an opinion on the place to use to perform the Lua reinit ?
> 
> Oh with H2 it's even simpler, streams are distinct from each other
> so we don't reuse them and the issue doesn't exist :-)
> 
> Does this mean I should take Patrick's patch ?


Technically, the patch works, but  the reset of the Lua session was done
in the stream. This reset is done in the proto_http.c, so I’m not sure
that’s the correct place for this reset.

If the proto_http is the only one place which known the stop of one
request and the start of another, there is the right place, otherwise ... ?

I don’t known.

A+
Thierry

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Thierry Fournier]

Hi

[...]

>> I miss also the relation between oprators and between the content
>> of operators. I mean AND or OR. How I understand your example:
>> 
>> +local filter = {
>> +  lt={{"gpc0", 1}, {"gpc1", 2}},
>> +  gt={{"conn_rate", 3}},
>> +  eq={{"conn_cur", 4}}
>> +}
>> 
>> Are you sure that the syntax =
>> is a good format ? Maybe something like the following, with the operator
>> as argument between the two operands. lines are implicitly OR, and columns
>> are AND:
>> 
>> +local filter = {
>> +  {{"gpc0", "lt", 1}, {"gpc1", "lt", 2}},
>> +  {{"conn_rate", "gt", 3}},
>> +  {{"conn_cur", "eq", 4}}
>> +}
> Actually, I was playing with some other ideas, and it was useful to be
> able to "preselect" filter operators.
> However, the CLI doesn't even support more than one, maybe we don't need
> to complicate too much. Maybe we can simplify to this:
> 
>  local filter = {
>{"gpc0", "lt", 1},
>{"gpc1", "lt", 2},
>{"conn_rate", "gt", 3},
>{"conn_cur", "eq", 4}
>  }
> 
> The default operator would be AND, and we would not support other
> operators (to keep the things simple). e.g. example use case for the
> filter would be to filter out on gpc0 > X AND gpc1 > Y
> 
> If this sounds good, I can update/simplify the code.


Ok, it sounds good. I think this kind of syntax is easily understandable
and it allow a good way for filtering values.


>> Idea of extension for the future: Maybe it will be safe to compile
>> sticktable filter during the initialisation of the Lua code, to avoid
>> runtime errors ?
> I'm returning runtime errors since it can be easy to mix up data from
> the client side (most probably data would come as json table, then
> transformed to Lua table)


ok

[...]


>> Line 274 of your patch, I don't see any HA_SPIN_LOCK(STK_TABLE_LOCK
>> I don't known very well the thread, so maybe there are useles, maybe no.
> hlua_stktable_lookup() uses stktable_lookup_key() which does have locks,
> so I guess that it should be fine then?


sure !


[...]


>> l.365, 369: The user doesn't have context about the error. there are the
>> first entry of the table, the second ? Which operator doesn't exists ?
>> 
>> L.380, 384: Which line is wrong ?
> Yes, it is somwehat cryptic. I've tried to avoid returning user supplied
> data in the error messages. We can revisit this if/when we change the
> filter table format.


ok


>> L.431: Your release the lock, so the next element relative to the current
>> "n", can disappear and the ebmb_next() can return wrong memory.
> I was under impression that we only have to acquire lock and increment
> ref_cnt (so we can be sure our current node n is not deleted)
> ebmb_next() is called only when we're holding lock, first and every
> other iteration, i.e.
> 
>  HA_SPIN_LOCK(STK_TABLE_LOCK, >lock);
>  eb = ebmb_first(>keys);
>  for (n = eb; n; n = ebmb_next(n)) {
>  ...
>  ts->ref_cnt++;
>  HA_SPIN_UNLOCK(STK_TABLE_LOCK, >lock);
>  ...
> 
>  HA_SPIN_LOCK(STK_TABLE_LOCK, >lock);
>  }
> 
> Or I didn't get your point?


ok, you probably right. 


Thierry

Re: BUG: Tw is negative with lua sleep

[Thierry Fournier]

> On 22 Aug 2018, at 06:00, Patrick Hemmer  wrote:
> 
> 
> 
> On 2018/7/18 09:03, Frederic Lecaille wrote:
>> Hello Patrick, 
>> 
>> On 07/17/2018 03:59 PM, Patrick Hemmer wrote: 
>>> Ping? 
>>> 
>>> -Patrick 
>>> 
>>> On 2018/6/22 15:10, Patrick Hemmer wrote: 
 When using core.msleep in lua, the %Tw metric is a negative value. 
 
 For example with the following config: 
 haproxy.cfg: 
 global 
 lua-load /tmp/haproxy.lua 
 
 frontend f1 
 mode http 
 bind :8000 
 default_backend b1 
 log 127.0.0.1:1234 daemon 
 log-format Ta=%Ta\ Tc=%Tc\ Td=%Td\ Th=%Th\ Ti=%Ti\ Tq=%Tq\ 
 TR=%TR\ Tr=%Tr\ Tt=%Tt\ Tw=%Tw 
 
 backend b1 
 mode http 
 http-request use-service lua.foo 
 
 haproxy.lua: 
 core.register_service("foo", "http", function(applet) 
 core.msleep(100) 
 applet:set_status(200) 
 applet:start_response() 
 end) 
 
 The log contains: 
 Ta=104 Tc=0 Td=0 Th=0 Ti=0 Tq=104 TR=104 Tr=104 Tt=104 Tw=-104 
 
 ^ TR also looks wrong, as it did not take 104ms to receive the full 
 request. 
 
 This is built from the commit before current master: d8fd2af 
 
 -Patrick 
>>> 
>> 
>> The patch attached to this mail fixes this issue at least for %TR field. 
>> 
>> But I am not sure at all it is correct or if there is no remaining issues. 
>> For instance the LUA tcp callback also updates the tv_request log field. 
>> 
>> So, let's wait for Thierry's validation.


Hi,

Applets should be considered as server independent from HAProxy, so applet 
should not
change HAProxy information like log times.

I guess that your patch works, and the function hlua_applet_tcp_fct() should 
follow
the same way.

unfortunately I do not have free time to test all of this changes.

Thierry


>> Regards. 
>> 
> 
> Any update on this?
> 
> -Patrick
> 

Re: [PATCH] MEDIUM: reset lua transaction between http requests

[Thierry Fournier]

Hi,

Your patch make sense, that’s the right appoach, but I have a doubt about
the place to use for doing the reinitialization. 

I add Willy in this thread in order to have http2 advisor.

Before the 1.8 the Lua context was reinitialized with the stream because
the stream was reinitialized between each http request in a keepalive
session.

With http2 I guess that this behavior change. So, Willy, do you have
an opinion on the place to use to perform the Lua reinit ?

Thierry


> On 22 Aug 2018, at 16:09, Patrick Hemmer  wrote:
> 
> Not sure if this is the right approach, but this addresses the issue for me.
> This should be backported to 1.8.
> 
> -Patrick
> <0001-MEDIUM-reset-lua-transaction-between-http-requests.patch>

Re: BUG: LUA txn:get_priv() scoped to connection, not transaction

[Thierry Fournier]

Hi Patrick,

Could you retry adding the keyword “local” before data. Unfortunately, by 
default, Lua variables are global.


> core.register_action("test", { "http-req" }, function(txn)
> local data = txn:get_priv()
> if not data then
> data = 0
> end
> data = data + 1
> print(string.format("set to %d", data))
> txn:set_priv(data)
> end)


BR,
Thierry


> On 22 Aug 2018, at 05:57, Patrick Hemmer  wrote:
> 
> There is a bug in the current stable haproxy (1.8.13) where the LUA function 
> txn:get_priv() is returning data stored from other transactions. This was 
> discovered as we have code that triggers on certain requests, and it was 
> triggering on requests it should not have been.
> 
> You can reproduce with this config:
> global
> lua-load haproxy.lua
> 
> defaults
> mode http
> 
> frontend f1
> bind :8000
> default_backend b1
> http-request lua.test
> 
> backend b1
> http-request use-service lua.fakeserv
> 
> And this lua file:
> core.register_action("test", { "http-req" }, function(txn)
> data = txn:get_priv()
> if not data then
> data = 0
> end
> data = data + 1
> print(string.format("set to %d", data))
> txn:set_priv(data)
> end)
> 
> core.register_service("fakeserv", "http", function(applet)
> applet:set_status(200)
> applet:start_response()
> end)
> 
> And this curl command:
> curl http://localhost:8000  http://localhost:8000 
> 
> 
> Which provides this output:
> set to 1
> set to 2
> 
> 
> 
> Version information:
> HA-Proxy version 1.8.13 2018/07/30
> Copyright 2000-2018 Willy Tarreau  
> 
> 
> Build options :
> TARGET  = osx
> CPU = generic
> CC  = gcc
> CFLAGS  = -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement 
> -fwrapv -fno-strict-overflow -Wno-address-of-packed-member 
> -Wno-null-dereference -Wno-unused-label
> OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
> 
> Default settings :
> maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 
> 200
> 
> Built with OpenSSL version : OpenSSL 1.1.0h  27 Mar 2018
> Running on OpenSSL version : OpenSSL 1.1.0h  27 Mar 2018
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
> Built with Lua version : Lua 5.3.4
> Built with transparent proxy support using:
> Encrypted password support via crypt(3): yes
> Built with PCRE version : 8.42 2018-03-20
> Running on PCRE version : 8.42 2018-03-20
> PCRE library supports JIT : no (USE_PCRE_JIT not set)
> Built with zlib version : 1.2.11
> Running on zlib version : 1.2.11
> Compression algorithms supported : identity("identity"), 
> deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
> Built with network namespace support.
> 
> Available polling systems :
>  kqueue : pref=300,  test result OK
>  poll : pref=200,  test result OK
>  select : pref=150,  test result OK
> Total: 3 (3 usable), will use kqueue.
> 
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [TRACE] trace
> 
> 
> -Patrick

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Thierry Fournier]

Hi Adis,

Thanks for this patch, it is a very useful class.

Some remark about the documentation and the formats:


js:function:: StickTable.info()
js:function:: StickTable.lookup(key)

Maybe the specification and an example of the returnes values
will be welcome, because I guess that the table keys are hardcoded.
Something like :

   {
  type =  -- list of given string,
  length = ,
  ...
   }

or

   {
  expire =  -- Specifiy the unit (ms or ?)
   }

js:function:: StickTable.dump([filter])

The exact list of allowed operators will helps the user to use
your class. It seems that string or regexes operators are not
allowed, only the integer operator are taken in account. This
list is "eq", "ne", "le", "lt", "ge", "gt".

Same remarqk for allowed data type. Maybe a link to the HAProxy
documentation will be sufficient for the data type.

I see in the code that the filters can exceed 4 entries. This
limitation must be written in the doc.

I miss also the relation between oprators and between the content
of operators. I mean AND or OR. How I understand your example:

+local filter = {
+  lt={{"gpc0", 1}, {"gpc1", 2}},
+  gt={{"conn_rate", 3}},
+  eq={{"conn_cur", 4}}
+}

( gpc0 >= 1 AND gpc1 >= 2 ) OR (conn_rate > 3) OR (conn_cur = 4)
   or
( gpc0 >= 1 OR gpc1 >= 2 ) OR (conn_rate > 3) OR (conn_cur = 4)
   or 
( gpc0 >= 1 AND gpc1 >= 2 ) AND (conn_rate > 3) AND (conn_cur = 4)
   or
( gpc0 >= 1 OR gpc1 >= 2 ) AND (conn_rate > 3) AND (conn_cur = 4)

Are you sure that the syntax =
is a good format ? Maybe something like the following, with the operator
as argument between the two operands. lines are implicitly OR, and columns
are AND:

+local filter = {
+  {{"gpc0", "lt", 1}, {"gpc1", "lt", 2}},
+  {{"conn_rate", "gt", 3}},
+  {{"conn_cur", "eq", 4}}
+}

It is read:

   ( gpc0 >= 1 OR gpc1 >= 2 ) AND (conn_rate > 3) AND (conn_cur = 4)

You can also implement a parser for natural language, in order to understand
directly the previous string ? This algoritm is very simple to implement:

   https://en.wikipedia.org/wiki/Shunting-yard_algorithm

Idea of extension for the future: Maybe it will be safe to compile
sticktable filter during the initialisation of the Lua code, to avoid
runtime errors ?


Other point with the doc of this function, you must specify that the
execution of this function can be very long for millions of entry, even
if a filter is specified because the stick table is entirely scanned.


some remarks about the code and the logs:
-

The line 182 of the patch contains space in place of tab.

Line 274 of your patch, I don't see any HA_SPIN_LOCK(STK_TABLE_LOCK
I don't known very well the thread, so maybe there are useles, maybe no.

Line 311: I see the decrement of the refcount without ATOMIC function
and whitout lock. Once again, I don't known very well the thread but
I send a warning. Maybe locks are useles, maybe no.

Line 286 of your patch. It seems that smp.flags is used uninitialized.
Maybe you should apply this change:

   -smp.flags |= SMP_F_CONST;
   +smp.flags = SMP_F_CONST;

l.365, 369: The user doesn't have context about the error. there are the
first entry of the table, the second ? Which operator doesn't exists ?

L.380, 384: Which line is wrong ?

L.431: Your release the lock, so the next element relative to the current
"n", can disappear and the ebmb_next() can return wrong memory.

That's all.
Thierry





> On 20 Aug 2018, at 15:15, Adis Nezirovic  wrote:
> 
> On Mon, Aug 20, 2018 at 02:11:13PM +0200, Adis Nezirovic wrote:
>> Hi guys,
>> 
>> I've attached a patch to add stick table support to Lua. Operations are
>> mostly similar to "show table" functionality from admin socket, namely:
>> 
>> - Show basic table info
>> - Key lookup
>> - Table dump, optionally using data/column filter
>> 
>> One side note, the code provides support for multiple filters
>> (4 by default) while CLI doesn't support that, nor it complains about
>> multiple "  " clauses.
>> 
>> Also, if this patch is accepted, maybe we can use provided helper
>> functions in other places in the code.
>> 
> 
> It's always funny to reply to self, right sending email to public I've
> spotted a bug. New patch attached.
> 
> SMT_T_SINT should be treated as ordinary signed integer, and shoud use
> lua_pushinteger() on it?
> 
> On many places in the code it is noted as "64" bit integer, but in
> stick_table.c it is defined as 32bit integer:
> 
> struct stktable_type stktable_types[SMP_TYPES] = {
>   [SMP_T_SINT] = { "integer", 0, 4 },
> 
> 
> Best regards,
> Adis
> <0001-MEDIUM-lua-Add-stick-table-support-for-Lua-read-only.patch>

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Thierry Fournier]

> On 20 Aug 2018, at 15:15, Adis Nezirovic  wrote:
> 
> On Mon, Aug 20, 2018 at 02:11:13PM +0200, Adis Nezirovic wrote:
>> Hi guys,
>> 
>> I've attached a patch to add stick table support to Lua. Operations are
>> mostly similar to "show table" functionality from admin socket, namely:
>> 
>> - Show basic table info
>> - Key lookup
>> - Table dump, optionally using data/column filter
>> 
>> One side note, the code provides support for multiple filters
>> (4 by default) while CLI doesn't support that, nor it complains about
>> multiple "  " clauses.


Hi Adis,

This is a great feature. I look this ASAP.


>> Also, if this patch is accepted, maybe we can use provided helper
>> functions in other places in the code.
>> 
> 
> It's always funny to reply to self, right sending email to public I've
> spotted a bug. New patch attached.
> 
> SMT_T_SINT should be treated as ordinary signed integer, and shoud use
> lua_pushinteger() on it?


There are a function which convert haproxy type in Lua types: hlua_arg2lua(),
and SINT is converted with lua_pushinteger(). I guess that stick tables SINT
are the same.


> On many places in the code it is noted as "64" bit integer, but in
> stick_table.c it is defined as 32bit integer:
> 
> struct stktable_type stktable_types[SMP_TYPES] = {
>   [SMP_T_SINT] = { "integer", 0, 4 },


The lua_pushinteger function takes a lua_Integer which is defined as known C
type (int, long or long long). So, when the function lua_pushinteger() is
called, the compilator perform a conversion between the type passed as argument
and the expected type. So I’m confident with the good behavior.

br,
Thierry


> Best regards,
> Adis
> <0001-MEDIUM-lua-Add-stick-table-support-for-Lua-read-only.patch>

Re: Performance of using lua calls for map manipulation on every request

[Thierry Fournier]

Hi,

The Lua overhead is very low. On my laptop I reach easyly 18 000 HTTP
requests by seconds with basic Lua processing. I guess that your code
will not have significant impact on perfs.

Note that the function:

> txn.http:req_get_headers()["host"][0]

Is consume more CPU than 

   txn.f[‘req.fhdr’](‘host’)

or

   txn.sf[‘req.fhdr’](‘host’)

Other point: I’m not sure that the split() function exists.

Thierry


> On 27 Jul 2018, at 14:38, Sachin Shetty  wrote:
> 
> Hi,
> 
> We are doing about 10K requests/minute on a single haproxy server, we have 
> enough CPUs and memory. Right now each requests looks up a map for backend 
> info. It works well. 
> 
> Now we  need to build some expire logic around the map. Like ignore some 
> entries in the map entries after some time. I could do this in lua, but it 
> woud mean that every request would make a lua call to look up a map value and 
> make a decision.
> 
> My lua method looks like this:
> 
> function get_proxy_from_map(txn)
> local host = txn.http:req_get_headers()["host"][0]
> local value = proxy_map_v2:lookup(host)
> if value then
> local values = split(value, ",")
> local proxy = values[1]
> local time = values[2]
> if os.time() > tonumber(time) then
> core.Alert("Expired: returning nil: " .. host)
> return
> else
> return proxy
> end
> end
> return
> end
> 
> 
> Any suggestions on how this would impact performance, our tests looks ok. 
> 
> Thanks
> Sachin

Re: SSL: double free on reload

[Thierry Fournier]

On Tue, 17 Jul 2018 10:10:58 +0200
Willy Tarreau  wrote:

> Hi again Nenad,
> 
> On Tue, Jul 17, 2018 at 05:18:45AM +0200, Willy Tarreau wrote:
> > Hi Nenad,
> > 
> > On Tue, Jul 17, 2018 at 03:37:37AM +0200, Nenad Merdanovic wrote:
> > > Ugh, this was a long time ago. [FROM MEMORY] The element should not be
> > > duplicated as far as I can remember. The references are stored in an 
> > > ebtree
> > > in order to prevent duplication and to provide consistent view when 
> > > updated
> > > dynamically.
> > 
> > OK. Then maybe the elements are freed after scanning the ebtree as well,
> > and we're meeting the node again after it was freed. I'll run a test
> > with the memory debugging options to see if it crashes on first dereference.
> 
> OK I found it, the same keys_ref is indeed assigned to multiple bind_confs :
> 
> static int bind_parse_tls_ticket_keys(char **args, int cur_arg, struct proxy 
> *px, struct bind_conf *conf, char **err)
> {
> ...
>   keys_ref = tlskeys_ref_lookup(args[cur_arg + 1]);
>   if(keys_ref) {
>   conf->keys_ref = keys_ref;
>   return 0;
>   }
> 
> So we clearly end up with a normal double-free. I'm checking what's the best
> solution to fix this now, as we don't have flags nor any such thing in the
> bind_conf to indicate that it must or must not be freed. We can't duplicate
> the allocation either because it's used for the updates. I think the cleanest
> solution will be to add a refcount to the tls_keys_ref entry.
> 
> Then I'm proposing the attached patch which works for me and is obvious
> enough to make valgrind happy as well.
> 
> Could you guys please give it a try to confirm ? I'll then merge it.


i, the patch works for me with backport to the 1.8 version. It is on
productio stage.

Thanks,
Thierry



> Thanks,
> Willy


-- 
Thierry Fournier
Web Performance & Security Expert
m: +33 6 68 69 21 85  | e: thierry.fourn...@ozon.io
w: http://www.ozon.io/| b: http://blog.ozon.io/

Re: Using LUA to redirect a connection based-upon initial content and to redirect upon target disconnection

[thierry . fournier]

Hi,

On Sun, 15 Jul 2018 17:14:01 +0100
Alistair Lowe  wrote:

> Hi folks,
> 
> I'm looking to use LUA in HAProxy to enable two use cases:
> 
>1. Redirect a connection based upon inspecting initial traffic for a
>server name.
>2. Redirect a connection when the target server closes its connection.
> 
> My understanding is that both of these scenarios should be possible if I
> inspect a connection for its duration, however I'm also wishing to build
> something high-performance and wish for connections to remain primarily in
> TCP pass-through mode, most of the time.
> 
> My questions are:
> 
>1. What's the most CPU friendly way to inspect a connection for a
>time-limited period (i.e. what configuration and commands in LUA should I
>be looking at etc).


I'm not sure to understand your question. You want to analyse received data
before taking a decision ? the Lua receive function doesn't consume CPU while
is waiting for data. Once data are available the function receive() returns
and ou can process these data.


>2. Is there anyway to trigger a LUA action upon TCP active close from
>the target?


If you're using http, once the first bytes are send to the client, you can't
send you own response. So the only one case is when th server close without
sending anything.

Maybe you can use a "tcp-response content" Lua function and call dup() function
(https://www.arpalert.org/src/haproxy-lua-api/1.8/index.html#Channel.dup). If
you receive data, it's ok, if the channel is close you receive 'nil' reponse,
and you can send your redirect.

I'm not sure that the dup() function wait for data, and I'm not sure that
HAPRoxy call the Lua function if the client close the connection.

Thierry

Re: SSL: double free on reload

[Thierry Fournier]

On Mon, 16 Jul 2018 08:00:48 +0200
Willy Tarreau  wrote:

> Hi Thierry,
> 
> On Fri, Jul 06, 2018 at 04:28:22PM +0200, Thierry Fournier wrote:
> > Hi list,
> > 
> > I caught a double-free whien I reload haproxy-1.8:
> > 
> > writev(2, [{"*** Error in `", 14}, {"/opt/o3-haproxy/sbin/haproxy", 
> > 28}, {"': ", 3}, {"double free or corruption (!prev)", 33}, {": 0x", 4}, 
> > {"1cec2ab0", 16}, {" ***\n", 5}], 7) = 103
> > 
> > Decoded:
> > 
> > *** Error in `/opt/o3-haproxy/sbin/haproxy': double free or corruption 
> > (!prev): 0x1cec2ab0 ***
> > 
> > Gdb says:
> > 
> >#0  0x7f4bac88b067 in __GI_raise (sig=sig@entry=6) at 
> > ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> >#1  0x7f4bac88c448 in __GI_abort () at abort.c:89
> >#2  0x7f4bac8c91b4 in __libc_message (do_abort=do_abort@entry=1, 
> >fmt=fmt@entry=0x7f4bac9be210 "*** Error in `%s': %s: 0x%s ***\n")
> >at ../sysdeps/posix/libc_fatal.c:175
> >#3  0x7f4bac8ce98e in malloc_printerr (action=1, 
> >str=0x7f4bac9be318 "double free or corruption (!prev)", 
> > ptr=) at malloc.c:4996
> >#4  0x7f4bac8cf696 in _int_free (av=, p= > out>, have_lock=0) at malloc.c:3840
> >#5  0x0042af56 in ssl_sock_destroy_bind_conf 
> > (bind_conf=0x1d27e810) at src/ssl_sock.c:4819
> >#6  0x004b1390 in deinit () at src/haproxy.c:2240
> >#7  0x0041b83c in main (argc=, 
> > argv=0x7ffc22f6b4d8) at src/haproxy.c:3094
> > 
> > I use the last 1.8.12 version.
> 
> This one looks a bit strange. I looked at it a little bit and it corresponds
> to the line "free(bind_conf->keys_ref->tlskeys);". Unfortunately, there is no
> other line in the code appearing to perfom a free on this element, and when
> passing through this code the key_ref is destroyed and properly nulled. I
> checked if it was possible for this element not to be allocated and I don't
> see how that could happen either. Thus I'm seeing only three possibilities :
> 
>   - this element was duplicated and appears at multiple places (multiple list
> elements) leading to a real double free
> 
>   - there is a memory corruption somewhere possibly resulting in this element
> being corrupted and not in fact victim of a double free
> 
>   - I can't read code and there is another free that I failed to detect.
> 
> Are you able to trigger this on a trivial config ? Maybe it only happens
> when certain features you have in your config are enabled ?


Reproduced ! unfortunately, I can't reproduce it without systemd. Check the
tls-keys path. With relative path, you must force the start path in the 
systemd config file, or give the fullpath.

The bug seems to be linked with multiple bind line. The followng has no sense,
but the bug appens (on by original conf, I use multi process, avec each bind
line is associated with one process).

Maybe each bind line is duplicated on each process, the tls-key is commun for
each lines, and double-free when the second bind try to release memory.

I guess that systemd is not a cause of the crash, but if I start the process
with -Ws on command line, and I sent kill -USER2, the bug is not trigerred.

test.cfg:
-

   global

   frontend frt
  bind *:443  ssl crt default.pem tls-ticket-keys tls-keys
  bind *:443  ssl crt default.pem tls-ticket-keys tls-keys

tls-keys


   WRGMXEZMeqZzeY7bJTLsfWvrlBKszxDuZ+2WlSP3YFOqUq4dbzBpH+8nvwforYej
   b2dwxCxZsV02/8bmEv+q/QjMllu/4bOSCYFWn6CuTtwiQExG8SLYnwBMevOUjVpL
   cOGgEy6YK4K3h8rS9jSEiu8xWjHP4iMT+IRhHkwYaKPmgwbmvARzvoPkMDnyw5gq


/lib/systemd/system/test.service:
-
   [Service]
   LimitCORE=infinity
   Environment="PIDFILE=/run/test.pid"
   WorkingDirectory=/etc/o3-haproxy
   ExecStart=/opt/o3-haproxy/sbin/haproxy -Ws -f test.cfg
   ExecReload=/bin/kill -USR2 $MAINPID


Thierry
-- 
Thierry Fournier
Web Performance & Security Expert
m: +33 6 68 69 21 85  | e: thierry.fourn...@ozon.io
w: http://www.ozon.io/| b: http://blog.ozon.io/

SSL: double free on reload

[Thierry Fournier]

Hi list,

I caught a double-free whien I reload haproxy-1.8:

writev(2, [{"*** Error in `", 14}, {"/opt/o3-haproxy/sbin/haproxy", 28}, 
{"': ", 3}, {"double free or corruption (!prev)", 33}, {": 0x", 4}, 
{"1cec2ab0", 16}, {" ***\n", 5}], 7) = 103

Decoded:

*** Error in `/opt/o3-haproxy/sbin/haproxy': double free or corruption 
(!prev): 0x1cec2ab0 ***

Gdb says:

   #0  0x7f4bac88b067 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
   #1  0x7f4bac88c448 in __GI_abort () at abort.c:89
   #2  0x7f4bac8c91b4 in __libc_message (do_abort=do_abort@entry=1, 
   fmt=fmt@entry=0x7f4bac9be210 "*** Error in `%s': %s: 0x%s ***\n")
   at ../sysdeps/posix/libc_fatal.c:175
   #3  0x7f4bac8ce98e in malloc_printerr (action=1, 
   str=0x7f4bac9be318 "double free or corruption (!prev)", ptr=) at malloc.c:4996
   #4  0x7f4bac8cf696 in _int_free (av=, p=, 
have_lock=0) at malloc.c:3840
   #5  0x0042af56 in ssl_sock_destroy_bind_conf (bind_conf=0x1d27e810) 
at src/ssl_sock.c:4819
   #6  0x004b1390 in deinit () at src/haproxy.c:2240
   #7  0x0041b83c in main (argc=, argv=0x7ffc22f6b4d8) 
at src/haproxy.c:3094

I use the last 1.8.12 version.

Thierry


-- 
Thierry Fournier
Web Performance & Security Expert
m: +33 6 68 69 21 85  | e: thierry.fourn...@ozon.io
w: http://www.ozon.io/| b: http://blog.ozon.io/

BUG / CLOSE-WAIT / INFINITE-LOOP with applets. How reproducing.

[Thierry FOURNIER]

Hi List,

It have a bug with internal applets. Maybe this bug impacts other parts
like these subject "Connections stuck in CLOSE_WAIT state with h2".

when the applet requires more data and doesn't read the input buffer
buffer, and when the client stops waiting and break the connection, a
CLOSE-WAIT state appears and never disappear.
 
I propose a simple applet to reproduce the bug. I join the C
code. To build haproxy with this code, just edit Makefile and add

   OPTIONS_OBJS += bug39.o

near the line 445.



The conf juste use the applet. The bug works with http and tcp.

Start haproxy like this:

   ./haproxy -d -f bug39.1.conf

   or

   ./haproxy -d -f bug39.2.conf

And curl like this (for http an tcp cases):

   curl --max-time 1 http://127.0.0.1:8080/


Thierry
#include 

#include 
#include 
#include 

static void bug39_http_fct(struct appctx *ctx)
{
	struct stream_interface *si = ctx->owner;
	fprintf(stderr, "%s\n", __FUNCTION__);

	if (si->state == SI_ST_DIS || si->state == SI_ST_CLO)
		return;

	/* alway wait data, and never read data from input. */
	si_applet_cant_get(si);
}

static void bug39_http_release(struct appctx *ctx)
{
	fprintf(stderr, "%s\n", __FUNCTION__);
}

static int bug39_http_init(struct appctx *ctx, struct proxy *px,
   struct stream *strm)
{
	fprintf(stderr, "%s\n", __FUNCTION__);
	return 1;
}

static enum act_parse_ret bug39_parse(const char **args, int *cur_arg,
  struct proxy *px,
  struct act_rule *rule, char **err)
{
	rule->applet.obj_type = OBJ_TYPE_APPLET;
	rule->applet.name = "";
	rule->applet.init = bug39_http_init;
	rule->applet.fct  = bug39_http_fct;
	rule->applet.release  = bug39_http_release;
	rule->applet.timeout  = 1;
	return ACT_RET_PRS_OK;
}

static struct action_kw_list bug39_akl = { { }, {
	{ "bug39", bug39_parse },
	{ /* end */ }
}};

__attribute__((constructor))
static void __bug39_init(void)
{
	fprintf(stderr, "%s\n", __FUNCTION__);
	service_keywords_register(_akl);
}


bug39.1.conf
Description: Binary data


bug39.2.conf
Description: Binary data

[PATCH] Close-wait bug in Lua applets

[Thierry FOURNIER]

Hi,

It have a bug in the Lua HTTP parser which is a trigger for close-wait
behavior.

I thing that the close-wait state is not due to the parser bug. I will
sent anoter email with a method to reproduce it.

This patch must be backported from 1.6 to 1.8.


Thierry
>From e0790d51546e7f032076852f2dd4ea7e5a301f10 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER 
Date: Sat, 30 Jun 2018 10:37:33 +0200
Subject: [PATCH] BUG/MEDIUM: lua: possible CLOSE-WAIT state with '\n' headers

The Lua parser doesn't takes in account end-of-headers containing
only '\n'. It expects always '\r\n'. If a '\n' is processes the Lua
parser considers it miss 1 byte, and wait indefinitely for new data.

When the client reaches their timeout, it closes the connection.
This close is not detected and the connection keep in CLOSE-WAIT
state.

I guess that this patch fix only a visible part of the problem.
If the Lua HTTP parser wait for data, the timeout server or the
connectio closed by the client may stop the applet.

How reproduce the problem:

HAProxy conf:

   global
  lua-load bug38.lua
   frontend frt
  timeout client 2s
  timeout server 2s
  mode http
  bind *:8080
  http-request use-service lua.donothing

Lua conf

   core.register_service("donothing", "http", function(applet) end)

Client request:

   echo -ne 'GET / HTTP/1.1\n\n' | nc 127.0.0.1 8080

Look for CLOSE-WAIT in the connection with "netstat" or "ss". I
use this script:

   while sleep 1; do ss | grep CLOSE-WAIT; done

This patch must be backported in 1.6, 1.7 and 1.8

Workaround: enable the "hard-stop-after" directive, and perform
periodic reload.
---
 src/hlua.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/hlua.c b/src/hlua.c
index f733cd48d..daa540813 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -,13 +,13 @@ static void hlua_applet_http_fct(struct appctx *ctx)
 			len2 = 0;
 		if (ret == 0)
 			len1 = 0;
-		if (len1 + len2 < strm->txn->req.eoh + 2) {
+		if (len1 + len2 < strm->txn->req.eoh + strm->txn->req.eol) {
 			si_applet_cant_get(si);
 			return;
 		}
 
 		/* skip the requests bytes. */
-		co_skip(si_oc(si), strm->txn->req.eoh + 2);
+		co_skip(si_oc(si), strm->txn->req.eoh + strm->txn->req.eol);
 	}
 
 	/* Executes The applet if it is not done. */
-- 
2.16.3

>From e0790d51546e7f032076852f2dd4ea7e5a301f10 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER 
Date: Sat, 30 Jun 2018 10:37:33 +0200
Subject: [PATCH] BUG/MEDIUM: lua: possible CLOSE-WAIT state with '\n' headers

The Lua parser doesn't takes in account end-of-headers containing
only '\n'. It expects always '\r\n'. If a '\n' is processes the Lua
parser considers it miss 1 byte, and wait indefinitely for new data.

When the client reaches their timeout, it closes the connection.
This close is not detected and the connection keep in CLOSE-WAIT
state.

I guess that this patch fix only a visible part of the problem.
If the Lua HTTP parser wait for data, the timeout server or the
connectio closed by the client may stop the applet.

How reproduce the problem:

HAProxy conf:

   global
  lua-load bug38.lua
   frontend frt
  timeout client 2s
  timeout server 2s
  mode http
  bind *:8080
  http-request use-service lua.donothing

Lua conf

   core.register_service("donothing", "http", function(applet) end)

Client request:

   echo -ne 'GET / HTTP/1.1\n\n' | nc 127.0.0.1 8080

Look for CLOSE-WAIT in the connection with "netstat" or "ss". I
use this script:

   while sleep 1; do ss | grep CLOSE-WAIT; done

This patch must be backported in 1.6, 1.7 and 1.8

Workaround: enable the "hard-stop-after" directive, and perform
periodic reload.
---
 src/hlua.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/hlua.c b/src/hlua.c
index f733cd48d..daa540813 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -,13 +,13 @@ static void hlua_applet_http_fct(struct appctx *ctx)
 			len2 = 0;
 		if (ret == 0)
 			len1 = 0;
-		if (len1 + len2 < strm->txn->req.eoh + 2) {
+		if (len1 + len2 < strm->txn->req.eoh + strm->txn->req.eol) {
 			si_applet_cant_get(si);
 			return;
 		}
 
 		/* skip the requests bytes. */
-		co_skip(si_oc(si), strm->txn->req.eoh + 2);
+		co_skip(si_oc(si), strm->txn->req.eoh + strm->txn->req.eol);
 	}
 
 	/* Executes The applet if it is not done. */
-- 
2.16.3

Re: [Patch] Re: Segfault with haproxy 1.8.10

[Thierry Fournier]

> On 26 Jun 2018, at 13:56, Willy Tarreau  wrote:
> 
> Hi Thierry,
> 
> On Tue, Jun 26, 2018 at 10:20:38AM +0200, thierry.fourn...@arpalert.org wrote:
>> BR,Hi,
>> 
>> I found the bug. The segfault is unavoidable, because is trigged if an
>> entry doesn't exists in the stick-tables. At the start, the stick table
>> is empty.
>> 
>> It is a regression introduced in 1.8.10 by this patch:
>> 
>>   BUG/MEDIUM: stick-tables: Decrement ref_cnt in table_* converters
>>   commit d7bd88009d88dd413e01bc0baa90d6662a3d7718
>>   Author: Daniel Corbett 
>>   Date:   Sun May 27 09:47:12 2018 -0400
> 
> Oh crap :-(  I noticed the memory leaks in the first version but I did
> not notice this one, which is pretty visible nonetheless given the test
> on !!ts 2 lines above :-(


It happens ...


>> I join a patch.
>> 
>> Daniel, could you check the compliance with your original patch ?
> 
> Your patch is obviously good, I've just merged it.


great.


>> William, The backport in 1.8 stable is trivial.
> 
> William / Christopher, given that 1.8.10 is very recent and few people
> had the time to upgrade, I think it would make sense to issue 1.8.11
> to limit the risks. I don't like such bugs which crash at runtime
> while you're drinking coffee after the upgrade.


Arggg ... I just created my own maintenance branch with the patch for my
customers and own projets :-)

Thierry

[Patch] Re: Segfault with haproxy 1.8.10

[thierry . fournier]

BR,Hi,

I found the bug. The segfault is unavoidable, because is trigged if an
entry doesn't exists in the stick-tables. At the start, the stick table
is empty.

It is a regression introduced in 1.8.10 by this patch:

   BUG/MEDIUM: stick-tables: Decrement ref_cnt in table_* converters
   commit d7bd88009d88dd413e01bc0baa90d6662a3d7718
   Author: Daniel Corbett 
   Date:   Sun May 27 09:47:12 2018 -0400

I join a patch.

Daniel, could you check the compliance with your original patch ?
William, The backport in 1.8 stable is trivial.

BR,
Thierry


On Mon, 25 Jun 2018 23:03:37 +0200
Willy Tarreau  wrote:

> On Mon, Jun 25, 2018 at 10:45:51PM +0200, Thierry Fournier wrote:
> > Just for information, If someone is working on this bug, I think
> > that I found the origin of the crash. I check impact and the
> > validity of the patch, and them I submit a patch
> 
> Ah cool, thank you, we'll have a fairly busy week and I didn't expect
> to have the time to look at this crash this week :-(
> 
> Cheers,
> Willy
> 
>From 92622852bccce39afbc63320ee7cad4df0586388 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER 
Date: Mon, 25 Jun 2018 22:35:20 +0200
Subject: [PATCH] BUG/MAJOR: Stick-tables crash with segfault when the key is
 not in the stick-table

When a lookup is done on a key not present in the stick-table the "st"
pointer is NULL and it is used to return the converter result, but it
is used untested with stktable_release().

This regression was introduced in 1.8.10 here:

   BUG/MEDIUM: stick-tables: Decrement ref_cnt in table_* converters
   commit d7bd88009d88dd413e01bc0baa90d6662a3d7718
   Author: Daniel Corbett 
   Date:   Sun May 27 09:47:12 2018 -0400

Minimal conf for reproducong the problem:

   frontend test
  mode http
  stick-table type ip size 1m expire 1h store gpc0
  bind *:8080
  http-request redirect location /a if { src,in_table(test) }

The segfault is triggered using:

   curl -i http://127.0.0.1:8080/

This patch must be backported in 1.8
---
 src/stick_table.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/stick_table.c b/src/stick_table.c
index 101a4e253..429465455 100644
--- a/src/stick_table.c
+++ b/src/stick_table.c
@@ -875,7 +875,8 @@ static int sample_conv_in_table(const struct arg *arg_p, struct sample *smp, voi
 	smp->data.type = SMP_T_BOOL;
 	smp->data.u.sint = !!ts;
 	smp->flags = SMP_F_VOL_TEST;
-	stktable_release(t, ts);
+	if (ts)
+		stktable_release(t, ts);
 	return 1;
 }
 
-- 
2.16.3

Re: Segfault with haproxy 1.8.10

[Thierry Fournier]

Just for information, If someone is working on this bug, I think
that I found the origin of the crash. I check impact and the
validity of the patch, and them I submit a patch

Thierry

> On 25 Jun 2018, at 11:07, Thierry Fournier  
> wrote:
> 
> Hi,
> 
> I freshly compile haproxy-1.9.10, and after the start, I display a lot of 
> segfaults.
> 
> #0  stktable_release (t=t@entry=0x274a5a8, ts=0x0) at src/stick_table.c:419
> #1  0x0049a0d6 in sample_conv_in_table (arg_p=, 
> smp=0x7fffc6ed0d70, private=)
>at src/stick_table.c:876
> #2  0x004d1554 in sample_process (px=px@entry=0x32cbae0, 
> sess=sess@entry=0x36e13b0,
>strm=strm@entry=0x365d0d0, opt=opt@entry=6, expr=0x3294540, 
> p=p@entry=0x7fffc6ed0d70) at src/sample.c:1082
> #3  0x004f99fc in acl_exec_cond (cond=0x3707690, px=0x32cbae0, 
> sess=sess@entry=0x36e13b0,
>strm=strm@entry=0x365d0d0, opt=6, opt@entry=2) at src/acl.c:1148
> #4  0x004e6c7d in tcp_inspect_request (s=s@entry=0x365d0d0, 
> req=req@entry=0x365d0e0, an_bit=an_bit@entry=2)
>at src/tcp_rules.c:148
> #5  0x00487deb in process_stream (t=t@entry=0x8163d80) at 
> src/stream.c:1902
> #6  0x00508b1b in process_runnable_tasks () at src/task.c:229
> #7  0x004ba44b in run_poll_loop () at src/haproxy.c:2403
> #8  run_thread_poll_loop (data=data@entry=0x27d97e0) at src/haproxy.c:2470
> #9  0x0041b5a5 in main (argc=, argv=0x7fffc6ed1498) at 
> src/haproxy.c:3074
> 
> src/stick_table.c:419
> 
> 415 /* Just decrease the ref_cnt of the current session */
> 416 void stktable_release(struct stktable *t, struct stksess *ts)
> 417 {
> 418 HA_SPIN_LOCK(STK_TABLE_LOCK, >lock);
> 419 ts->ref_cnt--;
> 420 HA_SPIN_UNLOCK(STK_TABLE_LOCK, >lock);
> 421 }
> 
> (gdb) p ts
> $1 = (struct stksess *) 0x0
> 
> There is basic haproxy 1.8.10 (ec17d7a98f30326918219ba876fcfc56f6ad6823) 
> compiled with these options:
> 
> make -j 2 -C haproxy-1.8 \
>DEBUG="-DDEBUG_EXPR" \
>TARGET=linux2628 \
>USE_THREAD=1 \
>USE_REGPARM=1 \
>USE_SYSTEMD=1 \
>USE_LINUX_TPROXY=1 \
>USE_OPENSSL=yes \
>USE_PCRE=yes \
>USE_ZLIB=yes \
>USE_LUA=1 \
>USE_51DEGREES=1 \
>51DEGREES_SRC=/opt/Device-Detection/src/pattern
> 
> Tell me if you want the core & binary file.
> 
> Thierry

Segfault with haproxy 1.8.10

[Thierry Fournier]

Hi,

I freshly compile haproxy-1.9.10, and after the start, I display a lot of 
segfaults.

#0  stktable_release (t=t@entry=0x274a5a8, ts=0x0) at src/stick_table.c:419
#1  0x0049a0d6 in sample_conv_in_table (arg_p=, 
smp=0x7fffc6ed0d70, private=)
at src/stick_table.c:876
#2  0x004d1554 in sample_process (px=px@entry=0x32cbae0, 
sess=sess@entry=0x36e13b0,
strm=strm@entry=0x365d0d0, opt=opt@entry=6, expr=0x3294540, 
p=p@entry=0x7fffc6ed0d70) at src/sample.c:1082
#3  0x004f99fc in acl_exec_cond (cond=0x3707690, px=0x32cbae0, 
sess=sess@entry=0x36e13b0,
strm=strm@entry=0x365d0d0, opt=6, opt@entry=2) at src/acl.c:1148
#4  0x004e6c7d in tcp_inspect_request (s=s@entry=0x365d0d0, 
req=req@entry=0x365d0e0, an_bit=an_bit@entry=2)
at src/tcp_rules.c:148
#5  0x00487deb in process_stream (t=t@entry=0x8163d80) at 
src/stream.c:1902
#6  0x00508b1b in process_runnable_tasks () at src/task.c:229
#7  0x004ba44b in run_poll_loop () at src/haproxy.c:2403
#8  run_thread_poll_loop (data=data@entry=0x27d97e0) at src/haproxy.c:2470
#9  0x0041b5a5 in main (argc=, argv=0x7fffc6ed1498) at 
src/haproxy.c:3074

src/stick_table.c:419

 415 /* Just decrease the ref_cnt of the current session */
 416 void stktable_release(struct stktable *t, struct stksess *ts)
 417 {
 418 HA_SPIN_LOCK(STK_TABLE_LOCK, >lock);
 419 ts->ref_cnt--;
 420 HA_SPIN_UNLOCK(STK_TABLE_LOCK, >lock);
 421 }

(gdb) p ts
$1 = (struct stksess *) 0x0

There is basic haproxy 1.8.10 (ec17d7a98f30326918219ba876fcfc56f6ad6823) 
compiled with these options:

make -j 2 -C haproxy-1.8 \
DEBUG="-DDEBUG_EXPR" \
TARGET=linux2628 \
USE_THREAD=1 \
USE_REGPARM=1 \
USE_SYSTEMD=1 \
USE_LINUX_TPROXY=1 \
USE_OPENSSL=yes \
USE_PCRE=yes \
USE_ZLIB=yes \
USE_LUA=1 \
USE_51DEGREES=1 \
51DEGREES_SRC=/opt/Device-Detection/src/pattern

Tell me if you want the core & binary file.

Thierry

Re: Variables

[Thierry Fournier]

Hi,

There are no mean to display the memory used. It is a good idea to do somme 
accounting about variable usage.

Default limit is “unlimited”.

Thierry


> On 23 Jun 2018, at 19:51, mlist  wrote:
> 
> Hi,
>  
> how to debug/show variables memory consumption to correctly tune size with:
>  
> tune.vars….
>  
> What are the default limits ?
>  
> 
> 
> 
>  
> 
>  
> mlist
> 
>  
> APKAPPA s.r.l. sede legale Via F. Albani, 21 20149 Milano | p.iva/vat no. 
> IT-08543640158
> sede amministrativa e operativa Reggio Emilia (RE) via M. K. Gandhi, 24/A 
> 42123 - sede operativa Magenta (MI) via Milano 89/91 20013
> www.apkappa.it 
>  
>  
> 
> Ai sensi e per gli effetti della Legge sulla tutela della riservatezza 
> personale (DL.gs. 196/03 e collegate), questa mail è destinata unicamente 
> alle persone sopra indicate e le informazioni in essa contenute sono da 
> considerarsi strettamente riservate. 
> This email is confidential, do not use the contents for any purpose 
> whatsoever nor disclose them to anyone else. If you are not the intended 
> recipient, you should not copy, modify, distribute or take any action in 
> reliance on it. If you have received this email in error, please notify the 
> sender and delete this email from your system.

Re: [PATCH] Re: Random crash (segfault, double free, ...) with a mix of SSL + cipherlist hash

[Thierry Fournier]

> On 18 Jun 2018, at 15:38, Emmanuel Hocdet  wrote:
> 
> 
>> Le 18 juin 2018 à 15:30, Thierry Fournier  a 
>> écrit :
>> 
>> 
>> 
>>> On 18 Jun 2018, at 14:37, Emmanuel Hocdet  wrote:
>>> 
>>>> 
>>>> Le 18 juin 2018 à 10:43, Thierry Fournier  
>>>> a écrit :
>>>> 
>>>> 
>>>>> On 18 Jun 2018, at 10:33, Willy Tarreau  wrote:
>>>>> 
>>>>> On Sun, Jun 17, 2018 at 09:44:50PM +0200, thierry.fourn...@arpalert.org 
>>>>> wrote:
>>>>>> Finally, I got it ! It works with luck because we have 1 bug in Haproxy
>>>>>> and 1 error (I suppose) in a OpenSSL compatibility layer.
>>>>> (...)
>>>>>> I join two patch. The first which fix the cipher capture must be
>>>>>> backported to 1.8, for the second patch wich fix the app data
>>>>>> compatibility, I dont known (at least 1.8).
>>>>> 
>>>>> Good job! I imagine you didn't have a funny week-end playing with this 
>>>>> one :-/
>>>> 
>>>> 
>>>> Yes, including the Friday :-) But I hope this path improve stability. If 
>>>> someone
>>>> have time and is interested by the subject, it may be interesting to see 
>>>> in the
>>>> OpenSSL code if the slot 0 used without reservation works fine, or works 
>>>> because
>>>> we have luck.
>>>> 
>>> 
>>> It work find because slot 0 is natively reserved for old *_{set, 
>>> get}_app_data API compatibility.
>> 
>> 
>> Ok, thanks. So the classifcation BUG/MAJOR can be changed for BUG/MEDIUM
>> because it impacts only the usage of SSL join with the cipherlist hash.
>> Too late :-)
>> 
> 
> I think it should not be a bug at all (second patch), and set of ex_data 
> without reservation
> (first patch and my patch) should be the only sources of bugs.
> 

I don’t known. In fact it works, so it is not a bug. But, when I use the
reservation for an ex_data slot, it returns the slot 0, and this slot is
used for the compatibility layer and can be crush some data. I conclude
that is a bug in Openssl. The reservation function must give a slot
starting to 1.

Maybe, the recommendation is to not mix the compatibility functions
like set_app_data() with *ex_data*() functions. But, I don’t see 
anything about this.

thierry

Re: [PATCH] Re: Random crash (segfault, double free, ...) with a mix of SSL + cipherlist hash

[Thierry Fournier]

> On 18 Jun 2018, at 14:37, Emmanuel Hocdet  wrote:
> 
>> 
>> Le 18 juin 2018 à 10:43, Thierry Fournier  a 
>> écrit :
>> 
>> 
>>> On 18 Jun 2018, at 10:33, Willy Tarreau  wrote:
>>> 
>>> On Sun, Jun 17, 2018 at 09:44:50PM +0200, thierry.fourn...@arpalert.org 
>>> wrote:
>>>> Finally, I got it ! It works with luck because we have 1 bug in Haproxy
>>>> and 1 error (I suppose) in a OpenSSL compatibility layer.
>>> (...)
>>>> I join two patch. The first which fix the cipher capture must be
>>>> backported to 1.8, for the second patch wich fix the app data
>>>> compatibility, I dont known (at least 1.8).
>>> 
>>> Good job! I imagine you didn't have a funny week-end playing with this one 
>>> :-/
>> 
>> 
>> Yes, including the Friday :-) But I hope this path improve stability. If 
>> someone
>> have time and is interested by the subject, it may be interesting to see in 
>> the
>> OpenSSL code if the slot 0 used without reservation works fine, or works 
>> because
>> we have luck.
>> 
> 
> It work find because slot 0 is natively reserved for old *_{set, 
> get}_app_data API compatibility.


Ok, thanks. So the classifcation BUG/MAJOR can be changed for BUG/MEDIUM
because it impacts only the usage of SSL join with the cipherlist hash.
Too late :-)

Thierry

Re: [PATCH] Re: Random crash (segfault, double free, ...) with a mix of SSL + cipherlist hash

[Thierry Fournier]

> On 18 Jun 2018, at 10:33, Willy Tarreau  wrote:
> 
> On Sun, Jun 17, 2018 at 09:44:50PM +0200, thierry.fourn...@arpalert.org wrote:
>> Finally, I got it ! It works with luck because we have 1 bug in Haproxy
>> and 1 error (I suppose) in a OpenSSL compatibility layer.
> (...)
>> I join two patch. The first which fix the cipher capture must be
>> backported to 1.8, for the second patch wich fix the app data
>> compatibility, I dont known (at least 1.8).
> 
> Good job! I imagine you didn't have a funny week-end playing with this one :-/


Yes, including the Friday :-) But I hope this path improve stability. If someone
have time and is interested by the subject, it may be interesting to see in the
OpenSSL code if the slot 0 used without reservation works fine, or works because
we have luck.

BR,
Thierry

Re: [PATCH] BUG/MINOR: lua: Segfaults with wrong usage of types.

[Thierry Fournier]

I read the ML too quicky, thiss is the right patch.
Thanks.
Thierry

> On 15 Jun 2018, at 15:06, Frederic Lecaille  wrote:
> 
> On 06/15/2018 02:28 PM, Frederic Lecaille wrote:
>> On 06/15/2018 02:15 PM, Frederic Lecaille wrote:
>>> On 06/14/2018 11:05 PM, Patrick Hemmer wrote:
 Haproxy segfaults if you pass the wrong argument type to a converter.
 Example:
 
 haproxy.cfg:
  global
  lua-load /tmp/haproxy.lua
 
  frontend f1
  mode http
  bind :8000
  default_backend b1
 
  http-request lua.foo
 
  backend b1
  mode http
  server s1 127.0.0.1:8080
 
 haproxy.lua:
  core.register_action("foo", { "http-req" }, function(txn)
  txn.sc:ipmask(txn.f:src(), 24, 112)
  end)
 
 Result:
  * thread #1, queue = 'com.apple.main-thread', stop reason = 
 EXC_BAD_ACCESS (code=1, address=0x18)
  frame #0: 0x7fffc9fcbf56 
 libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell + 182
 libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell:
  ->  0x7fffc9fcbf56 <+182>: movb   (%rsi,%r8), %cl
  0x7fffc9fcbf5a <+186>: movb   %cl, (%rdi,%r8)
  0x7fffc9fcbf5e <+190>: subq   $0x1, %rdx
  0x7fffc9fcbf62 <+194>: je 0x7fffc9fcbf78; <+216>
  Target 0: (haproxy) stopped.
  (lldb) bt
  * thread #1, queue = 'com.apple.main-thread', stop reason = 
 EXC_BAD_ACCESS (code=1, address=0x18)
* frame #0: 0x7fffc9fcbf56 
 libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell + 182
  frame #1: 0x7fffc9e7442e libsystem_c.dylib`__memcpy_chk + 22
  frame #2: 0x00010002ec46 
 haproxy`hlua_lua2arg_check(L=0x00010120d298, first=3, 
 argp=0x7fff5fbfe690, mask=196, p=0x000101817000) at hlua.c:749
  frame #3: 0x00010001fa00 
 haproxy`hlua_run_sample_conv(L=0x00010120d298) at hlua.c:3393
  frame #4: 0x00010032400b haproxy`luaD_precall + 747
  frame #5: 0x0001003343c6 haproxy`luaV_execute + 3158
  frame #6: 0x000100323429 haproxy`luaD_rawrunprotected + 89
  frame #7: 0x000100324516 haproxy`lua_resume + 278
  frame #8: 0x00010001b199 
 haproxy`hlua_ctx_resume(lua=0x000101205080, yield_allowed=1) at 
 hlua.c:1080
  frame #9: 0x000100027de8 
 haproxy`hlua_action(rule=0x00010101b180, px=0x000101817000, 
 sess=0x00010120cb70, s=0x00010120cc00, flags=2) at hlua.c:6198
  frame #10: 0x000100044bcd 
 haproxy`http_req_get_intercept_rule(px=0x000101817000, 
 rules=0x000101817048, s=0x00010120cc00, 
 deny_status=0x7fff5fbfee78) at proto_http.c:2760
  frame #11: 0x000100046182 
 haproxy`http_process_req_common(s=0x00010120cc00, 
 req=0x00010120cc10, an_bit=16, px=0x000101817000) at 
 proto_http.c:3461
  frame #12: 0x000100094c50 
 haproxy`process_stream(t=0x00010120cf40, context=0x00010120cc00, 
 state=9) at stream.c:1905
  frame #13: 0x00010016179f haproxy`process_runnable_tasks at 
 task.c:362
  frame #14: 0x0001000ea0eb haproxy`run_poll_loop at 
 haproxy.c:2403
  frame #15: 0x0001000e7c74 
 haproxy`run_thread_poll_loop(data=0x7fff5fbff3a4) at haproxy.c:2464
  frame #16: 0x0001000e4a49 haproxy`main(argc=3, 
 argv=0x7fff5fbff590) at haproxy.c:3082
  frame #17: 0x7fffc9db9235 libdyld.dylib`start + 1
 
 Issue goes away if you change the lua txn.sc:ipmask() line to:
  txn.sc:ipmask(txn.f:src(), '24', '112')
 
 Reproduced with current master (9db0fed) and lua version 5.3.4.
 
 -Patrick
>>> 
>>> It seems the patch attached to this mail fixes this issue. It at least make 
>>> the varnishtest test file pass.
>>> 
>>> Must be checked by Thierry.
>> Should have mentionned that I could not reproduce this issue without 
>> compiling the thread support (USE_THREAD=1).
> 
> There is potentially the same issue in hlua_run_sample_conv(). See the 
> updated patch attached to this mail.
> 
> 
> 
> 
> <0001-BUG-MINOR-lua-Segfaults-with-wrong-usage-of-types.patch>

Re: BUG: segfault with lua sample converters & wrong arg types

[Thierry Fournier]

Thanks. the patch make sense.
Willy, could you apply ?


> On 15 Jun 2018, at 14:15, Frederic Lecaille  wrote:
> 
> On 06/14/2018 11:05 PM, Patrick Hemmer wrote:
>> Haproxy segfaults if you pass the wrong argument type to a converter.
>> Example:
>> haproxy.cfg:
>> global
>> lua-load /tmp/haproxy.lua
>> frontend f1
>> mode http
>> bind :8000
>> default_backend b1
>> http-request lua.foo
>> backend b1
>> mode http
>> server s1 127.0.0.1:8080
>> haproxy.lua:
>> core.register_action("foo", { "http-req" }, function(txn)
>> txn.sc:ipmask(txn.f:src(), 24, 112)
>> end)
>> Result:
>> * thread #1, queue = 'com.apple.main-thread', stop reason = 
>> EXC_BAD_ACCESS (code=1, address=0x18)
>> frame #0: 0x7fffc9fcbf56 
>> libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell + 182
>> libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell:
>> ->  0x7fffc9fcbf56 <+182>: movb   (%rsi,%r8), %cl
>> 0x7fffc9fcbf5a <+186>: movb   %cl, (%rdi,%r8)
>> 0x7fffc9fcbf5e <+190>: subq   $0x1, %rdx
>> 0x7fffc9fcbf62 <+194>: je 0x7fffc9fcbf78; <+216>
>> Target 0: (haproxy) stopped.
>> (lldb) bt
>> * thread #1, queue = 'com.apple.main-thread', stop reason = 
>> EXC_BAD_ACCESS (code=1, address=0x18)
>>   * frame #0: 0x7fffc9fcbf56 
>> libsystem_platform.dylib`_platform_memmove$VARIANT$Haswell + 182
>> frame #1: 0x7fffc9e7442e libsystem_c.dylib`__memcpy_chk + 22
>> frame #2: 0x00010002ec46 
>> haproxy`hlua_lua2arg_check(L=0x00010120d298, first=3, 
>> argp=0x7fff5fbfe690, mask=196, p=0x000101817000) at hlua.c:749
>> frame #3: 0x00010001fa00 
>> haproxy`hlua_run_sample_conv(L=0x00010120d298) at hlua.c:3393
>> frame #4: 0x00010032400b haproxy`luaD_precall + 747
>> frame #5: 0x0001003343c6 haproxy`luaV_execute + 3158
>> frame #6: 0x000100323429 haproxy`luaD_rawrunprotected + 89
>> frame #7: 0x000100324516 haproxy`lua_resume + 278
>> frame #8: 0x00010001b199 
>> haproxy`hlua_ctx_resume(lua=0x000101205080, yield_allowed=1) at 
>> hlua.c:1080
>> frame #9: 0x000100027de8 
>> haproxy`hlua_action(rule=0x00010101b180, px=0x000101817000, 
>> sess=0x00010120cb70, s=0x00010120cc00, flags=2) at hlua.c:6198
>> frame #10: 0x000100044bcd 
>> haproxy`http_req_get_intercept_rule(px=0x000101817000, 
>> rules=0x000101817048, s=0x00010120cc00, 
>> deny_status=0x7fff5fbfee78) at proto_http.c:2760
>> frame #11: 0x000100046182 
>> haproxy`http_process_req_common(s=0x00010120cc00, 
>> req=0x00010120cc10, an_bit=16, px=0x000101817000) at 
>> proto_http.c:3461
>> frame #12: 0x000100094c50 
>> haproxy`process_stream(t=0x00010120cf40, context=0x00010120cc00, 
>> state=9) at stream.c:1905
>> frame #13: 0x00010016179f haproxy`process_runnable_tasks at 
>> task.c:362
>> frame #14: 0x0001000ea0eb haproxy`run_poll_loop at haproxy.c:2403
>> frame #15: 0x0001000e7c74 
>> haproxy`run_thread_poll_loop(data=0x7fff5fbff3a4) at haproxy.c:2464
>> frame #16: 0x0001000e4a49 haproxy`main(argc=3, 
>> argv=0x7fff5fbff590) at haproxy.c:3082
>> frame #17: 0x7fffc9db9235 libdyld.dylib`start + 1
>> Issue goes away if you change the lua txn.sc:ipmask() line to:
>> txn.sc:ipmask(txn.f:src(), '24', '112')
>> Reproduced with current master (9db0fed) and lua version 5.3.4.
>> -Patrick
> 
> It seems the patch attached to this mail fixes this issue. It at least make 
> the varnishtest test file pass.
> 
> Must be checked by Thierry.
> 
> 
> Fred.
> <0001-BUG-MINOR-lua-Segfaults-with-wrong-usage-of-types.patch>

[PATCH] Re: Random crash (segfault, double free, ...) with a mix of SSL + cipherlist hash

[thierry . fournier]

Finally, I got it ! It works with luck because we have 1 bug in Haproxy
and 1 error (I suppose) in a OpenSSL compatibility layer.

First point is some OpenSSL macro, in the ssl.h file:

   #define SSL_set_app_data(s,arg) (SSL_set_ex_data(s,0,(char*)arg))
   #define SSL_get_app_data(s) (SSL_get_ex_data(s,0))

These macro uses the function SSL_get_ex_data ans SSL_set_ex_data with
the index 0, but this index is not reserved with the function
SSL_get_ex_new_index()

When I reserve a slot for the cipherlist hash, I have the number 0, this
produce a collision an the set_app_data pointer is crushed by the
cipherlist data.

Luckily, the cipherlist hash is declared in the SSL_CTX space in place
of SSL space. The first consequence was a memory leak because the
allocated struct is never cleared. The second consequence is an index
number > 0.

 - With this index number > 0, the data doesn't crush app_data, but
   there stored in a non-reserved storage => various crash cause

 - When I fix the reservation, The app_data are crushed, and we have
   systematic segfault.

I join two patch. The first which fix the cipher capture must be
backported to 1.8, for the second patch wich fix the app data
compatibility, I dont known (at least 1.8).

I join also the backports for 1.8 (there are trivial backport, with ery
minor conflict)

Thierry




On Sun, 17 Jun 2018 03:01:54 +0200
Thierry FOURNIER  wrote:

> Hi,
> 
> When I use SSL requests and the cipherlist hash enabled, HAProxy
> randomly crash:
> 
>  - segfault
>  - double free
>  - munmap_chunk(): invalid pointer
> 
> I think that is a memory crush.
> 
> I read the "cipherlist hash" code, and I put some printf, I do not
> detect any memory override.
> 
> When I comment the following line, the bug disappear
> 
>SSL_set_ex_data(ssl, ssl_capture_ptr_index, capture);
> 
> The crash happens with many versions of openssl:
> 
>  - 1.0.2j (home build)
>  - 1.0.1t-1+deb7u4
>  - 1.0.1t-1+deb8u8
>  - 1.0.2g-1ubuntu4.12
> 
> cipherlist hash is available from 1.8. The bug appears with current 1.8
> and current 1.9dev.
> 
> I join some files:
> 
>  - bug36.build.sh   : build script
>  - bug36.run.sh : run haproxy command
>  - bug36.request.sh : curl request
>  - bug36.conf   : minimal conf which reproduce the problem
>  - bug36.pem: ramdom self signed certificate
> 
> Just execute some requests, and the bug is reproduced.
> 
> BR,
> Thierry
>From a99fcb5af1a079ab5403f4cf7d2cf9345e4daf03 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER 
Date: Sun, 17 Jun 2018 21:33:01 +0200
Subject: [PATCH 1/2] BUG/MAJOR: Random crash with cipherlist capture

The cipher list capture struct is stored in the SSL memory space,
but the slot is reserved in the SSL_CTX memory space. This causes
ramdom crashes.

This patch should be backported to 1.8
---
 src/ssl_sock.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 9fb2bb151..599c8c3ec 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -8805,7 +8805,7 @@ static void __ssl_sock_init(void)
 #if (OPENSSL_VERSION_NUMBER >= 0x1000200fL && !defined OPENSSL_NO_TLSEXT && !defined OPENSSL_IS_BORINGSSL && !defined LIBRESSL_VERSION_NUMBER)
 	sctl_ex_index = SSL_CTX_get_ex_new_index(0, NULL, NULL, NULL, ssl_sock_sctl_free_func);
 #endif
-	ssl_capture_ptr_index = SSL_CTX_get_ex_new_index(0, NULL, NULL, NULL, ssl_sock_capture_free_func);
+	ssl_capture_ptr_index = SSL_get_ex_new_index(0, NULL, NULL, NULL, ssl_sock_capture_free_func);
 	sample_register_fetches(_fetch_keywords);
 	acl_register_keywords(_kws);
 	bind_register_keywords(_kws);
-- 
2.16.3

>From 54ce55a1203e732d266e72332bfea19d30f45487 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER 
Date: Sun, 17 Jun 2018 21:37:05 +0200
Subject: [PATCH 2/2] BUG/MAJOR: OpenSSL context is stored in non-reserved
 memory slot

We never saw unexplicated crash with SSL, so I suppose that we are
luck, or the slot 0 is always reserved. Anyway the usage of the macro
SSL_get_app_data() and SSL_set_app_data() seem wrong. This patch change
the deprecated functions SSL_get_app_data() and SSL_set_app_data()
by the new functions SSL_get_ex_data() and SSL_set_ex_data(), and
it reserves the slot in the SSL memory space.

For information, this is the two declaration which seems wrong or
incomplete in the OpenSSL ssl.h file. We can see the usage of the
slot 0 whoch is hardcoded, but never reserved.

   #define SSL_set_app_data(s,arg) (SSL_set_ex_data(s,0,(char *)arg))
   #define SSL_get_app_data(s)  (SSL_get_ex_data(s,0))

This patch must be backported at least in 1.8, maybe in other versions.
---
 src/ssl_sock.c | 22 --
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 599c8c3ec..2707a40a9 100644
--- a/src/ssl_s

Random crash (segfault, double free, ...) with a mix of SSL + cipherlist hash

[Thierry FOURNIER]

Hi,

When I use SSL requests and the cipherlist hash enabled, HAProxy
randomly crash:

 - segfault
 - double free
 - munmap_chunk(): invalid pointer

I think that is a memory crush.

I read the "cipherlist hash" code, and I put some printf, I do not
detect any memory override.

When I comment the following line, the bug disappear

   SSL_set_ex_data(ssl, ssl_capture_ptr_index, capture);

The crash happens with many versions of openssl:

 - 1.0.2j (home build)
 - 1.0.1t-1+deb7u4
 - 1.0.1t-1+deb8u8
 - 1.0.2g-1ubuntu4.12

cipherlist hash is available from 1.8. The bug appears with current 1.8
and current 1.9dev.

I join some files:

 - bug36.build.sh   : build script
 - bug36.run.sh : run haproxy command
 - bug36.request.sh : curl request
 - bug36.conf   : minimal conf which reproduce the problem
 - bug36.pem: ramdom self signed certificate

Just execute some requests, and the bug is reproduced.

BR,
Thierry
#!/bin/bash
make -j8 \
	TARGET=linux2428 \
	USE_DL=1 \
	USE_OPENSSL=1 \
	$*
#!/bin/bash
curl -i -k https://127.0.0.1:/
curl -i -k https://127.0.0.1:/
#!/bin/bash
./haproxy -d -f bug36.conf


bug36.pem
Description: Binary data


bug36.conf
Description: Binary data

Re: [Patch] Lua / Increase error verbosity

[Thierry Fournier]

thanks.

> On 11 Jun 2018, at 11:12, William Lallemand  wrote:
> 
> On Fri, Jun 08, 2018 at 06:28:10PM +0200, Thierry Fournier wrote:
>> 
>> 
>>> On 8 Jun 2018, at 18:21, Willy Tarreau  wrote:
>>> 
>>> On Fri, Jun 08, 2018 at 01:09:04PM +0200, Thierry Fournier wrote:
>>>> Hi,
>>>> 
>>>> runtime errors doesn't provides the error line, in particular the memory 
>>>> error (out of memory)
>>>> 
>>>> Please find a Lua patch in attachment.
>>>> It provides backtrace condensed in one line.
>>> 
>>> Indeed it's much better. Given that it doesn't change any feature and
>>> only increases verbosity in case of problem, do you want it to be
>>> backported to 1.8 as well ?
>> 
>> 
>> It should great !
>> 
>> Thierry
> 
> Backported to 1.8.
> 
> -- 
> William Lallemand
> 

Re: [Patch] Lua / Increase error verbosity

[Thierry Fournier]

> On 8 Jun 2018, at 18:21, Willy Tarreau  wrote:
> 
> On Fri, Jun 08, 2018 at 01:09:04PM +0200, Thierry Fournier wrote:
>> Hi,
>> 
>> runtime errors doesn't provides the error line, in particular the memory 
>> error (out of memory)
>> 
>> Please find a Lua patch in attachment.
>> It provides backtrace condensed in one line.
> 
> Indeed it's much better. Given that it doesn't change any feature and
> only increases verbosity in case of problem, do you want it to be
> backported to 1.8 as well ?


It should great !

Thierry

[Patch] Lua / Increase error verbosity

[Thierry Fournier]

Hi,

runtime errors doesn’t provides the error line, in particular the memory error 
(out of memory)

Please find a Lua patch in attachment.
It provides backtrace condensed in one line.

Thierry



0001-MINOR-lua-Increase-debug-information.patch
Description: Binary data

Re: [PATCH] Mod security

[Thierry Fournier]

Hi Willy,

Could you apply ?

Thierry

On Thu, 31 May 2018 18:49:03 +0100
David CARLIER  wrote:

> Hi list,
> 
> Here a small diff for fixing few typos.
> 
> Cheers !


-- 
Thierry Fournier
Web Performance & Security Expert
m: +33 6 68 69 21 85  | e: thierry.fourn...@ozon.io
w: http://www.ozon.io/| b: http://blog.ozon.io/

Re: [PATCH] Re: 1.8.8 & 1.9dev, lua, xref_get_peer_and_lock hang / 100% cpu usage after restarting haproxy a few times

[Thierry FOURNIER]

On Sat, 26 May 2018 19:47:54 +0200
PiBa-NL <piba.nl@gmail.com> wrote:

> Hi Thierry,
> 
> Op 25-5-2018 om 15:40 schreef Thierry FOURNIER:
> > On Fri, 18 May 2018 22:17:00 +0200
> > PiBa-NL <piba.nl@gmail.com> wrote:
> >
> >> Hi Thierry,
> >>
> >> Op 18-5-2018 om 20:00 schreef Thierry FOURNIER:
> >>> Hi Pieter,
> >>>
> >>> Could you test the attached patch ? It seems to fix the problem, but I
> >>> have some doubts about the reliability of the patch.
> >>>
> >>> Thierry
> >> The crash seems 'fixed' indeed.. but the lua scipt below now takes
> >> 5seconds instead of 150ms.
> >>
> >> Regards,
> >> PiBa-NL (Pieter)
> >>
> >> con = core.tcp()
> >> con:connect_ssl("216.58.212.132",443) --google: 216.58.212.132
> >> request = [[GET / HTTP/1.0
> >>
> >> ]]
> >> con:send(request)
> >> res = con:receive("*a")
> >> con:close()
> > One bug can hide another bug :-) I catch both. Could you test ?
> >
> > If the result is positive I join also the backport for 1.6 and 1.7
> >
> > Thierry
> 
> Thanks, seems both the hang and the 2nd uncovered task schedule issue 
> are fixed now (google website response is received/processed fast 
> again). I've done some testing, and installed the updated/patched 
> version on my production box last night. At the moment it still works 
> properly. Activated my lua healthchecker and mailer tasks and enabled 3 
> threads again.. Lets see how it go's :), but as said for now it seems to 
> work alright.
> 
> Does the second issue you found and fixed clear the initial 'doubts 
> about the reliability' of the first one.? Or did you have a particular 
> possibly problematic scenario in mind that i could try and check for?
> 
> For the moment i think it is more 'reliable / stable' with the patches 
> than without. So in that regard i think they could be merged.
> 
> There seems to be a issue with tcp:settimeout() though. But ill put that 
> in a different mail-threat as im not sure its related and the issues 
> where this thread started are fixed.


I tried a case never encountered before. When the data sent is bigger than
an haproxy buffer, the send function is absolutely bugged, so I submit
some additional patches.

BR,
Thierry



> Regards,
> 
> PiBa-NL (Pieter)
> 
>From 4f8fb1ea81e9e33005f52754775b6dada035bf22 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER <thierry.fourn...@ozon.io>
Date: Fri, 25 May 2018 14:38:57 +0200
Subject: [PATCH 1/5] BUG/MEDIUM: lua/socket: wrong scheduling for sockets

The appctx pointer inherit from pre-thread dev. The thread gives new
link system called xref which maintain a link between the socket ignitor
and the applet. appctx is given at once at the start fo the function.

Inheritage import wrong method for getting the appctx. This implies
the wakeup of wrong applet, and the socket are no longer responsive.

This behavior is hidden by another inherited error whic is fixed in the
next patch.

This patch remove all wrong appctx affectations.

This patch must be backported in 1.6, 1.7 and 1.8
---
 src/hlua.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/src/hlua.c b/src/hlua.c
index 4de5db5ac..03e961c37 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -1799,7 +1799,6 @@ no_peer:
 
 connection_empty:
 
-	appctx = objt_appctx(s->si[0].end);
 	if (!notification_new(>com, >ctx.hlua_cosocket.wake_on_read, hlua->task)) {
 		xref_unlock(>xref, peer);
 		WILL_LJMP(luaL_error(L, "out of memory"));
@@ -1951,7 +1950,6 @@ static int hlua_socket_write_yield(struct lua_State *L,int status, lua_KContext
 	 * the request buffer if its not required.
 	 */
 	if (s->req.buf->size == 0) {
-		appctx = hlua->task->context;
 		if (!channel_alloc_buffer(>req, >buffer_wait))
 			goto hlua_socket_write_yield_return;
 	}
@@ -1959,7 +1957,6 @@ static int hlua_socket_write_yield(struct lua_State *L,int status, lua_KContext
 	/* Check for avalaible space. */
 	len = buffer_total_space(s->req.buf);
 	if (len <= 0) {
-		appctx = objt_appctx(s->si[0].end);
 		if (!notification_new(>com, >ctx.hlua_cosocket.wake_on_write, hlua->task)) {
 			xref_unlock(>xref, peer);
 			WILL_LJMP(luaL_error(L, "out of memory"));
-- 
2.16.3

>From cce5f285c78c3bacd24212ccfca45a460dc50090 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER <thierry.fourn...@ozon.io>
Date: Fri, 25 May 2018 15:03:50 +0200
Subject: [PATCH 2/5] BUG/MAJOR: lua: Dead lock with sockets

In some cases, when we are waiting for data and the socket
timeout expires, we have a dead lock. The Lua socket locks
the applet socket, and call for a notify. The notify
immediat

Re: [PATCH] Re: 1.8.8 & 1.9dev, lua, xref_get_peer_and_lock hang / 100% cpu usage after restarting haproxy a few times

[Thierry Fournier]

> On 26 May 2018, at 19:47, PiBa-NL <piba.nl@gmail.com> wrote:
> 
> Hi Thierry,
> 
> Op 25-5-2018 om 15:40 schreef Thierry FOURNIER:
>> On Fri, 18 May 2018 22:17:00 +0200
>> PiBa-NL <piba.nl@gmail.com> wrote:
>> 
>>> Hi Thierry,
>>> 
>>> Op 18-5-2018 om 20:00 schreef Thierry FOURNIER:
>>>> Hi Pieter,
>>>> 
>>>> Could you test the attached patch ? It seems to fix the problem, but I
>>>> have some doubts about the reliability of the patch.
>>>> 
>>>> Thierry
>>> The crash seems 'fixed' indeed.. but the lua scipt below now takes
>>> 5seconds instead of 150ms.
>>> 
>>> Regards,
>>> PiBa-NL (Pieter)
>>> 
>>> con = core.tcp()
>>> con:connect_ssl("216.58.212.132",443) --google: 216.58.212.132
>>> request = [[GET / HTTP/1.0
>>> 
>>> ]]
>>> con:send(request)
>>> res = con:receive("*a")
>>> con:close()
>> One bug can hide another bug :-) I catch both. Could you test ?
>> 
>> If the result is positive I join also the backport for 1.6 and 1.7
>> 
>> Thierry
> 
> Thanks, seems both the hang and the 2nd uncovered task schedule issue are 
> fixed now (google website response is received/processed fast again). I've 
> done some testing, and installed the updated/patched version on my production 
> box last night. At the moment it still works properly. Activated my lua 
> healthchecker and mailer tasks and enabled 3 threads again.. Lets see how it 
> go's :), but as said for now it seems to work alright.
> 
> Does the second issue you found and fixed clear the initial 'doubts about the 
> reliability' of the first one.? Or did you have a particular possibly 
> problematic scenario in mind that i could try and check for?


Yes. On the first version, I just tried something without
confidence, and this version I read all the socket code,
so I’m confident.


> For the moment i think it is more 'reliable / stable' with the patches than 
> without. So in that regard i think they could be merged.
> 
> There seems to be a issue with tcp:settimeout() though. But ill put that in a 
> different mail-threat as im not sure its related and the issues where this 
> thread started are fixed.


I saw this problem. I tried to understand it, but without success.
The timeout applied seems to be the hardcoded default timeout. I
check this later.

BR,
Thierry

[PATCH] lua socket / Read maount of data

[Thierry Fournier]

Hi,

Actually, I'm working with Lua and Redis, and I found a bug with the
socket function. It impacts the read of a great amount of data when
these data arriving in a lot of network packet.

the Lua user ask for 1 bytes and it receive many more data.

The patch in attachment.

Thierry
>From 933c5120a5f91c09ca426f210fb475e03f3598ca Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER <thierry.fourn...@ozon.io>
Date: Fri, 25 May 2018 16:27:44 +0200
Subject: [PATCH] BUG/MEDIUM: lua/socket: Length required read doesn't work

The limit of data read works only if all the data is in the
input buffer. Otherwise (if the data arrive in chunks), the
total amount of data is not taken in acount.

Only the current read data are compared to the expected amout
of data.

This patch must be backported from 19 to 1.6
---
 src/hlua.c | 12 +++-
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/src/hlua.c b/src/hlua.c
index 02e0beea4..15e9f7e5c 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -1676,6 +1676,7 @@ __LJMP static int hlua_socket_receive_yield(struct lua_State *L, int status, lua
 	struct stream_interface *si;
 	struct stream *s;
 	struct xref *peer;
+	int missing_bytes;
 
 	/* Check if this lua stack is schedulable. */
 	if (!hlua || !hlua->task)
@@ -1745,11 +1746,12 @@ __LJMP static int hlua_socket_receive_yield(struct lua_State *L, int status, lua
 		if (nblk == 0) /* No data avalaible. */
 			goto connection_empty;
 
-		if (len1 > wanted) {
+		missing_bytes = wanted - socket->b.n;
+		if (len1 > missing_bytes) {
 			nblk = 1;
-			len1 = wanted;
-		} if (nblk == 2 && len1 + len2 > wanted)
-			len2 = wanted - len1;
+			len1 = missing_bytes;
+		} if (nblk == 2 && len1 + len2 > missing_bytes)
+			len2 = missing_bytes - len1;
 	}
 
 	len = len1;
@@ -1771,7 +1773,7 @@ __LJMP static int hlua_socket_receive_yield(struct lua_State *L, int status, lua
 	 */
 	if (wanted == HLSR_READ_ALL)
 		goto connection_empty;
-	else if (wanted >= 0 && len < wanted)
+	else if (wanted >= 0 && socket->b.n < wanted)
 		goto connection_empty;
 
 	/* Return result. */
-- 
2.16.3

[PATCH] Re: 1.8.8 & 1.9dev, lua, xref_get_peer_and_lock hang / 100% cpu usage after restarting haproxy a few times

[Thierry FOURNIER]

On Fri, 18 May 2018 22:17:00 +0200
PiBa-NL <piba.nl@gmail.com> wrote:

> Hi Thierry,
> 
> Op 18-5-2018 om 20:00 schreef Thierry FOURNIER:
> > Hi Pieter,
> >
> > Could you test the attached patch ? It seems to fix the problem, but I
> > have some doubts about the reliability of the patch.
> >
> > Thierry
> The crash seems 'fixed' indeed.. but the lua scipt below now takes 
> 5seconds instead of 150ms.
> 
> Regards,
> PiBa-NL (Pieter)
> 
> con = core.tcp()
> con:connect_ssl("216.58.212.132",443) --google: 216.58.212.132
> request = [[GET / HTTP/1.0
> 
> ]]
> con:send(request)
> res = con:receive("*a")
> con:close()

One bug can hide another bug :-) I catch both. Could you test ?

If the result is positive I join also the backport for 1.6 and 1.7

Thierry
>From 498b9d4e76b599e533ff31d0d3c14dd1726d00f5 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER <thierry.fourn...@ozon.io>
Date: Fri, 25 May 2018 14:38:57 +0200
Subject: [PATCH 1/2] BUG/MEDIUM: lua/socket: wrong scheduling for sockets

The appctx pointer inherit from pre-thread dev. The thread gives new
link system called xref which maintain a link between the socket ignitor
and the applet. appctx is given at once at the start fo the function.

Inheritage import wrong method for getting the appctx. This implies
the wakeup of wrong applet, and the socket are no longer responsive.

This behavior is hidden by another inherited error whic is fixed in the
next patch.

This patch remove all wrong appctx affectations.

This patch must be backported in 1.6, 1.7 and 1.8
---
 src/hlua.c | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/src/hlua.c b/src/hlua.c
index a476bcc3d..f8d3836ef 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -1689,6 +1689,8 @@ __LJMP static int hlua_socket_receive_yield(struct lua_State *L, int status, lua
 	if (!socket->s)
 		goto connection_closed;
 
+	appctx = objt_appctx(socket->s->si[0].end);
+
 	oc = >s->res;
 	if (wanted == HLSR_READ_LINE) {
 		/* Read line. */
@@ -1785,7 +1787,6 @@ connection_closed:
 
 connection_empty:
 
-	appctx = objt_appctx(socket->s->si[0].end);
 	if (!hlua_com_new(hlua, >ctx.hlua.wake_on_read))
 		WILL_LJMP(luaL_error(L, "out of memory"));
 	WILL_LJMP(hlua_yieldk(L, 0, 0, hlua_socket_receive_yield, TICK_ETERNITY, 0));
@@ -1894,6 +1895,8 @@ static int hlua_socket_write_yield(struct lua_State *L,int status, lua_KContext
 		return 1;
 	}
 
+	appctx = objt_appctx(socket->s->si[0].end);
+
 	/* Update the input buffer data. */
 	buf += sent;
 	send_len = buf_len - sent;
@@ -1906,7 +1909,6 @@ static int hlua_socket_write_yield(struct lua_State *L,int status, lua_KContext
 	 * the request buffer if its not required.
 	 */
 	if (socket->s->req.buf->size == 0) {
-		appctx = hlua->task->context;
 		if (!channel_alloc_buffer(>s->req, >buffer_wait))
 			goto hlua_socket_write_yield_return;
 	}
@@ -1914,7 +1916,6 @@ static int hlua_socket_write_yield(struct lua_State *L,int status, lua_KContext
 	/* Check for avalaible space. */
 	len = buffer_total_space(socket->s->req.buf);
 	if (len <= 0) {
-		appctx = objt_appctx(socket->s->si[0].end);
 		if (!hlua_com_new(hlua, >ctx.hlua.wake_on_write))
 			WILL_LJMP(luaL_error(L, "out of memory"));
 		goto hlua_socket_write_yield_return;
-- 
2.16.3

>From d61022902ff1489f9343a3f430ad1b2b9efd23c7 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER <thierry.fourn...@ozon.io>
Date: Fri, 25 May 2018 15:03:50 +0200
Subject: [PATCH 2/2] BUG/MAJOR: lua: Dead lock with sockets

In some cases, when we are waiting for data and the socket
timeout expires, we have a dead lock. The Lua socket locks
the applet socket, and call for a notify. The notify
immediately executes code and try to acquire the same lock,
so ... dead lock.

stream_int_notify() cant be used because it wakeup the applet
task only if the stream have changes. The changes are forces
by Lua, but not repported on the stream.

stream_int_update_applet() cant be used because the deadlock.

So, I inconditionnaly wakeup the applet. This wake is performed
asynchronously, and will call a stream_int_notify().

This patch must be backported in 1.6, 1.7 and 1.8
---
 src/hlua.c | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/src/hlua.c b/src/hlua.c
index f8d3836ef..de5afa6c5 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -1759,8 +1759,7 @@ __LJMP static int hlua_socket_receive_yield(struct lua_State *L, int status, lua
 	bo_skip(oc, len + skip_at_end);
 
 	/* Don't wait anything. */
-	stream_int_notify(>s->si[0]);
-	stream_int_update_applet(>s->si[0]);
+	appctx_wakeup(appctx);
 
 	/* If the pattern reclaim to read all the data
 	 * in the connection, got out.
@@ -1942,8 +1941,7 @@ static int hlua_socket_write_yield(struct lua_State *L,int status,

Re: [PATCH] lua & threads

[Thierry Fournier]

> On 22 May 2018, at 19:03, Willy Tarreau <w...@1wt.eu> wrote:
> 
> Hi Thierry,
> 
> On Mon, May 21, 2018 at 07:58:01PM +0200, Thierry Fournier wrote:
>> Hi,
>> 
>> You will two patches in attachment.
>> 
>> - The first fix some Lua error messages
> 
> thanks, I've merged this one already.
> 
>> - The second fix a build error. This second should be reviewed because, I'm 
>> not
>>   so proud of solution :-) Note that this build error happens for compilation
>>   without threads on macosx.
> 
> In my opinion this one looks wrong. Apparently there's a special case for
> all_threads_mask when set to zero to indicate that no threads are enabled,
> and it bypasses any such checks, which is better than setting it to ULONG_MAX.
> 
> I *suspect* it doesn't have any impact for now, except that since code relies
> on !all_threads_mask it can progressively spread and break again later. So
> please check by setting it to 0UL and if it works that's OK.


I do not observe error during runtime, my only one problem is the
compilation. I don’t understand the impact of these modification,
and so I can’t test, because I don’t known the impact on the
polling.

The only one function impacted is “done_update_polling()” in proto/fd.h
which hangs during compilation without threads.

In other way, I so not like the remplacement of a variable by a define.
It can be create hard situation in the future, like which will work only
in thread case.

   long *ptr = _threads_mask


My patch have for goal to shows the error and not fix it.

BR,
Thierry

[PATCH] lua & threads

[Thierry Fournier]

Hi,

You will two patches in attachment.

 - The first fix some Lua error messages

 - The second fix a build error. This second should be reviewed because, I’m not
   so proud of solution :-) Note that this build error happens for compilation
   without threads on macosx.

BR,
Thierry



0001-MINOR-lua-Improve-error-message.patch
Description: Binary data


0002-BUG-MINOR-thread-build-The-variable-all_threads_mask.patch
Description: Binary data

Re: 1.8.8 & 1.9dev, lua, xref_get_peer_and_lock hang / 100% cpu usage after restarting haproxy a few times

[Thierry FOURNIER]

Hi Pieter,

Could you test the attached patch ? It seems to fix the problem, but I
have some doubts about the reliability of the patch.

Thierry




On Fri, 11 May 2018 20:13:25 +0200
PiBa-NL <piba.nl@gmail.com> wrote:

> Hi Thierry,
> 
> Okay found a simple reproduction with tcploop with a 6 second delay in 
> there and a short sleep before calling kqueue.
> 
> ./tcploop 81 L W N20 A R S:"response1\r\n" R P6000 S:"response2\r\n" R [ 
> F K ]
> 
>   gettimeofday(_poll, NULL);
> +    usleep(100);
>   status = kevent(kqueue_fd[tid], // int kq
> 
> Together with the attached config the issue is reproduced every time the 
> /myapplet url is requested.
> 
> Output as below:
> :stats.clihdr[0007:]: Accept-Language: 
> nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7
> [info] 130/195936 (76770) : Wait for it..
> [info] 130/195937 (76770) : Wait response 2..
>    xref_get_peer_and_lock xref->peer == 1
> 
> Hope this helps to come up with a solution..
> 
> Thanks in advance,
> PiBa-NL (Pieter)
> 
> Op 9-5-2018 om 19:47 schreef PiBa-NL:
> > Hi Thierry,
> >
> > Op 9-5-2018 om 18:30 schreef Thierry Fournier:
> >> It seems a dead lock, but you observe a loop. 
> > Effectively it is a deadlock, it keeps looping over these few lines of 
> > code below from xref.h 
> > <http://git.haproxy.org/?p=haproxy.git;a=blob_plain;f=include/common/xref.h;hb=29d698040d6bb56b29c036aeba05f0d52d8ce94b>..
> >  
> > The XCHG just swaps the 2 values (both are '1') and continues on, then 
> > the local==BUSY check is true it loops and swaps 1 and 1 again, and 
> > the circle continues..
> >
> > Thanks for looking into it :) Ill try and get 'simpler' reproduction 
> > with some well placed sleep() as you suggest.
> > Regards,
> > PiBa-NL
> >
> > http://git.haproxy.org/?p=haproxy.git;a=blob;f=include/common/xref.h;h=6dfa7b62758dfaebe12d25f66aaa858dc873a060;hb=29d698040d6bb56b29c036aeba05f0d52d8ce94b
> >  
> >
> 
>From ea4ad78120759351f203797ca263b8247edd88a5 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER <thierry.fourn...@ozon.io>
Date: Fri, 18 May 2018 16:22:20 +0200
Subject: [PATCH 1/2] BUG/MEDIUM: lua: Dead-lock with socket

In some cases, ... fill later
---
 src/hlua.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/src/hlua.c b/src/hlua.c
index 8cc305138..15a5397e1 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -1781,7 +1781,6 @@ __LJMP static int hlua_socket_receive_yield(struct lua_State *L, int status, lua
 	co_skip(oc, len + skip_at_end);
 
 	/* Don't wait anything. */
-	stream_int_notify(>si[0]);
 	stream_int_update_applet(>si[0]);
 
 	/* If the pattern reclaim to read all the data
@@ -2004,7 +2003,6 @@ static int hlua_socket_write_yield(struct lua_State *L,int status, lua_KContext
 	}
 
 	/* update buffers. */
-	stream_int_notify(>si[0]);
 	stream_int_update_applet(>si[0]);
 
 	s->req.rex = TICK_ETERNITY;
-- 
2.16.3

Re: Show: h-app-proxy – Application server inside haproxy

[Thierry FOURNIER]

On Fri, 18 May 2018 16:30:46 +0200
Tim Düsterhus <t...@bastelstu.be> wrote:

> Thierry,
> 
> Am 18.05.2018 um 12:47 schrieb Thierry FOURNIER:
> > Hi,
> > 
> > This is a great usage of Lua. I add a link on my own blog:
> > 
> >http://blog.arpalert.org/p/interesting-links.html
> 
> Thank you! Don't forget to fix the typo in your redis connection pool I
> mentioned in my mail.


Thanks, I missed it. Done.

> 
> Best regards
> Tim Düsterhus

Re: Modifying TCP payload via LUA action & yield issues

[thierry . fournier]

Hi,

I can reproduce the bug, forgot my previous workaround proposition and just add 
an inspect delay:

   
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-tcp-request%20inspect-delay
   tcp-request inspect-delay 10s

If some actions can't be executed because it have no data, haproxy wait
XX seconds for more data. In this case, the yield is allowed during at
mean 10s.

Thierry



On Fri, 18 May 2018 14:22:20 +0200
thierry.fourn...@arpalert.org wrote:

> Could you change your configuration to test a workaround ?
> 
> After the line:
> 
>tcp-request content lua.enrich_query if !has_client_version
> 
> Add a fake action:
> 
>tcp-request set-var(req.wa) int(0)
> 
> Tell me if this work around fix the issue.
> 
> Thierry
> 
> 
> On Tue, 15 May 2018 01:11:46 +
> Grant Byers  wrote:
> 
> > Hi all,
> > 
> > Is it at all feasible to modify a TCP payload via a custom LUA action
> > in a stable manner? ie, something like this;
> > 
> > 
> > function enrich_query(txn)
> > --- Duplicate request. :set() will replace the request, if possible
> > local req = txn.req:dup()
> > 
> > -- if error or client closes connection, quit
> > if req == nil then return end
> > 
> > --- Add an identifier for haproxy and include client's IP address.
> > if txn.req:set("-Vha1.8," .. txn.f:src() .. " " .. req) == -1 then
> > txn.Error(txn, "Failed to enrich query")
> > end
> > end
> > 
> > core.register_action('enrich_query', {'tcp-req'}, enrich_query)
> > 
> > 
> > Then called as an action in the frontend;
> > 
> > # Don't enrich if query already contains client identification
> > acl has_client_version req.payload(0,0) -m sub -- -V --client
> > 
> > # Enrich our query
> > tcp-request content lua.enrich_query if !has_client_version
> > WAIT_END
> > 
> > 
> > The reason I ask is that I randomly get yield errors, I assume during
> > the txn.req:set() ;
> > 
> > Lua function 'enrich_query': yield not allowed.
> > 
> > 
> > I have experimented with different values of tune.lua.forced-yield with
> > little success. Sometimes this occurs very infrequently, other times,
> > very frequently.
> > 
> > FYI - I have experimented with PROXY, but our backend application is
> > using a java API that doesn't have native PROXY support (although
> > there's been a patch pending for about 5 years). Have also experimented
> > with PROXY and Cloudflare's "mmproxy". That works well, but i've been
> > asked to use LUA to extend haproxy if possible.
> > 
> > 
> > Thanks,
> > Grant
> 

Re: Modifying TCP payload via LUA action & yield issues

[thierry . fournier]

Could you change your configuration to test a workaround ?

After the line:

   tcp-request content lua.enrich_query if !has_client_version

Add a fake action:

   tcp-request set-var(req.wa) int(0)

Tell me if this work around fix the issue.

Thierry


On Tue, 15 May 2018 01:11:46 +
Grant Byers  wrote:

> Hi all,
> 
> Is it at all feasible to modify a TCP payload via a custom LUA action
> in a stable manner? ie, something like this;
> 
> 
> function enrich_query(txn)
> --- Duplicate request. :set() will replace the request, if possible
> local req = txn.req:dup()
> 
> -- if error or client closes connection, quit
> if req == nil then return end
> 
> --- Add an identifier for haproxy and include client's IP address.
> if txn.req:set("-Vha1.8," .. txn.f:src() .. " " .. req) == -1 then
> txn.Error(txn, "Failed to enrich query")
> end
> end
> 
> core.register_action('enrich_query', {'tcp-req'}, enrich_query)
> 
> 
> Then called as an action in the frontend;
> 
> # Don't enrich if query already contains client identification
> acl has_client_version req.payload(0,0) -m sub -- -V --client
> 
> # Enrich our query
> tcp-request content lua.enrich_query if !has_client_version
> WAIT_END
> 
> 
> The reason I ask is that I randomly get yield errors, I assume during
> the txn.req:set() ;
> 
> Lua function 'enrich_query': yield not allowed.
> 
> 
> I have experimented with different values of tune.lua.forced-yield with
> little success. Sometimes this occurs very infrequently, other times,
> very frequently.
> 
> FYI - I have experimented with PROXY, but our backend application is
> using a java API that doesn't have native PROXY support (although
> there's been a patch pending for about 5 years). Have also experimented
> with PROXY and Cloudflare's "mmproxy". That works well, but i've been
> asked to use LUA to extend haproxy if possible.
> 
> 
> Thanks,
> Grant

Re: Modifying TCP payload via LUA action & yield issues

[Thierry FOURNIER]

Hi Grant,

It seems to be a bug. The action should allow yield.

Could you have a method for reproducing the bug ?

Thierry


On Tue, 15 May 2018 01:11:46 +
Grant Byers  wrote:

> Hi all,
> 
> Is it at all feasible to modify a TCP payload via a custom LUA action
> in a stable manner? ie, something like this;
> 
> 
> function enrich_query(txn)
> --- Duplicate request. :set() will replace the request, if possible
> local req = txn.req:dup()
> 
> -- if error or client closes connection, quit
> if req == nil then return end
> 
> --- Add an identifier for haproxy and include client's IP address.
> if txn.req:set("-Vha1.8," .. txn.f:src() .. " " .. req) == -1 then
> txn.Error(txn, "Failed to enrich query")
> end
> end
> 
> core.register_action('enrich_query', {'tcp-req'}, enrich_query)
> 
> 
> Then called as an action in the frontend;
> 
> # Don't enrich if query already contains client identification
> acl has_client_version req.payload(0,0) -m sub -- -V --client
> 
> # Enrich our query
> tcp-request content lua.enrich_query if !has_client_version
> WAIT_END
> 
> 
> The reason I ask is that I randomly get yield errors, I assume during
> the txn.req:set() ;
> 
> Lua function 'enrich_query': yield not allowed.
> 
> 
> I have experimented with different values of tune.lua.forced-yield with
> little success. Sometimes this occurs very infrequently, other times,
> very frequently.
> 
> FYI - I have experimented with PROXY, but our backend application is
> using a java API that doesn't have native PROXY support (although
> there's been a patch pending for about 5 years). Have also experimented
> with PROXY and Cloudflare's "mmproxy". That works well, but i've been
> asked to use LUA to extend haproxy if possible.
> 
> 
> Thanks,
> Grant

Re: Show: h-app-proxy – Application server inside haproxy

[Thierry FOURNIER]

Hi,

This is a great usage of Lua. I add a link on my own blog:

   http://blog.arpalert.org/p/interesting-links.html

Thierry


On Sat, 12 May 2018 11:23:31 +0200
Aleksandar Lazic  wrote:

> Hi Tim.
> 
> Am 11.05.2018 um 20:57 schrieb Tim Düsterhus:
> > Hi list,
> > 
> > I recently experimented with the Lua API to check out it's capabilities
> > and wanted to show off the results:
> > 
> > I implemented a very simple short URL service entirely in haproxy with
> > Redis as it's backend. No backend service needed :-)
> 
> Cool stuff ;-)
> Thanks for sharing.
> 
> > Thanks to Thierry for his Redis Connection Pool implementation:
> > http://blog.arpalert.org/2018/02/haproxy-lua-redis-connection-pool.html
> > 
> > Thierry, note that you made a small typo in your pool: r.release(conn)
> > in renew should read r:release(conn).
> > 
> > Blog post  : https://bl.duesterhus.eu/20180511/
> > GitHub : https://github.com/TimWolla/h-app-roxy
> > Live Demo  : https://bl.duesterhus.eu/20180511/demo/DWhxJf2Gpt
> > Hacker News: https://news.ycombinator.com/item?id=17049715
> >
> > Best regards
> > Tim Düsterhus
> > 
> > PS: Don't use this at home or at work even :-)
> 
> ;-)
> 
> Best regards
> Aleks
> 

Re: [PATCH] BUG/MINOR: lua: Socket.send threw runtime error: 'close' needs 1 arguments.

[thierry . fournier]

Hi Sada,

Thanks for the patch,
Willy, can you apply ?

BR,
Thierry


On Fri, 11 May 2018 11:48:18 -0700
sada  wrote:

> Function `hlua_socke_close` expected exactly one argument on the Lua stack.
> But when `hlua_socket_close` was called from `hlua_socket_write_yield`,
> Lua stack had 3 arguments. So `hlua_socket_close` threw the exception with
> message "'close' needs 1 arguments".
> 
> Introduced new helper function `hlua_socket_close_helper`, which removed the
> Lua stack argument count check and only checked if the first argument was
> a socket.
> 
> This fix should be backported to 1.8, 1.7 and 1.6.
> ---
>  src/hlua.c | 14 ++
>  1 file changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/src/hlua.c b/src/hlua.c
> index d07e8d67..8cc30513 100644
> --- a/src/hlua.c
> +++ b/src/hlua.c
> @@ -1629,14 +1629,12 @@ __LJMP static int hlua_socket_gc(lua_State *L)
>  /* The close function send shutdown signal and break the
>   * links between the stream and the object.
>   */
> -__LJMP static int hlua_socket_close(lua_State *L)
> +__LJMP static int hlua_socket_close_helper(lua_State *L)
>  {
>   struct hlua_socket *socket;
>   struct appctx *appctx;
>   struct xref *peer;
>  
> - MAY_LJMP(check_args(L, 1, "close"));
> -
>   socket = MAY_LJMP(hlua_checksocket(L, 1));
>  
>   /* Check if we run on the same thread than the xreator thread.
> @@ -1659,6 +1657,14 @@ __LJMP static int hlua_socket_close(lua_State *L)
>   return 0;
>  }
>  
> +/* The close function calls close_helper.
> + */
> +__LJMP static int hlua_socket_close(lua_State *L)
> +{
> + MAY_LJMP(check_args(L, 1, "close"));
> + return hlua_socket_close_helper(L);
> +}
> +
>  /* This Lua function assumes that the stack contain three parameters.
>   *  1 - USERDATA containing a struct socket
>   *  2 - INTEGER with values of the macro defined below
> @@ -1990,7 +1996,7 @@ static int hlua_socket_write_yield(struct lua_State 
> *L,int status, lua_KContext
>   if (len == -1)
>   s->req.flags |= CF_WAKE_WRITE;
>  
> - MAY_LJMP(hlua_socket_close(L));
> + MAY_LJMP(hlua_socket_close_helper(L));
>   lua_pop(L, 1);
>   lua_pushinteger(L, -1);
>   xref_unlock(>xref, peer);
> -- 
> 2.17.0
> 
> 
>From 7034eef0ed9afefde6a2ff94ce6acfa63caa1832 Mon Sep 17 00:00:00 2001
From: sada 
Date: Fri, 11 May 2018 11:48:18 -0700
Subject: [PATCH] BUG/MINOR: lua: Socket.send threw runtime error: 'close'
 needs 1 arguments.

Function `hlua_socke_close` expected exactly one argument on the Lua stack.
But when `hlua_socket_close` was called from `hlua_socket_write_yield`,
Lua stack had 3 arguments. So `hlua_socket_close` threw the exception with
message "'close' needs 1 arguments".

Introduced new helper function `hlua_socket_close_helper`, which removed the
Lua stack argument count check and only checked if the first argument was
a socket.

This fix should be backported to 1.8, 1.7 and 1.6.
---
 src/hlua.c | 14 ++
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/src/hlua.c b/src/hlua.c
index 3845a11bb..cb683fa19 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -1629,14 +1629,12 @@ __LJMP static int hlua_socket_gc(lua_State *L)
 /* The close function send shutdown signal and break the
  * links between the stream and the object.
  */
-__LJMP static int hlua_socket_close(lua_State *L)
+__LJMP static int hlua_socket_close_helper(lua_State *L)
 {
 	struct hlua_socket *socket;
 	struct appctx *appctx;
 	struct xref *peer;
 
-	MAY_LJMP(check_args(L, 1, "close"));
-
 	socket = MAY_LJMP(hlua_checksocket(L, 1));
 
 	/* Check if we run on the same thread than the xreator thread.
@@ -1659,6 +1657,14 @@ __LJMP static int hlua_socket_close(lua_State *L)
 	return 0;
 }
 
+/* The close function calls close_helper.
+ */
+__LJMP static int hlua_socket_close(lua_State *L)
+{
+	MAY_LJMP(check_args(L, 1, "close"));
+	return hlua_socket_close_helper(L);
+}
+
 /* This Lua function assumes that the stack contain three parameters.
  *  1 - USERDATA containing a struct socket
  *  2 - INTEGER with values of the macro defined below
@@ -1992,7 +1998,7 @@ static int hlua_socket_write_yield(struct lua_State *L,int status, lua_KContext
 		if (len == -1)
 			s->req.flags |= CF_WAKE_WRITE;
 
-		MAY_LJMP(hlua_socket_close(L));
+		MAY_LJMP(hlua_socket_close_helper(L));
 		lua_pop(L, 1);
 		lua_pushinteger(L, -1);
 		xref_unlock(>xref, peer);
-- 
2.16.3

SPOE patchs

[Thierry FOURNIER]

Hi,

In attachment two patches for SPOE.
The first fix an error message, and the second fix a mistake in the protocol.

BR,
Thierry
>From c0274215ed91e90e127bda790b785b698fb149e7 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER <thierry.fourn...@ozon.io>
Date: Thu, 10 May 2018 16:41:26 +0200
Subject: [PATCH 1/3] BUG/MINOR: spoe: Mistake in error message about SPOE
 configuration

The announced accepted chars are "[a-zA-Z_-.]", but
the real accepted alphabet is "[a-zA-Z0-9_.]".

Numbers are supported and "-" is not supported.

This patch should be backported to 1.8 and 1.7
---
 src/flt_spoe.c | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/flt_spoe.c b/src/flt_spoe.c
index 6ac48fc7d..cc6c55e29 100644
--- a/src/flt_spoe.c
+++ b/src/flt_spoe.c
@@ -3535,7 +3535,7 @@ cfg_parse_spoe_agent(const char *file, int linenum, char **args, int kwm)
 			tmp = args[2];
 			while (*tmp) {
 if (!isalnum(*tmp) && *tmp != '_' && *tmp != '.') {
-	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z_-.] chars.\n",
+	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z0-9_.] chars.\n",
 		 file, linenum, args[0], args[1]);
 	err_code |= ERR_ALERT | ERR_FATAL;
 	goto out;
@@ -3569,7 +3569,7 @@ cfg_parse_spoe_agent(const char *file, int linenum, char **args, int kwm)
 			tmp = args[2];
 			while (*tmp) {
 if (!isalnum(*tmp) && *tmp != '_' && *tmp != '.') {
-	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z_-.] chars.\n",
+	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z0-9_.] chars.\n",
 		 file, linenum, args[0], args[1]);
 	err_code |= ERR_ALERT | ERR_FATAL;
 	goto out;
@@ -3593,7 +3593,7 @@ cfg_parse_spoe_agent(const char *file, int linenum, char **args, int kwm)
 			tmp = args[2];
 			while (*tmp) {
 if (!isalnum(*tmp) && *tmp != '_' && *tmp != '.') {
-	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z_-.] chars.\n",
+	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z0-9_.] chars.\n",
 		 file, linenum, args[0], args[1]);
 	err_code |= ERR_ALERT | ERR_FATAL;
 	goto out;
@@ -3617,7 +3617,7 @@ cfg_parse_spoe_agent(const char *file, int linenum, char **args, int kwm)
 			tmp = args[2];
 			while (*tmp) {
 if (!isalnum(*tmp) && *tmp != '_' && *tmp != '.') {
-	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z_-.] chars.\n",
+	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z0-9_.] chars.\n",
 		 file, linenum, args[0], args[1]);
 	err_code |= ERR_ALERT | ERR_FATAL;
 	goto out;
-- 
2.16.3

>From 67d525fec42cd9739a093d0e53415b5209d1ee4e Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER <thierry.fourn...@ozon.io>
Date: Fri, 18 May 2018 12:25:39 +0200
Subject: [PATCH 3/3] BUG/MEDIUM: spoe: Flags are not encoded in network order

The flags are direct copy of the "unsigned int" in the network stream,
so the stream contains a 32 bits field encoded with the host endian.
 - This is not reliable for stream betwen different architecture host
 - For x86, the bits doesn't correspond to the documentation.

This patch add some precision in the documentation and put the bitfield
in the stream usig network butes order.

Warning: this patch can break compatibility with existing agents.

This patch should be backported in all version supporing SPOE

Original network capture:

   12:28:16.181343 IP 127.0.0.1.46782 > 127.0.0.1.12345: Flags [P.], seq 134:168, ack 59, win 342, options [nop,nop,TS val 2855241281 ecr 2855241281], length 34
   0x:  4500 0056 6b94 4000 4006 d10b 7f00 0001  E..Vk.@.@...
   0x0010:  7f00 0001 b6be 3039 a3d1 ee54 7d61 d6f7  ..09...T}a..
   0x0020:  8018 0156 fe4a  0101 080a aa2f 8641  ...V.J.../.A
   0x0030:  aa2f 8641  001e 0301   010f  ./.A
  ^^
   0x0040:  6368 6563 6b2d 636c 6965 6e74 2d69 7001  check-client-ip.
   0x0050:  0006 7f00 0001   ..

Fixed network capture:

   12:24:26.948165 IP 127.0.0.1.46706 > 127.0.0.1.12345: Flags [P.], seq 4066280627:4066280661, ack 3148908096, win 342, options [nop,nop,TS val 2855183972 ecr 2855177690], length 34
   0x:  4500 0056 0538 4000 4006 3768 7f00 0001  E..V.8@.@.7h
   0x0010:  7f00 0001 b672 3039 f25e 84b3 bbb0 8640  .r09.^.@
   0x0020:  8018 0156 fe4a  0101 080a aa2e a664  ...V.J.d
   0x0030:  aa2e 8dda  001e 0300  0114 010f  
  ^^
   0x0040:  6368 6563 6b2d 636c 6965 6e74 2d69 7001  check-client-ip.
   0x0050:  0006 7f00 0001   ..
---
 contrib/spoa_ex

[PATCH] Badd SPOE message

[Thierry FOURNIER]

Hi List and Christopher,

A little patch in attachment. I tagged it as BUG because the bahavior
doesn't correspond to the error message and I read the code to
understand the problem.

So, it is very minor, and maybe I would be classed as DOC ?

br,
Thierry
>From b8051783f6133b31cb3d0474ad542ab859d27853 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER <thierry.fourn...@ozon.io>
Date: Thu, 10 May 2018 16:41:26 +0200
Subject: [PATCH] BUG/MINOR: spoe: Mistake in error message about SPOE
 configuration

The announced accepted chars are "[a-zA-Z_-.]", but
the real accepted alphabet is "[a-zA-Z0-9_.]".

Numbers are supported and "-" is not supported.

This patch should be backported to 1.8 and 1.7
---
 src/flt_spoe.c | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/flt_spoe.c b/src/flt_spoe.c
index 6ac48fc7d..cc6c55e29 100644
--- a/src/flt_spoe.c
+++ b/src/flt_spoe.c
@@ -3535,7 +3535,7 @@ cfg_parse_spoe_agent(const char *file, int linenum, char **args, int kwm)
 			tmp = args[2];
 			while (*tmp) {
 if (!isalnum(*tmp) && *tmp != '_' && *tmp != '.') {
-	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z_-.] chars.\n",
+	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z0-9_.] chars.\n",
 		 file, linenum, args[0], args[1]);
 	err_code |= ERR_ALERT | ERR_FATAL;
 	goto out;
@@ -3569,7 +3569,7 @@ cfg_parse_spoe_agent(const char *file, int linenum, char **args, int kwm)
 			tmp = args[2];
 			while (*tmp) {
 if (!isalnum(*tmp) && *tmp != '_' && *tmp != '.') {
-	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z_-.] chars.\n",
+	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z0-9_.] chars.\n",
 		 file, linenum, args[0], args[1]);
 	err_code |= ERR_ALERT | ERR_FATAL;
 	goto out;
@@ -3593,7 +3593,7 @@ cfg_parse_spoe_agent(const char *file, int linenum, char **args, int kwm)
 			tmp = args[2];
 			while (*tmp) {
 if (!isalnum(*tmp) && *tmp != '_' && *tmp != '.') {
-	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z_-.] chars.\n",
+	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z0-9_.] chars.\n",
 		 file, linenum, args[0], args[1]);
 	err_code |= ERR_ALERT | ERR_FATAL;
 	goto out;
@@ -3617,7 +3617,7 @@ cfg_parse_spoe_agent(const char *file, int linenum, char **args, int kwm)
 			tmp = args[2];
 			while (*tmp) {
 if (!isalnum(*tmp) && *tmp != '_' && *tmp != '.') {
-	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z_-.] chars.\n",
+	ha_alert("parsing [%s:%d]: '%s %s' only supports [a-zA-Z0-9_.] chars.\n",
 		 file, linenum, args[0], args[1]);
 	err_code |= ERR_ALERT | ERR_FATAL;
 	goto out;
-- 
2.16.3

Re: Haproxy SSO

[thierry . fournier]

On Wed, 9 May 2018 22:02:49 +0100
Andrew Smalley  wrote:

> Hi Thierry
> 
> I saw the packetengine here
> https://www.haproxy.com/documentation/aloha/9-5/packetshield/sso/


Ok. There are "HAProxy Technologies" softwares. Do not hesitate
to contact the company for more information. These components are
not available with opensource HAProxy.

Thierry


> It looks like it's a HAPEE thing only thou.   "sudo apt install
> hapee--spoa-sso" part way down the page
> Andruw Smalley
> 
> Loadbalancer.org Ltd.
> 
> www.loadbalancer.org
> +1 888 867 9504 / +44 (0)330 380 1064
> asmal...@loadbalancer.org
> 
> Leave a Review | Deployment Guides | Blog
> 
> 
> On 9 May 2018 at 22:01,   wrote:
> > On Wed, 9 May 2018 21:51:13 +0100
> > Andrew Smalley  wrote:
> >
> >> Hi Thierry,
> >>
> >> I split the thread as I changed subject to SSO part way through, I
> >> apologize for that.
> >>
> >> Your references to SPOA/SPOE Engines were liked very much. I see the
> >> SPOA examples in the source code just now in the link you provided
> >>
> >> https://www.mail-archive.com/haproxy@formilux.org/msg29093.html
> >>
> >> However the HAproxy makes reference to "packetengine", can I ask what
> >> this is (is it the python part of SSO)
> >
> >
> > I don't known. Where do you see a reference to "packetengine" ?
> >
> > Thierry
> >
> >
> >> Andruw Smalley
> >>
> >> Loadbalancer.org Ltd.
> >>
> >> www.loadbalancer.org
> >> +1 888 867 9504 / +44 (0)330 380 1064
> >> asmal...@loadbalancer.org
> >>
> >> Leave a Review | Deployment Guides | Blog
> >>
> 

Re: Haproxy SSO

[thierry . fournier]

On Wed, 9 May 2018 21:51:13 +0100
Andrew Smalley  wrote:

> Hi Thierry,
> 
> I split the thread as I changed subject to SSO part way through, I
> apologize for that.
> 
> Your references to SPOA/SPOE Engines were liked very much. I see the
> SPOA examples in the source code just now in the link you provided
> 
> https://www.mail-archive.com/haproxy@formilux.org/msg29093.html
> 
> However the HAproxy makes reference to "packetengine", can I ask what
> this is (is it the python part of SSO)


I don't known. Where do you see a reference to "packetengine" ?

Thierry


> Andruw Smalley
> 
> Loadbalancer.org Ltd.
> 
> www.loadbalancer.org
> +1 888 867 9504 / +44 (0)330 380 1064
> asmal...@loadbalancer.org
> 
> Leave a Review | Deployment Guides | Blog
> 

Re: WAF with HA Proxy.

[thierry . fournier]

On Thu, 10 May 2018 02:07:24 +0530
DHAVAL JAISWAL  wrote:

> I would prefer to keep this in front of HAProxy. So that any request comes
> first it will pass through he WAF standard rules and then it will come
> inside.


HAProxy is a very robust component. It block protocol attacks which doesn't
respect HTTP protocol and forward other attacks. In other way, it can block
basic attacks with simple ACL (attacks like http://../../../etc/passwd).

With HAProxy in front component, you can process loadbalancing on your WAFs.
This is useful because WAFs use more CPU than loadbalancers.

BR,
Thierry


> Could you please help me with some more documentation, configuration about
> this. How would I achieve it.
> 
> 
> 
> On Thu, May 10, 2018 at 12:14 AM, Malcolm Turnbull  > wrote:
> 
> > Dhaval,
> >
> > As far as I'm concerned almost everyone on the planet uses mod_security...
> > But most use it with apache & some use it with Nginx...
> > So you can either put it on all of your web servers...
> > Or Put it in-front of HAProxy...
> > Or make an HAProxy[1] sandwich (which is what we do at Loadbalancer.org[2])
> >
> > [1] https://www.haproxy.com/blog/scalable-waf-protection-with-
> > haproxy-and-apache-with-modsecurity/
> > [2] https://www.loadbalancer.org/blog/blocking-invalid-range-
> > headers-using-modsecurity-and-haproxy-ms15-034-cve-2015-1635/
> >
> >
> > Malcolm Turnbull
> >
> > Loadbalancer.org Ltd.
> >
> > www.loadbalancer.org
> >
> >  +44 (0)330 380 1064
> > malc...@loadbalancer.org
> >
> >
> >
> >
> > On 9 May 2018 at 19:21, DHAVAL JAISWAL  wrote:
> > > Looking for open source.
> > >
> > > On Wed, May 9, 2018 at 11:10 PM, Mark Lakes 
> > > wrote:
> > >>
> > >> For commercial purposes, see Signal Sciences Next Gen WAF solution:
> > >> https://www.signalsciences.com/waf-web-application-firewall/
> > >>
> > >>
> > >>
> > >> Mark Lakes
> > >> Sr Software Engineer
> > >> (555) 555-
> > >> Winner: InfoWorld Technology of the Year 2018
> > >>
> > >>
> > >> On Wed, May 9, 2018 at 2:23 AM, DHAVAL JAISWAL 
> > wrote:
> > >>>
> > >>> I am looking for WAF solution with HA Proxy.
> > >>>
> > >>> One which I come to know is with HA Proxy version 1.8.8 + mode
> > security.
> > >>> However, I feel its still on early stage.
> > >>>
> > >>> Any other recommendation for WAF with HA Proxy.
> > >>>
> > >>>
> > >>> --
> > >>> Thanks & Regards
> > >>> Dhaval Jaiswal
> > >>
> > >>
> > >
> > >
> > >
> > > --
> > > Thanks & Regards
> > > Dhaval Jaiswal
> >
> 
> 
> 
> -- 
> Thanks & Regards
> Dhaval Jaiswal

Re: WAF with HA Proxy.

[thierry . fournier]

On Wed, 9 May 2018 21:10:48 +0100
Andrew Smalley <asmal...@loadbalancer.org> wrote:

> Hello Thierry
> 
> Thank you for your response saying it is the SPOE engine that does
> mod_security integration and not the almost correct SPOA that I said.


No, you're right: SPOA is the Agent and the ModSec implemention is an
SPOA. SPOE is the Engine.


> Can I ask how haproxy does the SSO with the SPOE/SPOA Engine?


The SPOE/SPOA is designed for this kind of usage, but I don't heard
about any SPOA soft which implements this kind of functionnality.

I propose four ways:

 - Not easy, but reliable: copy/paste the C SPOA demo agent and modify
   it to perform SSO authentication according with your needs.

 - Easy, but with questionable reliability (because recent dev): I
   submit a few days ago a generic SPOA daemon whoch executes Python
   scripts. Unfortunately, I based my dev on a old HAProxy version
   (1.6 or 1.7), and the agent is not compatible with all SPOP
   (P=Protocol) feature, but i works with 1.8 and 1.9.
  https://www.mail-archive.com/haproxy@formilux.org/msg29093.html
   Once python is executed, you can done authentication with any backend.

 - Hard and not reliable (because new dev): Internal haproxy dev (based
   on the same way than SPOE and Lua socket) which communicates with
   SASL. SASL seems great for SSO authentication: it can process many
   authentication method (HTTP Basic, HTTP Digest) and use many backend:
   PAM, files, passwd, ldap, ...)

 - Easy with some protocols and reliable. Use Lua and socket to
   establish authentication protocol with another server. But some
   limitations prevent the usage of some libraries. The libldap is
   not usable. The usable libs are libs using luasocket, but which
   can be modificated for using haproxy sockets (its the same API
   than luasocket).

BR,
Thierry


> 
> 
> Andruw Smalley
> 
> Loadbalancer.org Ltd.
> 
> www.loadbalancer.org
> +1 888 867 9504 / +44 (0)330 380 1064
> asmal...@loadbalancer.org
> 
> Leave a Review | Deployment Guides | Blog
> 
> 
> On 9 May 2018 at 21:04, Thierry Fournier <thierry.fourn...@arpalert.org> 
> wrote:
> > Hi,
> >
> > I confirm: the modsecurity i done throught SPOE.
> >
> > The limitation are:
> >
> > The limit of the body size analysed is the size of HAProxy buffer (default
> > 16kB, but for my own usage, I configure 1MB)
> >
> >
> > The response is not analysed.
> >
> >
> > BR,
> > Thierry
> >
> >
> > On 9 May 2018, at 21:40, Andrew Smalley <asmal...@loadbalancer.org> wrote:
> >
> > Hi Mark
> >
> > Actually as far as I understand the Haproxy implementation of
> > mod_security integration is not with Lua but with SPOA
> >
> > https://www.haproxy.org/download/1.7/doc/SPOE.txt
> > Andruw Smalley
> >
> > Loadbalancer.org Ltd.
> >
> > www.loadbalancer.org
> > +1 888 867 9504 / +44 (0)330 380 1064
> > asmal...@loadbalancer.org
> >
> > Leave a Review | Deployment Guides | Blog
> >
> >
> > On 9 May 2018 at 20:36, Mark Lakes <mla...@signalsciences.com> wrote:
> >
> > RIght, via lua module it integrates with haproxy.
> > -mark
> >
> >
> >
> >
> > Mark Lakes
> > Sr Software Engineer
> > (555) 555-
> > Winner: InfoWorld Technology of the Year 2018
> >
> >
> > On Wed, May 9, 2018 at 11:43 AM, Jonathan Matthews <cont...@jpluscplusm.com>
> > wrote:
> >
> >
> > On Wed, 9 May 2018 at 18:43, Mark Lakes <mla...@signalsciences.com> wrote:
> >
> >
> > For commercial purposes, see Signal Sciences Next Gen WAF solution:
> > https://www.signalsciences.com/waf-web-application-firewall/
> >
> >
> >
> > That page says it supports "Nginx, Nginx Plus, Apache and IIS". Does it
> > integrate with HAProxy? Via what mechanism?
> >
> > J
> >
> > --
> > Jonathan Matthews
> > London, UK
> > http://www.jpluscplusm.com/contact.html
> >
> >
> >
> >
> >
> 

Re: WAF with HA Proxy.

[Thierry Fournier]

Hi,

I confirm: the modsecurity i done throught SPOE.

The limitation are:

The limit of the body size analysed is the size of HAProxy buffer (default 
16kB, but for my own usage, I configure 1MB)

The response is not analysed.

BR,
Thierry

> On 9 May 2018, at 21:40, Andrew Smalley  wrote:
> 
> Hi Mark
> 
> Actually as far as I understand the Haproxy implementation of
> mod_security integration is not with Lua but with SPOA
> 
> https://www.haproxy.org/download/1.7/doc/SPOE.txt
> Andruw Smalley
> 
> Loadbalancer.org Ltd.
> 
> www.loadbalancer.org
> +1 888 867 9504 / +44 (0)330 380 1064
> asmal...@loadbalancer.org
> 
> Leave a Review | Deployment Guides | Blog
> 
> 
> On 9 May 2018 at 20:36, Mark Lakes  wrote:
>> RIght, via lua module it integrates with haproxy.
>> -mark
>> 
>> 
>> 
>> 
>> Mark Lakes
>> Sr Software Engineer
>> (555) 555-
>> Winner: InfoWorld Technology of the Year 2018
>> 
>> 
>> On Wed, May 9, 2018 at 11:43 AM, Jonathan Matthews 
>> wrote:
>>> 
>>> On Wed, 9 May 2018 at 18:43, Mark Lakes  wrote:
 
 For commercial purposes, see Signal Sciences Next Gen WAF solution:
 https://www.signalsciences.com/waf-web-application-firewall/
>>> 
>>> 
>>> That page says it supports "Nginx, Nginx Plus, Apache and IIS". Does it
>>> integrate with HAProxy? Via what mechanism?
>>> 
>>> J
>>> 
>>> --
>>> Jonathan Matthews
>>> London, UK
>>> http://www.jpluscplusm.com/contact.html
>> 
>> 
> 

Re: 1.8.8 & 1.9dev, lua, xref_get_peer_and_lock hang / 100% cpu usage after restarting haproxy a few times

[Thierry Fournier]

> On 9 May 2018, at 18:30, Thierry Fournier <thierry.fourn...@arpalert.org> 
> wrote:
> 
> 
> 
>> On 8 May 2018, at 00:33, PiBa-NL <piba.nl@gmail.com> wrote:
>> 
>> Hi List, Thierry,
>> 
>> Actually this is not limited to restarts, and also happens with 1.9dev. It 
>> now happens while haproxy was running for a while and no restart was 
>> attempted while running/debugging in my NetBeans IDE..
>> 
>> Root cause imo is that hlua_socket_receive_yield and hlua_socket_release 
>> both try and acquire the same lock.
>> 
>> 
>> For debugging purposes ive added some code in hlua_socket_receive_yield(..) 
>> before the stream_int_notify:
>> 
>>struct channel *ic2 = si_ic(si);
>>struct channel *oc2 = si_oc(si);
>>ha_warning("hlua_socket_receive_yield calling notify peer:%9x  
>> si[0].state:%d oc2.flag:%09x ic2.flag:%09x\n", peer, s->si[0].state, 
>> oc2->flags, ic2->flags);
>>stream_int_notify(>si[0]);
>> 
>> And:
>> static void hlua_socket_release(struct appctx *appctx)
>> {
>>struct xref *peer;
>>if (appctx->ctx.hlua_cosocket.xref.peer > 1)
>>ha_warning("hlua_socket_release peer: %9x %9x\n", 
>> appctx->ctx.hlua_cosocket.xref, appctx->ctx.hlua_cosocket.xref.peer->peer);
>>else
>>ha_warning("hlua_socket_release peer: %9x 0\n", 
>> appctx->ctx.hlua_cosocket.xref);
>> 
>> 
>> And also added code in xref_get_peer_and_lock(..):
>> static inline struct xref *xref_get_peer_and_lock(struct xref *xref)
>> {
>>if (xref->peer == 1) {
>>printf("  xref_get_peer_and_lock xref->peer == 1 \n");
>>}
>> 
>> 
>> This produces the logging:
>> 
>> [WARNING] 127/001127 (36579) : hlua_socket_receive_yield calling notify 
>> peer:  2355590  si[0].state:7 oc2.flag:0c000c220 ic2.flag:00084a024
>> [WARNING] 127/001127 (36579) : hlua_socket_release peer: 1 0
>>  xref_get_peer_and_lock xref->peer == 1 
>> 
>> When xref_get_peer_and_lock is called with a parameter xref->peer value of 1 
>> then it looks like it keeps swapping 1 and 1 until it is not 1, that never 
>> happens..
>> 
>> As for the oc2.flags it contains the CF_SHUTW_NOW.. of which im still not 
>> 100% when that flag is exactly set to get a foolproof reproduction.. but it 
>> happens on pretty much a daily basis for me in production and in test i can 
>> now usually trigger it after a few testruns with no actual traffic passing 
>> along within the first minute of running (healthchecks are performed on 
>> several backend, and a mail or 2 is send by the lua code during this startup 
>> period..).. with the full production config..
>> 
>> Below the stacktrace that comes with it..
>> 
>> xref_get_peer_and_lock (xref=0x802355590) at 
>> P:\Git\haproxy\include\common\xref.h:37
>> hlua_socket_release (appctx=0x802355500) at P:\Git\haproxy\src\hlua.c:1595
>> si_applet_release (si=0x8023514c8) at 
>> P:\Git\haproxy\include\proto\stream_interface.h:233
>> stream_int_shutw_applet (si=0x8023514c8) at 
>> P:\Git\haproxy\src\stream_interface.c:1504
>> si_shutw (si=0x8023514c8) at 
>> P:\Git\haproxy\include\proto\stream_interface.h:320
>> stream_int_notify (si=0x8023514c8) at 
>> P:\Git\haproxy\src\stream_interface.c:465
>> hlua_socket_receive_yield (L=0x80223b388, status=1, ctx=0) at 
>> P:\Git\haproxy\src\hlua.c:1789
>> ?? () at null:
>> ?? () at null:
>> lua_resume () at null:
>> hlua_ctx_resume (lua=0x8022cb800, yield_allowed=1) at 
>> P:\Git\haproxy\src\hlua.c:1022
>> hlua_process_task (task=0x80222a500) at P:\Git\haproxy\src\hlua.c:5556
>> process_runnable_tasks () at P:\Git\haproxy\src\task.c:232
>> run_poll_loop () at P:\Git\haproxy\src\haproxy.c:2401
>> run_thread_poll_loop (data=0x802242080) at P:\Git\haproxy\src\haproxy.c:2463
>> main (argc=4, argv=0x7fffea80) at P:\Git\haproxy\src\haproxy.c:3053
>> 
>> I don't yet have a any idea about the direction of a possible fix.. :(..
>> Issue is that probably the hlua_socket_release should happen, but it doesnt 
>> know what socket / peer it should release at that point.. its in the local 
>> peer variable of the hlua_socket_receive_yield funtion.. Should it be 
>> 'unlocked' before calling stream_int_notify?
> 
> 
> This part of the code is tricky.
> 
> Just a precision:
> 
> - I write the “socket” session an haproxy classic session which is created by
>   Lua code for performing netwo

Re: 1.8.8 & 1.9dev, lua, xref_get_peer_and_lock hang / 100% cpu usage after restarting haproxy a few times

[Thierry Fournier]

> On 8 May 2018, at 00:33, PiBa-NL  wrote:
> 
> Hi List, Thierry,
> 
> Actually this is not limited to restarts, and also happens with 1.9dev. It 
> now happens while haproxy was running for a while and no restart was 
> attempted while running/debugging in my NetBeans IDE..
> 
> Root cause imo is that hlua_socket_receive_yield and hlua_socket_release both 
> try and acquire the same lock.
> 
> 
> For debugging purposes ive added some code in hlua_socket_receive_yield(..) 
> before the stream_int_notify:
> 
> struct channel *ic2 = si_ic(si);
> struct channel *oc2 = si_oc(si);
> ha_warning("hlua_socket_receive_yield calling notify peer:%9x  
> si[0].state:%d oc2.flag:%09x ic2.flag:%09x\n", peer, s->si[0].state, 
> oc2->flags, ic2->flags);
> stream_int_notify(>si[0]);
> 
> And:
> static void hlua_socket_release(struct appctx *appctx)
> {
> struct xref *peer;
> if (appctx->ctx.hlua_cosocket.xref.peer > 1)
> ha_warning("hlua_socket_release peer: %9x %9x\n", 
> appctx->ctx.hlua_cosocket.xref, appctx->ctx.hlua_cosocket.xref.peer->peer);
> else
> ha_warning("hlua_socket_release peer: %9x 0\n", 
> appctx->ctx.hlua_cosocket.xref);
> 
> 
> And also added code in xref_get_peer_and_lock(..):
> static inline struct xref *xref_get_peer_and_lock(struct xref *xref)
> {
> if (xref->peer == 1) {
> printf("  xref_get_peer_and_lock xref->peer == 1 \n");
> }
> 
> 
> This produces the logging:
> 
> [WARNING] 127/001127 (36579) : hlua_socket_receive_yield calling notify peer: 
>  2355590  si[0].state:7 oc2.flag:0c000c220 ic2.flag:00084a024
> [WARNING] 127/001127 (36579) : hlua_socket_release peer: 1 0
>   xref_get_peer_and_lock xref->peer == 1 
> 
> When xref_get_peer_and_lock is called with a parameter xref->peer value of 1 
> then it looks like it keeps swapping 1 and 1 until it is not 1, that never 
> happens..
> 
> As for the oc2.flags it contains the CF_SHUTW_NOW.. of which im still not 
> 100% when that flag is exactly set to get a foolproof reproduction.. but it 
> happens on pretty much a daily basis for me in production and in test i can 
> now usually trigger it after a few testruns with no actual traffic passing 
> along within the first minute of running (healthchecks are performed on 
> several backend, and a mail or 2 is send by the lua code during this startup 
> period..).. with the full production config..
> 
> Below the stacktrace that comes with it..
> 
> xref_get_peer_and_lock (xref=0x802355590) at 
> P:\Git\haproxy\include\common\xref.h:37
> hlua_socket_release (appctx=0x802355500) at P:\Git\haproxy\src\hlua.c:1595
> si_applet_release (si=0x8023514c8) at 
> P:\Git\haproxy\include\proto\stream_interface.h:233
> stream_int_shutw_applet (si=0x8023514c8) at 
> P:\Git\haproxy\src\stream_interface.c:1504
> si_shutw (si=0x8023514c8) at 
> P:\Git\haproxy\include\proto\stream_interface.h:320
> stream_int_notify (si=0x8023514c8) at 
> P:\Git\haproxy\src\stream_interface.c:465
> hlua_socket_receive_yield (L=0x80223b388, status=1, ctx=0) at 
> P:\Git\haproxy\src\hlua.c:1789
> ?? () at null:
> ?? () at null:
> lua_resume () at null:
> hlua_ctx_resume (lua=0x8022cb800, yield_allowed=1) at 
> P:\Git\haproxy\src\hlua.c:1022
> hlua_process_task (task=0x80222a500) at P:\Git\haproxy\src\hlua.c:5556
> process_runnable_tasks () at P:\Git\haproxy\src\task.c:232
> run_poll_loop () at P:\Git\haproxy\src\haproxy.c:2401
> run_thread_poll_loop (data=0x802242080) at P:\Git\haproxy\src\haproxy.c:2463
> main (argc=4, argv=0x7fffea80) at P:\Git\haproxy\src\haproxy.c:3053
> 
> I don't yet have a any idea about the direction of a possible fix.. :(..
> Issue is that probably the hlua_socket_release should happen, but it doesnt 
> know what socket / peer it should release at that point.. its in the local 
> peer variable of the hlua_socket_receive_yield funtion.. Should it be 
> 'unlocked' before calling stream_int_notify?


This part of the code is tricky.

Just a precision:

 - I write the “socket” session an haproxy classic session which is created by
   Lua code for performing network I/O.

 - I write “Lua” session for a classic HAProxy session which executes “Lua”, and
   this “Lua” code communicates with the “socket” session.

This lock have two goal: maintain the link between the Lua session and the 
“socket”
session in a coherent state and ensure the access of the “socket” buffer from 
the
Lua session. When the lock is acquired, the “socket” session can’t be cleaned.

I resume:

 - “Lua” session is executed

 - hlua_socket_receive_yield() acquire a lock

 - hlua_socket_receive_yield() call stream_int_notify()
   NOTE: the function hlua_socket_receive_yield() can’t close the “socket” 
session

 - stream_int_notify() is executed from “Lua” session.

 - stream_int_notify() try to close the “socket” session

 - stream_int_notify() try to acquire the lock which is already acquired by the
   caller function hlua_socket_receive_yield().

It 

Re: [Lua] Using txn.c:

[thierry . fournier]

On Tue, 8 May 2018 21:26:49 +0200
Baptiste  wrote:

> On Tue, May 8, 2018 at 8:17 PM, Baptiste  wrote:
> 
> > Hi All, Thierry,
> >
> > I'm trying to use the converter 'table_http_req_cnt()' from a Lua script,
> > but I'm not successful and so I wonder how I'm supposed to use the
> > converter class (txn.c:)...
> >
> > The documentation misses an example:
> > https://www.arpalert.org/src/haproxy-lua-api/1.8/index.
> > html#converters-class
> >
> > Like for the fetches class:
> > https://www.arpalert.org/src/haproxy-lua-api/1.8/index.html#fetches-class
> >
> > Any help would be appreciated. (HAProxy 1.8, HTTP action context, where I
> > want to pass a string to the converter table_http_req_cnt to read some
> > data from my table).
> >
> > Baptiste
> >
> 
> 
> Replying to myself with the solution: the sample must be set as the first
> argument when calling the converter.
> IE: local myvar = txn.c:table_http_req_cnt(,)
> 
> Hope this helps.


Hi Baptiste,

Lua converters functions are automatically generated from registered
haproxy converters, So converters can't have specific documentation.
The only one rules are:

 - The first argument is the input data.
 - Next arguments are the arguments described in classic HAProxy doc.

So, the following HAProxy configuration example:

   str(test),conv(33)

I written like this in Lua:

   txn.c:conv("test", 33)

If you have some useful example, do not hesitate to improve the
documentation.

BR,
Thierry

Re: [PATCH] BUG/MINOR: lua: schedule socket task when lua connect() is called

[Thierry Fournier]

> On 6 May 2018, at 06:32, Willy Tarreau  wrote:
> 
> Hi Pieter,
> 
> On Sun, May 06, 2018 at 12:00:14AM +0200, PiBa-NL wrote:
>> The parameters like server-address, port and timeout should be set before
>> process_stream task is called to avoid the stream being 'closed' before it
>> got initialized properly. This is most clearly visible when running with
>> tune.lua.forced-yield=1.. So scheduling the task should not be done when 
>> creating the lua socket, but when connect is called. The error 
>> "socket: not yet initialised, you can't set timeouts." would then appear.
>> 
>> Below code for example also shows this issue, as the sleep will
>> yield the lua code:
>>  local con = core.tcp()
>>  core.sleep(1)
>>  con:settimeout(10)
> 
> It seems to make sense indeed. I'll let Thierry check as he's the one who
> really knows if there's a possible impact behind this.


I don’t see any impact, and I don’t remember the reason of the task woken
during init and not during connect().

I agree with Willy, this patch make sense.

In 1.6 version, the task is woken during the “hlua_socket_new” by the function
“stream_new”. I test your code and ... segfault :-) The bug fix for the version
1.6 and 1.7 are available in attachment. The version 1.8 was not affected.

Note that the following code:

>>  local con = core.tcp()
>>  core.sleep(1)
>>  con:settimeout(10)

Can’t work in 1.6 and 1.7. It is bad :-( the function core.tcp() starts the task
without Lua requirement (this is done in “stream_new()” function) and the new
tcp connection disappear during the sleep(). I don’t see any work around. With
1.6, core.tcp() and conn:connect*() must be executed at the same time without
Lua yield :-(

The patch was not complex. I guess that I must remove the task from the run 
queue,
and apply your patch to wake up the task after connect. But the workaround is 
easy,
and I prefer to not modify this code. Any opinion about this ?

From the 1.8 version, the task_wakeup is no longer called in the function
“stream_new()”. The bug exists from the begining of the Lua socket, and it 
changes
to its new form from the commit:

   87787acf724eeaf413393b5fce0047ad74356815
   MEDIUM: stream: make stream_new() allocate its own task

So, I trust your patch, and it must be applied on the master branch and on the
1.8 branch.

BR,
Thierry



0001-BUG-MEDIUM-hlua-sockets-Segfault-raises-if-the-socke.1.6.patch
Description: Binary data


0001-BUG-MEDIUM-hlua-sockets-Segfault-raises-if-the-socke.1.7.patch
Description: Binary data

Re: [PATCH] MINOR: Add server name & puid to LUA Server class.

[Thierry Fournier]

> On 3 May 2018, at 18:49, Willy Tarreau <w...@1wt.eu> wrote:
> 
> On Thu, May 03, 2018 at 03:35:41PM +0200, Thierry Fournier wrote:
>> Ok, I understand ... I hesitate. I think that is better to keep
>> the original type, but the consistency is a good reason to convert
>> to string.
>> 
>> I doubt that integer operations (add, sub, ...) will be performed on
>> this id. So maybe, we can keep string for the consistency.
>> 
>> If anyone have an opinion, do not hesitate...
> 
> OK then I picked it. However I noticed something wrong that I fixed
> and which is wrong in other parts of the code :
> 
>  char buffer[10];
>  (...)
>  snprintf(buffer, sizeof(buffer), "%d", srv->puid);
>  lua_pushstring(L, buffer);
> 
> With UIDs larger than 9 snprintf() will fail and we'll push a
> random string in the buffer. So I fixed it to set the buffer size to
> 12 ("-2147483648"). I'll have to fix the other places in the same code
> part for this as well.
> 
> It's extremely unlikely anyone will notice it since few people use manual
> IDs, but if some automated tool builds these IDs using ranges, this will
> definitely fail.


Thanks Willy.
Thierry

Re: [PATCH] BUG/MINOR: lua: Socket.send causing 'close' needs 1 arguments.

[Thierry Fournier]

> On 3 May 2018, at 19:54, Sadasiva Gujjarlapudi <s...@signalsciences.com> 
> wrote:
> 
> `check_args` checked for the "exact number of" arguments available on the Lua 
> stack as
> given in the second param(nb).


You’re right ! Good catch.

Can you provide the patch with this lot of explanation for the commit ?

Like this: 
http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h=05657bd24ebaf20e5c508a435be9a0830591f033

Thanks,
Thierry


> So `close` was expecting exactly "one" argument on the Lua stack.
> But when `close` was called from xxx_write_yield(), Lua stack had 3 arguments.
> 
> So close threw the exception with message "'close' needs 1 arguments".
> 
> "close_helper" function removed the Lua stack argument count check and only 
> checked
> if the first argument was a socket. 
> 
> 
> void check_args(lua_State *L, int nb, char *fcn)
> {
>   if (lua_gettop(L) == nb)
>   return;
>   WILL_LJMP(luaL_error(L, "'%s' needs %d arguments", fcn, nb));
> }
> ----
> 
> Please let me know if you need more information.
> Thanks,
> Sada.
> 
> 
> On Thu, May 3, 2018 at 12:16 AM, Thierry Fournier <tfourn...@arpalert.org> 
> wrote:
> This function checks that one argument are present in the stack, at the
> bottom of the stack (position 1).
> 
>   MAY_LJMP(check_args(L, 1, "close"));
> 
> The stack is kept between calls, so the position 1 will contains anything.
> The content of the position 1 in the stack is a struct associated with the
> socket object.
> 
> The function close is called byt the function xxx_write() which is a callback
> of the Lua function “send()”. This send function have the same same first
> argument then the Lua function "close()”, so the stack contains a socket 
> object
> at position 1.
> 
> I’m sure because the following line, just after the stack check gets
> this argument:
> 
>   socket = MAY_LJMP(hlua_checksocket(L, 1));
> 
> If your patch works, this is a proof that the stack contains expected value
> at position 1, so removing the stack check doesn’t fix anything.
> 
> Maybe I miss something, or you encounters a bug caused by anything else.
> Do you have a method for reproducing a bug ?
> 
> BR,
> Thierry
> 
> 
> 
> > On 3 May 2018, at 08:56, Thierry Fournier <tfourn...@arpalert.org> wrote:
> > 
> > Hi,
> > 
> > I’m not sure about your patch because the values in the stack are kept.
> > I need to be focus to read it, and unfortunately I’m very busy.
> > I try to check this morning, during the french strikes... 
> > 
> > Thierry
> > 
> > 
> >> On 25 Apr 2018, at 20:26, Sadasiva Gujjarlapudi <s...@signalsciences.com> 
> >> wrote:
> >> 
> >> cc: Haproxy
> >> 
> >> `hlua_socket_close` expected exactly one argument on stack.
> >> But when it was called from `hlua_socket_write_yield`, it found more than 
> >> one argument on stack and threw an error.
> >> 
> >> This patch introduced new helper function `hlua_socket_close_helper`, 
> >> which doesn't check arguments count.
> >> Please let me know if you need more information.
> >> 
> >> Thanks,
> >> Sada.
> >> 
> >> 
> >> On Wed, Apr 25, 2018 at 10:51 AM, Thierry Fournier 
> >> <tfourn...@arpalert.org> wrote:
> >> Hi,
> >> 
> >> do you have a little bit of context. It is not easy to read a patch without
> >> the associated explanation.
> >> 
> >> Thanks,
> >> Thierry
> >> 
> >> 
> >> 
> >>> On 14 Apr 2018, at 01:56, sada <s...@signalsciences.com> wrote:
> >>> 
> >>> ---
> >>> src/hlua.c | 14 ++
> >>> 1 file changed, 10 insertions(+), 4 deletions(-)
> >>> 
> >>> diff --git a/src/hlua.c b/src/hlua.c
> >>> index 60cf8f94..0585a1e7 100644
> >>> --- a/src/hlua.c
> >>> +++ b/src/hlua.c
> >>> @@ -1629,14 +1629,12 @@ __LJMP static int hlua_socket_gc(lua_State *L)
> >>> /* The close function send shutdown signal and break the
> >>> * links between the stream and the object.
> >>> */
> >>> -__LJMP static int hlua_socket_close(lua_State *L)
> >>> +__LJMP static int hlua_socket_close_helper(lua_State *L)
> >>> {
> >>> struct hlua_socket *socket;
> >>> struct appctx *appctx;
> >>> struct xref *peer;
> >>> 
> >>> -   MAY_LJMP(check_args(L

Re: [PATCH] MINOR: Add server name & puid to LUA Server class.

[Thierry Fournier]

> On 3 May 2018, at 14:27, Patrick Hemmer <phem...@stormcloud9.net> wrote:
> 
> 
> 
> On 2018/5/3 02:52, Thierry Fournier wrote:
>>> On 2 May 2018, at 16:49, Willy Tarreau <w...@1wt.eu>
>>>  wrote:
>>> 
>>> Hi Thierry,
>>> 
>>> when you have a moment, could you please give a quick look at these
>>> patches from Patrick so that I know if I can merge them or not ? There
>>> are 2 other ones on the list.
>>> 
>> 
>> Hi Willy and Patrick,
>> 
>> I check it. I don’t understand why you convert the puid in string.
>> You could add directly the ouid integer as is in a Lua variable with
>> the function lua_pushinteger().
>> 
>> Thierry
>> 
> I did this for consistency, as this is how Proxy.uuid behaves. Even though it 
> could return an integer, it converts it to a string and returns that instead.


Ok, I understand ... I hesitate. I think that is better to keep
the original type, but the consistency is a good reason to convert
to string.

I doubt that integer operations (add, sub, ...) will be performed on
this id. So maybe, we can keep string for the consistency.

If anyone have an opinion, do not hesitate...

Thierry


> 
>>> Thanks,
>>> Willy
>>> 
>>> On Sun, Apr 29, 2018 at 02:23:48PM -0400, Patrick Hemmer wrote:
>>> 
>>>> ---
>>>> doc/lua-api/index.rst |  8 
>>>> src/hlua_fcn.c| 14 ++
>>>> 2 files changed, 22 insertions(+)
>>>> 
>>>> 
>>>> 
>>>> diff --git a/doc/lua-api/index.rst b/doc/lua-api/index.rst
>>>> index 7a77e46ee..cdb285957 100644
>>>> --- a/doc/lua-api/index.rst
>>>> +++ b/doc/lua-api/index.rst
>>>> @@ -925,6 +925,14 @@ Server class
>>>> 
>>>>   This class provides a way for manipulating servers and retrieving 
>>>> information.
>>>> 
>>>> +.. js:attribute:: Server.name
>>>> +
>>>> +  Contain the name of the server.
>>>> +
>>>> +.. js:attribute:: Server.puid
>>>> +
>>>> +  Contain the proxy unique identifier of the server.
>>>> +
>>>> .. js:function:: Server.is_draining(sv)
>>>> 
>>>>   Return true if the server is currently draining sticky connections.
>>>> diff --git a/src/hlua_fcn.c b/src/hlua_fcn.c
>>>> index a8d53d45b..280d8e5af 100644
>>>> --- a/src/hlua_fcn.c
>>>> +++ b/src/hlua_fcn.c
>>>> @@ -490,6 +490,8 @@ int hlua_listener_get_stats(lua_State *L)
>>>> 
>>>> int hlua_fcn_new_server(lua_State *L, struct server *srv)
>>>> {
>>>> +  char buffer[10];
>>>> +
>>>>lua_newtable(L);
>>>> 
>>>>/* Pop a class sesison metatable and affect it to the userdata. */
>>>> @@ -498,6 +500,18 @@ int hlua_fcn_new_server(lua_State *L, struct server 
>>>> *srv)
>>>> 
>>>>lua_pushlightuserdata(L, srv);
>>>>lua_rawseti(L, -2, 0);
>>>> +
>>>> +  /* Add server name. */
>>>> +  lua_pushstring(L, "name");
>>>> +  lua_pushstring(L, srv->id);
>>>> +  lua_settable(L, -3);
>>>> +
>>>> +  /* Add server puid. */
>>>> +  lua_pushstring(L, "puid");
>>>> +  snprintf(buffer, sizeof(buffer), "%d", srv->puid);
>>>> +  lua_pushstring(L, buffer);
>>>> +  lua_settable(L, -3);
>>>> +
>>>>return 1;
>>>> }
>>>> 
>>>> 
>>>> 
>> 
> 

Re: LUA Converters should be global

[Thierry Fournier]

Hi Patrick,

You’re right. This question was already ask.

I can’t move the converters in the global context because
some converters linked with a session. for example:

 - capture-req
 - capture-res
 - set-var
 - unset-var

Maybe we can add a flag to each converter to known if they use a session
or no, and import the sessionless converter in a global object ?

Sometime ago, I wrote this message. I do not yet test the proposed
solution, but you can try:

=

Hi, Maybe I'm wrong, but it seems that the server is choosed after the
Lua executions (action or sample fetch), so it is not possible to known
the chossen serveur during Lua phase.

In other way, the choice of the server is not easy and it is not easiy
predictible.

The converters are static functions, and they can run during the init
phase, but there are not accessible. Maybe it have an ugly solution
that consist to create a fake object Converter and use it. The
following code is just a guideline, it is not tested, and the Lua
syntax is not checked.

  -- Get the metable of converters searching in in the Global object
  -- I assume the variable meta_converter contains this metatable
  meta_converter = ...

  -- Create new object which is an array containing specific content
  -- in the slot "0"
  convs[0] = 0
  set_metatable(convs, meta_converter)

  -- Now conv is a Converter object, and maybe it can execute some
  -- converters.
  convs:crc32("test")

I'm afraid that this method doesn't work or in the worst case produce
segfault, but you can try.

In other way, if you are able to procude some line of C, you can export
the hash function from haproxy in a file. These function are autonomous
and doesn't have dependencies. You create your own Lua library
containing these two functions. You will find easyly tutorials.

BR,
Thierry

=


BR,
Thierry


> On 30 Apr 2018, at 22:19, Patrick Hemmer  wrote:
> 
> Right now in LUA, all the HAProxy converters are accessible through the 
> `Converters` class. However this class is only accessible through the TXN 
> class. Can we get this changed so that the Converters class is a global?
> 
> My intent is to be able to call a builtin HAProxy converter from within a 
> custom LUA converter. However the LUA converter prototype does not receive a 
> TXN, so I have no access to the the Converters class. Converters are supposed 
> to be stateless, so I see no reason why they should be accessible only from 
> within the TXN class.
> 
> -Patrick

Re: [PATCH] BUG/MINOR: lua: Socket.send causing 'close' needs 1 arguments.

[Thierry Fournier]

Hi,

I’m not sure about your patch because the values in the stack are kept.
I need to be focus to read it, and unfortunately I’m very busy.
I try to check this morning, during the french strikes... 

Thierry


> On 25 Apr 2018, at 20:26, Sadasiva Gujjarlapudi <s...@signalsciences.com> 
> wrote:
> 
> cc: Haproxy
> 
> `hlua_socket_close` expected exactly one argument on stack.
> But when it was called from `hlua_socket_write_yield`, it found more than one 
> argument on stack and threw an error.
> 
> This patch introduced new helper function `hlua_socket_close_helper`, which 
> doesn't check arguments count.
> Please let me know if you need more information.
> 
> Thanks,
> Sada.
> 
> 
> On Wed, Apr 25, 2018 at 10:51 AM, Thierry Fournier <tfourn...@arpalert.org> 
> wrote:
> Hi,
> 
> do you have a little bit of context. It is not easy to read a patch without
> the associated explanation.
> 
> Thanks,
> Thierry
> 
> 
> 
>> On 14 Apr 2018, at 01:56, sada <s...@signalsciences.com> wrote:
>> 
>> ---
>> src/hlua.c | 14 ++
>> 1 file changed, 10 insertions(+), 4 deletions(-)
>> 
>> diff --git a/src/hlua.c b/src/hlua.c
>> index 60cf8f94..0585a1e7 100644
>> --- a/src/hlua.c
>> +++ b/src/hlua.c
>> @@ -1629,14 +1629,12 @@ __LJMP static int hlua_socket_gc(lua_State *L)
>> /* The close function send shutdown signal and break the
>> * links between the stream and the object.
>> */
>> -__LJMP static int hlua_socket_close(lua_State *L)
>> +__LJMP static int hlua_socket_close_helper(lua_State *L)
>> {
>>  struct hlua_socket *socket;
>>  struct appctx *appctx;
>>  struct xref *peer;
>> 
>> -MAY_LJMP(check_args(L, 1, "close"));
>> -
>>  socket = MAY_LJMP(hlua_checksocket(L, 1));
>> 
>>  /* Check if we run on the same thread than the xreator thread.
>> @@ -1659,6 +1657,14 @@ __LJMP static int hlua_socket_close(lua_State *L)
>>  return 0;
>> }
>> 
>> +/* The close function calls close_helper.
>> + */
>> +__LJMP static int hlua_socket_close(lua_State *L)
>> +{
>> +MAY_LJMP(check_args(L, 1, "close"));
>> +return hlua_socket_close_helper(L);
>> +}
>> +
>> /* This Lua function assumes that the stack contain three parameters.
>> *  1 - USERDATA containing a struct socket
>> *  2 - INTEGER with values of the macro defined below
>> @@ -1990,7 +1996,7 @@ static int hlua_socket_write_yield(struct lua_State 
>> *L,int status, lua_KContext
>>  if (len == -1)
>>  s->req.flags |= CF_WAKE_WRITE;
>> 
>> -MAY_LJMP(hlua_socket_close(L));
>> +MAY_LJMP(hlua_socket_close_helper(L));
>>  lua_pop(L, 1);
>>  lua_pushinteger(L, -1);
>>  xref_unlock(>xref, peer);
>> -- 
>> 2.17.0
>> 
> 
> 

Re: 1.9dev LUA shows partial results from print_r(core.get_info()) after adding headers ?

[Thierry Fournier]

Hi,

I can’t reproduce the bug. I even installed a FreeBSD :-) I add Willy in
copy, maybe he will reproduce it.

Thierry


> On 28 Apr 2018, at 01:36, PiBa-NL  wrote:
> 
> Hi Thierry,
> 
> Op 27-4-2018 om 1:54 schreef PiBa-NL:
>> Hi Thierry,
>> 
>> Op 26-4-2018 om 12:25 schreef thierry.fourn...@arpalert.org:
>>> Your trace shows a corrupted tree. Maybe it is due to the freebsd
>>> architecture and the corruption is no reproductible on Linux ? I do not have
>>> freebsd for testing.
>>> 
>>> Regards,
>>> Thierry
>> 
>> My 'best' reproduction scenario involves around 50 to 100 request or 
>> sometimes less requests total made from my Chrome browser. (over a vpn 
>> connection which adds a little latency.. a ping/reply to the haproxy server 
>> takes +-17ms , maybe that helps reproduce.. :/ )
>> I've changed the lua function 'print_r' to not write anything in its wr() 
>> function, to rule out the the freebsd console / ssh session to be causing 
>> the issue. It stays the same though.
>> 
>> Also been adding some printf statements to how the eb_tree is used, which i 
>> think might be interesting..
>>printf("eb32sc_insert(%d)\n",new);
>>printf("eb32sc_next(%d)  leaf_p: %d \n",eb32, eb32->node.leaf_p);
>>printf("eb32sc_delete(%d)\n",eb32);
>> 
>> The pattern i see here is that usually a task is inserted, the next task is 
>> looked up, and the task gets deleted again.
>> Last round before the crash i see that the task is inserted, then deleted 
>> and then afterwards the next task is being retrieved for that same 'item'.. 
>> Which fails.. Perhaps because it was the last task to run.?. And there is 
>> nolonger a higher root to jump higher up the tree.?. I must admit i don't 
>> fully grasp how the tree is traversed exactly.. I seem to see that at least 
>> for the first task to be put into the tree there is some special handling.. 
>> On the other side, if it aint in the tree, is there technically still a 
>> 'next' item??
>> 
>> Below the last part of that logging, and also attached the complete log from 
>> the start.. Perhaps it gives a clue.?.
>> 
>> Regards,
>> 
>> PiBa-NL (Pieter)
>> 
>> eb32sc_insert(35826016)
>> process_runnable_tasks() active_tasks_mask &= ~tid_bit
>> eb32sc_next(35826016)  leaf_p: 8606272
>> eb32sc_delete(35826016)
>> task_wakeup  35826016  32
>> active_tasks_mask |= t->thread_mask  (35826016)
>> eb32sc_insert(35826016)
>> task_wakeup  35826656  8
>> active_tasks_mask |= t->thread_mask  (35826656)
>> eb32sc_insert(35826656)
>> process_runnable_tasks() active_tasks_mask &= ~tid_bit
>> eb32sc_next(35826656)  leaf_p: 35826656
>> eb32sc_delete(35826656)
>> 0013:TestSite.srvrep[0006:]: HTTP/1.1 200 OK
>> 0013:TestSite.srvhdr[0006:]: Refresh: 1
>> 0013:TestSite.srvhdr[0006:]: Server: haproxy/webstats
>> 0013:TestSite.srvhdr[0006:]: Content-Type: text/html
>> 0013:TestSite.srvhdr[0006:]: Content-Length: 1778
>> eb32sc_delete(35826016)
>> 0013:TestSite.srvcls[0006:]
>> eb32sc_next(35826016)  leaf_p: 0
>> Segmentation fault (core dumped)
>> 
> Tried to dig a little further.. pretty sure this are the steps to the issue. 
> The exact reproduction probably is rather timing sensitive though.
> A tcp connection 'stream task' gets closed, before the applet was done 
> running and this removes the applet task from the tree. This applet task is 
> however what the 'rq_next' is pointing to..
> 
> Below stack seems to be what removes and frees the applet task.. Not exactly 
> sure if this is the exact one causing the crash a little later its not 
> related to below 'log' anyhow..
> 
> hlua_applet_http_release (ctx=0x802238780) at P:\Git\haproxy\src\hlua.c:6668
> si_applet_release (si=0x8022accf0) at 
> P:\Git\haproxy\include\proto\stream_interface.h:234
> stream_int_shutw_applet (si=0x8022accf0) at 
> P:\Git\haproxy\src\stream_interface.c:1506
> si_shutw (si=0x8022accf0) at 
> P:\Git\haproxy\include\proto\stream_interface.h:321
> process_stream (t=0x80222a8c0) at P:\Git\haproxy\src\stream.c:2161
> process_runnable_tasks () at P:\Git\haproxy\src\task.c:236
> run_poll_loop () at P:\Git\haproxy\src\haproxy.c:2404
> run_thread_poll_loop (data=0x8022420a0) at P:\Git\haproxy\src\haproxy.c:2469
> main (argc=5, argv=0x7fffea68) at P:\Git\haproxy\src\haproxy.c:3060
> 
> 
>*Below some printf output, mostly containing function names, and a few 
> extra inside the process_runnable_tasks loop to show what was the current and 
> next task...
> active_tasks_mask |= t->thread_mask  (35827936)
> eb32sc_insert(35827936)
> process_runnable_tasks() active_tasks_mask &= ~tid_bit
> find rq_next, current task 35827936  rq_current:35827936
> eb32sc_next(35827936)  leaf_p: 8610368
> eb32sc_delete(35827936)
> process task 35827936  rq_next:0
> task_wakeup  35827936  32
> active_tasks_mask |= t->thread_mask  (35827936)
> eb32sc_insert(35827936)
> process_runnable_tasks() active_tasks_mask &= ~tid_bit
> find 

Re: [PATCH] MINOR: Add server name & puid to LUA Server class.

[Thierry Fournier]

> On 2 May 2018, at 16:49, Willy Tarreau  wrote:
> 
> Hi Thierry,
> 
> when you have a moment, could you please give a quick look at these
> patches from Patrick so that I know if I can merge them or not ? There
> are 2 other ones on the list.


Hi Willy and Patrick,

I check it. I don’t understand why you convert the puid in string.
You could add directly the ouid integer as is in a Lua variable with
the function lua_pushinteger().

Thierry

> 
> Thanks,
> Willy
> 
> On Sun, Apr 29, 2018 at 02:23:48PM -0400, Patrick Hemmer wrote:
>> ---
>> doc/lua-api/index.rst |  8 
>> src/hlua_fcn.c| 14 ++
>> 2 files changed, 22 insertions(+)
>> 
>> 
> 
>> diff --git a/doc/lua-api/index.rst b/doc/lua-api/index.rst
>> index 7a77e46ee..cdb285957 100644
>> --- a/doc/lua-api/index.rst
>> +++ b/doc/lua-api/index.rst
>> @@ -925,6 +925,14 @@ Server class
>> 
>>  This class provides a way for manipulating servers and retrieving 
>> information.
>> 
>> +.. js:attribute:: Server.name
>> +
>> +  Contain the name of the server.
>> +
>> +.. js:attribute:: Server.puid
>> +
>> +  Contain the proxy unique identifier of the server.
>> +
>> .. js:function:: Server.is_draining(sv)
>> 
>>  Return true if the server is currently draining sticky connections.
>> diff --git a/src/hlua_fcn.c b/src/hlua_fcn.c
>> index a8d53d45b..280d8e5af 100644
>> --- a/src/hlua_fcn.c
>> +++ b/src/hlua_fcn.c
>> @@ -490,6 +490,8 @@ int hlua_listener_get_stats(lua_State *L)
>> 
>> int hlua_fcn_new_server(lua_State *L, struct server *srv)
>> {
>> +char buffer[10];
>> +
>>  lua_newtable(L);
>> 
>>  /* Pop a class sesison metatable and affect it to the userdata. */
>> @@ -498,6 +500,18 @@ int hlua_fcn_new_server(lua_State *L, struct server 
>> *srv)
>> 
>>  lua_pushlightuserdata(L, srv);
>>  lua_rawseti(L, -2, 0);
>> +
>> +/* Add server name. */
>> +lua_pushstring(L, "name");
>> +lua_pushstring(L, srv->id);
>> +lua_settable(L, -3);
>> +
>> +/* Add server puid. */
>> +lua_pushstring(L, "puid");
>> +snprintf(buffer, sizeof(buffer), "%d", srv->puid);
>> +lua_pushstring(L, buffer);
>> +lua_settable(L, -3);
>> +
>>  return 1;
>> }
>> 
>> 
> 

Re: [PATCH] BUG/MINOR: lua: Socket.send causing 'close' needs 1 arguments.

[Thierry Fournier]

This function checks that one argument are present in the stack, at the
bottom of the stack (position 1).

  MAY_LJMP(check_args(L, 1, "close"));

The stack is kept between calls, so the position 1 will contains anything.
The content of the position 1 in the stack is a struct associated with the
socket object.

The function close is called byt the function xxx_write() which is a callback
of the Lua function “send()”. This send function have the same same first
argument then the Lua function "close()”, so the stack contains a socket object
at position 1.

I’m sure because the following line, just after the stack check gets
this argument:

  socket = MAY_LJMP(hlua_checksocket(L, 1));

If your patch works, this is a proof that the stack contains expected value
at position 1, so removing the stack check doesn’t fix anything.

Maybe I miss something, or you encounters a bug caused by anything else.
Do you have a method for reproducing a bug ?

BR,
Thierry



> On 3 May 2018, at 08:56, Thierry Fournier <tfourn...@arpalert.org> wrote:
> 
> Hi,
> 
> I’m not sure about your patch because the values in the stack are kept.
> I need to be focus to read it, and unfortunately I’m very busy.
> I try to check this morning, during the french strikes... 
> 
> Thierry
> 
> 
>> On 25 Apr 2018, at 20:26, Sadasiva Gujjarlapudi <s...@signalsciences.com> 
>> wrote:
>> 
>> cc: Haproxy
>> 
>> `hlua_socket_close` expected exactly one argument on stack.
>> But when it was called from `hlua_socket_write_yield`, it found more than 
>> one argument on stack and threw an error.
>> 
>> This patch introduced new helper function `hlua_socket_close_helper`, which 
>> doesn't check arguments count.
>> Please let me know if you need more information.
>> 
>> Thanks,
>> Sada.
>> 
>> 
>> On Wed, Apr 25, 2018 at 10:51 AM, Thierry Fournier <tfourn...@arpalert.org> 
>> wrote:
>> Hi,
>> 
>> do you have a little bit of context. It is not easy to read a patch without
>> the associated explanation.
>> 
>> Thanks,
>> Thierry
>> 
>> 
>> 
>>> On 14 Apr 2018, at 01:56, sada <s...@signalsciences.com> wrote:
>>> 
>>> ---
>>> src/hlua.c | 14 ++
>>> 1 file changed, 10 insertions(+), 4 deletions(-)
>>> 
>>> diff --git a/src/hlua.c b/src/hlua.c
>>> index 60cf8f94..0585a1e7 100644
>>> --- a/src/hlua.c
>>> +++ b/src/hlua.c
>>> @@ -1629,14 +1629,12 @@ __LJMP static int hlua_socket_gc(lua_State *L)
>>> /* The close function send shutdown signal and break the
>>> * links between the stream and the object.
>>> */
>>> -__LJMP static int hlua_socket_close(lua_State *L)
>>> +__LJMP static int hlua_socket_close_helper(lua_State *L)
>>> {
>>> struct hlua_socket *socket;
>>> struct appctx *appctx;
>>> struct xref *peer;
>>> 
>>> -   MAY_LJMP(check_args(L, 1, "close"));
>>> -
>>> socket = MAY_LJMP(hlua_checksocket(L, 1));
>>> 
>>> /* Check if we run on the same thread than the xreator thread.
>>> @@ -1659,6 +1657,14 @@ __LJMP static int hlua_socket_close(lua_State *L)
>>> return 0;
>>> }
>>> 
>>> +/* The close function calls close_helper.
>>> + */
>>> +__LJMP static int hlua_socket_close(lua_State *L)
>>> +{
>>> +   MAY_LJMP(check_args(L, 1, "close"));
>>> +   return hlua_socket_close_helper(L);
>>> +}
>>> +
>>> /* This Lua function assumes that the stack contain three parameters.
>>> *  1 - USERDATA containing a struct socket
>>> *  2 - INTEGER with values of the macro defined below
>>> @@ -1990,7 +1996,7 @@ static int hlua_socket_write_yield(struct lua_State 
>>> *L,int status, lua_KContext
>>> if (len == -1)
>>> s->req.flags |= CF_WAKE_WRITE;
>>> 
>>> -   MAY_LJMP(hlua_socket_close(L));
>>> +   MAY_LJMP(hlua_socket_close_helper(L));
>>> lua_pop(L, 1);
>>> lua_pushinteger(L, -1);
>>> xref_unlock(>xref, peer);
>>> -- 
>>> 2.17.0
>>> 
>> 
>> 
> 

Re: [PATCH 1/1] DOC/MINOR: clean up LUA documentation re: servers & array/table.

[Thierry Fournier]

Hi Patrick,

Thanks for cleaning the Lua doc.

Willy, could you apply these patches ?
Theses patch could be backported from 1.8 to 1.6, but it is not so important.

Thierry


> On 2 May 2018, at 03:30, Patrick Hemmer  wrote:
> 
> 
> * A few typos
> * Fix definitions of values which are tables, not arrays.
> * Consistent US English naming for "server" instead of "serveur".
> ---
> doc/lua-api/index.rst | 88
> ++-
> 1 file changed, 45 insertions(+), 43 deletions(-)
> 
> 
> <0001-DOC-MINOR-clean-up-LUA-documentation-re-servers-arra.patch>

Re: 1.9dev LUA shows partial results from print_r(core.get_info()) after adding headers ?

[thierry . fournier]

On Wed, 25 Apr 2018 22:13:46 +0200
PiBa-NL <piba.nl@gmail.com> wrote:

> Hi Thierry,
> 
> Op 25-4-2018 om 11:19 schreef Thierry Fournier:
> > I extracted the part which dumps the ‘core.get_info()’, and I can’t 
> > reproduce
> > the segfault. I attach the extracted code. I use le lastest master branch.
> >
> I'm testing on master branch as well.
> I started over with the extracted bug29.conf and bug29.lua configuration 
> and the coredump still happens for me.
> 
> I do request all pages below more or less simultaneously with Chrome 
> browser, and then after a few seconds the crash happens..
>    http://haproxy:8000/webrequest/
>    http://haproxy:8000/webrequest/
>    http://haproxy:8000/webrequest/
>    http://haproxy:8000/haproxy?stats
> 
> I've also requested both urls with 'wrk' and then it does *not* crash 
> after several thousand requests to both.. There is something strange 
> there.. Though chrome does of course request the /favicon.ico sends way 
> more headers..
> 
> FYI im using :
> FreeBSD freebsd11 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 
> 21 02:08:28 UTC 2017 
> r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
> Also when using nokqueue the issue still happens.. (for some things that 
> helps.. not today.)
> 
> Attached the exact sequence of events when it happened fast. And the 
> coredump backtrace belonging with that output.
> Anything i can try to narrow it down further.? Or perhaps leave it for 
> now.?. as long as i dont output tons of info on console after the last 
> tcp.send it seams to work okay for now..


I can't reproduce it. I tried with many request on the Lua page (1000/s),
in the same time any request on the stat page (1600/s) and request throught
the proxy (1000/s - with an haproxy wich return 302).

HAProxy doesn't want to crash.

Your trace shows a corrupted tree. Maybe it is due to the freebsd
architecture and the corruption is no reproductible on Linux ? I do not have
freebsd for testing.

Regards,
Thierry

Re: 1.9dev LUA core.tcp() cannot be used from different threads

[Thierry Fournier]

> On 26 Apr 2018, at 11:49, Christopher Faulet  wrote:
> 
> Le 25/04/2018 à 20:51, PiBa-NL a écrit :
>> Hi Christopher, Thierry,
>> Op 25-4-2018 om 11:30 schreef Christopher Faulet:
>>> Oh, these tasks can be created before the threads creation... Ok, so
>>> maybe the right way to fix the bug is to registered these tasks
>>> without specific affinity and set it on the current thread the first
>>> time the tasks are woken up.
>>> 
>>> Here is an updated (and untested) patch. Pieter, could you check it
>>> please ?
>>> 
>>> Thierry, is there any way to create cosockets and applets from outside
>>> a lua's task and then manipulate them in the task's context ?
>>> 
>> Thanks, works for me.
>> I've only tested the last patch from Christopher, and that seems to do
>> the trick nicely for my situation.
>> If you guys come up with a different approach i'm happy to try that as
>> well instead.
> 
> Thanks Pieter for you feedback. Thierry, is this patch seems good for you ? 
> Can we merge it ?


Yes. Thanks Christopher.

Thierry

Re: [PATCH] BUG/MINOR: lua: Socket.send causing 'close' needs 1 arguments.

[Thierry Fournier]

Hi,do you have a little bit of context. It is not easy to read a patch withoutthe associated explanation.Thanks,ThierryOn 14 Apr 2018, at 01:56, sada  wrote:---src/hlua.c | 14 ++1 file changed, 10 insertions(+), 4 deletions(-)diff --git a/src/hlua.c b/src/hlua.cindex 60cf8f94..0585a1e7 100644--- a/src/hlua.c+++ b/src/hlua.c@@ -1629,14 +1629,12 @@ __LJMP static int hlua_socket_gc(lua_State *L)/* The close function send shutdown signal and break the* links between the stream and the object.*/-__LJMP static int hlua_socket_close(lua_State *L)+__LJMP static int hlua_socket_close_helper(lua_State *L){	struct hlua_socket *socket;	struct appctx *appctx;	struct xref *peer;-	MAY_LJMP(check_args(L, 1, "close"));-	socket = MAY_LJMP(hlua_checksocket(L, 1));	/* Check if we run on the same thread than the xreator thread.@@ -1659,6 +1657,14 @@ __LJMP static int hlua_socket_close(lua_State *L)	return 0;}+/* The close function calls close_helper.+ */+__LJMP static int hlua_socket_close(lua_State *L)+{+	MAY_LJMP(check_args(L, 1, "close"));+	return hlua_socket_close_helper(L);+}+/* This Lua function assumes that the stack contain three parameters.*  1 - USERDATA containing a struct socket*  2 - INTEGER with values of the macro defined below@@ -1990,7 +1996,7 @@ static int hlua_socket_write_yield(struct lua_State *L,int status, lua_KContext		if (len == -1)			s->req.flags |= CF_WAKE_WRITE;-		MAY_LJMP(hlua_socket_close(L));+		MAY_LJMP(hlua_socket_close_helper(L));		lua_pop(L, 1);		lua_pushinteger(L, -1);		xref_unlock(>xref, peer);-- 2.17.0

Re: [PATCH] BUG/MINOR: lua: Socket.send causing 'close' needs 1 arguments.

[Thierry Fournier]

Hi. I miss this email, I will chek it ASAP.
Thierry

> On 19 Apr 2018, at 19:53, Sadasiva Gujarlapudi  
> wrote:
> 
> Hi,
> Did you get a chance to review the patch.
> 
> Thanks,
> Sada.
> 
> On Fri, Apr 13, 2018 at 4:56 PM, sada  wrote:
> ---
>  src/hlua.c | 14 ++
>  1 file changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/src/hlua.c b/src/hlua.c
> index 60cf8f94..0585a1e7 100644
> --- a/src/hlua.c
> +++ b/src/hlua.c
> @@ -1629,14 +1629,12 @@ __LJMP static int hlua_socket_gc(lua_State *L)
>  /* The close function send shutdown signal and break the
>   * links between the stream and the object.
>   */
> -__LJMP static int hlua_socket_close(lua_State *L)
> +__LJMP static int hlua_socket_close_helper(lua_State *L)
>  {
> struct hlua_socket *socket;
> struct appctx *appctx;
> struct xref *peer;
> 
> -   MAY_LJMP(check_args(L, 1, "close"));
> -
> socket = MAY_LJMP(hlua_checksocket(L, 1));
> 
> /* Check if we run on the same thread than the xreator thread.
> @@ -1659,6 +1657,14 @@ __LJMP static int hlua_socket_close(lua_State *L)
> return 0;
>  }
> 
> +/* The close function calls close_helper.
> + */
> +__LJMP static int hlua_socket_close(lua_State *L)
> +{
> +   MAY_LJMP(check_args(L, 1, "close"));
> +   return hlua_socket_close_helper(L);
> +}
> +
>  /* This Lua function assumes that the stack contain three parameters.
>   *  1 - USERDATA containing a struct socket
>   *  2 - INTEGER with values of the macro defined below
> @@ -1990,7 +1996,7 @@ static int hlua_socket_write_yield(struct lua_State 
> *L,int status, lua_KContext
> if (len == -1)
> s->req.flags |= CF_WAKE_WRITE;
> 
> -   MAY_LJMP(hlua_socket_close(L));
> +   MAY_LJMP(hlua_socket_close_helper(L));
> lua_pop(L, 1);
> lua_pushinteger(L, -1);
> xref_unlock(>xref, peer);
> -- 
> 2.17.0
> 
> 

Re: 1.9dev LUA shows partial results from print_r(core.get_info()) after adding headers ?

[Thierry Fournier]

Hi,


> On 24 Apr 2018, at 21:46, PiBa-NL  wrote:
> 
> Hi Tim,
> 
> Op 24-4-2018 om 14:36 schreef Tim Düsterhus:
>> Hi
>> 
>> Am 23.04.2018 um 22:36 schrieb PiBa-NL:
>>> Is there a bug in my script, or is it more likely that 'something' needs
>>> fixing in the lua api / interaction?
>> I poked around a bit: The cause in this case is the Content-Length
>> header. It causes that haproxy does not use chunked encoding for the output.
>> 
>> My suspicion is some kind of "race condition". It looks like that the
>> applet function does not get scheduled any more, once all data is sent
>> over the wire and thus the output to stdout is not printed in all cases.


It makes sense. Lua processing is regularly interrupted. The interrupt
condition is a number of instructions of Lua executes (this is for preserving
fast processing for other HAProxy task - We don’t want to be blocked anytime
in heavy processing Lua function). Maybe the script is interrupted during the
dump ?

The print_r function doesn’t use HAProxy things. It just write data on stdout
with classic Lua function. So, we don’t have buffer problem or something like
that.

Note: +1 for your usage of the print_r() function :-)


>> I could not reproduce the issue if I added another `applet:send()` below
>> the second print_r. I also could not reproduce the issue if the
>> Content-Length header specifies a length *greater* than the actual
>> length of the content. I could however reproduce it, if the
>> Content-Length header specifies a length *smaller* than the actual
>> length of the content.
>> 
>> Best regards
>> Tim Düsterhus
> 
> Thanks for investigating, i did a little more of my own also :) and got some 
> results (also a crash..).
> As you say found i do string.len(response)+1 and then another send("1") in 
> the end it suddenly runs fine.
> 
> Running from gdb directly allowed me to get a readable backtrace of the crash 
> that seems to happen due to the print_r of the core.get_info() ..
> Requesting the stats page from 1 and the coreinfo from 3 browserwindows.


I extracted the part which dumps the ‘core.get_info()’, and I can’t reproduce
the segfault. I attach the extracted code. I use le lastest master branch.

Note that I do not have the date of this email because my email server is
broken and it date is stick in 2002 (if i set the hour the servers stops…)
:-(



bug29.conf
Description: Binary data


bug29.lua
Description: Binary data

> When just requesting the core.get_info() and NOT printing it to core.Info() 
> it runs fine for a long time.
> Somehow this output method seems related to some serous issue at least when 
> combined with large output from lua.. Perhaps there is a deeper issue?.?
> Technically its probably not advisable to dump such large output's on the 
> console.. But still to crash on that, i didn't expect it, and dumping info on 
> console is the only way i can easily 'debug' the scripts interaction with 
> haproxy..
> 
> Anyhow hoping it can be fixed or even maybe just some string can be truncated 
> at its maximum buffer length.?.
> 
> Regards,
> 
> PiBa-NL (Pieter)
> 
> global
>   nbthread 1
>   lua-load /root/haproxytest/print_r.lua
>   lua-load /root/haproxytest/smtpmailqueue/smtpmailqueue.lua
>   lua-load /root/haproxytest/serverhealthchecker/serverhealthchecker.lua
>   lua-load /root/haproxytest/serverhealth_smtpmail.lua
> 
> defaults
> mode http
> timeout connect 5s
> timeout client 30s
> timeout server 60s
> 
> frontend TestSite
> bind *:80
> 
> acl webrequest path -m beg /webrequest
> http-request use-service lua.testitweb-webrequest if webrequest
> 
> stats enable
> stats admin if TRUE
> stats refresh 1s
> 
> # prevent overloading yourself with loopback requests..
> acl isloopback src 127.0.0.0/8
> http-request deny if isloopback
> 
> default_backend myservers
> 
> backend myservers
> server localSRVa 127.0.0.1:80 check
> server localSRVb 127.0.0.1:81 check inter 20s
> server localSRVc 127.0.0.1:82 check
> 
> # serverhealth_smtpmail.lua ##
> 
> local smtpmailer = Smtpmailqueue("luamailer",5)
> smtpmailer:setserver("127.0.0.1","25")
> 
> local checknotifier = function(subject, message, serverstatecounters, 
> allstates)
> smtpmailer:addmail("haproxy@domain.local","itguy@domain.local","[srv-checker]"..subject,
>  message.."\r\n"..serverstatecounters.."\r\n\r\n"..allstates)
> end
> local mychecker = Serverhealthchecker("hapchecker",3,2,checknotifier)
> 
> testitweb = {}
> testitweb.webrequest = function(applet)
> if string.match(applet['path'],"/webrequest/mailstat") then
> return smtpmailer:webstats(applet)
> end
> 
>   if string.match(applet['path'],"/webrequest/coreinfo") then
> core.Info("# CORE 1")
> local cor = core.get_info()
> print_r(cor)
> core.Info("# CORE 1 ^")
> 
> local resp = ""
> 

Re: [PATCH] BUG/MEDIUM: lua: Fix segmentation fault if a Lua task exits

[Thierry Fournier]

Hi Tim,

Thanks for the patch. You got it ! (and I discover the --scissors :-) )

Willy, You can apply the patch in attachment. This patch should be
backported on 1.8. (1.6 and 1.7 are not impacted by the bug). The
backport is simple.

BR,
Thierry


0001-BUG-MEDIUM-lua-Fix-segmentation-fault-if-a-Lua-task-.patch
Description: Binary data


> On 24 Apr 2018, at 13:56, Tim Duesterhus  wrote:
> 
> Pieter,
> 
> try the attached patch, please.
> 
> Apply with `git am --scissors` to automatically cut the commit message.
> -- >8 --
> Subject: [PATCH] BUG/MEDIUM: lua: Fix segmentation fault if a Lua task exits
> 
> PiBa-NL reported that haproxy crashes with a segmentation fault
> if a function registered using `core.register_task` returns.
> 
> An example Lua script that reproduces the bug is:
> 
>  mytask = function()
>   core.Info("Stopping task")
>  end
>  core.register_task(mytask)
> 
> The Valgrind output is as follows:
> 
>  ==6759== Process terminating with default action of signal 11 (SIGSEGV)
>  ==6759==  Access not within mapped region at address 0x20
>  ==6759==at 0x5B60AA9: lua_sethook (in 
> /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
>  ==6759==by 0x430264: hlua_ctx_resume (hlua.c:1009)
>  ==6759==by 0x43BB68: hlua_process_task (hlua.c:5525)
>  ==6759==by 0x4FED0A: process_runnable_tasks (task.c:231)
>  ==6759==by 0x4B2256: run_poll_loop (haproxy.c:2397)
>  ==6759==by 0x4B2256: run_thread_poll_loop (haproxy.c:2459)
>  ==6759==by 0x41A7E4: main (haproxy.c:3049)
> 
> Add the missing `task = NULL` for the `HLUA_E_OK` case. The error cases
> have been fixed as of 253e53e661c49fb9723535319cf511152bf09bc7 which
> first was included in haproxy v1.8-dev3. This bugfix should be backported
> to haproxy 1.8.
> ---
> src/hlua.c | 1 +
> 1 file changed, 1 insertion(+)
> 
> diff --git a/src/hlua.c b/src/hlua.c
> index aeb0e2d4..5c265d16 100644
> --- a/src/hlua.c
> +++ b/src/hlua.c
> @@ -5530,6 +5530,7 @@ static struct task *hlua_process_task(struct task *task)
>   hlua_ctx_destroy(hlua);
>   task_delete(task);
>   task_free(task);
> + task = NULL;
>   break;
> 
>   case HLUA_E_AGAIN: /* co process or timeout wake me later. */
> -- 
> 2.17.0
> 

Re: 1.9dev LUA core.tcp() cannot be used from different threads

[Thierry Fournier]

Hi Pieter,

Note: The word “task" have sense in HAProxy architecture, and in Lua 
terminology. Its not
easy to write an explanation. So, the task used by Lua will be called "Lua task"

This error is unexpected. It is produced when Lua socket is started on a thread 
an continue
its execution on other thread. Technically, the core.tcp() creates a fake proxy 
to manage
the tcp connexion. This proxy is attached to a task, and the task is attached 
to a thread.
This thread must be the same than the thread of Lua Task.

The task which process HTTP or TCP trafic are sticky to one thread. But, the 
Lua task ...
no. The patch seems easy to write, but I’m afraid by the context.

For Christopher. I need an advisor :-)

In the file hlua.c at line 5602, I start a task with the value 
MAX_THREADS_MASK. I must
fix one thread, but sometimes this code is executed before effective starting, 
and before
start of threads. Maybe, if “tid” is no set, using 0x01, or random bit ? This 
is ugly...

Any idea ?

Pieter: I attach a temporary patch just for validating the source of the 
problem. Could
you try it ?

BR,
Thierry


0001-TEMP-BUG-MINOR-Lua-Task-are-not-attached-to-a-thread.patch
Description: Binary data


> On 23 Apr 2018, at 23:07, PiBa-NL  wrote:
> 
> Hi List, Thierry (LUA maintainer), Christopher (Multi-Threading),
> 
> When im making a tcp connection to a (mail) server from a lua task this error 
> pops up randomly when using 'nbthread 4', the error luckily seems pretty self 
> explanatory, but ill leave that to the threading and lua experts to come up 
> with a fix ;) i think somehow the script or at least its socket commands must 
> be forced to always be executed on the same thread? or perhaps there is 
> another way..
> 
> Also i do wonder how far lua is safe to use at all in a multithreaded 
> program. Or would that become impossible to keep safe.?. But thats a bit 
> offtopic perhaps..
> 
> Line 240: recieve = mailer.receive(mailer, "*l")
> [ALERT] 110/232212 (678) : Lua task: runtime error: 
> /root/haproxytest/test.lua:240: connect: cannot use socket on other thread.
> 
> Line 266:  local mailer = core.tcp()
> Line 267:ret = mailer.connect(mailer, self.mailserver, 
> self.mailserverport)
> [ALERT] 110/232321 (682) : Lua task: runtime error: 
> /root/haproxytest/test.lua:267: connect: cannot use socket on other thread.
> 
> Let me know if there is a patch or something else i can test/check. Or should 
> configure differently.?.
> Thanks in advance.
> 
> Regards,
> 
> PiBa-NL (Pieter)
> 
>  haproxy.conf & lua scripts
> Basically the serverhealth_smtpmail_haproxy.conf 
> 
>  and the files it links to are here:
> https://github.com/PiBa-NL/MyPublicProjects/tree/master/haproxy/lua-scripts
> 
> p.s.
> The 'mailer' code if anyone is interested that was used is written in some 
> 'libraries' ive committed on github link, maybe they are of use to someone 
> else as well :) comments and fixes are welcome ;).. They are 'first versions' 
> but seem functional with limited testing sofar :).
> 

Re: Grrrr.... warnings

[Thierry Fournier]

You can rewrite haproxy with Lua. It was announced one year and one
day ago. It’s time ... :-)

Thierry

-

> On 1 Apr 2018, at 23:24, Willy Tarreau  wrote:
> 
> Can someone tell me how I'm supposed to work around this one ?
> 
> gcc -Iinclude -Iebtree -Wall  -O2 -march=native -g -fno-strict-aliasing 
> -Wdeclaration-after-statement -fwrapv -fno-strict-overflow
> -Wno-unused-label  -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36
> -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY 
> -DCONFIG_HAP_CRYPT -DUSE_ZLIB  -DENABLE_POLL -DENABLE_EPOLL 
> -DUSE_CPU_AFFINITY -DASSUME_SPLICE_WORKS -DUSE_ACCEPT4 -DNETFILTER 
> -DUSE_THREAD -DUSE_OPENSSL  -DUSE_SYSCALL_FUTEX -DUSE_PCRE -I/usr/include  
> -DCONFIG_HAPROXY_VERSION=\"1.9-dev0-495298-299\" 
> -DCONFIG_HAPROXY_DATE=\"2018/03/21\" \
>  -DBUILD_TARGET='"linux2628"' \
>  -DBUILD_ARCH='""' \
>  -DBUILD_CPU='"native"' \
>  -DBUILD_CC='"gcc"' \
>  -DBUILD_CFLAGS='"-O2 -march=native -g -fno-strict-aliasing 
> -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label 
> -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36"' \
>  -DBUILD_OPTIONS='"USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1"' \
>   -c -o src/haproxy.o src/haproxy.c
> 
> src/haproxy.c:1:1: warning: gcc has detected the presence of C code in this 
> file. This language is potentially unsafe as it lets the developer do things 
> that the compiler doesn't understand. Please upgrade to a new language, 
> support will be deprecated in gcc-9. [-Wc-deprecated].
> 
> Thanks,
> Willy
> 

Re: lua socket api settimeout in seconds vs. milliseconds

[Thierry Fournier]

Thank Mark,

Willy, could you apply the attached patch.

For information, I updated doc and add comment in the commit message.

Thierry


0001-MINOR-lua-allow-socket-api-settimeout-to-accept-inte.patch
Description: Binary data


> On 27 Mar 2018, at 06:49, Mark Lakes <mla...@signalsciences.com> wrote:
> 
> Hi Thierry, remade patch for support of number other than integer in 
> hlua_socket_settimeout() ...from latest source so should be able to be 
> applied.
> 
> Desc: instead of accepting only integers, allow user to specify float and 
> double when hlua_socket_settimeout() is called. Round up user specified value 
> and validate against maximum integer to prevent overflow.
> 
> Mark Lakes
> Signal Sciences | www.signalsciences.com |
> 
> 
> On Thu, Mar 8, 2018 at 1:24 AM, Thierry Fournier <tfourn...@arpalert.org> 
> wrote:
> Hi Mark,
> 
> Thanks, it seems perfect. But, I can’t apply on current master branch, the
> patch is rejected.
> 
> I forgot 3 things while my first read:
> 
>  - The Lua error are not trigerred with a return 1 (the return 1 is a bug
>in the original function), but with something like that:
> 
>WILL_LJMP(luaL_error(L, "settimeout: cannot set negatives values"));
> 
>The return became useless because the luaL_error() function never returns.
>(it soes a longjmp call).
> 
>  - I think that timeouts < 1s are allowed. The cli refuse these ones,
>but the HAProxy core is ready to work with timeout less than 1s. Note
>that, When you remove this check, negative timeouts could be accepted
>(the check < 1000 chek also for negative values) and this is a bug.
> 
>  - The doc (doc/lua-api/index.rst) should be updated (actuelly, line 1899:
>.. js:function:: Socket.settimeout(socket, value [, mode])). It will be
>necessary to precise that the fucntion accept Number in place of Integer
>and, number with fractional part are allowed.
> 
> I join two of my fixes. The third patch is yours, check if you agree with
> the content and the commit message.
> 
> BR,
> Thierry
> 
> 
> 
> 
> 
> 
> 
> > On 8 Mar 2018, at 01:17, Mark Lakes <mla...@signalsciences.com> wrote:
> >
> > Hi Thierry, thanks for feedback. Addressed concerns in the new attached 
> > patch.
> >
> > http://w3.impa.br/~diego/software/luasocket/tcp.html#settimeout
> >
> > Description: instead of hlua_socket_settimeout() accepting only integers, 
> > allow user to specify float and
> > double as well. Convert to milliseconds much like cli_parse_set_timeout but 
> > also sanity check the value.
> >
> > -mark
> >
> >
> > On Wed, Mar 7, 2018 at 9:55 AM, Thierry Fournier <tfourn...@arpalert.org> 
> > wrote:
> > Hi Mark,
> >
> > Thanks for the patch. I don’t like usage of floating point, but the
> > luasocket documentation says that the settimeout() function accept only
> > second. In this case, the usage of floating point seems be to be a good
> > way.
> >
> > Can you split in a second commit the fix of comments from the effective
> > patch, and avoid this kind of changes:
> >
> >-int tmout;
> >+inttmout;
> >
> > Just because, this kind of changes are useless, and it add noisy
> > information in the patch.
> >
> > A last point: could you explain int the message of the patch the
> > goal of these patch. To avoid a search, this is the link of the official
> > luasocket setimeout function:
> >
> > http://w3.impa.br/~diego/software/luasocket/tcp.html#settimeout
> >
> > Thanks
> > Thierry
> >
> >
> > > On 7 Mar 2018, at 18:16, Mark Lakes <mla...@signalsciences.com> wrote:
> > >
> > > In regards to earlier conversation, herein is a patch attached for the 
> > > feature.
> > > From the mail archive:
> > > https://www.mail-archive.com/haproxy@formilux.org/msg27806.html
> > > https://www.mail-archive.com/haproxy@formilux.org/msg27807.html
> > >
> > > Mark Lakes
> > > Signal Sciences | www.signalsciences.com |
> > >
> > > conversation participants:
> > > Willy Tarreau
> > > Adis Nezirovic
> > > Nick Galbreath
> > >
> > > - Last conversation and decision agreement --
> > > Nick Galbreath Thu, 09 Nov 2017 20:44:28 -0800
> > >
> > > thanks wily.
> > >
> > > re: " CONTRIBUTING in the sources directory," -
> > >
> > > yes, that is what I was looking for!  thanks for the tip.
> > >

Segfault / 1.8.5 / 9a083d1428b

[Thierry Fournier]

Hi,

I just install the 1.8.5 and I encountered some of coredump. Below one of
these (Unfortunately, I remove the others before checking their content)


#0  0x004a715c in h2_process_mux (h2c=) at
src/mux_h2.c:2125
2125 h2s->cs->data_cb->send(h2s->cs);


(gdb) bt
#0  0x004a715c in h2_process_mux (h2c=) at
src/mux_h2.c:2125
#1  h2_send (conn=) at src/mux_h2.c:2243
#2  0x004fa0da in conn_fd_handler (fd=) at
src/connection.c:128
#3  0x00509414 in fd_process_cached_events () at src/fd.c:271
#4  0x004b7aa1 in run_poll_loop () at src/haproxy.c:2426
#5  run_thread_poll_loop (data=data@entry=0xdf1570) at src/haproxy.c:2461
#6  0x0041b3fd in main (argc=, argv=0x7ffe6074b528)
at src/haproxy.c:3050


(gdb) print h2s
$1 = (struct h2s *) 0x68ce4a0

(gdb) print h2s->cs
$2 = (struct conn_stream *) 0x6e5bf30

(gdb) print h2s->cs->data_cb
$3 = (const struct data_cb *) 0x0


Tell me if you want the binary and the core, I will sent you.

Thierry

[PATCH] server-template and allowed chars in the name

[Thierry Fournier]

Hi list,

The name of the "server" directive allows [A-Za-Z0-9_.:-] (see
function invalid_char()).

The name allowed for the "server-template" directive only allows [A-Za-Z_.-],
the chars [0-9:] disappear.

This limitation forces the modifications of all the server names which
contains numbers and forces to review the monitoring system. This is not
great.

It have a good reason to reject chars [0-9:]  ?
If it is not a case, I join a patch which fix the syntax for [0-9], not for
':'

Thanks,
Thierry


0001-MINOR-servers-Support-alphanumeric-characters-for-th.patch
Description: Binary data

Re: lua socket api settimeout in seconds vs. milliseconds

[Thierry Fournier]

> On 8 Mar 2018, at 22:41, Tim Düsterhus <t...@bastelstu.be> wrote:
> 
> Thierry,
> 
> Am 08.03.2018 um 21:15 schrieb Thierry Fournier:
>> Hey, the example of use of socket.http in attachment of your original
>> commit is great !
> 
> If you are curious what I built with that and in case you missed my list
> mail advertising the project:
> https://github.com/TimWolla/haproxy-auth-request
> 


Interesting. I add a link on my blog: 
http://blog.arpalert.org/p/interesting-links.html


>> 3 new patch in attachement to consider for the initial subject of
>> this thread.
> 
> I took a look at it:
> 
>> The LuaSocket documentation tell anything about the returned value,
>> but the effective code set an integer of value one.
> 
> Should this read: "The LuaSocket documentation __does not__ tell
> anything about the returned value …"?


Yes. I dont see anything about the returned value.


> There's a few more regular typos (spelling of words) in the commit
> messages, but that specific part is misleadingly worded, thus I wanted
> to make you aware of it.


ok, thanks.

BR,
Thierry