Re: Error "line too long" after adding some conditions

[Adis Nezirovic]

On 5/11/21 4:15 PM, Tom wrote:

Using haproxy 2.1.3:
I've configured multiple conditions to a "use_backend" directive like this:

use_backend example_192.168.1.30 if host_test.example.com || 
host_a01.test.example.com || host_a02.test.example.com || 
host_a03.test.example.com || host_a04.test.example.com || ... || ...


When I add some more conditions, then I got an error after reloading the 
daemon:


"line too long, truncating at word 65, position 783:..."


I'm also interested in technical reason for this limitation :), waiting 
for smarter people to speak ;)



How can I configure more conditions in the "use_backend"-directive above 
without having the error?


Maybe you can use named acl, repeat it multiple times:

"With named ACLs, specifying the same ACL name multiple times will cause 
a logical OR of the conditions, so the last block can also be expressed as:


acl evil path_beg /evil/
acl evil path_end /evil
http-request deny if evil
"

Example from our blog post:

https://www.haproxy.com/blog/introduction-to-haproxy-acls/

Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

Re: HAPROXY CAN NOT POINT IN TO PORT 5000 OF PATRONI

[Adis Nezirovic]

On 4/22/21 11:39 AM, Maciej Zdeb wrote:

Hi,
try removing those two lines from config:
option httpchk
http-check expect status 200

it is postgres (tcp backend), you should not expect http response on 
health check.


As far as I can see, this is Patroni style HA setup, where Patroni 
services manage Postgres and detect the current master (i.e. writeable 
server). Hence the http checks (on different port)


https://github.com/zalando/patroni/blob/master/haproxy.cfg

It's probably a problem in general Patroni setup, but that's out of the 
scope of this list. Revert to the original (patroni) provided haproxy 
configuration and check if the backend servers are up. (i.e. open the 
"stats" frontend endpoint in your browser).



Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

Re: Logging down output from the a Lua script

[Adis Nezirovic]

On 3/4/21 9:47 PM, Mihaly Zachar wrote:

If I do this:
applet:set_var('txn.myvar', 'myvar_value')

Then in the HAProxy layer I can reach the variable with %[var(txn.myvar)]
So it DOES work !
But Is this safe ? Did I do it well or I was just lucky ?


Actions expose 'txn', while services expose full 'applet' object, so I 
do think it works as intended, it's not an accident. You are using Lua 
service for redirection?


Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

Re: Logging down output from the a Lua script

[Adis Nezirovic]

On 3/4/21 12:22 PM, Mihaly Zachar wrote:

Sorry, the above might be misunderstood.

I would like to log from the frontend, because some timer values are 
available only there.
So I know that I can send log from the Lua script using core.log() but I 
need to have the information in the frontend.
If you use Lua for action or service (i.e. you have txn object readily 
available in the script), you can use txn:set_var(name, value) to 
communicate stuff back to regular HAProxy layer, and then log the value 
of the variable with rest of the stuff.


Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

Re: [PATCH v2] BUG/MEDIUM: stats: add missing INF_BUILD_INFO definition

[Adis Nezirovic]

On 1/15/21 5:01 PM, Willy Tarreau wrote:

OK thanks. Adis, do you have the ability to recheck for your test case,
or do you want me to pick it right now ?

Thanks,
Willy



Sorry, I've missed that patch from William. That works for me, my Lua 
test case passes.


(maybe I'll invest some time to contribute a few basic Lua tests, so 
things get noted faster).



Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

[PATCH] BUG/MEDIUM: stats: Add missing INF_BUILD_INFO definition

[Adis Nezirovic]

Hi guys,

Recent prometheus exporter patch broke stats (specifically Lua 
core.get_info() maybe other stuff)



The offending commit is:

commit 5a982a71656ce885be4b1d4b90b8db31204788a1
Author: William Dauchy 
Date:   Fri Jan 8 13:18:06 2021 +0100

MINOR: contrib/prometheus-exporter: export build_info


I've attached a trivial patch which fixes the issue (I simply add 
missing definitions, but build info is hardcoded to 1, since I don't 
know where should I get that from).



Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com
>From 6823e172b71590d4c540bddaa36add217828644c Mon Sep 17 00:00:00 2001
From: Adis Nezirovic 
Date: Wed, 13 Jan 2021 19:02:33 +0100
Subject: [PATCH] BUG/MEDIUM: stats: Add missing INF_BUILD_INFO definition

It was introduced in 5a982a7 "MINOR: contrib/prometheus-exporter: ..."
Lua function core.get_info() was broken, probably some other stuff.
---
 src/stats.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/stats.c b/src/stats.c
index 4ddcd7e41..c9359e346 100644
--- a/src/stats.c
+++ b/src/stats.c
@@ -148,6 +148,7 @@ const struct name_desc info_fields[INF_TOTAL_FIELDS] = {
 	[INF_BYTES_OUT_RATE] = { .name = "BytesOutRate",.desc = "Number of bytes emitted by current worker process over the last second" },
 	[INF_DEBUG_COMMANDS_ISSUED]  = { .name = "DebugCommandsIssued", .desc = "Number of debug commands issued on this process (anything > 0 is unsafe)" },
 	[INF_CUM_LOG_MSGS]   = { .name = "CumRecvLogs", .desc = "Total number of log messages received by log-forwarding listeners on this worker process since started" },
+	[INF_BUILD_INFO] = { .name = "Build info",  .desc = "Build info" },
 };

 const struct name_desc stat_fields[ST_F_TOTAL_FIELDS] = {
@@ -3911,6 +3912,7 @@ int stats_fill_info(struct field *info, int len)
 	info[INF_BYTES_OUT_RATE] = mkf_u64(FN_RATE, (unsigned long long)read_freq_ctr(_32bps) * 32);
 	info[INF_DEBUG_COMMANDS_ISSUED]  = mkf_u32(0, debug_commands_issued);
 	info[INF_CUM_LOG_MSGS]   = mkf_u32(FN_COUNTER, cum_log_messages);
+	info[INF_BUILD_INFO] = mkf_u32(FN_GAUGE, 1);

 	return 1;
 }
--
2.30.0

Re: do we want to keep CentOS 6 builds?

[Adis Nezirovic]

On 12/2/20 9:45 PM, Willy Tarreau wrote:

On Wed, Dec 02, 2020 at 10:19:47PM +0500,  ??? wrote:

seems, CentOS 6 packages were removed from mirrors

https://cirrus-ci.com/task/5915513668763648


I've never understood why some distros do something that stupid. They
even prevent some people from setting up a backup server in emergency.

So does this mean we'll drop this one ?


For what it's worth, after EOL, data is moved to CentOS vault:

https://vault.centos.org/6.10/

Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

Re: Count uniq Client ips

[Adis Nezirovic]

On 10/15/20 2:08 PM, Aleksandar Lazic wrote:

Hi.

I though maybe the peers could help me when I yust add the client IP with the 
URL but I'm not sure if I can query the peers store in a efficient way.

The target is to know how much concurrent IP's request the a specific URL.

Could lua be a solution.


Hey Aleks,

I'm not sure Lua would be the right solution for your situation, 
counting stuff is tricky.


However, I think Redis has INCR, you you can store per URL counters and 
maybe (just maybe) use Lua action in HAProxy to write to Redis.


Obviously, you'd need to look out for performance, added latency etc, 
but it would be a start.
You can then access Redis outside of the HAProxy context and observe the 
counters.


Just my 2c, hope it helps you (like you helped many people on this list)

Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

set-path impact on ACL eval and on set-var

[Adis Nezirovic]

Hi guys,

Here is a funny config where I've encountered something unexpected.

The theme is URL rewriting (let's say for some API),
after getting the original URI, extracting fields from path, we want to 
construct new path and save extracted fields into query args (we use 
haproxy vars as storage)


1)
If you run the attached haproxy config as is, everything works as expected

curl -v localhost:8080/a/1/b/2 -o /dev/null 2>&1 |grep x-field

< x-field: /a/1/b/2?a=1=2


curl -v localhost:8080/a/1/d/2 -o /dev/null 2>&1 |grep x-field

< x-field: /a/1/d/2?a=1


2) However, since want to replace path as well, let's
uncomment first set-path from the config file
curl -v localhost:8080/a/1/b/2 -o /dev/null 2>&1 |grep x-field



< x-field: /a/1/b/2?a=1

Not expected!

If we move that set-path statement after set-query (Case 2) it starts 
working fine again.



What's happening? Looks like set-var was lazily evaluated only after 
we've used it in set-query lines guarded by acl.


3) If we replace both set-query lines with

 http-request set-query "a=%[var(txn.a)]=%[var(txn.b)]"

It starts working again, but we'll always see "empty" b in the output:

curl -v localhost:8080/a/1/b2/2 -o /dev/null 2>&1 |grep x-field

< x-field: /x/y/z?a=1=



P.S.
For reference, here is the relevant config part:

  http-request set-var(txn.a) path,field(3,/)

  acl is_b path,field(4,/) -m str "b"

  http-request set-var(txn.b) path,field(5,/) if is_b

  # Case 1, we can't see b value if we set-path here

  # http-request set-path /x/y/z

  http-request set-query "a=%[var(txn.a)]" if !is_b

  http-request set-query "a=%[var(txn.a)]=%[var(txn.b)]" if is_b

  # Case 2, but we can see b value now

  # http-request set-path /x/y/z


Best regards,
Adis
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com
global
log /dev/log local1

defaults
log global
timeout connect 5s
timeout client 10s
timeout server 10s
timeout http-request 1s
mode http

listen blah
bind 127.0.0.1:8080
http-request set-var(txn.a) path,field(3,/)
acl is_b path,field(4,/) -m str "b"
http-request set-var(txn.b) path,field(5,/) if is_b

# Case 1, we can't see b value if we set-path here
http-request set-path /x/y/z
#http-request set-query "a=%[var(txn.a)]" if !is_b
#http-request set-query "a=%[var(txn.a)]=%[var(txn.b)]" if is_b
http-request set-query "a=%[var(txn.a)]=%[var(txn.b)]"
# Case 2, but we can see b value now
#http-request set-path /x/y/z

http-request set-var(txn.url) url
http-response set-header X-Field %[var(txn.url)]

server srv1 haproxy.org:80

Re: Several CVEs in Lua 5.4

[Adis Nezirovic]

On 7/29/20 11:16 AM, Froehlich, Dominik wrote:

Hi Lukas,

Thanks for the reply.
My query goes along the lines of which Lua version is compatible with HAproxy 
and contains fixes to those CVEs.
I could not find a specific instruction as to which Lua version can be used to 
build HAproxy / has been tested for production use.

We are consuming a bundled version (currently HAproxy 1.9.15 with Lua 5.3.5) 
but I don't know if it is safe to bump the Lua version only.


I don't think HAProxy works with Lua 5.4 (yet), there should be another 
recent thread about it. So you are stuck at 5.3.x branch for now.


Best regards,
Adis

Re: [PATCH 0/6] Lua variable handling

[Adis Nezirovic]

Tim,





First, not a a direct comment on your patches, but before I forget, I'd 
like to point out that there is one more thing which need fixing in 
regard to Lua variables:




  TXN.set_var(TXN, var, value)



where "var" is noted as:

  The var name according with the HAProxy syntax



However, nothing in the code checks that variable name is formed 
correctly (with correct prefix, such as "txn."), or that the rest of the 
variable name is correct.




Lua code using set_var() with wrong name silently fails, the variable is 
not set, and nothing is logged

.


With your proposal set_var() now can return result, and we could even 
log warning for "variable name syntax error".



---8<---

Now back on topic. Instead of adding more parameters to set_var(), I'd 
prefer a warning instead.




If someone is using set_var() from Lua, and that variable is never used 
or set in the rest of the config, we would know that. Additionally, one 
can set "zero-warning" option to prevent abuse by Lua scripts or to 
prevent bugs.


For your use case, maybe you could use maps from Lua? They should be 
backed by ebtree and be faster right?



Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

Re: [PATCH] Fix dumping of stick table entries for k/v pairs.

[Adis Nezirovic]

On 5/7/20 2:44 PM, William Lallemand wrote:

On Thu, May 07, 2020 at 02:39:30PM +0200, Christopher Faulet wrote:

Sorry for the delay and thanks for the reminder. I don't really know how stick
table works internally. But you should probably test the dict_entry pointer is
not NULL before accessing its value. It is how it is done in
stable_dump_entry_to_buffer(). If it ok for you, I can amend the commit to do
the same. Let me know.
Ah yes, that's obvious, please amend, I was trimming "unecessary" 
temporary variables, and was left only with unguarded typecast :-)




Also, it should probably be tagged 'BUG/MEDIUM' instead of 'MEDIUM'
This one too, it is clearly a bug which causes user script in Lua to 
fail during StickTable:dump() (and similar)



Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

Re: [PATCH] Fix dumping of stick table entries for k/v pairs.

[Adis Nezirovic]

Hello guys, any comments on the patch?

Bump, adding Thierry to the conversation.


Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

Re: [PATCH] MEDIUM: cli: Allow multiple filter entries for show table

[Adis Nezirovic]

On 1/22/20 6:17 PM, Willy Tarreau wrote:

Strangely, while I was certain I had build-tested the original one, apparently
I failed as it didn't build for two errors, which I fixed in a subsequent patch.
I'm seeing that your patches still seem to rely on this bug (data_op < 0) which
normally cannot compile so I find this surprising.


I've using clang during the testing, and apparently -Wextra doesn't 
enable it on clang (I'm using v9.0), only -pedantic helps:


clang -Wall -Wextra -Wpedantic test.c

test.c:6:12: warning: ordered comparison between pointer and zero ('char 
*' and 'int') is an extension [-Wpedantic]


Sorry about that, after being away from C for a while, it's hard to 
switch on some brain, instead of letting the compiler to catch stupid 
bugs like this one.




I still merged the first one of these two because I think it's OK, however,
could you please add a bit of commit message to the second one ? As a rule
of thumb, the commit message should be used to "sell" me your patch and to
sell it to whoever would be hesitating in backporting it or not. Thus I think
that here the benefits and/or impacts are not obvious from this one-liner :


Here is me doing my best, selling you the good stuff, and the reason to 
backport the whole feature (so we can detect the multi-filter from CLI, 
and not rely on haproxy version or similar), see the attached patch.



Best regards,
Adis
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com
>From 8437fd4ac1a9cc5a9d88daad3a23e0d2a2eaeb4f Mon Sep 17 00:00:00 2001
From: Adis Nezirovic 
Date: Wed, 22 Jan 2020 16:50:27 +0100
Subject: [PATCH] MINOR: cli: Report location of errors or any extra data for
 "show table"

When using multiple filters with "show table", it can be useful to
report which filter entry failed

  > show table MY_TABLE data.gpc0 gt 0 data.gpc0a lt 1000
  Filter entry #2: Unknown data type

  > show table MY_TABLE data.gpc0 gt 0 data.gpc0 lt 1000a
  Filter entry #2: Require a valid integer value to compare against

We now also catch garbage data after the filter

  > show table MY_TABLE data.gpc0 gt 0 data.gpc0 lt 1000 data.gpc0 gt 1\
data.gpc0 gt 10 a
  Detected extra data in filter, 16th word of input, after '10'

Even before multi-filter feature we've also silently accepted garbage
after the input, hiding potential bugs

  > show table MY_TABLE data.gpc0 gt 0 data.gpc0
or
  > show table MY_TABLE data.gpc0 gt 0 a

In both cases, only first filter entry would be used, silently ignoring
extra filter entry or garbage data.

Last, but not the least, it is now possible to detect multi-filter
feature from cli with something like the following:

  > show table MY_TABLE data.blah
  Filter entry #1: Unknown data type
---
 src/stick_table.c | 13 +
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/src/stick_table.c b/src/stick_table.c
index 1b397e59e..1e7d4f3a8 100644
--- a/src/stick_table.c
+++ b/src/stick_table.c
@@ -3601,6 +3601,7 @@ static int table_process_entry_per_key(struct appctx *appctx, char **args)
 static int table_prepare_data_request(struct appctx *appctx, char **args)
 {
 	int i;
+	char *err = NULL;
 
 	if (appctx->ctx.table.action != STK_CLI_ACT_SHOW && appctx->ctx.table.action != STK_CLI_ACT_CLR)
 		return cli_err(appctx, "content-based lookup is only supported with the \"show\" and \"clear\" actions\n");
@@ -3611,17 +3612,21 @@ static int table_prepare_data_request(struct appctx *appctx, char **args)
 		/* condition on stored data value */
 		appctx->ctx.table.data_type[i] = stktable_get_data_type(args[3+3*i] + 5);
 		if (appctx->ctx.table.data_type[i] < 0)
-			return cli_err(appctx, "Unknown data type\n");
+			return cli_dynerr(appctx, memprintf(, "Filter entry #%i: Unknown data type\n", i + 1));
 
 		if (!((struct stktable *)appctx->ctx.table.target)->data_ofs[appctx->ctx.table.data_type[i]])
-			return cli_err(appctx, "Data type not stored in this table\n");
+			return cli_dynerr(appctx, memprintf(, "Filter entry #%i: Data type not stored in this table\n", i + 1));
 
 		appctx->ctx.table.data_op[i] = get_std_op(args[4+3*i]);
 		if (appctx->ctx.table.data_op[i] < 0)
-			return cli_err(appctx, "Require and operator among \"eq\", \"ne\", \"le\", \"ge\", \"lt\", \"gt\"\n");
+			return cli_dynerr(appctx, memprintf(, "Filter entry #%i: Require and operator among \"eq\", \"ne\", \"le\", \"ge\", \"lt\", \"gt\"\n", i + 1));
 
 		if (!*args[5+3*i] || strl2llrc(args[5+3*i], strlen(args[5+3*i]), >ctx.table.value[i]) != 0)
-			return cli_err(appctx, "Require a valid integ

Re: [PATCH] MEDIUM: cli: Allow multiple filter entries for show table

[Adis Nezirovic]

Hello,

Here it is, I've introduced a bug in last patch (since it is recent I've 
put "MINOR", even if it might have bigger impact in prod.


Also, a patch for better error diagnostics, and to report any extra data 
in the filter.


Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com
>From 030ee62e99d0ede6ea2e0bae16d4621cdd77da19 Mon Sep 17 00:00:00 2001
From: Adis Nezirovic 
Date: Wed, 22 Jan 2020 16:16:48 +0100
Subject: [PATCH 1/2] BUG/MINOR: cli: Missing arg offset for filter data
 values.

We don't properly check for missing data values for additional filter
entries, passing out of bounds index to args[], then passing to strlen.

Introduced in commit 1a693fc2: (MEDIUM: cli: Allow multiple filter
entries for "show table")
---
 src/stick_table.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/stick_table.c b/src/stick_table.c
index 1393b1ff3..beec279c2 100644
--- a/src/stick_table.c
+++ b/src/stick_table.c
@@ -3620,7 +3620,7 @@ static int table_prepare_data_request(struct appctx *appctx, char **args)
 		if (appctx->ctx.table.data_op < 0)
 			return cli_err(appctx, "Require and operator among \"eq\", \"ne\", \"le\", \"ge\", \"lt\", \"gt\"\n");
 
-		if (!*args[5] || strl2llrc(args[5+3*i], strlen(args[5+3*i]), >ctx.table.value[i]) != 0)
+		if (!*args[5+3*i] || strl2llrc(args[5+3*i], strlen(args[5+3*i]), >ctx.table.value[i]) != 0)
 			return cli_err(appctx, "Require a valid integer value to compare against\n");
 	}
 
-- 
2.25.0

>From d193f62ff73f01deb1eae77c0d3f319af50858ad Mon Sep 17 00:00:00 2001
From: Adis Nezirovic 
Date: Wed, 22 Jan 2020 16:50:27 +0100
Subject: [PATCH 2/2] MINOR: cli: Report location of error or any extra data
 for "show table"

---
 src/stick_table.c | 13 +
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/src/stick_table.c b/src/stick_table.c
index beec279c2..b7d5d145b 100644
--- a/src/stick_table.c
+++ b/src/stick_table.c
@@ -3601,6 +3601,7 @@ static int table_process_entry_per_key(struct appctx *appctx, char **args)
 static int table_prepare_data_request(struct appctx *appctx, char **args)
 {
 	int i;
+	char *err = NULL;
 
 	if (appctx->ctx.table.action != STK_CLI_ACT_SHOW && appctx->ctx.table.action != STK_CLI_ACT_CLR)
 		return cli_err(appctx, "content-based lookup is only supported with the \"show\" and \"clear\" actions\n");
@@ -3611,17 +3612,21 @@ static int table_prepare_data_request(struct appctx *appctx, char **args)
 		/* condition on stored data value */
 		appctx->ctx.table.data_type[i] = stktable_get_data_type(args[3+3*i] + 5);
 		if (appctx->ctx.table.data_type[i] < 0)
-			return cli_err(appctx, "Unknown data type\n");
+			return cli_dynerr(appctx, memprintf(, "Filter entry #%i: Unknown data type\n", i + 1));
 
 		if (!((struct stktable *)appctx->ctx.table.target)->data_ofs[appctx->ctx.table.data_type[i]])
-			return cli_err(appctx, "Data type not stored in this table\n");
+			return cli_dynerr(appctx, memprintf(, "Filter entry #%i: Data type not stored in this table\n", i + 1));
 
 		appctx->ctx.table.data_op[i] = get_std_op(args[4+3*i]);
 		if (appctx->ctx.table.data_op < 0)
-			return cli_err(appctx, "Require and operator among \"eq\", \"ne\", \"le\", \"ge\", \"lt\", \"gt\"\n");
+			return cli_dynerr(appctx, memprintf(, "Filter entry #%i: Require and operator among \"eq\", \"ne\", \"le\", \"ge\", \"lt\", \"gt\"\n", i + 1));
 
 		if (!*args[5+3*i] || strl2llrc(args[5+3*i], strlen(args[5+3*i]), >ctx.table.value[i]) != 0)
-			return cli_err(appctx, "Require a valid integer value to compare against\n");
+			return cli_dynerr(appctx, memprintf(, "Filter entry #%i: Require a valid integer value to compare against\n", i + 1));
+	}
+
+	if (*args[3+3*i]) {
+		return cli_dynerr(appctx, memprintf(, "Detected extra data in filter, %ith word of input, after '%s'\n", 3+3*i + 1, args[2+3*i]));
 	}
 
 	/* OK we're done, all the fields are set */
-- 
2.25.0

[PATCH] MEDIUM: cli: Allow multiple filter entries for show table

[Adis Nezirovic]

Hello,

Here is a patch to extend 'show table' with multiple filters. We already 
have this functionality in Lua (also adapted here to use the same 
definition for number of filters)


The current code  is somewhat relaxed about the extra data (garbage) 
after the

  'data.  '
part, and we've kept that behavior. For forward compatibility, it would 
be nice if we can introduce either additional 'haproxy -vv' info line, 
or CLI command (e.g. 'show cli filters'), which would return the 
supported number of filter entries.


Error handling could also be improved (we don't show which filter entry 
is wrong, if we have more than one), but I wanted to first check with 
you guys if we can add "snprintf" style error function to the cli, in 
addition to existing  cli_dynerr()/cli_dynmsg()



Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com
>From fdfcf4e124db1cffcce301116198bab9deec6583 Mon Sep 17 00:00:00 2001
From: Adis Nezirovic 
Date: Thu, 16 Jan 2020 15:19:29 +0100
Subject: [PATCH] MEDIUM: cli: Allow multiple filter entries for "show table"

For complex stick tables with many entries/columns, it can be beneficial
to filter using multiple criteria. The maximum number of filter entries
can be controlled by defining STKTABLE_FILTER_LEN during build time.

This patch can be backported to older releases.
---
 doc/management.txt|   4 +-
 include/common/defaults.h |   5 ++
 include/types/applet.h|   6 +--
 src/hlua_fcn.c|   7 ++-
 src/stick_table.c | 111 +-
 5 files changed, 76 insertions(+), 57 deletions(-)

diff --git a/doc/management.txt b/doc/management.txt
index 521a67112..24969be88 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -2569,7 +2569,7 @@ show table
 >>> # table: front_pub, type: ip, size:204800, used:171454
 >>> # table: back_rdp, type: ip, size:204800, used:0
 
-show table  [ data.   ] | [ key  ]
+show table  [ data.   [data. ...]] | [ key  ]
   Dump contents of stick-table . In this mode, a first line of generic
   information about the table is reported as with "show table", then all
   entries are dumped. Since this can be quite heavy, it is possible to specify
@@ -2588,6 +2588,8 @@ show table  [ data.   ] | [ key  ]
 - lt : match entries whose data is less than this value
 - gt : match entries whose data is greater than this value
 
+  In this form, you can use multiple data filter entries, up to a maximum
+  defined during build time (4 by default).
 
   When the key form is used the entry  is shown.  The key must be of the
   same type as the table, which currently is limited to IPv4, IPv6, integer,
diff --git a/include/common/defaults.h b/include/common/defaults.h
index fbacca481..e86c9bce1 100644
--- a/include/common/defaults.h
+++ b/include/common/defaults.h
@@ -116,6 +116,11 @@
 #define STKTABLE_EXTRA_DATA_TYPES 0
 #endif
 
+// max # of stick-table filter entries that can be used during dump
+#ifndef STKTABLE_FILTER_LEN
+#define STKTABLE_FILTER_LEN 4
+#endif
+
 // max # of loops we can perform around a read() which succeeds.
 // It's very frequent that the system returns a few TCP segments at a time.
 #ifndef MAX_READ_POLL_LOOPS
diff --git a/include/types/applet.h b/include/types/applet.h
index dfb489c9f..76a598d21 100644
--- a/include/types/applet.h
+++ b/include/types/applet.h
@@ -150,9 +150,9 @@ struct appctx {
 			void *target;		/* table we want to dump, or NULL for all */
 			struct stktable *t;	/* table being currently dumped (first if NULL) */
 			struct stksess *entry;	/* last entry we were trying to dump (or first if NULL) */
-			long long value;	/* value to compare against */
-			signed char data_type;	/* type of data to compare, or -1 if none */
-			signed char data_op;	/* operator (STD_OP_*) when data_type set */
+			long long value[STKTABLE_FILTER_LEN];	 /* value to compare against */
+			signed char data_type[STKTABLE_FILTER_LEN];  /* type of data to compare, or -1 if none */
+			signed char data_op[STKTABLE_FILTER_LEN];/* operator (STD_OP_*) when data_type set */
 			char action;/* action on the table : one of STK_CLI_ACT_* */
 		} table;
 		struct {
diff --git a/src/hlua_fcn.c b/src/hlua_fcn.c
index 7ef708ccb..f8024aa84 100644
--- a/src/hlua_fcn.c
+++ b/src/hlua_fcn.c
@@ -39,7 +39,6 @@ static int class_listener_ref;
 static int class_regex_ref;
 static int class_stktable_ref;
 
-#define MAX_STK_FILTER_LEN 4
 #define STATS_LEN (MAX((int)ST_F_TOTAL_FIELDS, (int)INF_TOTAL_FIELDS))
 
 static THREAD_LOCAL struct field stats[STATS_LEN];
@@ -682,7 +681,7 @@ int hlua_stktable_dump(lua_State *L)
 	int op;
 	int dt;
 	long long val;
-	struct stk_filter filter[MAX_STK_FILTER_LEN];
+	struct stk_filter filter[STKTABLE_FILTER_LEN];
 	int filter_count = 0;
 	int i;
 	int ski

Re: [PATCH] MINOR: lua: Add lua-prepend-path configuration option

[Adis Nezirovic]

On 1/8/20 10:54 PM, Tim Duesterhus wrote:

List,

while working on updating my haproxy-auth-request [1] to make use of
haproxy-lua-http [2] instead of the crappy lua-socket [3] I realized
that it would be difficult for the administrator to use the script
afterwards.

The reason is that `require` in Lua attempts to search the modules
either in the current working directory (which is different from
the filename of the script executing the `require` and which most
likely is not anywhere near HAProxy's configuration folder) or within
Lua's library path which is compiled into Lua (in my case this is
something like `/usr/share/lua/5.3/`).

Add a `lua-prepend-path` configuration option that prepend the given
string to Lua's path, allowing scripts loaded with `lua-load` to find
libraries within that path.

Example configuration:

global
lua-prepend-path /etc/haproxy/lua-modules/?/init.lua
lua-prepend-path /etc/haproxy/lua-modules/?.lua
lua-load /etc/haproxy/lua-modules/auth-request.lua



Hey Tim,

For what is worth, we solve situations like this with regular script, 
loaded before main script:


global
  lua-load /path/to/config.lua
  lua-load /path/to/app.lua


Inside config.lua you can override your app configuration options like this

  conf = { opt1 = "val" }


Then, app.lua contains the following pattern

  if not conf then
conf = { ... }  -- defaults
  else
if not conf.opt1 then conf.opt1 = "val" end
...
  end

And now for the main topic, if you need to set any paths for Lua 
scripts/libraries in non standard locations, you can directly use 
package.path in config.lua script


  package.path = package.path .. ';/some/path/?.lua'


That way, there is no need for users to edit app script, and they can 
override configuration or Lua path settings. IMO, we don't need another 
directive in HAProxy just to set Lua paths.



Best regards
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

Re: [PATCH 2/2] BUG/MINOR: lua: Make the arrays in the list of headers 1-indexed

[Adis Nezirovic]

On 9/26/19 3:11 PM, Tim Duesterhus wrote:

Lua arrays start at the index 1 by convention:


However, it is customary in Lua to start arrays with index 1. The Lua
libraries adhere to this convention; so, if your arrays also start with 1,
you will be able to use their functions directly.

(https://www.lua.org/pil/11.1.html)

This patch makes the array 1-indexed. To preserve backwards compatibility
the first value for each header is duplicated into the index 0. While the
duplication might cause harm where a specific value order is expected or
required by HTTP standards (e.g. Transfer-Encoding) I consider the breakage
to be higher if index 0 ceases to exist.


Hi Tim,

While I agree that using zero based array indexing is a mistake (wearing 
my Lua hat), I don't think your proposal will make situation any better. 
Actually ot might hurt existing Lua code in unforeseen ways.


e.g. what happens if you have a Lua action or service which uses request 
headers for its own nefarious purposes (like copying headers and using 
them for new requests). It now sends duplicate headers (name, value) 
which some upstream servers might not like.


Second thing, with Lua 5.2(?), "one" based arrays can use '#' operator 
to get the array length. With your proposal, we'd have tables of length 
N, where # operator would report N - 1 length

(That's just confusing, we'd have "1" based arrays which are not "1" based)

The last, but not the least, I am not fond of modifying request data by 
the HAProxy Lua layer. Suddenly, existing Lua code would see duplicate 
header values out of blue.


In my book, an improvement would be a new function, let's call it 
hlua_http_get_headers_kv(), which returns table with header names as 
keys, and values would be either "string" or array in case of multiple 
header values (we can fold multiple values easily with Lua)...



Just my 2c.


Best regards,
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com

[PATCH] BUG/MINOR: Missing stat_field_names (since f21d17bb)

[Adis Nezirovic]

Hello guys,

We've noticed a recently introduced a bug introduced in commit f21d17bb 
where new stat fields were defined, without proper string names.


We've noticed this with Lua Proxy class, failed calls to get_stats(). It 
also affects CLI (show stat):


 ... cache_lookups,cache_hits,(null),(null),

The attached patch solves the issue.

Best regards
--
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com
>From 807d0285c4b2e436284219d99e3e8d6c5e2be004 Mon Sep 17 00:00:00 2001
From: Adis Nezirovic 
Date: Fri, 13 Sep 2019 11:43:03 +0200
Subject: [PATCH] BUG/MINOR: Missing stat_field_names (since f21d17bb)

Recently Lua code which uses Proxy class (get_stats method) stopped
working ("table index is nil from [C] method 'get_stats'")
It probably affects other codepaths too.

This should be backported do 2.0 and 1.9.
---
 src/stats.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/stats.c b/src/stats.c
index e59ad10bb..f44f5eef7 100644
--- a/src/stats.c
+++ b/src/stats.c
@@ -244,6 +244,8 @@ const char *stat_field_names[ST_F_TOTAL_FIELDS] = {
 	[ST_F_REUSE]  = "reuse",
 	[ST_F_CACHE_LOOKUPS]  = "cache_lookups",
 	[ST_F_CACHE_HITS] = "cache_hits",
+	[ST_F_SRV_ICUR]   = "srv_icur",
+	[ST_F_SRV_ILIM]   = "src_ilim"
 };
 
 /* one line of info */
-- 
2.23.0

Re: BUG: Lua tasks can't use client sockets after bf89ff3d

[Adis Nezirovic]

On Thu, Nov 29, 2018 at 09:03:34AM +0100, Willy Tarreau wrote:
> OK thanks, I'll take a look at it once I've flushed my pending stuff on
> H2+HTX :-(

Great, I had my morning coffee and visited my optometrist, so here is
a fixed test script (correctly setting Host header).

P.S.
Lua usually suffers trying to do things in tasks, I don't think this is
the first time something gets broken. Can we make reg test with Lua
script (maybe strip out LuaSocket requirement)?

Best regards,
-- 
Adis Nezirovic
Software Engineer
HAProxy Technologies - Powering your uptime!
375 Totten Pond Road, Suite 302 | Waltham, MA 02451, US
+1 (844) 222-4340 | https://www.haproxy.com
local http = require 'socket.http'
local ltn12 = require 'ltn12'

local function main(applet)
end

local function cron()
local headers = {
host = 'www.haproxy.org'
}
local content = {}

while true do
core.Debug('CRON')
local b, c, h = http.request {
method='GET',
headers=headers,
url = 'http://51.15.8.218',
sink = ltn12.sink.table(content),
redirect = false,
create = core.tcp
}
core.Debug(tostring(c))
core.msleep(1000)
end
end

core.register_service("cron", "http", main)
core.register_task(cron)
global
log /dev/log local0 debug
nbproc 1
daemon
lua-load cron.lua

defaults
log global
mode http
option httplog
timeout connect 5s
timeout client 10s
timeout server 10s

frontend cron
bind 127.0.0.1:9000
http-request use-service lua.cron

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Adis Nezirovic]

On Thu, Sep 27, 2018 at 04:52:29PM +0200, Thierry Fournier wrote:
> I Adis,
>
> Sorry for the delay, I processed a quick review, and all seems to be ok for 
> me!
>
> BR,
> Thierry

Great, happy to hear that, I hope guys will merge it soon.

Best regards,
Adis

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Adis Nezirovic]

On Mon, Sep 03, 2018 at 12:09:47PM +0200, Adis Nezirovic wrote:
> Hi Thierry,
>
> Have you had the time to review my patches?

Thierry,

Reviving this thread, do you have any objections about latest version of
the patch (Lua stick table patch)

Best regards,
Adis

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Adis Nezirovic]

On Fri, Aug 24, 2018 at 11:40:51PM +0200, Adis Nezirovic wrote:
> Thierry,
>
> Something for Monday :-)
>
> Latest version of the patch in attachment:
>
> - Filter table format is flattened/simplified
> - I've tried to address filter table format error messages
>   (what is the error, and which filter entry is wrong)
> - Fixed one bug: stktable_get_data_type() return value can be < 0
>   (i.e. filter table contains unknown data column)

Hi Thierry,

Have you had the time to review my patches?

Best regards,
Adis

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Adis Nezirovic]

Thierry,

Something for Monday :-)

Latest version of the patch in attachment:

- Filter table format is flattened/simplified
- I've tried to address filter table format error messages
  (what is the error, and which filter entry is wrong)
- Fixed one bug: stktable_get_data_type() return value can be < 0
  (i.e. filter table contains unknown data column)

I've run a few unit tests for my Lua code using stick tables, hitting
all methods and handling regular return values and filter table errors.
So far so good, it looks good on my side.

Best regards,
Adis
>From 6b702ff6f12f919ba4d2f42a7962fa2345272382 Mon Sep 17 00:00:00 2001
From: Adis Nezirovic 
Date: Fri, 13 Jul 2018 12:18:33 +0200
Subject: [PATCH] MEDIUM: lua: Add stick table support for Lua.

This ads support for accessing stick tables from Lua. The supported
operations are reading general table info, lookup by string/IP key, and
dumping the table.

Similar to "show table", a data filter is available during dump, and as
an improvement over "show table" it's possible to use up to 4 filter
expressions instead of just one (with implicit AND clause binding the
expressions). Dumping with/without filters can take a long time for
large tables, and should be used sparingly.
---
 doc/lua-api/index.rst |  72 
 include/types/hlua.h  |   1 +
 src/hlua_fcn.c| 399 ++
 3 files changed, 472 insertions(+)

diff --git a/doc/lua-api/index.rst b/doc/lua-api/index.rst
index 0c79766e..9a2e039a 100644
--- a/doc/lua-api/index.rst
+++ b/doc/lua-api/index.rst
@@ -852,6 +852,10 @@ Proxy class
   Contain a table with the attached servers. The table is indexed by server
   name, and each server entry is an object of type :ref:`server_class`.
 
+.. js:attribute:: Proxy.stktable
+
+  Contains a stick table object attached to the proxy.
+
 .. js:attribute:: Proxy.listeners
 
   Contain a table with the attached listeners. The table is indexed by listener
@@ -2489,6 +2493,74 @@ AppletTCP class
   :see: :js:func:`AppletTCP.unset_var`
   :see: :js:func:`AppletTCP.set_var`
 
+StickTable class
+
+
+.. js:class:: StickTable
+
+  **context**: task, action, sample-fetch
+
+  This class can be used to access the HAProxy stick tables from Lua.
+
+.. js:function:: StickTable.info()
+
+  Returns stick table attributes as a Lua table. See HAProxy documentation for
+  "stick-table" for canonical info, or check out example bellow.
+
+  :returns: Lua table
+
+  Assume our table has IPv4 key and gpc0 and conn_rate "columns":
+
+.. code-block:: lua
+
+  {
+expire=,  # Value in ms
+size=,# Maximum table size
+used=,# Actual number of entries in table
+data={ # Data columns, with types as key, and periods as values
+ (-1 if type is not rate counter)
+  conn_rate=,
+  gpc0=-1
+},
+length=,  # max string length for string table keys, key length
+   # otherwise
+nopurge=, # purge oldest entries when table is full
+type="ip"  # can be "ip", "ipv6", "integer", "string", "binary"
+  }
+
+.. js:function:: StickTable.lookup(key)
+
+   Returns stick table entry for given 
+
+   :param string key: Stick table key (IP addresses and strings are supported)
+   :returns: Lua table
+
+.. js:function:: StickTable.dump([filter])
+
+   Returns all entries in stick table. An optional filter can be used
+   to extract entries with specific data values. Filter is a table with valid
+   comparison operators as keys followed by data type name and value pairs.
+   Check out the HAProxy docs for "show table" for more details. For the
+   reference, the supported operators are:
+ "eq", "ne", "le", "lt", "ge", "gt"
+
+   For large tables, execution of this function can take a long time (for
+   HAProxy standards). That's also true when filter is used, so take care and
+   measure the impact.
+
+   :param table filter: Stick table filter
+   :returns: Stick table entries (table)
+
+   See below for example filter, which contains 4 entries (or comparisons).
+   (Maximum number of filter entries is 4, defined in the source code)
+
+.. code-block:: lua
+
+local filter = {
+  {"gpc0", "gt", 30}, {"gpc1", "gt", 20}}, {"conn_rate", "le", 10}
+}
+
+
 External Lua libraries
 ==
 
diff --git a/include/types/hlua.h b/include/types/hlua.h
index 5a8173f3..2e453351 100644
--- a/include/types/hlua.h
+++ b/include/types/hlua.h
@@ -25,6 +25,7 @@
 #define CLASS_SERVER   "Server"
 #define CLASS_LISTENER "Listener"
 #define CLASS_REGEX"Regex"
+#define CLASS_STKTABLE "StickTable"
 
 struct stream;
 
diff --git a/src/hlua_fcn.c b/src/hlua_fcn.c
index c

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Adis Nezirovic]

On Thu, Aug 23, 2018 at 03:43:59PM +0200, Willy Tarreau wrote:
> Does this mean I should merge Adis' patch or do you want to verify
> other things ? Just let me know.

Willy,

I'll submit new patch later today with simplified filter definitions and
then we can ask Thierry for final ack for the patch.

Best regards,
Adis

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Adis Nezirovic]

On Tue, Aug 21, 2018 at 11:32:34AM +0200, Thierry Fournier wrote:
> Some remark about the documentation and the formats:
> 
>
> js:function:: StickTable.info()
> js:function:: StickTable.lookup(key)
>
> Maybe the specification and an example of the returnes values
> will be welcome, because I guess that the table keys are hardcoded.
> Something like :

That makes sense, I've updated the docs.

> js:function:: StickTable.dump([filter])
>
> The exact list of allowed operators will helps the user to use
> your class. It seems that string or regexes operators are not
> allowed, only the integer operator are taken in account. This
> list is "eq", "ne", "le", "lt", "ge", "gt".
Since only integers are available as data columns, we can only use
numeric operators. Noted in the docs now, with pointer to "show table".

> Same remarqk for allowed data type. Maybe a link to the HAProxy
> documentation will be sufficient for the data type.

> I see in the code that the filters can exceed 4 entries. This
> limitation must be written in the doc.
Noted.

> I miss also the relation between oprators and between the content
> of operators. I mean AND or OR. How I understand your example:
>
> +local filter = {
> +  lt={{"gpc0", 1}, {"gpc1", 2}},
> +  gt={{"conn_rate", 3}},
> +  eq={{"conn_cur", 4}}
> +}
>
> Are you sure that the syntax =
> is a good format ? Maybe something like the following, with the operator
> as argument between the two operands. lines are implicitly OR, and columns
> are AND:
>
> +local filter = {
> +  {{"gpc0", "lt", 1}, {"gpc1", "lt", 2}},
> +  {{"conn_rate", "gt", 3}},
> +  {{"conn_cur", "eq", 4}}
> +}
Actually, I was playing with some other ideas, and it was useful to be
able to "preselect" filter operators.
However, the CLI doesn't even support more than one, maybe we don't need
to complicate too much. Maybe we can simplify to this:

  local filter = {
{"gpc0", "lt", 1},
{"gpc1", "lt", 2},
{"conn_rate", "gt", 3},
{"conn_cur", "eq", 4}
  }

The default operator would be AND, and we would not support other
operators (to keep the things simple). e.g. example use case for the
filter would be to filter out on gpc0 > X AND gpc1 > Y

If this sounds good, I can update/simplify the code.

> Idea of extension for the future: Maybe it will be safe to compile
> sticktable filter during the initialisation of the Lua code, to avoid
> runtime errors ?
I'm returning runtime errors since it can be easy to mix up data from
the client side (most probably data would come as json table, then
transformed to Lua table)


> Other point with the doc of this function, you must specify that the
> execution of this function can be very long for millions of entry, even
> if a filter is specified because the stick table is entirely scanned.
Noted.


> some remarks about the code and the logs:
> -
>
> The line 182 of the patch contains space in place of tab.
Sorry, I generally try to be careful with code style, to many different
vim profiles :-D. Fixef.

> Line 274 of your patch, I don't see any HA_SPIN_LOCK(STK_TABLE_LOCK
> I don't known very well the thread, so maybe there are useles, maybe no.
hlua_stktable_lookup() uses stktable_lookup_key() which does have locks,
so I guess that it should be fine then?

> Line 311: I see the decrement of the refcount without ATOMIC function
> and whitout lock. Once again, I don't known very well the thread but
> I send a warning. Maybe locks are useles, maybe no.
You are totally right, locks are needed when decrementing references.
Fixed.

> Line 286 of your patch. It seems that smp.flags is used uninitialized.
> Maybe you should apply this change:
>
>-  smp.flags |= SMP_F_CONST;
>+  smp.flags = SMP_F_CONST;
Fixed, and cleaned up a bit (unecessary 'char *key' variable)

> l.365, 369: The user doesn't have context about the error. there are the
> first entry of the table, the second ? Which operator doesn't exists ?
>
> L.380, 384: Which line is wrong ?
Yes, it is somwehat cryptic. I've tried to avoid returning user supplied
data in the error messages. We can revisit this if/when we change the
filter table format.

> L.431: Your release the lock, so the next element relative to the current
> "n", can disappear and the ebmb_next() can return wrong memory.
I was under impression that we only have to acquire lock and increment
ref_cnt (so we can be sure our cu

Re: [PATCH] MEDIUM: lua: Add stick table support for Lua

[Adis Nezirovic]

On Mon, Aug 20, 2018 at 02:11:13PM +0200, Adis Nezirovic wrote:
> Hi guys,
>
> I've attached a patch to add stick table support to Lua. Operations are
> mostly similar to "show table" functionality from admin socket, namely:
>
> - Show basic table info
> - Key lookup
> - Table dump, optionally using data/column filter
>
> One side note, the code provides support for multiple filters
> (4 by default) while CLI doesn't support that, nor it complains about
> multiple "  " clauses.
>
> Also, if this patch is accepted, maybe we can use provided helper
> functions in other places in the code.
>

It's always funny to reply to self, right sending email to public I've
spotted a bug. New patch attached.

SMT_T_SINT should be treated as ordinary signed integer, and shoud use
lua_pushinteger() on it?

On many places in the code it is noted as "64" bit integer, but in
stick_table.c it is defined as 32bit integer:

struct stktable_type stktable_types[SMP_TYPES] = {
[SMP_T_SINT] = { "integer", 0, 4 },


Best regards,
Adis
>From 4c699b0d57ba5d6d5463595a4bdaa2f2ae8c2c56 Mon Sep 17 00:00:00 2001
From: Adis Nezirovic 
Date: Fri, 13 Jul 2018 12:18:33 +0200
Subject: [PATCH] MEDIUM: lua: Add stick table support for Lua (read-only ops).

---
 doc/lua-api/index.rst |  46 +
 include/types/hlua.h  |   1 +
 src/hlua_fcn.c| 401 ++
 3 files changed, 448 insertions(+)

diff --git a/doc/lua-api/index.rst b/doc/lua-api/index.rst
index 0c79766e..63848661 100644
--- a/doc/lua-api/index.rst
+++ b/doc/lua-api/index.rst
@@ -852,6 +852,10 @@ Proxy class
   Contain a table with the attached servers. The table is indexed by server
   name, and each server entry is an object of type :ref:`server_class`.
 
+.. js:attribute:: Proxy.stktable
+
+  Contains a stick table object attached to the proxy.
+
 .. js:attribute:: Proxy.listeners
 
   Contain a table with the attached listeners. The table is indexed by listener
@@ -2489,6 +2493,48 @@ AppletTCP class
   :see: :js:func:`AppletTCP.unset_var`
   :see: :js:func:`AppletTCP.set_var`
 
+StickTable class
+
+
+.. js:class:: StickTable
+
+  This class is used with applets that requires the 'tcp' mode. The tcp applet
+  can be registered with the *core.register_service()* function. They are used
+  for processing a tcp stream like a server in back of HAProxy.
+
+.. js:function:: StickTable.info()
+
+  Returns relevant stick table attributes: type, length, size, used, nopurge,
+  expire and exact interval values for frequency data columns.
+
+  :returns: Lua table
+
+.. js:function:: StickTable.lookup(key)
+
+   Returns stick table entry for given 
+
+   :param string key: Stick table key (IP addresses and strings are supported)
+   :returns: Lua table
+
+.. js:function:: StickTable.dump([filter])
+
+   Returns all entries in stick table. An optional filter can be used
+   to extract entries with specific data values. Filter is a table with valid
+   comparison operators as keys followed by data type name and value pairs.
+   e.g.
+
+   :param table filter: Stick table filter
+   :returns: Stick table entries (table)
+
+.. code-block:: lua
+
+local filter = {
+  lt={{"gpc0", 1}, {"gpc1", 2}},
+  gt={{"conn_rate", 3}},
+  eq={{"conn_cur", 4}}
+}
+
+
 External Lua libraries
 ==
 
diff --git a/include/types/hlua.h b/include/types/hlua.h
index 5a8173f3..2e453351 100644
--- a/include/types/hlua.h
+++ b/include/types/hlua.h
@@ -25,6 +25,7 @@
 #define CLASS_SERVER   "Server"
 #define CLASS_LISTENER "Listener"
 #define CLASS_REGEX"Regex"
+#define CLASS_STKTABLE "StickTable"
 
 struct stream;
 
diff --git a/src/hlua_fcn.c b/src/hlua_fcn.c
index cebce224..f38b9f37 100644
--- a/src/hlua_fcn.c
+++ b/src/hlua_fcn.c
@@ -30,6 +30,7 @@
 #include 
 #include 
 #include 
+#include 
 
 /* Contains the class reference of the concat object. */
 static int class_concat_ref;
@@ -37,7 +38,9 @@ static int class_proxy_ref;
 static int class_server_ref;
 static int class_listener_ref;
 static int class_regex_ref;
+static int class_stktable_ref;
 
+#define MAX_STK_FILTER_LEN 4
 #define STATS_LEN (MAX((int)ST_F_TOTAL_FIELDS, (int)INF_TOTAL_FIELDS))
 
 static THREAD_LOCAL struct field stats[STATS_LEN];
@@ -49,6 +52,38 @@ int hlua_checkboolean(lua_State *L, int index)
return lua_toboolean(L, index);
 }
 
+/* Helper to push unsigned integers to Lua stack, respecting Lua limitations  
*/
+static int hlua_fcn_pushunsigned(lua_State *L, unsigned int val)
+{
+#if (LUA_MAXINTEGER == LLONG_MAX || ((LUA_MAXINTEGER == LONG_MAX) && 
(__WORDSIZE == 64)))
+   lua_pushinteger(L, val);
+#else
+   if (val > INT_MAX)
+   lua_pushnumber(L, (lua_Number)val);
+   else
+   lua_pushi

[PATCH] MEDIUM: lua: Add stick table support for Lua

[Adis Nezirovic]

Hi guys,

I've attached a patch to add stick table support to Lua. Operations are
mostly similar to "show table" functionality from admin socket, namely:

- Show basic table info
- Key lookup
- Table dump, optionally using data/column filter

One side note, the code provides support for multiple filters
(4 by default) while CLI doesn't support that, nor it complains about
multiple "  " clauses.

Also, if this patch is accepted, maybe we can use provided helper
functions in other places in the code.

Best regards,
Adis
>From 27d0c08a85e8a997fb96e65937c70e983e31c21e Mon Sep 17 00:00:00 2001
From: Adis Nezirovic 
Date: Fri, 13 Jul 2018 12:18:33 +0200
Subject: [PATCH] MEDIUM: lua: Add stick table support for Lua (read-only ops).

---
 doc/lua-api/index.rst |  46 +
 include/types/hlua.h  |   1 +
 src/hlua_fcn.c| 401 ++
 3 files changed, 448 insertions(+)

diff --git a/doc/lua-api/index.rst b/doc/lua-api/index.rst
index 0c79766e..63848661 100644
--- a/doc/lua-api/index.rst
+++ b/doc/lua-api/index.rst
@@ -852,6 +852,10 @@ Proxy class
   Contain a table with the attached servers. The table is indexed by server
   name, and each server entry is an object of type :ref:`server_class`.
 
+.. js:attribute:: Proxy.stktable
+
+  Contains a stick table object attached to the proxy.
+
 .. js:attribute:: Proxy.listeners
 
   Contain a table with the attached listeners. The table is indexed by listener
@@ -2489,6 +2493,48 @@ AppletTCP class
   :see: :js:func:`AppletTCP.unset_var`
   :see: :js:func:`AppletTCP.set_var`
 
+StickTable class
+
+
+.. js:class:: StickTable
+
+  This class is used with applets that requires the 'tcp' mode. The tcp applet
+  can be registered with the *core.register_service()* function. They are used
+  for processing a tcp stream like a server in back of HAProxy.
+
+.. js:function:: StickTable.info()
+
+  Returns relevant stick table attributes: type, length, size, used, nopurge,
+  expire and exact interval values for frequency data columns.
+
+  :returns: Lua table
+
+.. js:function:: StickTable.lookup(key)
+
+   Returns stick table entry for given 
+
+   :param string key: Stick table key (IP addresses and strings are supported)
+   :returns: Lua table
+
+.. js:function:: StickTable.dump([filter])
+
+   Returns all entries in stick table. An optional filter can be used
+   to extract entries with specific data values. Filter is a table with valid
+   comparison operators as keys followed by data type name and value pairs.
+   e.g.
+
+   :param table filter: Stick table filter
+   :returns: Stick table entries (table)
+
+.. code-block:: lua
+
+local filter = {
+  lt={{"gpc0", 1}, {"gpc1", 2}},
+  gt={{"conn_rate", 3}},
+  eq={{"conn_cur", 4}}
+}
+
+
 External Lua libraries
 ==
 
diff --git a/include/types/hlua.h b/include/types/hlua.h
index 5a8173f3..2e453351 100644
--- a/include/types/hlua.h
+++ b/include/types/hlua.h
@@ -25,6 +25,7 @@
 #define CLASS_SERVER   "Server"
 #define CLASS_LISTENER "Listener"
 #define CLASS_REGEX"Regex"
+#define CLASS_STKTABLE "StickTable"
 
 struct stream;
 
diff --git a/src/hlua_fcn.c b/src/hlua_fcn.c
index cebce224..49f7ef03 100644
--- a/src/hlua_fcn.c
+++ b/src/hlua_fcn.c
@@ -30,6 +30,7 @@
 #include 
 #include 
 #include 
+#include 
 
 /* Contains the class reference of the concat object. */
 static int class_concat_ref;
@@ -37,7 +38,9 @@ static int class_proxy_ref;
 static int class_server_ref;
 static int class_listener_ref;
 static int class_regex_ref;
+static int class_stktable_ref;
 
+#define MAX_STK_FILTER_LEN 4
 #define STATS_LEN (MAX((int)ST_F_TOTAL_FIELDS, (int)INF_TOTAL_FIELDS))
 
 static THREAD_LOCAL struct field stats[STATS_LEN];
@@ -49,6 +52,38 @@ int hlua_checkboolean(lua_State *L, int index)
return lua_toboolean(L, index);
 }
 
+/* Helper to push unsigned integers to Lua stack, respecting Lua limitations  
*/
+static int hlua_fcn_pushunsigned(lua_State *L, unsigned int val)
+{
+#if (LUA_MAXINTEGER == LLONG_MAX || ((LUA_MAXINTEGER == LONG_MAX) && 
(__WORDSIZE == 64)))
+   lua_pushinteger(L, val);
+#else
+   if (val > INT_MAX)
+   lua_pushnumber(L, (lua_Number)val);
+   else
+   lua_pushinteger(L, (int)val);
+#endif
+   return 1;
+}
+
+/* Helper to push unsigned long long to Lua stack, respecting Lua limitations  
*/
+static int hlua_fcn_pushunsigned_ll(lua_State *L, unsigned long long val) {
+#if (LUA_MAXINTEGER == LLONG_MAX || ((LUA_MAXINTEGER == LONG_MAX) && 
(__WORDSIZE == 64)))
+   /* 64 bits case, U64 is supported until LLONG_MAX */
+   if (val > LLONG_MAX)
+   lua_pushnumber(L, (lua_Number)val);
+   else
+   lua_pushinteger(L, val);
+#else
+   /* 32 bits case, U64 is supported until INT_MAX */
+   if

Re: JWT payloads break b64dec convertor

[Adis Nezirovic]

On 05/26/2018 04:27 PM, Jonathan Matthews wrote:
> Hello folks,
> 
> The payload (and other parts) of a JSON Web Token (JWT, a popular and
> growing auth standard: https://tools.ietf.org/html/rfc7519) is base64
> encoded.
> 
> Unfortunately, the payload encoding (specified in
> https://tools.ietf.org/html/rfc7515) is defined as the "URL safe"
> variant. This variant allows for the lossless omission of base64
> padding ("=" or "=="), which the haproxy b64dec convertor doesn't
> appear to be able cope with. The result of

Jonathan,

It's not just padding, urlsafe base64 replaces '+' with '-', and '/'
with '_'. For now, I guess the easiest way would be to write a simple
converter in Lua, which just returns the original string, and send
payload somewhere for further processing.

Best regards,
Adis

Re: lua socket api settimeout in seconds vs. milliseconds

[Adis Nezirovic]

On 11/07/2017 08:49 PM, Nick Galbreath wrote:
> I have a question regarding the socket:settimeout function.
> 
> It appears to only accept an integer value for seconds, but internally
> converts to milliseconds (see below).
> 
> Is there a reason to enforce whole seconds, or could this be relaxed to
> accept a lua number (float/double).?

Hi Nick,

HAProxy uses Lua 5.3 which has proper integers. There is already a
similar situation where we have both, seconds and milliseconds
(hlua_sleep/hlua_msleep), so maybe that's the best solution here too,
just add another variant using milliseconds.

Best regards,
Adis

Re: Possible bug in task_wakeup() impacts Lua tasks

[Adis Nezirovic]

On 10/17/2017 07:05 PM, Emeric Brun wrote:
> Hi Adis,
> 
> This patch should fix the issue more consistently.
> 
> Could you confirm?

It seems to work fine here, for the trivial test task and other stuff I
work with. Thanks!

Best regards,
Adis

Possible bug in task_wakeup() impacts Lua tasks

[Adis Nezirovic]

Hello guys,

After this commit:

  commit 0194897e540cec67d7d1e9281648b70efe403f08
  Author: Emeric Brun 
  Date:   Thu Mar 30 15:37:25 2017 +0200

  MAJOR: task: task scheduler rework.

basic Lua tasks don't work anymore.
e.g. this only gets called once:

  function cron()
  while true do
  core.Debug("Hello from Cron")
  core.sleep(1)
  end
  end
  core.register_task(cron)



The current code in task_wakeup() checks for TASK_RUNNING and decides
that it won't call __task_wakeup(), but when Lua task wakes up, it has
both, TASK_WOKEN_TIMER and TASK_RUNNING set.

My quick fix/workaround was to add an additional check:

  if (unlikely(!(t->state & TASK_WOKEN_TIMER) &&
   (t->state & TASK_RUNNING)))

But I might be missing something more fundamental (i.e. this is really
necessary for multithreaded stuff), maybe we need additional flags when
running task_wakeup from task handlers or threads.


Best regards,
Adis

[PATCH] Fix-bitwise-logic-for-hlua_server_check*

[Adis Nezirovic]

Hello guys,

Here is a fix for bug in
hlua_server_check_enable()/hlua_server_check_disable, the bitwise logic
in the code was inverted.

This can be applied to 1.7 too.

Best regards,
Adis
>From b51318ab44cb904f9689aebe27f61a41a26240cc Mon Sep 17 00:00:00 2001
From: Adis Nezirovic <aneziro...@haproxy.com>
Date: Wed, 26 Jul 2017 09:19:06 +0200
Subject: [PATCH] BUG/MINOR: lua: Fix bitwise logic for hlua_server_check_*
 functions.

The logical operations were inverted so enable/disable operations did
the oposite.
---
 src/hlua_fcn.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/hlua_fcn.c b/src/hlua_fcn.c
index 69926132..bc6bc914 100644
--- a/src/hlua_fcn.c
+++ b/src/hlua_fcn.c
@@ -662,7 +662,7 @@ int hlua_server_check_enable(lua_State *L)
 
 	sv = hlua_check_server(L, 1);
 	if (sv->check.state & CHK_ST_CONFIGURED) {
-		sv->check.state &= ~CHK_ST_ENABLED;
+		sv->check.state |= CHK_ST_ENABLED;
 	}
 	return 0;
 }
@@ -673,7 +673,7 @@ int hlua_server_check_disable(lua_State *L)
 
 	sv = hlua_check_server(L, 1);
 	if (sv->check.state & CHK_ST_CONFIGURED) {
-		sv->check.state |= CHK_ST_ENABLED;
+		sv->check.state &= ~CHK_ST_ENABLED;
 	}
 	return 0;
 }
-- 
2.13.3

Re: BUG: Lua service timeouts while sending data (after 0194897e540cec67d7d1e9281648b70efe403f08)

[Adis Nezirovic]

On 07/24/2017 06:36 PM, Willy Tarreau wrote:
> Hehe I've just committed the fixes a few minutes ago :-) We'had quite a
> long head scratching session with Thierry, Christopher and Emeric on
> this one. It's sometimes impressive how some sleeping bugs can patiently
> wait for a subtle change to join efforts to annoy us!
> 
> Just pull the latest master, you shouldn't face the problem anymore.

Yep, it totally works now, what are the odds for you to fix just right
now :-)

Thanks, great work!

Best regards,
Adis

BUG: Lua service timeouts while sending data (after 0194897e540cec67d7d1e9281648b70efe403f08)

[Adis Nezirovic]

Hello guys,

I've noticed that a Lua service timeouts in DATA phase, for outputs
equal or bigger than 8k (approx).

After the timeout (timeout client), it returns the full response.
(Termination state is cD--)

I've attached the minimal configuration and a Lua script to trigger the
problem. You might need to tweak the length of test string, I even have
different behavior depending on the Lua code:

-- variant a) loop with hardcoded bounds:
for i = 1, 7492 do

-- variant b) using local variable
local body_len = 7491
for i = 1, body_len do

Makeflags:
make TARGET=linux2628 USE_GETADDRINFO=1 USE_ZLIB=1 USE_OPENSSL=1
USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1


git bisect tells me the bug appears after the following commit:

commit 0194897e540cec67d7d1e9281648b70efe403f08
Author: Emeric Brun 
Date:   Thu Mar 30 15:37:25 2017 +0200

MAJOR: task: task scheduler rework.


Best regards,
Adis
local function http_response(applet, code, data, content_type)
applet:set_status(code)
applet:add_header("Content-Length", string.len(data))
applet:add_header("Content-Type", content_type)
applet:start_response()
applet:send(data)
end

local function main(applet)
local c = core.concat()

-- variant a) 
for i = 1, 7492 do
-- variant b)
-- local body_len = 7491
-- for i = 1, body_len do
c:add("#")
end

http_response(applet, 200, c:dump(), "text/plain")
end

core.register_service("block", "http", main)
global
log /dev/log local0 debug
lua-load /etc/haproxy/lua/block.lua

defaults
log global
mode http
option httplog
timeout connect 5s
timeout client 7s
timeout server 10s

listen block
bind 127.0.0.1:9001
http-request use-service lua.block

Re: [PATCH] Support proxies with identical names in Lua core.proxies

[Adis Nezirovic]

On 07/24/2017 01:30 PM, Thierry FOURNIER wrote:
> I think that the most reliable way is adding anoter tree. We keep the
> "proxies" tree base with existing, and we add two trees "frontends" and
> "backends" which contains respecticely the list of frontends and
> backends.

This would work for me too, maybe just add a note in docs regarding that
bug in core.proxies (masking frontends/backends with the same name).

>> This is just for convenience and uniformity, Proxy.servers/listeners
>> returns a table/hash of objects with names as keys, but for example when
>> I want to pass such object to some other Lua function I have to manually
>> copy the name (or wrap the object), since the object itself doesn't
>> expose name info.
> 
> 
> You're right, it will be better adding the name in the object. I will
> do this.

Great, that would be convenient, thanks.


Best regards,
Adis

Re: [PATCH] Support proxies with identical names in Lua core.proxies

[Adis Nezirovic]

On 07/20/2017 02:55 PM, Willy Tarreau wrote:
> So you can have :
>   0 or 1 "listen"
>   0 or 1 "frontend" + 0 or 1 "backend"
> 
> Just a few ideas come to my mind :
>   - is it possible to store arrays into arrays ? I mean, could we have
> for example core.proxies["foo"].side[FRONT|BACK] where each side is
> set only when the two differ ?
> 
>   - or is it to return a list (1 or 2 elements) ? That would be even more
> convenient to use. If you have "p = core.proxies[foo]" and can use p
> then next(p), you're sure to always scan everything.
> 
>   - otherwise I like the principle of being able to force the type using
> a prefix "@f:" or "@b:" in front of the name. But we must be careful
> not to insert duplicates. So probably by default only "listen" instances
> would be created without the "@" prefix. My initial idea was to be able
> to always return the plain form and point to whichever exists, but for
> enumeration it would require to keep only the "@" form, which might be
> more complicated.

We currently use a variant of second proposal, when working internally
with backend/frontends with same name, type/side should still be
indicated. Simple list/array might be easiest thing to implement.

>> Also, apply the same logic to the 'Listener' and 'Server' classes.
> 
> These ones do not support duplicate names so we should not need to impact
> them.

This is just for convenience and uniformity, Proxy.servers/listeners
returns a table/hash of objects with names as keys, but for example when
I want to pass such object to some other Lua function I have to manually
copy the name (or wrap the object), since the object itself doesn't
expose name info.


Best regards,
Adis

Re: [PATCH] Support proxies with identical names in Lua core.proxies

[Adis Nezirovic]

On 07/20/2017 12:17 PM, Thierry FOURNIER wrote:
> I understand the problem, but I can't accept this patch because it makes
> the proxies list unusable. Your patch remove the proxies names, and the
> user cannot have solution for knowning the real name of the proxies
> now called 1, 2, 3, ...
> 
> I propose 2 solutions:
> 
>  - Two new lists called "frontend" and "backend" with no name collision
>by design
> 
>  - a prefix "@f:" for fontend and "@b:" for backends and keep the name
>without prefix for the compatibility.
> 
> I prefer the two new lists.

Hello Thierry,

Yeah, dropping the name might be inconvenient, we were traversing the
proxy list and calling get_stats() on each object, and fetched "pxname"
attribute.

How about different approach, can we set "name" and "type" attribute for
Proxy objects? Maybe expose some other attributes (id, ...)

Also, apply the same logic to the 'Listener' and 'Server' classes.


Best regards,
Adis

[PATCH] Support proxies with identical names in Lua core.proxies

[Adis Nezirovic]

Hi guys,

While playing with Lua API I've noticed that core.proxies attribute
doesn't return all the proxies, more precisely the ones with same names
(e.g. for frontend and backend with the same name it would only return
the latter one).

Here is a patch which tries to fix that, avoiding using the proxy name
as a key for Lua table.
We use a plain integer offsets (starting from 1) to create numerical
array/table iterable by both pairs() and ipairs() in Lua.

The patch also applies cleanly to 1.7 branch.

Best regards,
Adis
>From f201afcd5ec9790a40907ab24e089db28a9cb6f1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Adis=20Nezirovi=C4=87?= 
Date: Mon, 5 Jun 2017 17:10:51 +0200
Subject: [PATCH] BUG/MEDIUM: lua: Support proxies with identical names in
 core.proxies

When we have frontends/backends with identical names, core.proxies
returns only the last one, using proxy->id (i.e. proxy name) to insert
the value in Lua table. We should add them to numerical array/table, to
be able to access all proxies.
---
 src/hlua_fcn.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/hlua_fcn.c b/src/hlua_fcn.c
index 8406bfe5..89bf0b3f 100644
--- a/src/hlua_fcn.c
+++ b/src/hlua_fcn.c
@@ -893,6 +893,7 @@ int hlua_proxy_shut_bcksess(lua_State *L)
 int hlua_fcn_post_init(lua_State *L)
 {
 	struct proxy *px;
+	int index;
 
 	/* get core array. */
 	if (lua_getglobal(L, "core") != LUA_TTABLE)
@@ -903,8 +904,9 @@ int hlua_fcn_post_init(lua_State *L)
 	lua_newtable(L);
 
 	/* List all proxies. */
+	index = 1;
 	for (px = proxy; px; px = px->next) {
-		lua_pushstring(L, px->id);
+		lua_pushinteger(L, index++);
 		hlua_fcn_new_proxy(L, px);
 		lua_settable(L, -3);
 	}
-- 
2.13.0