[PATCH] MINOR: sample: Add fetcher for getting all cookie names
Hi team, As discussed in https://www.mail-archive.com/haproxy@formilux.org/msg44161.html, I have attached a patch for adding a new fetcher for getting all the cookie names for request / response. I did check some relevant docs (like CONTRIBUTING, coding-style.txt etc.) for the guidelines before my first submission but might still miss something. Any feedback would be greatly appreciated. Thanks, Ruei-Bang 0001-MINOR-sample-Add-fetcher-for-getting-all-cookie-name.patch Description: 0001-MINOR-sample-Add-fetcher-for-getting-all-cookie-name.patch
Re: Missing doc entities in doc/internals
Hi Willy. On 2023-10-20 (Fr.) 23:21, Willy Tarreau wrote: Hi Alex, On Fri, Oct 20, 2023 at 11:11:59PM +0200, Aleksandar Lazic wrote: I can't find any doc about entities in the current git alex@alex-tuxedoinfinitybooks1517gen7 on 20/10/2023 at 23:06:19 /datadisk/git-repos/haproxy $ find . -iname "*entities"* alex@alex-tuxedoinfinitybooks1517gen7 on 20/10/2023 at 23:06:27 /datadisk/git-repos/haproxy $ What's my mistake to find this doc? No mistake, this file was so much outdated that not any single box on it existed in recent versions so it was about time for it to be removed. It was a bit heartbreaking, but killing a 15-years old doc because the architecture evolves is not that bad of a news :-) I feel your pain, got similar with appsession years ago :-) I suggest that you have a look at doc/internals/api/layers.txt first, then doc/internals/muxes.pdf whicih show the lower layers in boxes and what remains of the stream layer on top as the channel. Thanks. Will send a patch to fix the doc. Regards, willy Regards Alex
Re: Missing doc entities in doc/internals
Hi Alex, On Fri, Oct 20, 2023 at 11:11:59PM +0200, Aleksandar Lazic wrote: > I can't find any doc about entities in the current git > > alex@alex-tuxedoinfinitybooks1517gen7 on 20/10/2023 at 23:06:19 > /datadisk/git-repos/haproxy $ find . -iname "*entities"* > alex@alex-tuxedoinfinitybooks1517gen7 on 20/10/2023 at 23:06:27 > /datadisk/git-repos/haproxy $ > > What's my mistake to find this doc? No mistake, this file was so much outdated that not any single box on it existed in recent versions so it was about time for it to be removed. It was a bit heartbreaking, but killing a 15-years old doc because the architecture evolves is not that bad of a news :-) I suggest that you have a look at doc/internals/api/layers.txt first, then doc/internals/muxes.pdf whicih show the lower layers in boxes and what remains of the stream layer on top as the channel. Regards, willy
Missing doc entities in doc/internals
Hi. As I go thru the filter.txt now is this statement written. https://github.com/haproxy/haproxy/blob/master/doc/internals/api/filters.txt#L50C15-L50C23 ``` First of all, to fully understand how filters work and how to create one, it is best to know, at least from a distance, what is a proxy (frontend/backend), a stream and a channel in HAProxy and how these entities are linked to each other. doc/internals/entities.pdf is a good overview. ``` I can't find any doc about entities in the current git alex@alex-tuxedoinfinitybooks1517gen7 on 20/10/2023 at 23:06:19 /datadisk/git-repos/haproxy $ find . -iname "*entities"* alex@alex-tuxedoinfinitybooks1517gen7 on 20/10/2023 at 23:06:27 /datadisk/git-repos/haproxy $ What's my mistake to find this doc? Regards Alex
Re: [PATCH] MINOR: lua: Add a flag to disable logging to stderr
Following up on Aurélien's remarks (thanks for catching my forgetting
it!), here's an additional patch to update the LUA-specific documentation.
Could be kept standalone or merged into the first patch, but to avoid
re-submitting the patchset already, here it is standalone for now.
Tristan
From 67c3177c51044d288036044f7d17f6f037ac4f55 Mon Sep 17 00:00:00 2001
From: Tristan
Date: Fri, 20 Oct 2023 18:59:03 +0100
Subject: [PATCH] DOC: lua: update core.log and TXN.log documentation
Mention that their behaviour depends on the tunables
introduced in commit 45e6f27140 ("MINOR: lua: Add
flags to configure logging behaviour")
Rather than being entirely precise here and risk drift, simply
refer to the relevant tunables for details.
This should be backported wherever the aforementionned commit is
---
doc/lua-api/index.rst | 12
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/doc/lua-api/index.rst b/doc/lua-api/index.rst
index 8a29bc5b5..9b9dcbd2c 100644
--- a/doc/lua-api/index.rst
+++ b/doc/lua-api/index.rst
@@ -267,8 +267,10 @@ Core class
**context**: body, init, task, action, sample-fetch, converter
This function sends a log. The log is sent, according with the HAProxy
- configuration file, on the default syslog server if it is configured and on
- the stderr if it is allowed.
+ configuration file, to the loggers relevant to the current context and/or
+ stderr.
+
+ The exact behaviour depends on tune.lua.log and tune.lua.log-stderr tunables.
:param integer loglevel: Is the log level associated with the message. It is
a
number between 0 and 7.
@@ -2648,8 +2650,10 @@ TXN class
.. js:function:: TXN.log(TXN, loglevel, msg)
This function sends a log. The log is sent, according with the HAProxy
- configuration file, on the default syslog server if it is configured and on
- the stderr if it is allowed.
+ configuration file, to the loggers relevant to the current context and/or
+ stderr.
+
+ The exact behaviour depends on tune.lua.log and tune.lua.log-stderr tunables.
:param class_txn txn: The class txn object containing the data.
:param integer loglevel: Is the log level associated with the message. It is
--
2.41.0
Re: [PATCH] MINOR: lua: Add a flag to disable logging to stderr
> https://www.arpalert.org/haproxy-api.html (related txn:log() > documentation: > https://www.arpalert.org/src/haproxy-lua-api/2.9/index.html#core.log) Forgot https://www.arpalert.org/src/haproxy-lua-api/2.9/index.html#TXN.log as well (both txn:log(), core:log() and friends with explicit log level in the name such as txn.Alert() end up using the hlua_sendlog() function internally)
Re: [PATCH] Added support for Arrays in sample_conv_json_query in sample.c
On Mon, Sep 25, 2023 at 01:30:53PM +, Jens Popp wrote: > Method now returns the content of Json Arrays, if it is specified in > Json Path as String. The start and end character is a square bracket. Any > complex object in the array is returned as Json, so that you might get Arrays > of Array or objects. Only recommended for Arrays of simple types (e.g., > String or int) which will be returned as CSV String. Also updated > documentation and fixed issue with parenthesis and other changes from > comments. > Hello Jens, Thank you for you patch, I just pushed it into master. I also added a reg-test into the patch: https://github.com/haproxy/haproxy/commit/f216b45bb94475aafcdd855fbf358cc812eb1d33#diff-c29792e4360cebe8528660d3a2da5c0401201d3c0490bf72cfe7b6cef0cb34e3R101 Don't forget to add one next time you contribute so we can easily test if we broke the feature by accident! Regards, -- William Lallemand
Re: [PATCH] MINOR: lua: Add a flag to disable logging to stderr
Hi Tristan, Thanks for the nice work :) Just my 2 cents, in the second patch, since you change the default behavior, you forgot to update your comment from the 1st patch in Lua's doc according to the new behavior: > diff --git a/doc/lua.txt b/doc/lua.txt > index 8d5561668..8d244ab3a 100644 > --- a/doc/lua.txt > +++ b/doc/lua.txt > @@ -630,6 +630,10 @@ It displays a log during the HAProxy startup: > > [alert] 285/083533 (14465) : Hello World ! > > +Note: By default, logs originating from a LUA script are sent to the loggers > +applicable to the current context, if any, and additionally to stderr. See > +tune.lua.log and tune.lua.log-stderr for more information. Note that the lua.txt file is not used much now, and mainly serves as documentation for developers, not very useful for users. If you're willing to reach more people you should probably maintain the doc/lua-api/index.rst file which is used to generate the online documentation at https://www.arpalert.org/haproxy-api.html (related txn:log() documentation: https://www.arpalert.org/src/haproxy-lua-api/2.9/index.html#core.log) For the rest, I'll leave it to Willy :) Regards, Aurelien
Re: [PATCH] MINOR: lua: Add a flag to disable logging to stderr
Hi all again,
Here is the updated patch set after changes based on feedback received.
The change is now split across 2 patches.
Patch 0001 adding:
- tune.lua.log { on | off } (defaults to 'on') for usage of loggers
- tune.lua.log-stderr { on | auto | off } (defaults to 'on') for usage
of stderr
Effectively no change to current behaviour, hence fit for backport.
Patch 0002 changing:
- tune.lua.log-stderr default from 'on' to 'auto'
Changes behaviour, hence no backport.
I guess my only remaining question is whether we want regtests for this?
I'm be happy to write some if yes, but I know regtests runtime is a
concern and it'd require a bit of test infra work to make capturing
stderr output, so I just want to be sure it is desirable before :-)
Regards,
Tristan
From 45e6f271404d20479284fc3e0e1f7448e260e016 Mon Sep 17 00:00:00 2001
From: Tristan
Date: Fri, 20 Oct 2023 16:31:58 +0100
Subject: [PATCH] MINOR: lua: Add flags to configure logging behaviour
Until now, messages printed from LUA log functions were sent both to
the any logger configured for the current proxy, and additionally to
stderr (in most cases)
This introduces two flags to configure LUA log handling:
- tune.lua.log to use standard loggers or not
- tune.lua.log-stderr to use stderr, or not, or only conditionally
This addresses github feature request #2316
This can be backported to 2.8 as it doesn't change previous behaviour.
---
doc/configuration.txt | 22
doc/lua.txt | 4 +++
src/hlua.c| 80 +--
3 files changed, 103 insertions(+), 3 deletions(-)
diff --git a/doc/configuration.txt b/doc/configuration.txt
index 50fe882d0..7c4a585ec 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -1196,6 +1196,8 @@ The following keywords are supported in the "global"
section :
- tune.lua.service-timeout
- tune.lua.session-timeout
- tune.lua.task-timeout
+ - tune.lua.log
+ - tune.lua.log-stderr
- tune.max-checks-per-thread
- tune.maxaccept
- tune.maxpollevents
@@ -3192,6 +3194,26 @@ tune.lua.task-timeout
remain alive during of the lifetime of HAProxy. For example, a task used to
check servers.
+tune.lua.log { on | off }
+ Enables ('on') or disables ('off') logging the output of LUA scripts via the
+ loggers applicable to the current proxy, if any.
+
+ Defaults to 'on'.
+
+tune.lua.log-stderr { on | auto | off }
+ Enables ('on') or disables ('off') logging the output of LUA scripts via
+ stderr.
+ When set to 'auto', logging via stderr is conditionally 'on' if any of:
+
+- tune.lua.log is set to 'off'
+- the script is executed in a non-proxy context with no global logger
+- the script is executed in a proxy context with no logger attached
+
+ Please note that, when enabled, this logging is in addition to the logging
+ configured via tune.lua.log.
+
+ Defaults to 'on'.
+
tune.max-checks-per-thread
Sets the number of active checks per thread above which a thread will
actively try to search a less loaded thread to run the health check, or
diff --git a/doc/lua.txt b/doc/lua.txt
index 8d5561668..8d244ab3a 100644
--- a/doc/lua.txt
+++ b/doc/lua.txt
@@ -630,6 +630,10 @@ It displays a log during the HAProxy startup:
[alert] 285/083533 (14465) : Hello World !
+Note: By default, logs originating from a LUA script are sent to the loggers
+applicable to the current context, if any, and additionally to stderr. See
+tune.lua.log and tune.lua.log-stderr for more information.
+
Default path and libraries
--
diff --git a/src/hlua.c b/src/hlua.c
index e94325727..70a25e762 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -69,6 +69,20 @@
#include
#include
+/* Global LUA flags */
+/* log handling */
+#define HLUA_TUNE_LOG 0x0001 /* send to current logger */
+#define HLUA_TUNE_LOG_STDERR_ON 0x0010 /* send to stderr */
+#define HLUA_TUNE_LOG_STDERR_AUTO 0x0020 /* send to stderr if no logger in
use */
+#define HLUA_TUNE_LOG_STDERR_OFF 0x0040 /* never send to stderr */
+#define HLUA_TUNE_LOG_STDERR_MASK 0x0070
+
+/* default flag values */
+#define HLUA_TUNE_FLAGS_DFLT 0x0011
+
+/* flags made of HLUA_TUNE_* */
+static uint hlua_tune_flags = HLUA_TUNE_FLAGS_DFLT;
+
/* Lua uses longjmp to perform yield or throwing errors. This
* macro is used only for identifying the function that can
* not return because a longjmp is executed.
@@ -1366,8 +1380,9 @@ const char *hlua_show_current_location(const char *pfx)
return NULL;
}
-/* This function is used to send logs. It try to send on screen (stderr)
- * and on the default syslog server.
+/* This function is used to send logs. It tries to send them to:
+ * - the log target applicable in the current context, AND
+ * - stderr if not in quiet mode or explicitly disabled
*/
static inline void hlua_sendlog(struct proxy *px, int level, const char *msg)
{
@@ -1392,8 +1407,28 @@
Two URL's in the same HAProxy with different TLS version for incoming connections
Dear all, I have HAproxy 1.8.27 which has TLS 1.0/1.1/1.2/1.3 support. I have defined two URL's inside the haproxy.cfg: www1.example.com www2.example.com If I test the TLS support connection using openssl and nmap, I get the following: www1.example.com --> TLS 1.2 www2.example.com --> TLS 1.3 Both URL's are configured in the same way in haproxy.cfg, there is no TLS version forcing. If I use openssl with TLS 1.3 test, I get: >openssl s_client -connect www1.example.com:443 -tls1_3 CONNECTED(0278) 2043:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl\record\rec_layer_s3.c:1586:SSL alert number 40 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 7 bytes and written 268 bytes Verification: OK --- New, (NONE), Cipher is (NONE) This TLS version forbids renegotiation. Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) So why www1.example.com negotiates TLS 1.2 and www2.example.com negotiates TLS 1.3 ??? Thanks a lot!!!
Re: [PATCH] MINOR: lua: Add a flag to disable logging to stderr
On 20/10/2023 15:30, Tristan wrote:
ie in this snippet (hlua.c:1387):
static inline void hlua_sendlog(struct proxy *px, ...)
{
...
if (... && (!LIST_ISEMPTY(>loggers)))
return;
has the following results:
- locally from source => compiles happily
- locally from clone + patch => compile error
- in CI from clone + patch => compile error
When it fails, this is what I get back:
src/hlua.c:1417:86: error: ‘struct proxy’ has no member named ‘loggers’
1417 | if ((hlua_tune_flags & (HLUA_TUNE_LOG))
&& (!LIST_ISEMPTY(>loggers)))
| ^~
include/haproxy/list.h:102:28: note: in definition of macro ‘LIST_ISEMPTY’
102 | #define LIST_ISEMPTY(lh) ((lh)->n == (lh))
| ^~
Which would make sense if struct proxy had no such member, but it most
certainly has one...
I must be missing something obvious, which hopefully someone can point
out to me...
Well, Aurélien replied off-list and turns out my build scripts were not
using the same exact commit I was writing it against...
https://github.com/mangadex-pub/haproxy/commit/46c15afd27876c3e260fc0c70234e0aff70422de#r130521133
Thanks Aurélien, and :facepalm: to me
Tristan
Re: [PATCH] MINOR: lua: Add a flag to disable logging to stderr
Hi again Willy,
On 18/10/2023 07:47, Willy Tarreau wrote:
[...]
maybe we can have a 3rd value "auto" which would be the default
and which would only log to stderr if there's no other logger ? I
don't know if we have this info where it's used, though. Hmmm at
first glance we seem to have it by testing if either px->loggers
is non-empty or global.loggers is non-empty. Thus it could be the
nicest approach, having "on" by default in 2.8 and older and "auto"
on 2.9 maybe ?
I did go that route and thus split it into 2 patches (one for the flags
+ 'on' for that, and one that swaps from 'on' to 'auto').
Also from my reading of the config parser it seems to me like the global
loggers are always appended to the proxy's loggers if 'log global' is used.
So I opted to only check the proxy's loggers since I can see no
situations where one has 1) some global loggers and 2) a proxy that
expressly doesn't inherit log global and 3) nor any other proxy-specific
logger target.
Anyway, that small question aside, I'm seeing some pretty strange
discrepancy when compiling in different places.
ie in this snippet (hlua.c:1387):
static inline void hlua_sendlog(struct proxy *px, ...)
{
...
if (... && (!LIST_ISEMPTY(>loggers)))
return;
has the following results:
- locally from source => compiles happily
- locally from clone + patch => compile error
- in CI from clone + patch => compile error
When it fails, this is what I get back:
src/hlua.c:1417:86: error: ‘struct proxy’ has no member named ‘loggers’
1417 | if ((hlua_tune_flags & (HLUA_TUNE_LOG))
&& (!LIST_ISEMPTY(>loggers)))
|
^~
include/haproxy/list.h:102:28: note: in definition of macro ‘LIST_ISEMPTY’
102 | #define LIST_ISEMPTY(lh) ((lh)->n == (lh))
|^~
Which would make sense if struct proxy had no such member, but it most
certainly has one...
I must be missing something obvious, which hopefully someone can point
out to me...
Regards,
Tristan
nb: don't want to waste your time by forcing to review the patches
before I had a chance to properly test them, but if seeing them helps
more than the error, they are here atm:
https://github.com/mangadex-pub/haproxy/tree/93c92592bc693b53a27305152de80f544664f49d/haproxy/patches-dev
Re: Some filter discussion for the future
Hi. FYI: I have created a repo for the rs filter https://github.com/git001/hap-rs-filter feel free to participate/contribute :-) Regards Alex On 2023-10-19 (Do.) 22:53, Aleksandar Lazic wrote: Hi Tristan. On 2023-10-17 (Di.) 10:51, Tristan wrote: Hi Aleksandar, That is a welcome follow-up to the tangent we went on in the announce thread. Thanks :-) As there was the discussion about the future of the SPOE filter, let me start a discussion about some possible filter options. [...] The question which I have is how difficult is it to add a http filter based on httpclient similar to SPOE or FCGI filter. Another option is to add some language specific filter like haproxy-rs-api shown in this comment https://github.com/khvzak/mlua/issues/320#issuecomment-1762027351 . I personally find the latter much more appealing. If only because the http client is "just" a much more restricted version of it. httpclient was just something comes into my mind. Maybe it's a better approch to have a flt_http.c similar to flt_spoe.c then can we use the full feature set of HAP within the backend section. ¯\_(ツ)_/¯ And since I was the first (in that thread, certainly not everywhere) to complain about the current language of choice for extending HAProxy (LUA), I have to say again that a target "language" like WASM sounds like an ideal selection: - no need to pick/enforce/encourage a specific input language - plenty of languages already compile to it, and likely to continue trending up since browsers support it From my point of view looks WASM also very promising as feature Technology but also a little bit a hype, so let's see what the time brings. As I'm quite newbie at WASM I will mainly create any "echo all params" file in shell/perl/go/js or any other language and convert it to WASM :-). The Idea to add the http filter is that there are so many http based tools out there and with that could HAProxy use such tools based on http. That is true, but needing an HTTP API + the loss in efficiency sounds a bit painful. And very painful if the response isn't so easy to parse. Thinking of cases where XML decoding becomes relevant, for example SAML-related ones which are common for auth-related matters still. That's a valid argument. Any opinion on that? Well on my end I certainly want to see this too. That said Willy had a few counterpoints of relevance in that other thread that are worth addressing here: > WASM on the other hand would provide more performance and compile-time > checks but I fear that it could also bring new classes of issues such > as higher memory usage, higher latencies, and would make it less > convenient to deploy updates since these would require to be rebuilt. I'd say first that there are interpreters (and JITs) so the rebuild is not necessary. However, even if it was, I'm not sure that the buildless use-case has that much traction as long as the build doesn't have to happen on the LBs directly. For example I don't remember seeing complaints that SPOEs essentially require a build step. Yep. SPOE is fully integrate into HAP that's so nice on that Protocol. > Also we don't want to put too much of the application into the load balancer. That's a much more fundamental question however. This is your project, not mine, so your call. But I have to emphasize that one reason I use HAProxy is specifically because it's extremely configurable and allows me to offload a lot of application-related logic directly at the edge. Full Ack. In a more impersonal way, that is also a direction many are interested in in general. See things like https://blog.cloudflare.com/cloudflare-snippets-alpha/ which are essentially ACL-triggered filters in HAProxy terms. One example case I see up and again is tee-ing a request, for various reasons: - for silent A/B testing between 2 backends (ie tee to 1 control and 1 test) - for routing the request that triggers a cached response both to the cache and to something interested in it for statistics; so users gets fast response and you still ALSO get to count those requests And of course that has concerns related to memory used for buffering the content if there are 2 targets and thus you can't purely stream through. But in some places it has applicability I think. > But as I said I haven't had a look at the details so I don't know > if we can yield like in Lua, implement our own bindings for internal > functions, or limit the memory usage per request being processed. That is much more difficult for me to answer, so to save you some time these seem to be the 3 main C-embeddable runtimes at the time of writing: - https://github.com/bytecodealliance/wasm-micro-runtime - https://github.com/wasm3/wasm3 - https://github.com/wasmerio/wasmer I had a look and however didn't see a way to control memory or force yielding... so it's not encouraging. But maybe I missed it. Well that's another option to add
