Re: [ANNOUNCE] haproxy-3.1.0
On Tue, Nov 26, 2024 at 09:57:11PM +0100, William Lallemand wrote: > Hi, > > On Tue, Nov 26, 2024 at 09:44:46PM +0100, Willy Tarreau wrote: > > > > > > This is interesting. I did not yet look into how exactly it works, but > > > perhaps this would also enabled structured logging with systemd-journald, > > > i.e. this issue: https://github.com/haproxy/haproxy/issues/771? > > > > Hmmm possibly. I don't even remember that issue and I suspect I missed > > it. I'm not seeing anything controversial in it so it should at least > > have got responses :-( > > > > I didn't see the issue either, but I'm seeing something controversial in it > :-) > I really don't want to bring back a dependency to libsystemd in HAProxy, so > we'll need to reimplement sd_journal_send() if we want to implement this. Ah I didn't notice. Probably that it's just another syscall again. And maybe instead we could directly use that FD with our regular send path. Just guessing. Willy
Re: [ANNOUNCE] haproxy-3.1.0
Hi, On Tue, Nov 26, 2024 at 09:44:46PM +0100, Willy Tarreau wrote: > > > > This is interesting. I did not yet look into how exactly it works, but > > perhaps this would also enabled structured logging with systemd-journald, > > i.e. this issue: https://github.com/haproxy/haproxy/issues/771? > > Hmmm possibly. I don't even remember that issue and I suspect I missed > it. I'm not seeing anything controversial in it so it should at least > have got responses :-( > I didn't see the issue either, but I'm seeing something controversial in it :-) I really don't want to bring back a dependency to libsystemd in HAProxy, so we'll need to reimplement sd_journal_send() if we want to implement this. https://www.freedesktop.org/software/systemd/man/latest/sd_journal_print.html https://systemd.io/JOURNAL_NATIVE_PROTOCOL/ Regards, -- William Lallemand
Re: [ANNOUNCE] haproxy-3.1.0
Hi Tim! On Tue, Nov 26, 2024 at 08:07:54PM +0100, Tim Düsterhus wrote: > Willy, > > On 11/26/24 16:48, Willy Tarreau wrote: > > HAProxy 3.1.0 was released on 2024/11/26. It added 41 new commits > > after version 3.1-dev14. > > Congratulations on the release. Thanks! > I've had regretfully little time / energy to > follow the development and the list in the past year or so (I still owe you > a response on the UUIDv7 patch), but it would not be a proper release > without an email of mine :-) No need to be sorry about that, we all have periods of more busy times and that's normal and expected. Simply don't be shy to come back more often once you have more time again. BTW I indeed remember a past discussion about UUIDv7, but no worries, don't add mental load for stuff that's not on a critical path. > I'm pretty sure we've had this already the last time and you told me that it > was fixed, but the "Changelog" column for 3.1 uses dashes instead of slashes > for the date separator. I just don't understand how I manage to do that each time, especially since I'm duplicating the line and restarting a script. I *suspect* that the full release script vs the dev release just use different formats, which might explain why it reproduces each time. But I could be wrong. I should check them because the date format is inconsistent between release date and the changelog. I used to use slashes a lot in the past until I realized that the dash is more universal and the digits more readable with a dash than a slash. I'll leave it as-is for now so that I can test the scripts. > I'm also noticing that the "Release date" column has > a tilde for 3.0 and 3.1, but now that the versions are relased it's no > longer an approximate date :-) At least this one is caused by my copy-paste of the first line which contains it! I'll adjust my note to put a mention about it ;-) > > - automatic encoding: logs can now be emitted in a structured way, > >using JSON or CBOR, with each field named after its original name or > >via an explicit alias described in the log-format line (using > >"%(...)", see sections 8.2.6 and 8.3.5). This naming will later also > >permit to perform some operations, matching, or on-the-fly > >transformations. > > This is interesting. I did not yet look into how exactly it works, but > perhaps this would also enabled structured logging with systemd-journald, > i.e. this issue: https://github.com/haproxy/haproxy/issues/771? Hmmm possibly. I don't even remember that issue and I suspect I missed it. I'm not seeing anything controversial in it so it should at least have got responses :-( I'll let Aurélien have a look, as he's definitely becoming the log expert :-) Cheers, Willy
Re: [ANNOUNCE] haproxy-3.1.0
Willy, On 11/26/24 16:48, Willy Tarreau wrote: HAProxy 3.1.0 was released on 2024/11/26. It added 41 new commits after version 3.1-dev14. Congratulations on the release. I've had regretfully little time / energy to follow the development and the list in the past year or so (I still owe you a response on the UUIDv7 patch), but it would not be a proper release without an email of mine :-) I'm pretty sure we've had this already the last time and you told me that it was fixed, but the "Changelog" column for 3.1 uses dashes instead of slashes for the date separator. I'm also noticing that the "Release date" column has a tilde for 3.0 and 3.1, but now that the versions are relased it's no longer an approximate date :-) - automatic encoding: logs can now be emitted in a structured way, using JSON or CBOR, with each field named after its original name or via an explicit alias described in the log-format line (using "%(...)", see sections 8.2.6 and 8.3.5). This naming will later also permit to perform some operations, matching, or on-the-fly transformations. This is interesting. I did not yet look into how exactly it works, but perhaps this would also enabled structured logging with systemd-journald, i.e. this issue: https://github.com/haproxy/haproxy/issues/771? Best regards Tim Düsterhus
[ANNOUNCE] haproxy-3.1.0
Hi, HAProxy 3.1.0 was released on 2024/11/26. It added 41 new commits after version 3.1-dev14. That's a final version as I like them, with two long-standing bugs fixed, a few cross-platform build issues addressed as well, and the rest being doc updates, CLI help messages and tiny code tidying patches. As with every release, you'll hear me say that this is the best ever release with tons of fixes and improvements, blah blah blah. So this time I took the time to review all of the 1119 commits since 3.0.0 to summarize everything below, and judge by yourself: First, due to the significant amount of time spent dealing with increasingly complex bug reports from recent versions during this development cycle, a large part of the time left for development was dedicated to improving the troubleshooting and observability in order to reduce the number of round trips between developers and users when chasing a problem, as well as to limit the guess work on the developers' side. The principle essentially consists in focusing on certain specific events in logs, traces, and even core dumps while minimizing the user impact of enabling the collection of these precious information. In this specific area, we can mention: - Log production and processing: - new last_entity and waiting_entity to indicate in the logs what internal entity was being processed, or blocked waiting for more data (filters, compression, Lua, request body, file:line of a rule etc). These allow to better understand what was happening when a timeout or error struck. - fc.debug_str / bc.debug_str: reports lots of internal information from the front and back connections (stream ID, connection flags, etc). These are mostly used by developers and can take a lot of room on the log line. See converter "when" below. - converter "when": allows to pass the contents of its input to the output only when a defined condition is met. The condition might be "error", "stopping", "acl:" and a few others, in order to define what's of interest to extend the log output. Combined with "fc/bc.debug_str", it allows to only emit detailed info when something abnormal happens (e.g. unusually long response). - glitch counters: now the HTTP/2 and QUIC multiplexers are able to count small anomalies that are not necessarily fatal for a connection but may be used to force a connection to be closed above a certain threshold. These can be retrieved at log time so that logs indicate a level of suspiciousness of the request. - connection-level error happening during data transfers are now reported upstream so that they can appear numerically in "fc_err" and "bc_err", or as full strings in "fc_err_str" and "bc_err_str", or as short strings (mostly errno-like) in the new "fc_err_name" and "bc_err_name". For example ERESET will indicate that recv() or send() faced an ECONNRESET status code by the operating system. This can be useful when the logs conflict with network capture, to detect that something else is acting in the OS itself (e.g. internal firewall rules, resource limits in the TCP stack, or bug at the kernel level). - CLI: - the "show info" command will now report the current and total number of streams, hence requests. It can help quickly detect if a slowdown is caused on the client side or the server side, as well as ease the production of activity metrics. - a new option "show-uri" of the command "show sess" to show the URI of a request in progress. This can help figure if a certain URL is mostly responsible for slow responses for example. - the number of calls to filters and other internal entities attached to a stream is now reported on "show sess". This is particularly convenient when using Lua filters or SPOE extensions, to figure if and where something odd is happening. - the "show dev" output will indicate many process-specific info such as permissions, capabilities, resource limits etc at both boot and runtime. These can sometimes explain some suspiciously low limits (e.g. users running in containers sometimes see their settings ignored for various reasons, now they'll be able to control after starting the process). - "show quic" now produces way more internal information about the internal state of the congestion control algorithm, and other dynamic metrics (window size, bytes in flight, counters etc). - Traces: - a new "traces" section permits to configure traces and to enable them at startup time if desired. The old "trace" keyword in the global section which was under experimental status is no longer accepted, and the traces do not require the experimental status anymore. The "trace" keyword now supports grouping operations between multiple s
