Re: [ANNOUNCE] haproxy-2.2-dev12
сб, 4 июл. 2020 г. в 11:13, Willy Tarreau : > Hi, > > HAProxy 2.2-dev12 was released on 2020/07/04. It added 72 new commits > after version 2.2-dev11. > > Yes, a 12th development release. But the good news is that it's just > here to help with testing because we've finally managed to address the > performance regression issue spotted by William Dauchy! And it was quite > a tough one, so it was a good decision to invested so many efforts on > this one before the release. > > To make a long story short, when you have many very fast servers, almost > all of a server's idle connections could be used then released by a thread > at once, and taken over by the next thread and so on, never leaving spare > ones for other threads. And this takeover would go through the global run > queue and cause contention there when using a moderate number of threads. > So that wouldn't affect low-performance users but definitely was a > performance killer for high performance ones dealing with tens to hundreds > of thousands of requests per second. > > The great thing is that by fixing all these issues we've had to implement > a few improvements that were anticipated for later and that this made the > internal infrastructure a bit better and further improved the overall > performance gap from 2.1. > > Since 2.2-dev11, the most user-visible changes are: > - fixed the performance regression above > > - addition of the new "pool-low-conn" server setting to improve > distribution > of idle connections on very fast servers (sub-millisecond response > time). > We've found that using twice the number of threads seems to provide > very > good performance. > > - added a few new fields in the stats page to report the number of idle > and used connections per server > > - new "show servers conn" command on the CLI to visualize the state of > used > and idle connections of a server, including per-thread > > - small change on the log-format processing: historically, multiple > spaces > were merged together as a single separator. This was OK for real logs, > but is a bit annoying when building headers, and very annoying for > error > pages. So this was changed so that only logs merge spaces. This should > probably be addressed in a more generic way later, but this was the > most > reasonble approach for this release. > > - the RFC5424 log format was missing the sub-second and timezone fields, > the former being highly recommended and the latter being mandatory. So > this was addressed right before having a new LTS version. I'm not much > tempted by backporting this to stable releases because that could > result > in visible changes that are not welcome in the middle of a stable > version, > that's why I asked to have it right now. > > - a few sample fetches and patterns were missing the trailing NUL > character > and wouldn't always match (I don't remember which ones, sorry). This > will > likely be backported as it was a bug. > > - threads are now disabled by default on OpenBSD which lacks thread-local > storage and fails to build. Clang seems to emulate it so users of clang > can enable USE_THREAD=1 if they want. > > - "show sess" would endlessly dump new streams when they arrive too fast. > It was a real pain so now it will only dump past the last stream known > at the moment the command is typed. This means that it may show less > streams than the total, but will not result in multi-gigabyte dumps > anymore. > > - for developers, building with DEBUG_MEM_STATS provides a new expert > command "debug dev memstats" which shows the total counts (calls and > sizes) of memory allocations per line of code. This is very cheap and > can be enabled on production servers if suspecting a memory leak > somewhere (and it served to spot a regression in a recent fix). > > In addition to this, William is finishing the addition of a sample fetch > to extract the equivalent of the TLS pre-master key for TLS 1.3, which > is needed in order to decrypt TLS traffic with Wireshark. It would be > useful to have this early so that those who upgrade can place that in > their logs if that can help them. > > Christopher addressed a few other low-importance bugs in the private > connections management. Since they were made available just before this > release and we've all been very tired by looking at these bugs over the > last weeks, I preferred that we let these cool down and we look at them > after a small rest; we've accidently broken enough stuff while working > on the fixes above, I didn't want to take the risk of creating new > breakage. It looks like 3 of these patches could be merged before the > release (they already affect previous versions) and the other ones could > be merged post-2.2 then backported once considered safe enough. > > Tim also had some post-2.2 fixes pending to improve free() calls and > remove some valgrind
Re: [ANNOUNCE] haproxy-2.2-dev12
сб, 4 июл. 2020 г. в 11:13, Willy Tarreau : > Hi, > > HAProxy 2.2-dev12 was released on 2020/07/04. It added 72 new commits > after version 2.2-dev11. > > Yes, a 12th development release. But the good news is that it's just > here to help with testing because we've finally managed to address the > performance regression issue spotted by William Dauchy! And it was quite > a tough one, so it was a good decision to invested so many efforts on > this one before the release. > > To make a long story short, when you have many very fast servers, almost > all of a server's idle connections could be used then released by a thread > at once, and taken over by the next thread and so on, never leaving spare > ones for other threads. And this takeover would go through the global run > queue and cause contention there when using a moderate number of threads. > So that wouldn't affect low-performance users but definitely was a > performance killer for high performance ones dealing with tens to hundreds > of thousands of requests per second. > > The great thing is that by fixing all these issues we've had to implement > a few improvements that were anticipated for later and that this made the > internal infrastructure a bit better and further improved the overall > performance gap from 2.1. > > Since 2.2-dev11, the most user-visible changes are: > - fixed the performance regression above > > - addition of the new "pool-low-conn" server setting to improve > distribution > of idle connections on very fast servers (sub-millisecond response > time). > We've found that using twice the number of threads seems to provide > very > good performance. > > - added a few new fields in the stats page to report the number of idle > and used connections per server > > - new "show servers conn" command on the CLI to visualize the state of > used > and idle connections of a server, including per-thread > > - small change on the log-format processing: historically, multiple > spaces > were merged together as a single separator. This was OK for real logs, > but is a bit annoying when building headers, and very annoying for > error > pages. So this was changed so that only logs merge spaces. This should > probably be addressed in a more generic way later, but this was the > most > reasonble approach for this release. > > - the RFC5424 log format was missing the sub-second and timezone fields, > the former being highly recommended and the latter being mandatory. So > this was addressed right before having a new LTS version. I'm not much > tempted by backporting this to stable releases because that could > result > in visible changes that are not welcome in the middle of a stable > version, > that's why I asked to have it right now. > > - a few sample fetches and patterns were missing the trailing NUL > character > and wouldn't always match (I don't remember which ones, sorry). This > will > likely be backported as it was a bug. > > - threads are now disabled by default on OpenBSD which lacks thread-local > storage and fails to build. Clang seems to emulate it so users of clang > can enable USE_THREAD=1 if they want. > > - "show sess" would endlessly dump new streams when they arrive too fast. > It was a real pain so now it will only dump past the last stream known > at the moment the command is typed. This means that it may show less > streams than the total, but will not result in multi-gigabyte dumps > anymore. > > - for developers, building with DEBUG_MEM_STATS provides a new expert > command "debug dev memstats" which shows the total counts (calls and > sizes) of memory allocations per line of code. This is very cheap and > can be enabled on production servers if suspecting a memory leak > somewhere (and it served to spot a regression in a recent fix). > > In addition to this, William is finishing the addition of a sample fetch > to extract the equivalent of the TLS pre-master key for TLS 1.3, which > is needed in order to decrypt TLS traffic with Wireshark. It would be > useful to have this early so that those who upgrade can place that in > their logs if that can help them. > > Christopher addressed a few other low-importance bugs in the private > connections management. Since they were made available just before this > release and we've all been very tired by looking at these bugs over the > last weeks, I preferred that we let these cool down and we look at them > after a small rest; we've accidently broken enough stuff while working > on the fixes above, I didn't want to take the risk of creating new > breakage. It looks like 3 of these patches could be merged before the > release (they already affect previous versions) and the other ones could > be merged post-2.2 then backported once considered safe enough. > > Tim also had some post-2.2 fixes pending to improve free() calls and > remove some valgrind
Re: [ANNOUNCE] haproxy-2.2-dev12
On Sat, Jul 04, 2020 at 02:42:53PM +0200, Tim Düsterhus wrote: > Willy, > > Am 04.07.20 um 12:11 schrieb Willy Tarreau: > > Thank you. I now rebased next and applied your series on top of it. I'm > > not going to review what goes into -next, or it defeats the purpose of > > using it as a pending queue. There's enough time so that anyone can > > suggest a change and even if some issues get merged, we have 6 months > > to spot them till next release. > > Okay, fair enough. Just make sure to read the backporting information > when moving next to master after the release. For some of the patches > backporting might make sense, for others not so much. > > But I suppose those are going to be treated just like regular BUGs and > will automatically be presented during the backporting sessions? Absolutely. We read each and every single commit message during backports to gauge whether it's relevant or not and if it depends on other patches or not (or what's the impact of not taking it if it's too painful/dangerous to backport). That's why I'm annoying with those commit messages :-) > 15 emails per hour is one every 4 minutes. If I'd be sending emails at > that rate I'd just use an actual chat :-) Except when they're to different people and on various subjects because you're responding to accumulated stuff while you're waiting for a test to finish or for someone to fix an issue that's blocking you. In short, e-mail allows asynchronous batch processing which is super important to limit inefficient context switching. Willy
Re: [ANNOUNCE] haproxy-2.2-dev12
Willy, Am 04.07.20 um 12:11 schrieb Willy Tarreau: > Thank you. I now rebased next and applied your series on top of it. I'm > not going to review what goes into -next, or it defeats the purpose of > using it as a pending queue. There's enough time so that anyone can > suggest a change and even if some issues get merged, we have 6 months > to spot them till next release. Okay, fair enough. Just make sure to read the backporting information when moving next to master after the release. For some of the patches backporting might make sense, for others not so much. But I suppose those are going to be treated just like regular BUGs and will automatically be presented during the backporting sessions? >> I had a little dispute with the limits I configured within my mail >> server. My own account is restricted to sending 15 emails per hour for >> abuse prevention in case my password ever leaks. So 2 patches did not >> make it through the first time. I hope I did not completely mess up >> threading when sending the last 2. My email limit is now up to 20 :-) > > I would be terribly handicaped with such a configuration, that would > force me to stop even more frequently and chat with coworkers :-) > 15 emails per hour is one every 4 minutes. If I'd be sending emails at that rate I'd just use an actual chat :-) That is of course unless I send largish patch bombs in an automated fashion. Best regards Tim Düsterhus
Re: [ANNOUNCE] haproxy-2.2-dev12
On Sat, Jul 04, 2020 at 11:57:08AM +0200, Tim Düsterhus wrote: > > It's up to you. I personally like to keep my topic branch till the last > > moment because that leaves more more freedom to revisit my patches later, > > but when I'm certain patches are definitive, I prefer to get them merged > > and to forget them. So you decide, and both are fine to me. > > > > Okay I sent them now. Details are in the cover letter of 'v2'. Thank you. I now rebased next and applied your series on top of it. I'm not going to review what goes into -next, or it defeats the purpose of using it as a pending queue. There's enough time so that anyone can suggest a change and even if some issues get merged, we have 6 months to spot them till next release. > I had a little dispute with the limits I configured within my mail > server. My own account is restricted to sending 15 emails per hour for > abuse prevention in case my password ever leaks. So 2 patches did not > make it through the first time. I hope I did not completely mess up > threading when sending the last 2. My email limit is now up to 20 :-) I would be terribly handicaped with such a configuration, that would force me to stop even more frequently and chat with coworkers :-) Have a nice week-end, Willy
Re: [ANNOUNCE] haproxy-2.2-dev12
Willy, Am 04.07.20 um 11:13 schrieb Willy Tarreau: >> Or you tell me to send it now and apply it to 'next'. > > It's up to you. I personally like to keep my topic branch till the last > moment because that leaves more more freedom to revisit my patches later, > but when I'm certain patches are definitive, I prefer to get them merged > and to forget them. So you decide, and both are fine to me. > Okay I sent them now. Details are in the cover letter of 'v2'. I had a little dispute with the limits I configured within my mail server. My own account is restricted to sending 15 emails per hour for abuse prevention in case my password ever leaks. So 2 patches did not make it through the first time. I hope I did not completely mess up threading when sending the last 2. My email limit is now up to 20 :-) Best regards Tim Düsterhus
Re: [ANNOUNCE] haproxy-2.2-dev12
On Sat, Jul 04, 2020 at 10:17:50AM +0200, William Dauchy wrote: > Hello Willy, > > On Sat, Jul 4, 2020 at 8:11 AM Willy Tarreau wrote: > > HAProxy 2.2-dev12 was released on 2020/07/04. It added 72 new commits > > after version 2.2-dev11. > > maybe the package is being uploaded but I can't find it on > http://www.haproxy.org/download/2.2/src/devel/ Ah this is because I'm stupid :-) On haproxy.org we run with *all* debugging and strict options available (including the expensive DEBUG_UAF), that's how I detected the issue when building with DEBUG_LOCK. And working on fixing this while I was typing the announce message, I simply forgot to type "publish-and-update" :-) It's fixed now. > > > - for developers, building with DEBUG_MEM_STATS provides a new expert > > command "debug dev memstats" which shows the total counts (calls and > > sizes) of memory allocations per line of code. This is very cheap and > > can be enabled on production servers if suspecting a memory leak > > somewhere (and it served to spot a regression in a recent fix). > > without forgetting the useful reset argument, "debug dev memstats > reset" which then allows you to see the allocation starting from a > specific moment. Yes, I thought it was really needed. Also please note that this doesn't count the free() calls as it uses #define and a few variables are called "free", so there's no difference and one has to use it as an aid and nothing else. But when you see 3 GB missing from your memory and you spot a line which totalled 3 GB since last reset, it definitely encourages to go into that direction first ;-) Cheers, Willy
Re: [ANNOUNCE] haproxy-2.2-dev12
Hi Tim, On Sat, Jul 04, 2020 at 10:58:12AM +0200, Tim Düsterhus wrote: > Willy, > > Am 04.07.20 um 08:10 schrieb Willy Tarreau: > > Tim also had some post-2.2 fixes pending to improve free() calls and > > remove some valgrind complaints on exit. > > > > Good you mention it, because I forgot to give a heads up: The series > grew a little bit further. My 'deinit' branch as of now is 15 commits > ahead of master. So I'll send an updated series once 2.2 is out. Perfect! > Or you tell me to send it now and apply it to 'next'. It's up to you. I personally like to keep my topic branch till the last moment because that leaves more more freedom to revisit my patches later, but when I'm certain patches are definitive, I prefer to get them merged and to forget them. So you decide, and both are fine to me. > While I'm here: > There's already one commit sitting in the next branch to be applied > after the release. Yep I remember, I think it was about strict limits enforcements. Cheers, Willy
Re: [ANNOUNCE] haproxy-2.2-dev12
Willy, Am 04.07.20 um 08:10 schrieb Willy Tarreau: > Tim also had some post-2.2 fixes pending to improve free() calls and > remove some valgrind complaints on exit. > Good you mention it, because I forgot to give a heads up: The series grew a little bit further. My 'deinit' branch as of now is 15 commits ahead of master. So I'll send an updated series once 2.2 is out. Or you tell me to send it now and apply it to 'next'. While I'm here: There's already one commit sitting in the next branch to be applied after the release. Best regards Tim Düsterhus
Re: [ANNOUNCE] haproxy-2.2-dev12
Hello Willy, On Sat, Jul 4, 2020 at 8:11 AM Willy Tarreau wrote: > HAProxy 2.2-dev12 was released on 2020/07/04. It added 72 new commits > after version 2.2-dev11. maybe the package is being uploaded but I can't find it on http://www.haproxy.org/download/2.2/src/devel/ > - for developers, building with DEBUG_MEM_STATS provides a new expert > command "debug dev memstats" which shows the total counts (calls and > sizes) of memory allocations per line of code. This is very cheap and > can be enabled on production servers if suspecting a memory leak > somewhere (and it served to spot a regression in a recent fix). without forgetting the useful reset argument, "debug dev memstats reset" which then allows you to see the allocation starting from a specific moment. -- William
[ANNOUNCE] haproxy-2.2-dev12
Hi, HAProxy 2.2-dev12 was released on 2020/07/04. It added 72 new commits after version 2.2-dev11. Yes, a 12th development release. But the good news is that it's just here to help with testing because we've finally managed to address the performance regression issue spotted by William Dauchy! And it was quite a tough one, so it was a good decision to invested so many efforts on this one before the release. To make a long story short, when you have many very fast servers, almost all of a server's idle connections could be used then released by a thread at once, and taken over by the next thread and so on, never leaving spare ones for other threads. And this takeover would go through the global run queue and cause contention there when using a moderate number of threads. So that wouldn't affect low-performance users but definitely was a performance killer for high performance ones dealing with tens to hundreds of thousands of requests per second. The great thing is that by fixing all these issues we've had to implement a few improvements that were anticipated for later and that this made the internal infrastructure a bit better and further improved the overall performance gap from 2.1. Since 2.2-dev11, the most user-visible changes are: - fixed the performance regression above - addition of the new "pool-low-conn" server setting to improve distribution of idle connections on very fast servers (sub-millisecond response time). We've found that using twice the number of threads seems to provide very good performance. - added a few new fields in the stats page to report the number of idle and used connections per server - new "show servers conn" command on the CLI to visualize the state of used and idle connections of a server, including per-thread - small change on the log-format processing: historically, multiple spaces were merged together as a single separator. This was OK for real logs, but is a bit annoying when building headers, and very annoying for error pages. So this was changed so that only logs merge spaces. This should probably be addressed in a more generic way later, but this was the most reasonble approach for this release. - the RFC5424 log format was missing the sub-second and timezone fields, the former being highly recommended and the latter being mandatory. So this was addressed right before having a new LTS version. I'm not much tempted by backporting this to stable releases because that could result in visible changes that are not welcome in the middle of a stable version, that's why I asked to have it right now. - a few sample fetches and patterns were missing the trailing NUL character and wouldn't always match (I don't remember which ones, sorry). This will likely be backported as it was a bug. - threads are now disabled by default on OpenBSD which lacks thread-local storage and fails to build. Clang seems to emulate it so users of clang can enable USE_THREAD=1 if they want. - "show sess" would endlessly dump new streams when they arrive too fast. It was a real pain so now it will only dump past the last stream known at the moment the command is typed. This means that it may show less streams than the total, but will not result in multi-gigabyte dumps anymore. - for developers, building with DEBUG_MEM_STATS provides a new expert command "debug dev memstats" which shows the total counts (calls and sizes) of memory allocations per line of code. This is very cheap and can be enabled on production servers if suspecting a memory leak somewhere (and it served to spot a regression in a recent fix). In addition to this, William is finishing the addition of a sample fetch to extract the equivalent of the TLS pre-master key for TLS 1.3, which is needed in order to decrypt TLS traffic with Wireshark. It would be useful to have this early so that those who upgrade can place that in their logs if that can help them. Christopher addressed a few other low-importance bugs in the private connections management. Since they were made available just before this release and we've all been very tired by looking at these bugs over the last weeks, I preferred that we let these cool down and we look at them after a small rest; we've accidently broken enough stuff while working on the fixes above, I didn't want to take the risk of creating new breakage. It looks like 3 of these patches could be merged before the release (they already affect previous versions) and the other ones could be merged post-2.2 then backported once considered safe enough. Tim also had some post-2.2 fixes pending to improve free() calls and remove some valgrind complaints on exit. With all the energy spent on the bugs above I couldn't work at all on the doc review I wanted to do. I'll try to do this shortly but it will not be as refined as I'd had expected. Anyway, I now consider 2.2 ready.
