Re: [ANNOUNCE] haproxy-2.5.2
Hi, On Thu, Feb 17, 2022 at 08:25:39AM +0100, Willy Tarreau wrote: > By the way Vincent, William figured that I missed a few patches during my > long backport session yesterday. I'll have to check with him if they > warrant another release or if they can wait for next one. Hence no need > to rush on new packages yet ;-) I'll keep you updated whatever the > outcome. > I'll probably emit a 2.5.3 this evening or tomorrow, some of the forgotten fixes could be bothersome for people trying to migrate in 2.5. -- William Lallemand
Re: [ANNOUNCE] haproxy-2.5.2
On Thu, Feb 17, 2022 at 07:57:30AM +0100, Vincent Bernat wrote: > ? 16 February 2022 22:15 +01, Willy Tarreau: > > > That's exactly the sense behind the word "maybe" above, to open the > > discussion :-) Those with large buffers can definitely see a > > difference. I've seen configs with WAF analysis using 1MB buffers, > > and there the extra CPU usage will be noticeable, maybe 5-10%. My > > impression is that the vast majority of users rely on distro packages > > and are not sensitive to performance (typically sites like haproxy.org > > where enabling everything has no measurable impact, when I'm lucky I > > see 1% CPU). Those who deal with high levels of traffic tend to be > > forced to follow stable updates more often, they'll typically build > > from the Git tree, and are also more at ease with debugging options. > > That was my reasoning, it may be wrong, and I perfectly understand > > your point which is equally valid. And I'm not even asking for a > > change, just saying "maybe it would be even better if". > > For Debian, being a binary distribution, we cannot be flexible with the > users. In the past, we were often told we were less performant than a > source distribution because we didn't enable this or this optimization. > Also, 1% CPU increase could also translate to increased latency. I agree. I know that the vast majority of us do not care, but I know a few places where that matters. Those who have to manage 100 LBs definitely don't want to go to 101 just because we changed an option (but arguably when performing major version upgrades, variations are larger than that in both directions). > As a comparison, we did not have memory cgroups in our kernels until the > overhead was reduced quite significantly when not using them. On our > side, we believe everyone is using Debian packages. ;-) Oh I'm not surprised! I'll work more on the runtime configuration of most of these settings, as I think the most expensive hence controversial ones are the ones which should easily support adding a runtime test. For the most sensitive parts (e.g. BUG_ON() in scheduler), that should still be addressed at build time but on a case-by-case basis. I'll come back trying to propose a better long-term solution for all this. By the way Vincent, William figured that I missed a few patches during my long backport session yesterday. I'll have to check with him if they warrant another release or if they can wait for next one. Hence no need to rush on new packages yet ;-) I'll keep you updated whatever the outcome. Cheers, Willy
Re: [ANNOUNCE] haproxy-2.5.2
❦ 16 February 2022 22:15 +01, Willy Tarreau: > That's exactly the sense behind the word "maybe" above, to open the > discussion :-) Those with large buffers can definitely see a > difference. I've seen configs with WAF analysis using 1MB buffers, > and there the extra CPU usage will be noticeable, maybe 5-10%. My > impression is that the vast majority of users rely on distro packages > and are not sensitive to performance (typically sites like haproxy.org > where enabling everything has no measurable impact, when I'm lucky I > see 1% CPU). Those who deal with high levels of traffic tend to be > forced to follow stable updates more often, they'll typically build > from the Git tree, and are also more at ease with debugging options. > That was my reasoning, it may be wrong, and I perfectly understand > your point which is equally valid. And I'm not even asking for a > change, just saying "maybe it would be even better if". For Debian, being a binary distribution, we cannot be flexible with the users. In the past, we were often told we were less performant than a source distribution because we didn't enable this or this optimization. Also, 1% CPU increase could also translate to increased latency. As a comparison, we did not have memory cgroups in our kernels until the overhead was reduced quite significantly when not using them. On our side, we believe everyone is using Debian packages. ;-) -- Be careful of reading health books, you might die of a misprint. -- Mark Twain
Re: [ANNOUNCE] haproxy-2.5.2
On Wed, Feb 16, 2022 at 09:57:45PM +0100, Christian Ruppert wrote: > On 2022-02-16 19:08, Vincent Bernat wrote: > > ? 16 February 2022 16:27 +01, Willy Tarreau: > > > > > Maybe that would even be a nice improvement for distros to provide > > > these > > > by default starting with 2.6 or maybe even 2.5. > > > > Why not enabling them directly on your side then? Are there some numbers > > on the performance impact of these options? I am a bit uncomfortable > > providing packages that perform slower than an upstream build. > > Do you want all those options to be enabled in distro packages or just some > specific? I don't know, as I mentioned in my previous response, it could be just some or even none for now, waiting for finer granularity. > Esp. for the ones that make up to 1-2% CPU usage I'd second > Vicent's idea of enabling it by default. So anybody has the option to > disable it, if 1-2% or perhaps some ns/µs delay really matters that much. The difficulty is that the ratio can vary based on some use cases (esp with buffer sizes), and we need to keep a sweet spot between performance and difficulty of deploying something for a particular user case. But once these are split and re-arranged, it could become easier to decide. I agree with Vincent in general about the fact that the distro should not deviate much from the original setup, and we've even changed some default options in the past to preserve this sane principle. For now I'm just trying to gauge interest and starting to put the focus on these possibilities for those who know they can easily afford a small perf drop and who think that it will not change anything for them, particularly if it shortens the life of the bugs they're facing. The granularity remains a bit too coarse right now to ask users to decide before testing, and for us to decide for all of them. Maybe we'll figure reasonable ways to turn some options to dynamic in the future (thinking about what's done with pools, I'm pretty sure that would be possible for almost half of the options, this would solve the problem). I'm still interested in this discussion and your opinions on this (and do not hesitate to violently disagree with me, my goal is to figure what's best for most users, while avoiding traps for newcomers). Cheers, Willy
Re: [ANNOUNCE] haproxy-2.5.2
Hi Vincent, On Wed, Feb 16, 2022 at 07:08:38PM +0100, Vincent Bernat wrote: > ? 16 February 2022 16:27 +01, Willy Tarreau: > > > Maybe that would even be a nice improvement for distros to provide these > > by default starting with 2.6 or maybe even 2.5. > > Why not enabling them directly on your side then? Are there some numbers > on the performance impact of these options? I am a bit uncomfortable > providing packages that perform slower than an upstream build. That's exactly the sense behind the word "maybe" above, to open the discussion :-) Those with large buffers can definitely see a difference. I've seen configs with WAF analysis using 1MB buffers, and there the extra CPU usage will be noticeable, maybe 5-10%. My impression is that the vast majority of users rely on distro packages and are not sensitive to performance (typically sites like haproxy.org where enabling everything has no measurable impact, when I'm lucky I see 1% CPU). Those who deal with high levels of traffic tend to be forced to follow stable updates more often, they'll typically build from the Git tree, and are also more at ease with debugging options. That was my reasoning, it may be wrong, and I perfectly understand your point which is equally valid. And I'm not even asking for a change, just saying "maybe it would be even better if". What I'd like to do for 2.6 and beyond would be to have multiple levels of protection/debugging classified by impacts. The vast majority of the BUG_ON() we have have absolutely zero impact. Some in the past were placed long after the code was written just to confirm that what was understood was correct. Thus we couldn't enable them by default. Then we started to place a lot more like plain assert() but still disabled by default to avoid affecting performance. And due to this raising concern about performance we don't put any into very sensitive places where it could help for the vast majority of users. So my goal would be to enable by default all those which have no visible impact, and let users easily change them in case of trouble. Similarly some of the DEBUG options will likely be enabled by default when the impact is tiny. Nowadays for example I think we can afford to lose 8 bytes in an allocated area to store the pointer to the last caller (especially for free). This might possibly save one week to one month of round trips in an issue, depending on the frequency of crashes for a given report. Once we manage to establish a balanced set of protection mechanisms and debugging options, we can better document the ones that can save the last few percent of performance or memory consumption, and the ones that improve the accuracy of bug reports. In this case maybe some users will more naturally enable some of them to get more solid reports (we all prefer to produce undisputable bug reports, as there's nothing more irritating than a developer expressing doubts about their validity). The options I mentioned today do not yet have this level of granularity, they will have an impact, albeit a small one, hence why I'd prefer to ask on a voluntary basis only. With some of the usual reporters, this is something that is regularly done when asked, and I think that openly indicating the costs and benefits around this allows us to progressively get out of a debug-centric model and start to look into the direction of a more generally proactive model. There will always be exceptions anyway, but finer grained control is necessary to enable such stuff by default in its current form. Cheers, Willy
Re: [ANNOUNCE] haproxy-2.5.2
❦ 16 February 2022 16:27 +01, Willy Tarreau: > Maybe that would even be a nice improvement for distros to provide these > by default starting with 2.6 or maybe even 2.5. Why not enabling them directly on your side then? Are there some numbers on the performance impact of these options? I am a bit uncomfortable providing packages that perform slower than an upstream build. -- Don't stop with your first draft. - The Elements of Programming Style (Kernighan & Plauger)
