Re: dev 2.2 High CPU Constantly
I was lucky with google cpu profiler https://github.com/gperftools/gperftools it could summarize cpu time per function. can you try it ? пт, 3 июл. 2020 г. в 23:20, Willy Tarreau : > Hi Igor, > > On Fri, Jul 03, 2020 at 12:52:35PM +0800, Igor Pav wrote: > > Hi William, Tried but still the same ;( > > That's bad. Do you know if your servers actually support 0rtt, and if > this 0rtt currently works between haproxy and the servers ? Because by > having the retry on 0rtt, there are two things which can have an impact > on your CPU usage: > - the alloc+memcpy() of the request buffer before sending it, in order > to be able to send it again if needed ; depending on your bandwidth > this may have an impact ; > > - if 0rtt constantly fails, haproxy would retry without it, so you > could actually be facing the double of the work on the request > processing. > > For the last one you should have a look at your stats page to see if the > retries column increases. You may also want to try without "allow-0rtt" > on the server lines and see if that fixes it. If so, we might be getting > closer (which doesn't mean I have anything in mind about it yet). > > Did 2.1 work fine for you on the same setup ? > > Thanks, > Willy > >
Re: dev 2.2 High CPU Constantly
Hi Igor, On Fri, Jul 03, 2020 at 12:52:35PM +0800, Igor Pav wrote: > Hi William, Tried but still the same ;( That's bad. Do you know if your servers actually support 0rtt, and if this 0rtt currently works between haproxy and the servers ? Because by having the retry on 0rtt, there are two things which can have an impact on your CPU usage: - the alloc+memcpy() of the request buffer before sending it, in order to be able to send it again if needed ; depending on your bandwidth this may have an impact ; - if 0rtt constantly fails, haproxy would retry without it, so you could actually be facing the double of the work on the request processing. For the last one you should have a look at your stats page to see if the retries column increases. You may also want to try without "allow-0rtt" on the server lines and see if that fixes it. If so, we might be getting closer (which doesn't mean I have anything in mind about it yet). Did 2.1 work fine for you on the same setup ? Thanks, Willy
Re: dev 2.2 High CPU Constantly
Hi William, Tried but still the same ;( On Fri, Jul 3, 2020 at 2:35 AM William Dauchy wrote: > > Hi Igor, > > On Thu, Jul 2, 2020 at 9:57 AM Igor Pav wrote: > > By using dev11, the CPU consumption drops a lot, but when connections > > reach ~1000, the CPU would still go high, remove the 0rtt-rejected > > from conf, the CPU becomes normal... > > if you have the opportunity to test the last HEAD of the repo it could > be useful. > I believe this could be fixed by last Willy commit: > https://git.haproxy.org/?p=haproxy.git;a=commit;h=18ed789ae262382e3fe89288132d709c114575aa > -- > William
Re: dev 2.2 High CPU Constantly
Hi Igor, On Thu, Jul 2, 2020 at 9:57 AM Igor Pav wrote: > By using dev11, the CPU consumption drops a lot, but when connections > reach ~1000, the CPU would still go high, remove the 0rtt-rejected > from conf, the CPU becomes normal... if you have the opportunity to test the last HEAD of the repo it could be useful. I believe this could be fixed by last Willy commit: https://git.haproxy.org/?p=haproxy.git;a=commit;h=18ed789ae262382e3fe89288132d709c114575aa -- William
Re: dev 2.2 High CPU Constantly
Hi Igor, On Thu, Jul 02, 2020 at 03:54:37PM +0800, Igor Pav wrote: > Hi Willy, > > By using dev11, the CPU consumption drops a lot, but when connections > reach ~1000, the CPU would still go high, remove the 0rtt-rejected > from conf, the CPU becomes normal... Thank your for the report, that is useful. At the moment I have no idea how 0-rtt is related to this, nor do I know the exact sequence that is executed in this case. But given that it's one of the retry-on settings, I'm suspecting that something prevents from getting away from the connect loop. I know that Christopher has already rechecked the retry code recently and didn't find a case where a retry could be performed without being decremented. So there's definitely something odd there. Note that we've addresses a number of remaining issues around CPU usage and connection reuse since dev11 but I don't see how they could be affected by your case above, and be careful if you test latest HEAD as apparently we broke something that I still can't reproduce :-/ Cheers, Willy
Re: dev 2.2 High CPU Constantly
Hi Willy, By using dev11, the CPU consumption drops a lot, but when connections reach ~1000, the CPU would still go high, remove the 0rtt-rejected from conf, the CPU becomes normal... On Fri, Jun 26, 2020 at 5:48 PM Willy Tarreau wrote: > > Hi Igor, > > On Fri, Jun 12, 2020 at 03:00:31PM +0800, Igor Pav wrote: > > Hi, are those log lines both in syslog? I didn't see it there. I'm > > using this simple setup for a forward HTTP proxy, sooner and later, > > CPU goes crazy. > > Sorry for this late reply. The "bogus stream" message William was talking > about is sent to stderr, so possibly it's logged by your service manager. > > Please note that the bug that caused this for William was addressed in dev10 > but that there's still a CPU usage issue floating around. So it would be > cool if you could try latest master (dev10+latest fixes) to check if you're > still facing this problem. If so, more info about your workload could help > us. > > Thanks! > Willy
Re: dev 2.2 High CPU Constantly
Hi Igor, On Fri, Jun 12, 2020 at 03:00:31PM +0800, Igor Pav wrote: > Hi, are those log lines both in syslog? I didn't see it there. I'm > using this simple setup for a forward HTTP proxy, sooner and later, > CPU goes crazy. Sorry for this late reply. The "bogus stream" message William was talking about is sent to stderr, so possibly it's logged by your service manager. Please note that the bug that caused this for William was addressed in dev10 but that there's still a CPU usage issue floating around. So it would be cool if you could try latest master (dev10+latest fixes) to check if you're still facing this problem. If so, more info about your workload could help us. Thanks! Willy
Re: dev 2.2 High CPU Constantly
Hi, are those log lines both in syslog? I didn't see it there. I'm using this simple setup for a forward HTTP proxy, sooner and later, CPU goes crazy. On Fri, Jun 12, 2020 at 12:24 AM William Dauchy wrote: > > Hello Igor, > > On Thu, Jun 11, 2020 at 5:25 PM Igor Pav wrote: > > We got a very high CPU constantly while using 2.2dev. Any suggestion? > > Thanks. > > Do you have more inputs of what you are doing to trigger that? > does it end at some point with an abort? e.g do you have in logs > things like "Thread X is about to kill the process". If so could you > provide the full logs of the error? > or do you have logs starting with "A bogus STREAM" -> in that case it > can be related to what we are currently looking at in > https://github.com/haproxy/haproxy/issues/662 > > Thanks, > -- > William
Re: dev 2.2 High CPU Constantly
Hello Igor, On Thu, Jun 11, 2020 at 5:25 PM Igor Pav wrote: > We got a very high CPU constantly while using 2.2dev. Any suggestion? Thanks. Do you have more inputs of what you are doing to trigger that? does it end at some point with an abort? e.g do you have in logs things like "Thread X is about to kill the process". If so could you provide the full logs of the error? or do you have logs starting with "A bogus STREAM" -> in that case it can be related to what we are currently looking at in https://github.com/haproxy/haproxy/issues/662 Thanks, -- William
