RES: RES: RES: RES: RES: RES: RES: RES: RES: High CPU Usage (HaProxy)
OK. The last point could slightly help in reducing the number of calls to kqueue and aggregate more events at once. But FreeBSD's kqueue is really fast so that should not change much. You really need to be able to pin the processes to certain CPUs, as well as the interrupts. Unfortunately I cannot be of any help here :-( But do you believe the CPU pinning will really make all this difference ? I know how to do it, using pthread, because I am used with it, just a few lines of code are able to make it.
RES: RES: RES: RES: RES: RES: RES: RES: High CPU Usage (HaProxy)
Hey, Willy. I've switch to haproxy 1.5 (last one available on the website), but the results didn't change much. However, I didn't try to run all the proxies in just one single process, to check the difference yet. -Mensagem original- De: Fred Pedrisa [mailto:fredhp...@hotmail.com] Enviada em: terça-feira, 5 de novembro de 2013 13:33 Para: 'Willy Tarreau' Cc: 'Lukas Tribus'; 'haproxy@formilux.org' Assunto: RES: RES: RES: RES: RES: RES: RES: RES: High CPU Usage (HaProxy) OK. Do you know if you have a single or multiple interrupts on your NICs, and if they're delivered to a single core, multiple cores, or floating around more or less randomly ? This is managed by FreeBSD, it currently have multiple queues and irq balance with msix. It seems that your numbers below tend to confirm this model. I still don't know why you have that high a context switch rate. Are you running with more processes than CPUs ? Also it looks like the system is mostly spending its time idling. Is it that haproxy is on the same CPU as the network's interrupts ? Then maybe it could make sense to start multiple processes and pin them to specific CPU cores, and do the same with the interrupts. Delivering 500-bytes large messages between two NICs via userspace experiences a high overhead and everything which could be saved must be saved (including CPU cache misses). Yes, if we have 40 processes running and 16 physical cores, I suppose this is more than the number of physical cores available right ? However, in FreeBSD we can't do that IRQ Assigning, like we can on linux. (As far I know). We are speaking about 100Kpps (input) and 140Kpps (output) 'approximately'. OK, so probably about 30k msg/s in each direction with their respective ACKs. That just makes me think it could possibly do better since we can do better with HTTP messages. Do you have enough concurrent connections to fill the wire and ensure that the system never waits for either a client or a server ? I'm assuming that OK given the values assigned to the file descriptors in your latest email, which were up to 1428. With such numbers and that small messages, it can make sense to use multiple processes if that's not the case yet. In theory yes, the connections are quick, because they are pure tcp applications and in other cases, http websites, but behind the pure tcp mode instead of http mode (not in all cases tho). Fred
RES: RES: RES: RES: RES: RES: RES: RES: High CPU Usage (HaProxy)
OK. Do you know if you have a single or multiple interrupts on your NICs, and if they're delivered to a single core, multiple cores, or floating around more or less randomly ? This is managed by FreeBSD, it currently have multiple queues and irq balance with msix. It seems that your numbers below tend to confirm this model. I still don't know why you have that high a context switch rate. Are you running with more processes than CPUs ? Also it looks like the system is mostly spending its time idling. Is it that haproxy is on the same CPU as the network's interrupts ? Then maybe it could make sense to start multiple processes and pin them to specific CPU cores, and do the same with the interrupts. Delivering 500-bytes large messages between two NICs via userspace experiences a high overhead and everything which could be saved must be saved (including CPU cache misses). Yes, if we have 40 processes running and 16 physical cores, I suppose this is more than the number of physical cores available right ? However, in FreeBSD we can't do that IRQ Assigning, like we can on linux. (As far I know). We are speaking about 100Kpps (input) and 140Kpps (output) 'approximately'. OK, so probably about 30k msg/s in each direction with their respective ACKs. That just makes me think it could possibly do better since we can do better with HTTP messages. Do you have enough concurrent connections to fill the wire and ensure that the system never waits for either a client or a server ? I'm assuming that OK given the values assigned to the file descriptors in your latest email, which were up to 1428. With such numbers and that small messages, it can make sense to use multiple processes if that's not the case yet. In theory yes, the connections are quick, because they are pure tcp applications and in other cases, http websites, but behind the pure tcp mode instead of http mode (not in all cases tho). Fred
Re: RES: RES: RES: RES: RES: RES: RES: RES: High CPU Usage (HaProxy)
On 05 нояб. 2013 г., at 19:33, Fred Pedrisa fredhp...@hotmail.com wrote: However, in FreeBSD we can't do that IRQ Assigning, like we can on linux. (As far I know). JFYI: you can assign IRQs to CPUs via cpuset -x irq (I can’t tell you if it is “like on linux” or not though).
