On Thu, Jul 3, 2014 at 8:27 PM, Maxime Brugidou <[email protected]> wrote: > > On Jul 3, 2014 7:19 PM, "Sasha Pachev" <[email protected]> wrote: >> >> Some ideas: >> >> - what happens if you up the size of the request ? > > We increased the request to more than 2500 bytes and it has an effect on the > soft interrupt CPU for the cpu 0 (the one where we pinned the IRQ, not the > one where we pinned haproxy). This makes sens since we don't use jumbo > frames and we probably double or tripled the number of RX packets. However > we arrive in the stunning situation where both cpu 0 and cpu 1 have around > 25 to 30% of soft interrupt (at least according to top). > > This does not affect at all the session rate, stuck to ~16k/s. This makes me > thing that the cpu 1 soft interrupt time is really abnormal and that the cpu > 0 si time is normal (normal = coming from a hard interrupt on eth0) right? > > >> - what results do you get with haproxy 1.4 ? > > Very similar result, same blocker around ~16k session/s with haproxy at 100% > CPU (same pattern with user, sys and si). It all looks like a kernel or > hardware or stupid config issue somewhere. > > >> - have you tried gdb -p $(pidof haproxy) --batch --ex "set pagination >> 0" --ex "thread apply all bt" ? Sometimes a few random stacktrace >> samples help you stumble upon an idea of where your bottleneck is. >> > > I ran a couple of them but without much success since i'm not very fluent in > this stuff :) from all the stacktraces i got nothing stood out and the code > didn't seem blocked somewhere special. > > Additional note: we get around 300k TIME_WAIT connections but use tw_reuse > so it shouldn't be an issue i think
Hi Maxime, First of all, a few questions: - what is the exact reference of your CPU ? - what is the frequency of your CPU ? - what is the command line you run on the "client" side (siege) - have you disabled irq-balance ? - what type of network interface are you using? (and which driver) - are you benchmarking in keep-alive mode or not? Then a few remarks: - you don't need to enable "USE_LINUX_SPLICE=1 USE_LIBCRYPT=1" at compilation time. These options are implicitly enabled by the target you chose (linux2628) - hyperthreading is not useful in your case, you can disable it - you can bind HAProxy process to a core using HAProxy's configuration directive 'cpu-map' (much better than taskset) - splicing is totally counter productive with a so small object, please disable it - option http-server-close was the best mode in HAProxy 1.4. In 1.5, you should prefer 'option http-keep-alive' which may help in case the client and the server are compatible with it - are you sure you CPU network IO binding was right? A few tests you could run: - disable logging and tell us if there is an improvement or not (if yes, how much) - make HAProxy to answer directly without accessing nginx and check if you doubled the performance or not (and report us the result) - run your nginx server on a separated server and tell us the performance you can reach Baptiste
