Re: Load testing HAProxy 2.2 on x86_64 and aarch64 VMs
чт, 23 июл. 2020 г. в 19:28, Martin Grigorov : > Hi Илья, > > I didn't have much success with Google Perf Tools. > I've tried both with adding '-lprofiler' to LDFLAGS when building HAProxy > and with LD_PRELOAD. > In both cases on ARM64 it fails with: terminated by signal SIGSEGV > (Address boundary error) > > On x86_64 there is no such issue but the produced result file is always > empty. > I can try on your hardware if you give me an access, > > $ ldd haproxy > linux-vdso.so.1 (0x7ffcfb5cd000) > libprofiler.so.0 => /usr/local/lib/libprofiler.so.0 > (0x7f2115507000)<< > libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 > (0x7f21154cc000) > libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x7f21154b) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f21154aa000) > librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f211549f000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > (0x7f211547c000) > libssl.so.1.1 => /home/ubuntu/opt/lib/libssl.so.1.1 > (0x7f21151e6000) > libcrypto.so.1.1 => /home/ubuntu/opt/lib/libcrypto.so.1.1 > (0x7f2114cf4000) > liblua5.3.so.0 => /usr/lib/x86_64-linux-gnu/liblua5.3.so.0 > (0x7f2114cb9000) > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f2114b6a000) > libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 > (0x7f2114abd000) > libpcreposix.so.3 => /usr/lib/x86_64-linux-gnu/libpcreposix.so.3 > (0x7f2114ab8000) > libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 > (0x7f2114a43000) > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 > (0x7f2114a28000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f2114836000) > libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 > (0x7f2114819000) > libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > (0x7f2114638000) > /lib64/ld-linux-x86-64.so.2 (0x7f211552e000) > liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 > (0x7f211460f000) > liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 > (0x7f21145ee000) > libgcrypt.so.20 => /usr/lib/x86_64-linux-gnu/libgcrypt.so.20 > (0x7f21144d) > libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 > (0x7f21144ad000) > > CPUPROFILE=/tmp/haproxy-load-x64.prof haproxy -p pid.txt -f haproxy.cfg > > The above command creates /tmp/haproxy-load-x64.prof but it is never > populated with any data. I've tried stopping HAProxy with signals INT, KILL > and TERM. > I even tried with env var CPUPROFILESIGNAL=12 to start/stop the profiler > manually but again no success. > > I could share with you reports from Linux 'perf' command. Just let me know > which events you'd be interested in! > perf is somewhat strange, I recall really weird reports from it. maybe something was wrong with my hands. yeah, let us have a look at perf from both arm64 and amd64 > > Regards, > Martin > > On Sat, Jul 18, 2020 at 11:40 AM Илья Шипицин > wrote: > >> Hello, Martin! >> >> Can you please compare load profiles using google perftools ? >> >> I never tried to use gperf on ARM64, also, my trial at Linaro is over, I >> do not have an access to any ARM64 anymore. >> in short, gperf can be found https://github.com/gperftools/gperftools >> >> please follow "CPU profiling part". >> >> it can collect cachegrind output, I attached example kcachegrind report >> (you can sort by "self" time). >> it would be interesting to compare amd64 <--> arm64 >> >> [image: Screenshot from 2020-07-18 13-35-27.png] >> >> >> пт, 10 июл. 2020 г. в 19:00, Martin Grigorov : >> >>> Hello HAProxy community, >>> >>> I wanted to compare how the newly released HAProxy 2.2 (Congrats!) >>> behaves under heavy load so I've ran some tests on my x86_64 and aarch64 >>> VMs: >>> >>> >>> https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566 >>> >>> Without much surprise the x86_64 VM gave better results! >>> >>> It is *not* a real use case scenario: the backends serve on GET / and >>> return "Hello World", without any file/network operations. >>> >>> What is interesting though is that I can get 120-160K reqs/sec when >>> hitting directly one of the backend servers, and only 20-40K reqs/sec when >>> using HAProxy as a load balancer in front of them. >>> >>> I'd be happy to re-run the tests with any kind of improvements you may >>> have! >>> >>> Regards, >>> Martin >>> >>
Re: Load testing HAProxy 2.2 on x86_64 and aarch64 VMs
Hi Илья, I didn't have much success with Google Perf Tools. I've tried both with adding '-lprofiler' to LDFLAGS when building HAProxy and with LD_PRELOAD. In both cases on ARM64 it fails with: terminated by signal SIGSEGV (Address boundary error) On x86_64 there is no such issue but the produced result file is always empty. $ ldd haproxy linux-vdso.so.1 (0x7ffcfb5cd000) libprofiler.so.0 => /usr/local/lib/libprofiler.so.0 (0x7f2115507000)<< libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x7f21154cc000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x7f21154b) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f21154aa000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f211549f000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f211547c000) libssl.so.1.1 => /home/ubuntu/opt/lib/libssl.so.1.1 (0x7f21151e6000) libcrypto.so.1.1 => /home/ubuntu/opt/lib/libcrypto.so.1.1 (0x7f2114cf4000) liblua5.3.so.0 => /usr/lib/x86_64-linux-gnu/liblua5.3.so.0 (0x7f2114cb9000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f2114b6a000) libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x7f2114abd000) libpcreposix.so.3 => /usr/lib/x86_64-linux-gnu/libpcreposix.so.3 (0x7f2114ab8000) libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x7f2114a43000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f2114a28000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f2114836000) libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x7f2114819000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x7f2114638000) /lib64/ld-linux-x86-64.so.2 (0x7f211552e000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x7f211460f000) liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x7f21145ee000) libgcrypt.so.20 => /usr/lib/x86_64-linux-gnu/libgcrypt.so.20 (0x7f21144d) libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x7f21144ad000) CPUPROFILE=/tmp/haproxy-load-x64.prof haproxy -p pid.txt -f haproxy.cfg The above command creates /tmp/haproxy-load-x64.prof but it is never populated with any data. I've tried stopping HAProxy with signals INT, KILL and TERM. I even tried with env var CPUPROFILESIGNAL=12 to start/stop the profiler manually but again no success. I could share with you reports from Linux 'perf' command. Just let me know which events you'd be interested in! Regards, Martin On Sat, Jul 18, 2020 at 11:40 AM Илья Шипицин wrote: > Hello, Martin! > > Can you please compare load profiles using google perftools ? > > I never tried to use gperf on ARM64, also, my trial at Linaro is over, I > do not have an access to any ARM64 anymore. > in short, gperf can be found https://github.com/gperftools/gperftools > > please follow "CPU profiling part". > > it can collect cachegrind output, I attached example kcachegrind report > (you can sort by "self" time). > it would be interesting to compare amd64 <--> arm64 > > [image: Screenshot from 2020-07-18 13-35-27.png] > > > пт, 10 июл. 2020 г. в 19:00, Martin Grigorov : > >> Hello HAProxy community, >> >> I wanted to compare how the newly released HAProxy 2.2 (Congrats!) >> behaves under heavy load so I've ran some tests on my x86_64 and aarch64 >> VMs: >> >> >> https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566 >> >> Without much surprise the x86_64 VM gave better results! >> >> It is *not* a real use case scenario: the backends serve on GET / and >> return "Hello World", without any file/network operations. >> >> What is interesting though is that I can get 120-160K reqs/sec when >> hitting directly one of the backend servers, and only 20-40K reqs/sec when >> using HAProxy as a load balancer in front of them. >> >> I'd be happy to re-run the tests with any kind of improvements you may >> have! >> >> Regards, >> Martin >> >
Re: Load testing HAProxy 2.2 on x86_64 and aarch64 VMs
Hi Илья, I will do it sometimes this week! Regards, Martin On Sat, Jul 18, 2020 at 11:40 AM Илья Шипицин wrote: > Hello, Martin! > > Can you please compare load profiles using google perftools ? > > I never tried to use gperf on ARM64, also, my trial at Linaro is over, I > do not have an access to any ARM64 anymore. > in short, gperf can be found https://github.com/gperftools/gperftools > > please follow "CPU profiling part". > > it can collect cachegrind output, I attached example kcachegrind report > (you can sort by "self" time). > it would be interesting to compare amd64 <--> arm64 > > [image: Screenshot from 2020-07-18 13-35-27.png] > > > пт, 10 июл. 2020 г. в 19:00, Martin Grigorov : > >> Hello HAProxy community, >> >> I wanted to compare how the newly released HAProxy 2.2 (Congrats!) >> behaves under heavy load so I've ran some tests on my x86_64 and aarch64 >> VMs: >> >> >> https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566 >> >> Without much surprise the x86_64 VM gave better results! >> >> It is *not* a real use case scenario: the backends serve on GET / and >> return "Hello World", without any file/network operations. >> >> What is interesting though is that I can get 120-160K reqs/sec when >> hitting directly one of the backend servers, and only 20-40K reqs/sec when >> using HAProxy as a load balancer in front of them. >> >> I'd be happy to re-run the tests with any kind of improvements you may >> have! >> >> Regards, >> Martin >> >
Re: Load testing HAProxy 2.2 on x86_64 and aarch64 VMs
Hello, Martin! Can you please compare load profiles using google perftools ? I never tried to use gperf on ARM64, also, my trial at Linaro is over, I do not have an access to any ARM64 anymore. in short, gperf can be found https://github.com/gperftools/gperftools please follow "CPU profiling part". it can collect cachegrind output, I attached example kcachegrind report (you can sort by "self" time). it would be interesting to compare amd64 <--> arm64 [image: Screenshot from 2020-07-18 13-35-27.png] пт, 10 июл. 2020 г. в 19:00, Martin Grigorov : > Hello HAProxy community, > > I wanted to compare how the newly released HAProxy 2.2 (Congrats!) behaves > under heavy load so I've ran some tests on my x86_64 and aarch64 VMs: > > > https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566 > > Without much surprise the x86_64 VM gave better results! > > It is *not* a real use case scenario: the backends serve on GET / and > return "Hello World", without any file/network operations. > > What is interesting though is that I can get 120-160K reqs/sec when > hitting directly one of the backend servers, and only 20-40K reqs/sec when > using HAProxy as a load balancer in front of them. > > I'd be happy to re-run the tests with any kind of improvements you may > have! > > Regards, > Martin >
Load testing HAProxy 2.2 on x86_64 and aarch64 VMs
Hello HAProxy community, I wanted to compare how the newly released HAProxy 2.2 (Congrats!) behaves under heavy load so I've ran some tests on my x86_64 and aarch64 VMs: https://medium.com/@martin.grigorov/compare-haproxy-performance-on-x86-64-and-arm64-cpu-architectures-bfd55d1d5566 Without much surprise the x86_64 VM gave better results! It is *not* a real use case scenario: the backends serve on GET / and return "Hello World", without any file/network operations. What is interesting though is that I can get 120-160K reqs/sec when hitting directly one of the backend servers, and only 20-40K reqs/sec when using HAProxy as a load balancer in front of them. I'd be happy to re-run the tests with any kind of improvements you may have! Regards, Martin
