subscribe
-- Martin Schmid Wolfwilerstrasse 57 CH-4626 Niederbuchsiten www.haeschmi.ch
[no subject]
-- Martin Schmid Wolfwilerstrasse 57 CH-4626 Niederbuchsiten www.haeschmi.ch
Partner of photobook printing
Dear Vendor, We are a professional printing service supplier specialized in printing all kinds of photobooks, hardcover books, softcover books, magazines, brochures and gift bags, etc for over ten years in China. Our advanced printing system (Heidelberg Printers) enables us to offer faster turnaround with minimal human error and lower overhead cost, which adds value to customers with consistently high-quality printing at favorable prices. We promise to give the best quality and favorable prices. Our price is about 60% of your local printing costs. Why?should you choose a right book printer? It's very important to choose a right book printer, because the?right printer can use the good paper materials and strict production process to control the quality of finished books. It requires you to just place an order, you don't need to check the quality and printing work at all, and all the work after it will be finished by us. Thus you can save your time and spirit, and will spend more time on your work. We believe that you will make a wise choice. Best Regards Peter
Re: About CPU usage
Hello Dmitry, On Thu, Sep 10, 2015 at 06:12:34PM +0300, Dmitry Sivachenko wrote: > Hello, > > I have a haproxy-1.5.11 with a single frontend passing requests to a single > backend in TCP mode (sample config). > Application establishes several long-living tcp connections and sends a lot > of small requests over them. > > In my test case I have 2 simultaneous TCP connections producing about 3 > MB/sec, 20 kpps input (as shown by netstat on backend machine) and > approximately the same output. > > haproxy process consumes about 20% of CPU core (I have a machine with 2 Intel > Xeon E5-2650 v2 @ 2.60GHz). > > In my understanding such CPU usage is rather high for the relatively low load. It's not necessarily high in fact, what you're observing is the system's capacity with no event aggregation. For each input message, you probably have one NIC IRQ, one TCP segment processed, one wake-up of haproxy, one poll() wake-up event, one recv() call, one send() call, one attempt to send a segment to the NIC, one update of the polled FDs, one call to poll(), one ACK in return, one NIC IRQ, one wake-up of the TCP stack to empty the send queue, then one CPU idle call. In practice at higher loads, the whole cycle above happends once for multiple messages/packets. That's why it's always very hard to predict the absolute maximum performance based on CPU usage alone. The graph of the load per packet rate generally represents an increasing sawtooth curve with smaller teeth as load increases. Since you're seeing only 20% CPU, in your case the load is probably low so you're at the beginning of the curve. If you want to get a very approximative idea of the maximum load you can handle with a single event at once, you can simulate this in haproxy (but you won't be able to do it in the system). You'll have to force the tune.maxpollevents value to 1, and stress the system. HAProxy will be in the worst possible case but that will give you an idea of the maximum performance in this situation, hence where you are on the curve with your current load. > Anything obvious I can tune? You really need to understand that current systems are designed for event aggregation all the way down to the hardware. NICs implement scatter-gather/LRO/TSO and IRQ mitigation. Systems implement NAPI, softirq and GRO/GSO. Poll() supports multiple events. HAproxy uses this as well. Without any form of even aggregation, the effective system's capacity can be much lower than the nominal performance (typically 10-20% of the nominal perf). So that's why the measured CPU usage doesn't grow linearly with the workload. By the way, sometimes even the CPU's ability to switch to idle mode comes with a latency cost, just like the calls to the scheduler's idle thread. There are situations where running a busy loop at nice +19 in the background results in lower latency on proxied HTTP requests! This explains why when you progressively increase the load, you'll see the processing time decrease at some points before starting to increase again. That's why I can only recommend you to find the limit for your workload (including the maximum latency you're willing to accept) and trust that limit. You may find that past a certain load, running IRQs on certain CPU cores and haproxy on another core sharing the same cache (L2 or L3) will bring you more power with lower latency. Don't move it to a different CPU socket though, the IPI (inter-processor-interrupt) can be very slow and the lack of cache sharing will cost a lot of performance and latency. Willy
Re: Multiple log entries with exact same Tq, Tc, Tr and Tt
Hello Dave,
On Fri, Sep 11, 2015 at 04:36:44PM -0400, Dave Stern wrote:
> We run haproxy 1.5.14 on Ubuntu 14.04 in AWS. It load balances connections
> from heroku to our backend, a neo4j cluster of multiple instances. It also
> terminates SSL and handles auth. Connections to the backend are over
> private network space within our AWS network and via unencrypted http.
>
> During a recent event with heavy load, we saw log entries with the same
> repeating pattern of multiple lines with the exact same Tq, Tc, Tr and Tt
> values, but with different requests from different clients routed to
> different backend pool members. We're not sure what's causing this or if it
> relates to performance issues we saw during our events. Here's some log
> lines with excerpted fields. We send a custom header with the query name
> for metrics, which I've replaced with generic entries. I also replaced the
> real client IP with 1.1.1.1, 2.2.2.2, etc.
>
>
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37480
> bs=db_production_read:production-04 hdrs="{query_1}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37481
> bs=db_production_read:production-02 hdrs="{query_2}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37482
> bs=db_production_read:production-03 hdrs="{query_3}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37484
> bs=db_production_read:production-04 hdrs="{query_4}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37483
> bs=db_production_read:production-02 hdrs="{query_5}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37485
> bs=db_production_read:production-03 hdrs="{query_6}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37486
> bs=db_production_read:production-05 hdrs="{query_7}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37488
> bs=db_production_read:production-03 hdrs="{query_8}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37487
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37488
> bs=db_production_read:production-04 hdrs="{query_10}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37489
> bs=db_production_read:production-02 hdrs="{query_11}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37490
> bs=db_production_read:production-03 hdrs="{query_12}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37491
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37492
> bs=db_production_read:production-04 hdrs="{query_13}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53958
> bs=db_production_read:production-02 hdrs="{query_10}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53959
> bs=db_production_read:production-03 hdrs="{query_11}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53960
> bs=db_production_read:production-04 hdrs="{query_12}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37489
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37493
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53961
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37490
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53962
> bs=db_production_write:production-01 hdrs="{query_14}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37491
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53963
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37495
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53964
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=3.3.3.3:53965
> bs=db_production_read:production-02 hdrs="{query_13}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37496
> bs=db_production_write:production-01 hdrs="{query_9}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37498
> bs=db_production_read:production-04 hdrs="{query_15}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37492
> bs=db_production_read:production-02 hdrs="{query_14}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=1.1.1.1:37499
> bs=db_production_write:production-01 hdrs="{query_13}"
> Sep 10 20:00:00 Tq=1060 Tw=0 Tc=1204 Tr=922 Tt=3186 cip=2.2.2.2:37493
>
Re: rate limiting according to "total time" - possible ?
Hi Roland, On Fri, Sep 11, 2015 at 05:11:11PM +0300, Roland RoLaNd wrote: > hello > i have haproxy directing traffic to a number of backends. > these backends can auto scale upon traffic; my goal is to change "maxcon" > depending on "total time" or "backend time" that a request took to complete. > for example: > if totaltime < 1 second ; maxcon = 1000if totaltime < 2 second: maxconn = > 500etc... > > the goal is to hold connections in queue till backend auto scaling is in > effect. > > Can i do the above scenario within haproxy config or a cron that checks > haproxy socket/totaltime and act accordingly is a better idea? > > do you have an alternative advice for me to accomplish that goal ? I could be wrong, but I think you're trying to re-implement by hand the dynamic rate limiting you can get using minconn,maxconn and fullconn. It dynamically increases or decreases the effective per-server maxconn depending on the total number of connections on all servers in the backend so that queues decrease when connection count increases. Willy
