Hi,
I've been running with haproxy (dev18 on Ubuntu 12.04) under load for a few
days, and then it stopped responding to most requests. Any suggestions to
help me figure out what happened?
Here's what I know:
* From my web browser, requests would time out to the haproxy server
* I have an Amazon ELB in front of haproxy, and it reported hundreds of
errors per minute (presumably from timeouts)
* the access log on haproxy shows less than 10 5XX errors for the affected
time period
* there doesn't seem to be any spike in usage around the affected time
period
* According to my ELB log, I'm getting about 500-1500 HTTP requests per
minute
I'm pretty sure the problem is my front end, not the backend
* I could access backends directly through the web browser
* the stats page showed no backends (or frontends for that matter) as
"down".
On the OS level
* CPU load has been averaging 15% for the last few days, but during this
period went up to about 50-60%.
My theories are:
* I hit a capacity limit in my configuration OR
* there's some kind of accumulating resource problem OR
* my server is under configured.
I'm running on an Amazon m1.small instance (1 core, 1.7G ram). Is that
sufficient?
My settings [in part]
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
user haproxy
group haproxy
daemon
stats socket /tmp/haproxy level admin
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
option http-server-close
maxconn 2000
contimeout 10000
clitimeout 180000
srvtimeout 180000
When I looked at the "show info" command (while the problem was occurring)
Name: HAProxy
Version: 1.5-dev18
Release_date: 2013/04/03
Nbproc: 1
Process_num: 1
Pid: 27906
Uptime: 1d 17h56m14s
Uptime_sec: 150974
Memmax_MB: 0
Ulimit-n: 8238
Maxsock: 8238
Maxconn: 4096
Hard_maxconn: 4096
Maxpipes: 0
CurrConns: 160
PipesUsed: 0
PipesFree: 0
ConnRate: 1
ConnRateLimit: 0
MaxConnRate: 93
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 204
Run_queue: 1
Idle_pct: 100
node: www1-new
description:
And the stats for my front end (excerpt from "show stats")
pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,
www,FRONTEND,,,162,260,2000,812931,4954376259,80496776875,0,0,89279,,,,,OPEN,,,,,,,,,1,25,0,,,,0,1,0,93,,,,0,3430598,379470,111021,1064,3449,,1,509,3925669,,,0,0,0,0,
WILL
--
Forio Online Simulations
Will Glass-Husain
415 440-7500x89
[email protected]
www.forio.com
