Hi folks - I've had a bit of an unexpected turn after upgrading one of
my load balancers from OpenBSD 5.0 to 5.2 today. Previously this
machine ran pound beautifully, so I'm surprised about the change.
Here's my setup:
- OpenBSD 5.2/i386, a very vanilla install
- OpenSSL 1.0.1e built with:
./config --prefix=/usr/local/openssl-1.0.1e
--openssldir=/usr/local/openssl-1.0.1e threads zlib shared
- ldconfig adjustmented appropriately via
shlib_dirs="/usr/local/lib /usr/local/openssl/lib"
- Pound 2.7a built from source with these configure flags:
--prefix=/usr/local --with-ssl=/usr/local/openssl --enable-cert1l
Yes, I verified that pound is using the right openssl lib:
# ldd /usr/local/sbin/pound
/usr/local/sbin/pound:
Start End Type Open Ref GrpRef Name
1c000000 3c007000 exe 1 0 0 /usr/local/sbin/pound
09b5c000 29b60000 rlib 0 1 0
/usr/local/lib/libpcreposix.so.1.5
01018000 21025000 rlib 0 1 0
/usr/local/openssl/lib/libssl.so.1.0.0
0e54b000 2e58a000 rlib 0 2 0
/usr/local/openssl/lib/libcrypto.so.1.0.0
0b978000 2b981000 rlib 0 1 0 /usr/lib/libm.so.7.0
0c5ac000 2c5b1000 rlib 0 2 0 /usr/lib/libpthread.so.16.0
0d70a000 2d738000 rlib 0 1 0 /usr/lib/libc.so.65.0
07de8000 27dfc000 rlib 0 1 0
/usr/local/lib/libpcre.so.3.0
07d20000 27d27000 rlib 0 2 0 /usr/lib/libz.so.4.1
048c7000 048c7000 rtld 0 1 0 /usr/libexec/ld.so
Pound is configured w/ ~80 listen directives, split HTTP and HTTPS.
Other config bits:
User "proxy"
Group "proxy"
RootJail "/var/pound"
Alive 3
TimeOut 120
LogLevel 5
Daemon 1
Control "/tmp/pound.sock"
DynScale 0
ConnTO 5
Threads 128
I ran some load against the machine via ab, hitting the pound on
127.0.0.1. No problems. When I switch some real traffic to the box (by
promoting the carp interface to master), pound seems to get stuck pretty
quickly and requests start to stall. If I look in top, pound is in
state "fdlock", and 'netstat -na' shows a bunch of nonzero entries in
the recv-q fields. CPU load is near-zero. And poundctl shows 0 queued
requests, though this command takes a while to return.
So it looks like something has deadlocked and pound isn't processing
requests. Eventually requests do appear to get processed, but extremely
slowly, and pound returns to the 'wait' state in top. If I then do
another request on localhost, it hangs again.
Using ktrace/kdump, I narrowed things down somewhat. The slowdown
appears to happen on the socket() call in this sequence:
15891/1004731 pound 1363487694.436047 GIO fd 1 read 114 bytes
... HTTP REQUEST SNIPPED HERE ...
15891/1004731 pound 1363487694.436055 RET read 114/0x72
15891/1004731 pound 1363487694.436086 CALL
gettimeofday(0x81a18b98,0x81a18b90)
15891/1004731 pound 1363487694.436096 STRU struct timeval { "Mar
16 22:34:54 2013".436093 }
15891/1004731 pound 1363487694.436103 RET gettimeofday 0
15891/1004731 pound 1363487694.436112 CALL gettimeofday(0x81a18b94,0)
15891/1004731 pound 1363487694.436121 STRU struct timeval { "Mar
16 22:34:54 2013".436119 }
15891/1004731 pound 1363487694.436127 RET gettimeofday 0
15891/1004731 pound 1363487694.436290 CALL
socket(PF_INET,SOCK_STREAM,IPPROTO_IP)
... 80 seconds go by...
15891/1004731 pound 1363487774.413972 RET socket 2
15891/1004731 pound 1363487774.413982 CALL
fcntl(0x2,F_GETFL,0x74be3f00)
15891/1004731 pound 1363487774.413990 RET fcntl 2
15891/1004731 pound 1363487774.413996 CALL
fcntl(0x2,F_SETFL,0x6<O_RDWR|O_NONBLOCK>)
15891/1004731 pound 1363487774.414003 RET fcntl 0
15891/1004731 pound 1363487774.414010 CALL
connect(0x2,0x7d8d35f0,0x10)
Does anybody have pound working on OpenBSD 5.2? Has anyone encountered
this problem or anything similar?
--
Dave Steinberg
http://www.geekisp.com/
http://www.steinbergcomputing.com/
http://www.redterror.net/
--
To unsubscribe send an email with subject unsubscribe to [email protected].
Please contact [email protected] for questions.
