Hi Cyril,
On Sat, Nov 17, 2012 at 11:22:43AM +0100, Cyril Bonté wrote:
> Le 17/11/2012 11:17, Cyril Bonté a écrit :
> >Hi Willy,
> >
> >Since yesterday, the git repository is unavailable, producing "504
> >Gateway Time-out".
>
> I realize I'm not clear. I'm talking about the git web interface
> (http://haproxy.1wt.eu/git/?p=haproxy.git), not the git repository
> itself (which is still available there :
> http://git.1wt.eu/git/haproxy.git/).
>
> > From http://demo.1wt.eu/ I see that there's always 2 active sessions on
> >the git backend (which is also the limit), but no traffic at all. The
> >page also shows haproxy has been updated recently. So, I wonder if
> >there's not a bug somewhere in the sessions counter. Can you have a look
> >at this ? Is there really 2 opened connections since yesterday ?
Thanks for reporting, I don't like this at all :
willy@rpx2:~$ netstat -atn|grep CLO
tcp 1 0 10.9.2.3:60081 88.191.124.161:49785 CLOSE_WAIT
tcp 29756 0 10.9.2.3:39541 10.9.3.3:80 CLOSE_WAIT
> show sess
0x199dd38: proto=tcpv4 src=88.191.124.161:49785 fe=http-in be=git srv=www ts=08
age=2d2h calls=9 rq[f=1808000h,i=0,an=2000h,rx=,wx=,ax=]
rp[f=2000000h,i=7000,an=100000h,rx=,wx=,ax=] s0=[7,8h,fd=1,ex=]
s1=[7,10ch,fd=2,ex=] exp=
0x197c978: proto=tcpv4 src=88.191.124.161:47488 fe=http-in be=git srv=www ts=08
age=2d38m calls=11 rq[f=1808000h,i=0,an=2000h,rx=,wx=,ax=]
rp[f=2000000h,i=7000,an=100000h,rx=,wx=,ax=] s0=[7,8h,fd=24,ex=]
s1=[7,10ch,fd=25,ex=] exp=
> show sess 0x199dd38
0x199dd38: id=5059, proto=tcpv4 source=88.191.124.161:49785
flags=0x4ce, conn_retries=3, srv_conn=0x1894b90, pend_pos=(nil)
frontend=http-in (id=2 mode=http), listener=IPv4-cached (id=2)
addr=10.9.2.3:60081
backend=git (id=4 mode=http) addr=10.9.2.3:44500
server=www (id=1) addr=10.9.3.3:80
task=0x199dc70 (state=0x08 nice=0 calls=9 exp=<NEVER>)
task created [15/Nov/2012:19:06:59.165697] (age=2d2h)
si[0]=0x199dee8 (state=7 flags=0x08 fd=1 exp=<NEVER>, et=0x000)
si[1]=0x199df50 (state=7 flags=0x10c fd=2 exp=<NEVER>, et=0x000)
txn=0x199dd60 (flags=0x18703000 meth=2 status=200 req.st=33 rsp.st=30)
req=0x199dcc8 (f=0x1808000 an=0x2000 i=0 o=0 pipe=0 fwd=0)
an_exp=<NEVER> rex=<NEVER> wex=<NEVER>
data=0x18f4818 p=0 next=0 total=683
res=0x19a0258 (f=0x2000000 an=0x100000 i=7000 o=0 pipe=0 fwd=0)
an_exp=<NEVER> rex=<NEVER> wex=<NEVER>
data=0x18f9538 p=292 next=0 total=7269
> show sess 0x197c978
0x197c978: id=8281, proto=tcpv4 source=88.191.124.161:47488
flags=0x4ce, conn_retries=3, srv_conn=0x1894b90, pend_pos=(nil)
frontend=http-in (id=2 mode=http), listener=IPv4-cached (id=2)
addr=10.9.2.3:60081
backend=git (id=4 mode=http) addr=10.9.2.3:39541
server=www (id=1) addr=10.9.3.3:80
task=0x197c8b0 (state=0x08 nice=0 calls=11 exp=<NEVER>)
task created [15/Nov/2012:20:49:06.160004] (age=2d39m)
si[0]=0x197cb28 (state=7 flags=0x08 fd=24 exp=<NEVER>, et=0x000)
si[1]=0x197cb90 (state=7 flags=0x10c fd=25 exp=<NEVER>, et=0x000)
txn=0x197c9a0 (flags=0x18703000 meth=2 status=200 req.st=33 rsp.st=30)
req=0x197ee98 (f=0x1808000 an=0x2000 i=0 o=0 pipe=0 fwd=0)
an_exp=<NEVER> rex=<NEVER> wex=<NEVER>
data=0x19a4eb0 p=0 next=0 total=363
res=0x197c908 (f=0x2000000 an=0x100000 i=7000 o=0 pipe=0 fwd=0)
an_exp=<NEVER> rex=<NEVER> wex=<NEVER>
data=0x1961e70 p=266 next=0 total=7243
So we have two stuck connections :-(
I'll have to investigate to understand what is happening there.
I have produced a core file to check the sessions state. Since this is
a very new issue, I fear that it might be related to the latest polling
changes to accomodate for the OpenSSL fun...
It's very interesting to note that the sessions got stuck 2 hours after
the process was restarted. Since the git web interface frequently causes
timeouts, I suspect that there is an improperly handled case :-/
I have just shut these two sessions down on the CLI and now it's working
again. I'll have to try to reproduce this and fix it ASAP.
Cheers,
Willy
