Hi there,
This morning I noticed interesting problems regarding haproxy
(1.3.14.2 here, yes I know archeology might be involved, I must
upgrade). I have to say that we had a blackout during a few hours
this night on our site, and servers restarted when power supply came
back. This may be one of the reasons, network problems might also be
involved.
The problem only happened on one of the remote backends (WAN setup).
An haproxy reload fixed the problem. FWIW note that the related URLs
were protected with Basic authentication.
Here are the offending logs (with name of host and frontend changed):
Apr 24 07:38:27 myhost haproxy[4499]: 127.0.0.1:51565
[24/Apr/2009:07:38:27.139] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 206/206/0/0 0/0 {} "<BADREQ>"
Apr 24 07:38:49 myhost haproxy[4499]: 127.0.0.1:44142
[24/Apr/2009:07:38:49.177] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 206/206/0/0 0/0 {} "<BADREQ>"
Apr 24 07:42:07 myhost haproxy[4499]: 127.0.0.1:48588
[24/Apr/2009:07:42:07.351] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 204/204/0/0 0/0 {} "<BADREQ>"
Apr 24 07:42:39 myhost haproxy[4499]: 127.0.0.1:49220
[24/Apr/2009:07:42:39.231] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 205/205/0/0 0/0 {} "<BADREQ>"
Apr 24 08:06:12 myhost haproxy[4499]: 127.0.0.1:55251
[24/Apr/2009:08:06:12.106] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 202/202/0/0 0/0 {} "<BADREQ>"
Apr 24 08:14:48 myhost haproxy[4499]: 127.0.0.1:59733
[24/Apr/2009:08:14:48.979] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 201/201/0/0 0/0 {} "<BADREQ>"
Apr 24 08:30:50 myhost haproxy[4499]: 127.0.0.1:50823
[24/Apr/2009:08:30:50.775] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 205/205/0/0 0/0 {} "<BADREQ>"
Apr 24 08:30:59 myhost haproxy[4499]: 127.0.0.1:51282
[24/Apr/2009:08:30:59.542] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 205/205/0/0 0/0 {} "<BADREQ>"
Apr 24 08:31:29 myhost haproxy[4499]: 127.0.0.1:52191
[24/Apr/2009:08:31:29.621] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 199/199/0/0 0/0 {} "<BADREQ>"
Apr 24 08:31:32 myhost haproxy[4499]: 127.0.0.1:52423
[24/Apr/2009:08:31:32.993] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 200/200/0/0 0/0 {} "<BADREQ>"
Apr 24 08:31:34 myhost haproxy[4499]: 127.0.0.1:52628
[24/Apr/2009:08:31:34.124] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 199/199/0/0 0/0 {} "<BADREQ>"
Apr 24 08:32:58 myhost haproxy[4499]: 127.0.0.1:55046
[24/Apr/2009:08:32:58.916] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 201/201/0/0 0/0 {} "<BADREQ>"
Apr 24 08:33:01 myhost haproxy[4499]: 127.0.0.1:55351
[24/Apr/2009:08:33:01.277] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 203/203/0/0 0/0 {} "<BADREQ>"
Apr 24 08:33:01 myhost haproxy[4499]: 127.0.0.1:55546
[24/Apr/2009:08:33:01.558] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 201/201/0/0 0/0 {} "<BADREQ>"
Apr 24 08:33:01 myhost haproxy[4499]: 127.0.0.1:55740
[24/Apr/2009:08:33:01.641] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 200/200/0/0 0/0 {} "<BADREQ>"
Apr 24 08:33:08 myhost haproxy[4499]: 127.0.0.1:56143
[24/Apr/2009:08:33:08.816] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 202/202/0/0 0/0 {} "<BADREQ>"
Apr 24 08:33:09 myhost haproxy[4499]: 127.0.0.1:56354
[24/Apr/2009:08:33:09.096] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 204/204/0/0 0/0 {} "<BADREQ>"
Apr 24 08:33:12 myhost haproxy[4499]: 127.0.0.1:56607
[24/Apr/2009:08:33:12.120] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 205/205/0/0 0/0 {} "<BADREQ>"
Apr 24 08:33:32 myhost haproxy[4499]: 127.0.0.1:57102
[24/Apr/2009:08:33:32.116] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 205/205/0/0 0/0 {} "<BADREQ>"
Apr 24 08:33:41 myhost haproxy[4499]: 127.0.0.1:47058
[24/Apr/2009:08:33:41.125] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 207/207/0/0 0/0 {} "<BADREQ>"
Apr 24 08:33:55 myhost haproxy[4499]: 127.0.0.1:47752
[24/Apr/2009:08:33:55.918] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 202/202/0/0 0/0 {} "<BADREQ>"
Apr 24 08:34:22 myhost haproxy[4499]: 127.0.0.1:48586
[24/Apr/2009:08:34:22.053] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 203/203/0/0 0/0 {} "<BADREQ>"
Apr 24 08:35:31 myhost haproxy[4499]: 127.0.0.1:50471
[24/Apr/2009:08:35:31.359] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 204/204/0/0 0/0 {} "<BADREQ>"
Apr 24 08:36:47 myhost haproxy[4499]: 127.0.0.1:52694
[24/Apr/2009:08:36:47.769] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 201/201/0/0 0/0 {} "<BADREQ>"
Apr 24 08:36:49 myhost haproxy[4499]: 127.0.0.1:52910
[24/Apr/2009:08:36:49.838] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 198/198/0/0 0/0 {} "<BADREQ>"
Apr 24 08:37:19 myhost haproxy[4499]: 127.0.0.1:53877
[24/Apr/2009:08:37:19.243] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 199/199/0/0 0/0 {} "<BADREQ>"
Apr 24 08:37:19 myhost haproxy[4499]: 127.0.0.1:54083
[24/Apr/2009:08:37:19.318] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 397/397/0/0 0/0 {} "<BADREQ>"
Apr 24 08:42:53 myhost haproxy[4499]: 127.0.0.1:55385
[24/Apr/2009:08:42:53.186] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 205/205/0/0 0/0 {} "<BADREQ>"
Apr 24 08:44:56 myhost haproxy[4499]: 127.0.0.1:45957
[24/Apr/2009:08:44:56.152] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 207/207/0/0 0/0 {} "<BADREQ>"
Apr 24 08:47:41 myhost haproxy[4499]: 127.0.0.1:51606
[24/Apr/2009:08:47:41.643] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 205/205/0/0 0/0 {} "<BADREQ>"
Apr 24 08:49:47 myhost haproxy[4499]: 127.0.0.1:47812
[24/Apr/2009:08:49:47.204] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 204/204/0/0 0/0 {} "<BADREQ>"
Apr 24 08:52:12 myhost haproxy[4499]: 127.0.0.1:52770
[24/Apr/2009:08:52:12.259] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 209/209/0/0 0/0 {} "<BADREQ>"
Apr 24 08:57:17 myhost haproxy[4499]: 127.0.0.1:53900
[24/Apr/2009:08:57:17.844] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 204/204/0/0 0/0 {} "<BADREQ>"
Apr 24 08:57:18 myhost haproxy[4499]: 127.0.0.1:54096
[24/Apr/2009:08:57:18.146] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 206/206/0/0 0/0 {} "<BADREQ>"
Apr 24 08:57:18 myhost haproxy[4499]: 127.0.0.1:54301
[24/Apr/2009:08:57:18.567] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 205/205/0/0 0/0 {} "<BADREQ>"
Apr 24 08:58:39 myhost haproxy[4499]: 127.0.0.1:56581
[24/Apr/2009:08:58:39.152] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 198/198/0/0 0/0 {} "<BADREQ>"
Apr 24 08:58:39 myhost haproxy[4499]: 127.0.0.1:56767
[24/Apr/2009:08:58:39.565] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 197/197/0/0 0/0 {} "<BADREQ>"
Apr 24 08:58:41 myhost haproxy[4499]: 127.0.0.1:41944
[24/Apr/2009:08:58:41.916] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 198/198/0/0 0/0 {} "<BADREQ>"
Apr 24 09:04:25 myhost haproxy[4499]: 127.0.0.1:35048
[24/Apr/2009:09:04:25.179] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 205/205/0/0 0/0 {} "<BADREQ>"
Apr 24 09:04:37 myhost haproxy[4499]: 127.0.0.1:35633
[24/Apr/2009:09:04:37.551] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 197/197/0/0 0/0 {} "<BADREQ>"
Apr 24 09:04:52 myhost haproxy[4499]: 127.0.0.1:36621
[24/Apr/2009:09:04:52.868] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 200/200/0/0 0/0 {} "<BADREQ>"
Apr 24 09:05:00 myhost haproxy[4499]: 127.0.0.1:37203
[24/Apr/2009:09:05:00.227] abc abc/<NOSRV> -1/-1/-1/-1/0 400 187 - -
PR-- 200/200/0/0 0/0 {} "<BADREQ>"
I read in another similar thread that upgrading haproxy might help?
BTW, I don't understand why the IP address is always 127.0.0.1, I'm
sure requests came from a remote IP. Maybe it's a side effect of what
is considered a "bad request"?
TIA!
--
Jean-Baptiste Quenot
http://jbq.caraldi.com/
