Hey Willy, LOL Then I was confused by other comments I got back when I posted about analyzing the logs the other day. :-)
Your right about syslog-ng, I would definitely recommend it to anyone also. Joe On Fri, Jan 13, 2012 at 1:54 AM, Willy Tarreau <w...@1wt.eu> wrote: > Hi Joe, > > On Thu, Jan 12, 2012 at 08:40:01PM -0500, Joseph Hardeman wrote: > > Hey Chris, > > > > What flavor of linux will you be putting syslog-ng on? Be sure the > > syslog-ng you install can handle multi-threading of its processes, so > > version 3.0 or newer I believe, otherwise it will eat up all of 1 CPU and > > could most certainly lose logs then if you have a lot of traffic going > > through haproxy. > > My experience with syslog-ng has already been extremely good since version > 1.4 around 10 years ago. I remember reaching 20000 logs per second with > zero losses on a pentium-3 933 MHz. You need to tune it to use large > buffers to cover disk latency, and that's all. Syslog-ng is an excellent > piece of software, which is why I always recommend it to everyone who needs > high logging rates. > > > We have it setup for one of our customers now, actually I just finished > > setting it up four days ago and I have syslog-ng splitting out logs per > > hour. I don't really see much in the way of missing logs, if anything > they > > now have more information than they were getting for the visits to their > > site from Google Analytics. > > > > But just as an idea, using "option httplog clf" in the listen section for > > mode http, yesterday I receiving around 12G of logs from a single haproxy > > box while today they are at 4.9G and the day isn't over yet. So today > may > > end up around 10G as the west coast is now getting off of work. And the > > clf option sends through less data than the normal option httplog so the > > amount of data is a bit lower than if you log normal logs from haproxy. > > This point surprizes me a little bit because CLF logs contain the same info > with more delimiters. Maybe they compress better but I'm surprized you find > them smaller. For instance : > > normal: > Jan 13 07:52:50 pcw haproxy[839]: 127.0.0.1:56837[13/Jan/2012:07:52:46.258] > echo echo/<NOSRV> 0/0/0/3325/3789 200 14 - - > ---- 0/0/0/0/0 0/0 "GET / HTTP/1.1" > clf: > Jan 13 07:52:34 pcw haproxy[834]: 127.0.0.1 - - [13/Jan/2012:06:52:31 > +0000] "GET / HTTP/1.1" 200 14 "-" "-" 56835 759 "echo" "echo" "<NOSRV>" 0 > 0 0 2285 2845 "----" 0 0 0 0 0 0 0 "-" "-" > > Regards, > Willy > >
