Hi Holger, On Sat, Mar 19, 2011 at 05:32:08PM +0100, Holger Just wrote: > Hi Roy, > > On 2011-03-18 22:21, Roy Smith wrote: > > Before I reinvent the wheel, has anybody already written code to parse > > haproxy log messages with Python? > > I have, although it's not _that_ fast. My approach requires about 1 > minutes per 100 MB gziped logs (with a roughly 10:1 compression). > > If your usecase matches on the features of halog, you should definitly > try that instead. It's written by Willy himself and is able to easily > maxout your streaming file I/O (meaning it is magnitudes faster than you > could ever do it in python itself)
in fact I'd like halog to be more commonly usable as a low-level "pre-parser", which means it would take care of extracting the useful information from the logs so that higher level scripts can process pre-digested information. Of course it will never be able to do everything, but if some scripts don't need all the lines of a log file, we should ensure that halog provides enough means to filter those lines out. For instance, right now you can already use halog to ensure that only valid parsable lines are returned. Most likely a number of other filtering options need to be added, we need to figure out which ones. Regards, Willy
