Our reqs are a bit beefier. I disagree on a few things. For the SAN/NAS stuff, just buy beefy servers (36-72TB systems are dirt cheap now). Iops won't be an issue that way either.
Also, instead of logrotate, I use a cron job which kicks off a threaded compression script, using xz -e -z -9 for beast mode compression using 16/40 cores max. 1.2B logs is like 20GB / day. Do the math on a 20-30TB mount and you're laughing. I'd love to see the audispatch work you've done. Also, in case you were wondering, I have a patent on my log-to-jpg-to-sql module, and the corresponding OCR extraction tool. Cheers, JB Original Message From:[email protected] Sent:June 22, 2016 3:26 PM To:[email protected] Reply-to:[email protected] Subject:Re: [rsyslog] Central logging solution I worked on something similar. Are your requirements as basic as what you have presented Abhilash? Do your requirements require centralizing audit log data? That should be handled through audispatch and I can advise you on how to do that. In fact I might even have a fully complete set of instructions for the audispatch and another for rsyslog. As for managing the volume of data (from 2000+ hosts) and data store, that would require some reaching out to NAS/SAN consultants for optimizing the storage configuration; and then if you set up the logrotation to be daily you can then write a script and put it into crontab to maintain the 18month auto-cropping of data from the volume based on a *find *command. That's my input, since I have had to do something similar but not at the same volume, or with the same storage requirements exactly. Let me know if you need more input. -------------------------- Warron French On Mon, Jun 20, 2016 at 2:04 PM, Joe Blow <[email protected]> wrote: > I feel like this is a troll....... > > 1. Turn on imptcp > 2. Turn on imudp > 3. Point your systems to rsyslog > 4. Profit > > Some other pieces of glue you'll need: > > Keepalived > Liblognormalize > Xz (for beast mode compression) > Drbd > Imagination > Iptables > > I would suggest you convert the log lines to jpgs, so that you can store > the base64 encoded jpg inside a non-searchable relational database of your > choice. > > Then you can sell your new "customer" an OCR appliance to search your > jpg-log-storage-appliance. > > Hope this helps with your homework. :) > > /troll > > Cheers, > > JB > > > Original Message > From:[email protected] > Sent:June 20, 2016 11:55 AM > To:[email protected] > Reply-to:[email protected] > Subject:[rsyslog] Central logging solution > > Hi Team, > > I am working on a Central Logging solution for our environment using Linux > native logging feature(rsyslog). The requirement is to store the logs from > all servers(Windows +all UNIX) and retain for 18 months.Our account got > around 2000 server(1200 Windows+ 800 UNIX(HPUX/AIX/Linux/Soalris) and > appliances. The customer wants a solution based on rsyslog.We also need to > retain the logs for 18 months on tamper proof storage. > > Please could anyone guide me ? Also pls share some good documentation > > -- > Abhilash.P.A > voice :+91 9663375151 > > " Known is a drop and unknown is an ocean" > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
