What were the logs you've compressed in that test? When testing with firewall logs (and logs which repeat a ton of data) it seemed the -9 compression was worth it, but I was testing xz -9, gzip -9, and bzip2 -9, not the 1-9 tests. Adding the -e did add a significant amount to compressing, but since xz was better than gzip and bzip2, it seemed like the better idea.
I love me some scientific method. Great work David. Cheers, JB Original Message From:[email protected] Sent:June 22, 2016 7:33 PM To:[email protected] Reply-to:[email protected] Subject:Re: [rsyslog] Central logging solution On Wed, 22 Jun 2016, Joe Blow wrote: > Our reqs are a bit beefier. I disagree on a few things. For the SAN/NAS > stuff, > just buy beefy servers (36-72TB systems are dirt cheap now). Iops won't be > an > issue that way either. I agree, the impact on both the performance and on the netapp for this much I/O can be significant, and while it can be done on central storage, it's FAR cheaper to just do it on local storage > Also, instead of logrotate, I use a cron job which kicks off a threaded > compression script, using xz -e -z -9 for beast mode compression using 16/40 > cores max. 1.2B logs is like 20GB / day. Do the math on a 20-30TB mount and > you're laughing. A cron job can also prune logs beyond a given date, or rotate them off to long-term storage (hello AWS Glacier, $0.01/GB/month with something like 14 9's of reliability) check your logs. I did a bunch of tests of xz from -1 to -9e and found the difference was far less than I expected: 10 min worth of logs, ~9.8GB netapp compression ~2:1 or ~5G of storage xz compression size and cores needed to compress 10 min of logs in 10 min note that these numbers are only approximate as they will vary as the log contents change, but this is probably within 20% or so of the cpu cost level size cores 1 242MB 1 2 209MB 1 3 197MB 1 4 257MB 2 5 227MB 3 6 200MB 4 7 198MB 5 8 197MB 5 9 196MB 5 1e 184MB 14 2e 171MB 16 3e 177MB 12 4e 163MB 18 5e 171MB 12 6e 158MB 21 7e 154MB 25 8e 152MB 35 9e 150MB 31 David Lang > I'd love to see the audispatch work you've done. > > Also, in case you were wondering, I have a patent on my log-to-jpg-to-sql > module, and the corresponding OCR extraction tool. > > Cheers, > > JB > > Original Message > From:[email protected] > Sent:June 22, 2016 3:26 PM > To:[email protected] > Reply-to:[email protected] > Subject:Re: [rsyslog] Central logging solution > > I worked on something similar. > > Are your requirements as basic as what you have presented Abhilash? > > Do your requirements require centralizing audit log data? That should be > handled through audispatch and I can advise you on how to do that. In fact > I might even have a fully complete set of instructions for the audispatch > and another for rsyslog. > > As for managing the volume of data (from 2000+ hosts) and data store, that > would require some reaching out to NAS/SAN consultants for optimizing the > storage configuration; and then if you set up the logrotation to be daily > you can then write a script and put it into crontab to maintain the 18month > auto-cropping of data from the volume based on a *find *command. > > That's my input, since I have had to do something similar but not at the > same volume, or with the same storage requirements exactly. > > Let me know if you need more input. > > -------------------------- > Warron French > > > On Mon, Jun 20, 2016 at 2:04 PM, Joe Blow <[email protected]> wrote: > >> I feel like this is a troll....... >> >> 1. Turn on imptcp >> 2. Turn on imudp >> 3. Point your systems to rsyslog >> 4. Profit >> >> Some other pieces of glue you'll need: >> >> Keepalived >> Liblognormalize >> Xz (for beast mode compression) >> Drbd >> Imagination >> Iptables >> >> I would suggest you convert the log lines to jpgs, so that you can store >> the base64 encoded jpg inside a non-searchable relational database of your >> choice. >> >> Then you can sell your new "customer" an OCR appliance to search your >> jpg-log-storage-appliance. >> >> Hope this helps with your homework. :) >> >> /troll >> >> Cheers, >> >> JB >> >> >> Original Message >> From:[email protected] >> Sent:June 20, 2016 11:55 AM >> To:[email protected] >> Reply-to:[email protected] >> Subject:[rsyslog] Central logging solution >> >> Hi Team, >> >> I am working on a Central Logging solution for our environment using Linux >> native logging feature(rsyslog). The requirement is to store the logs from >> all servers(Windows +all UNIX) and retain for 18 months.Our account got >> around 2000 server(1200 Windows+ 800 UNIX(HPUX/AIX/Linux/Soalris) and >> appliances. The customer wants a solution based on rsyslog.We also need to >> retain the logs for 18 months on tamper proof storage. >> >> Please could anyone guide me ? Also pls share some good documentation >> >> -- >> Abhilash.P.A >> voice :+91 9663375151 >> >> " Known is a drop and unknown is an ocean" >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
