David Lang, you also have valid points obviously, but what I wonder is if the server crashes where is the data going to be then. At least one centralized on a NAS/SAN solution a massive hit to the server won't be as destruction to the data on the NAS.
Plus, did he mention if this is a brand new stand up project requiring the ability to acquire a new hardware. I think I missed that part. \\Warron French from mobile On Jun 22, 2016 7:33 PM, "David Lang" <[email protected]> wrote: > On Wed, 22 Jun 2016, Joe Blow wrote: > > Our reqs are a bit beefier. I disagree on a few things. For the SAN/NAS >> stuff, just buy beefy servers (36-72TB systems are dirt cheap now). Iops >> won't be an issue that way either. >> > > I agree, the impact on both the performance and on the netapp for this > much I/O can be significant, and while it can be done on central storage, > it's FAR cheaper to just do it on local storage > > Also, instead of logrotate, I use a cron job which kicks off a threaded >> compression script, using xz -e -z -9 for beast mode compression using >> 16/40 cores max. 1.2B logs is like 20GB / day. Do the math on a 20-30TB >> mount and you're laughing. >> > > A cron job can also prune logs beyond a given date, or rotate them off to > long-term storage (hello AWS Glacier, $0.01/GB/month with something like 14 > 9's of reliability) > > check your logs. I did a bunch of tests of xz from -1 to -9e and found the > difference was far less than I expected: > > 10 min worth of logs, ~9.8GB > > netapp compression ~2:1 or ~5G of storage > > xz compression size and cores needed to compress 10 min of logs in 10 min > > note that these numbers are only approximate as they will vary as the log > contents change, but this is probably within 20% or so of the cpu cost > > level size cores > 1 242MB 1 > 2 209MB 1 > 3 197MB 1 > 4 257MB 2 > 5 227MB 3 > 6 200MB 4 > 7 198MB 5 > 8 197MB 5 > 9 196MB 5 > 1e 184MB 14 > 2e 171MB 16 > 3e 177MB 12 > 4e 163MB 18 > 5e 171MB 12 > 6e 158MB 21 > 7e 154MB 25 > 8e 152MB 35 > 9e 150MB 31 > > David Lang > > > I'd love to see the audispatch work you've done. >> >> Also, in case you were wondering, I have a patent on my log-to-jpg-to-sql >> module, and the corresponding OCR extraction tool. >> >> Cheers, >> >> JB >> >> Original Message >> From:[email protected] >> Sent:June 22, 2016 3:26 PM >> To:[email protected] >> Reply-to:[email protected] >> Subject:Re: [rsyslog] Central logging solution >> >> I worked on something similar. >> >> Are your requirements as basic as what you have presented Abhilash? >> >> Do your requirements require centralizing audit log data? That should be >> handled through audispatch and I can advise you on how to do that. In >> fact >> I might even have a fully complete set of instructions for the audispatch >> and another for rsyslog. >> >> As for managing the volume of data (from 2000+ hosts) and data store, that >> would require some reaching out to NAS/SAN consultants for optimizing the >> storage configuration; and then if you set up the logrotation to be daily >> you can then write a script and put it into crontab to maintain the >> 18month >> auto-cropping of data from the volume based on a *find *command. >> >> That's my input, since I have had to do something similar but not at the >> same volume, or with the same storage requirements exactly. >> >> Let me know if you need more input. >> >> -------------------------- >> Warron French >> >> >> On Mon, Jun 20, 2016 at 2:04 PM, Joe Blow <[email protected]> wrote: >> >> I feel like this is a troll....... >>> >>> 1. Turn on imptcp >>> 2. Turn on imudp >>> 3. Point your systems to rsyslog >>> 4. Profit >>> >>> Some other pieces of glue you'll need: >>> >>> Keepalived >>> Liblognormalize >>> Xz (for beast mode compression) >>> Drbd >>> Imagination >>> Iptables >>> >>> I would suggest you convert the log lines to jpgs, so that you can store >>> the base64 encoded jpg inside a non-searchable relational database of >>> your >>> choice. >>> >>> Then you can sell your new "customer" an OCR appliance to search your >>> jpg-log-storage-appliance. >>> >>> Hope this helps with your homework. :) >>> >>> /troll >>> >>> Cheers, >>> >>> JB >>> >>> >>> Original Message >>> From:[email protected] >>> Sent:June 20, 2016 11:55 AM >>> To:[email protected] >>> Reply-to:[email protected] >>> Subject:[rsyslog] Central logging solution >>> >>> Hi Team, >>> >>> I am working on a Central Logging solution for our environment using >>> Linux >>> native logging feature(rsyslog). The requirement is to store the logs >>> from >>> all servers(Windows +all UNIX) and retain for 18 months.Our account got >>> around 2000 server(1200 Windows+ 800 UNIX(HPUX/AIX/Linux/Soalris) and >>> appliances. The customer wants a solution based on rsyslog.We also need >>> to >>> retain the logs for 18 months on tamper proof storage. >>> >>> Please could anyone guide me ? Also pls share some good documentation >>> >>> -- >>> Abhilash.P.A >>> voice :+91 9663375151 >>> >>> " Known is a drop and unknown is an ocean" >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
