I was being a bit tongue in cheek regarding a patent which converts log lines to jpg, and then stores them in sql. That obviously wouldn't be the most performant or usable system, but I bet you could patent it. With omprog anything is possible :). Plus, I'm more of an ES guy myself.
That said, I pushed a 7 year old scavenged 8 core HP DL 385 with 4 disks in a raid 10 to around 1B / day. Once compressed we shipped them off to cheaper storage, not unlike what David suggested (we bought an $800 NAS at a store and hooked it via crossover). desperate times... well you know. My point is that for the writing/compressing, as long as you've got some decent local storage iops, you shouldn't run into too many problems. I've seen repurposed boxes go a long way, especially with logging. If you've got a higher iop "staging area" which could hold maybe 2 days of logs, you might be able to collect, process and ship off to long term storage (SAN/NAS/NFS/AWS Glacier) but I'd make sure you DR test like crazy, as compounding failures might leave you in a sticky situation. If you're trying for true HA/prod setup, I'd suggest looking at DRBD+pacemaker+corosync+crmsh+keepalived+rsyslog with local storage. Cheers, JB Original Message From:[email protected] Sent:June 22, 2016 8:01 PM To:[email protected] Reply-to:[email protected] Subject:Re: [rsyslog] Central logging solution Joe Blow having your own patent is very respectable. And your approach actually may be much better does it require excessive cost. What is the total LOE and TCO for your approach, verses using existing in-house equipment already? I will dig up my documentation on implementing audispatch and send it to you. I know the instructions work, and I collaborated with Steve on the implementation instructions. Plus a new dispatch is a tool natively available to Centos and redhat derivatives. \\Warron French from mobile On Jun 22, 2016 3:41 PM, "Joe Blow" <[email protected]> wrote: > Our reqs are a bit beefier. I disagree on a few things. For the SAN/NAS > stuff, just buy beefy servers (36-72TB systems are dirt cheap now). Iops > won't be an issue that way either. > > Also, instead of logrotate, I use a cron job which kicks off a threaded > compression script, using xz -e -z -9 for beast mode compression using > 16/40 cores max. 1.2B logs is like 20GB / day. Do the math on a 20-30TB > mount and you're laughing. > > I'd love to see the audispatch work you've done. > > Also, in case you were wondering, I have a patent on my log-to-jpg-to-sql > module, and the corresponding OCR extraction tool. > > Cheers, > > JB > > Original Message > From:[email protected] > Sent:June 22, 2016 3:26 PM > To:[email protected] > Reply-to:[email protected] > Subject:Re: [rsyslog] Central logging solution > > I worked on something similar. > > Are your requirements as basic as what you have presented Abhilash? > > Do your requirements require centralizing audit log data? That should be > handled through audispatch and I can advise you on how to do that. In fact > I might even have a fully complete set of instructions for the audispatch > and another for rsyslog. > > As for managing the volume of data (from 2000+ hosts) and data store, that > would require some reaching out to NAS/SAN consultants for optimizing the > storage configuration; and then if you set up the logrotation to be daily > you can then write a script and put it into crontab to maintain the 18month > auto-cropping of data from the volume based on a *find *command. > > That's my input, since I have had to do something similar but not at the > same volume, or with the same storage requirements exactly. > > Let me know if you need more input. > > -------------------------- > Warron French > > > On Mon, Jun 20, 2016 at 2:04 PM, Joe Blow <[email protected]> wrote: > > > I feel like this is a troll....... > > > > 1. Turn on imptcp > > 2. Turn on imudp > > 3. Point your systems to rsyslog > > 4. Profit > > > > Some other pieces of glue you'll need: > > > > Keepalived > > Liblognormalize > > Xz (for beast mode compression) > > Drbd > > Imagination > > Iptables > > > > I would suggest you convert the log lines to jpgs, so that you can store > > the base64 encoded jpg inside a non-searchable relational database of > your > > choice. > > > > Then you can sell your new "customer" an OCR appliance to search your > > jpg-log-storage-appliance. > > > > Hope this helps with your homework. :) > > > > /troll > > > > Cheers, > > > > JB > > > > > > Original Message > > From:[email protected] > > Sent:June 20, 2016 11:55 AM > > To:[email protected] > > Reply-to:[email protected] > > Subject:[rsyslog] Central logging solution > > > > Hi Team, > > > > I am working on a Central Logging solution for our environment using > Linux > > native logging feature(rsyslog). The requirement is to store the logs > from > > all servers(Windows +all UNIX) and retain for 18 months.Our account got > > around 2000 server(1200 Windows+ 800 UNIX(HPUX/AIX/Linux/Soalris) and > > appliances. The customer wants a solution based on rsyslog.We also need > to > > retain the logs for 18 months on tamper proof storage. > > > > Please could anyone guide me ? Also pls share some good documentation > > > > -- > > Abhilash.P.A > > voice :+91 9663375151 > > > > " Known is a drop and unknown is an ocean" > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
