On Wed, 22 Jun 2016, Joe Blow wrote:
What were the logs you've compressed in that test?
windows/linux/cisco/paloalto/application
this was a 10 min slice of my logs. It includes both the raw logs in the order
they arrived all mixed together, but also most of the logs broken out into
different categories.
So there could be several copies of any one log message, and since the logs were
rotated once/min, some of the files are very small.
some of the logs eaily compress at 100:1 others less so.
run a test on a sample of your logs (and do it again in a year to make sure
things haven't changed)
At this point I've basically abandoned bzip2, I don't fine any cases where it's
less cpu or more compression than is available via xz.
About the only time I still use gzip is for the logs compressed by rsyslog.
David Lang
When testing with firewall
logs (and logs which repeat a ton of data) it seemed the -9 compression was
worth it, but I was testing xz -9, gzip -9, and bzip2 -9, not the 1-9 tests.
Adding the -e did add a significant amount to compressing, but since xz was
better than gzip and bzip2, it seemed like the better idea.
I love me some scientific method. Great work David.
Cheers,
JB
Original Message
From:[email protected]
Sent:June 22, 2016 7:33 PM
To:[email protected]
Reply-to:[email protected]
Subject:Re: [rsyslog] Central logging solution
On Wed, 22 Jun 2016, Joe Blow wrote:
Our reqs are a bit beefier. I disagree on a few things. For the SAN/NAS stuff,
just buy beefy servers (36-72TB systems are dirt cheap now). Iops won't be an
issue that way either.
I agree, the impact on both the performance and on the netapp for this much I/O
can be significant, and while it can be done on central storage, it's FAR
cheaper to just do it on local storage
Also, instead of logrotate, I use a cron job which kicks off a threaded
compression script, using xz -e -z -9 for beast mode compression using 16/40
cores max. 1.2B logs is like 20GB / day. Do the math on a 20-30TB mount and
you're laughing.
A cron job can also prune logs beyond a given date, or rotate them off to
long-term storage (hello AWS Glacier, $0.01/GB/month with something like 14 9's
of reliability)
check your logs. I did a bunch of tests of xz from -1 to -9e and found the
difference was far less than I expected:
10 min worth of logs, ~9.8GB
netapp compression ~2:1 or ~5G of storage
xz compression size and cores needed to compress 10 min of logs in 10 min
note that these numbers are only approximate as they will vary as the log
contents change, but this is probably within 20% or so of the cpu cost
level size cores
1 242MB 1
2 209MB 1
3 197MB 1
4 257MB 2
5 227MB 3
6 200MB 4
7 198MB 5
8 197MB 5
9 196MB 5
1e 184MB 14
2e 171MB 16
3e 177MB 12
4e 163MB 18
5e 171MB 12
6e 158MB 21
7e 154MB 25
8e 152MB 35
9e 150MB 31
David Lang
I'd love to see the audispatch work you've done.
Also, in case you were wondering, I have a patent on my log-to-jpg-to-sql
module, and the corresponding OCR extraction tool.
Cheers,
JB
Original Message
From:[email protected]
Sent:June 22, 2016 3:26 PM
To:[email protected]
Reply-to:[email protected]
Subject:Re: [rsyslog] Central logging solution
I worked on something similar.
Are your requirements as basic as what you have presented Abhilash?
Do your requirements require centralizing audit log data? That should be
handled through audispatch and I can advise you on how to do that. In fact
I might even have a fully complete set of instructions for the audispatch
and another for rsyslog.
As for managing the volume of data (from 2000+ hosts) and data store, that
would require some reaching out to NAS/SAN consultants for optimizing the
storage configuration; and then if you set up the logrotation to be daily
you can then write a script and put it into crontab to maintain the 18month
auto-cropping of data from the volume based on a *find *command.
That's my input, since I have had to do something similar but not at the
same volume, or with the same storage requirements exactly.
Let me know if you need more input.
--------------------------
Warron French
On Mon, Jun 20, 2016 at 2:04 PM, Joe Blow <[email protected]> wrote:
I feel like this is a troll.......
1. Turn on imptcp
2. Turn on imudp
3. Point your systems to rsyslog
4. Profit
Some other pieces of glue you'll need:
Keepalived
Liblognormalize
Xz (for beast mode compression)
Drbd
Imagination
Iptables
I would suggest you convert the log lines to jpgs, so that you can store
the base64 encoded jpg inside a non-searchable relational database of your
choice.
Then you can sell your new "customer" an OCR appliance to search your
jpg-log-storage-appliance.
Hope this helps with your homework. :)
/troll
Cheers,
JB
Original Message
From:[email protected]
Sent:June 20, 2016 11:55 AM
To:[email protected]
Reply-to:[email protected]
Subject:[rsyslog] Central logging solution
Hi Team,
I am working on a Central Logging solution for our environment using Linux
native logging feature(rsyslog). The requirement is to store the logs from
all servers(Windows +all UNIX) and retain for 18 months.Our account got
around 2000 server(1200 Windows+ 800 UNIX(HPUX/AIX/Linux/Soalris) and
appliances. The customer wants a solution based on rsyslog.We also need to
retain the logs for 18 months on tamper proof storage.
Please could anyone guide me ? Also pls share some good documentation
--
Abhilash.P.A
voice :+91 9663375151
" Known is a drop and unknown is an ocean"
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
