On Wed, 22 Jun 2016, Joe Blow wrote:
Our reqs are a bit beefier. I disagree on a few things. For the SAN/NAS
stuff, just buy beefy servers (36-72TB systems are dirt cheap
now). Iops
won't be an issue that way either.
I agree, the impact on both the performance and on the netapp for this
much I/O can be significant, and while it can be done on central
storage,
it's FAR cheaper to just do it on local storage
Also, instead of logrotate, I use a cron job which kicks off a threaded
compression script, using xz -e -z -9 for beast mode compression using
16/40 cores max. 1.2B logs is like 20GB / day. Do the math on a
20-30TB
mount and you're laughing.
A cron job can also prune logs beyond a given date, or rotate them
off to
long-term storage (hello AWS Glacier, $0.01/GB/month with something
like 14
9's of reliability)
check your logs. I did a bunch of tests of xz from -1 to -9e and
found the
difference was far less than I expected:
10 min worth of logs, ~9.8GB
netapp compression ~2:1 or ~5G of storage
xz compression size and cores needed to compress 10 min of logs in
10 min
note that these numbers are only approximate as they will vary as
the log
contents change, but this is probably within 20% or so of the cpu cost
level size cores
1 242MB 1
2 209MB 1
3 197MB 1
4 257MB 2
5 227MB 3
6 200MB 4
7 198MB 5
8 197MB 5
9 196MB 5
1e 184MB 14
2e 171MB 16
3e 177MB 12
4e 163MB 18
5e 171MB 12
6e 158MB 21
7e 154MB 25
8e 152MB 35
9e 150MB 31
David Lang
I'd love to see the audispatch work you've done.
Also, in case you were wondering, I have a patent on my
log-to-jpg-to-sql
module, and the corresponding OCR extraction tool.
Cheers,
JB
Original Message
From:[email protected]
Sent:June 22, 2016 3:26 PM
To:[email protected]
Reply-to:[email protected]
Subject:Re: [rsyslog] Central logging solution
I worked on something similar.
Are your requirements as basic as what you have presented Abhilash?
Do your requirements require centralizing audit log data? That
should be
handled through audispatch and I can advise you on how to do that. In
fact
I might even have a fully complete set of instructions for the
audispatch
and another for rsyslog.
As for managing the volume of data (from 2000+ hosts) and data
store, that
would require some reaching out to NAS/SAN consultants for
optimizing the
storage configuration; and then if you set up the logrotation to be
daily
you can then write a script and put it into crontab to maintain the
18month
auto-cropping of data from the volume based on a *find *command.
That's my input, since I have had to do something similar but not
at the
same volume, or with the same storage requirements exactly.
Let me know if you need more input.
--------------------------
Warron French
On Mon, Jun 20, 2016 at 2:04 PM, Joe Blow <[email protected]>
wrote:
I feel like this is a troll.......
1. Turn on imptcp
2. Turn on imudp
3. Point your systems to rsyslog
4. Profit
Some other pieces of glue you'll need:
Keepalived
Liblognormalize
Xz (for beast mode compression)
Drbd
Imagination
Iptables
I would suggest you convert the log lines to jpgs, so that you can
store
the base64 encoded jpg inside a non-searchable relational database of
your
choice.
Then you can sell your new "customer" an OCR appliance to search your
jpg-log-storage-appliance.
Hope this helps with your homework. :)
/troll
Cheers,
JB
Original Message
From:[email protected]
Sent:June 20, 2016 11:55 AM
To:[email protected]
Reply-to:[email protected]
Subject:[rsyslog] Central logging solution
Hi Team,
I am working on a Central Logging solution for our environment using
Linux
native logging feature(rsyslog). The requirement is to store the
logs
from
all servers(Windows +all UNIX) and retain for 18 months.Our
account got
around 2000 server(1200 Windows+ 800 UNIX(HPUX/AIX/Linux/Soalris) and
appliances. The customer wants a solution based on rsyslog.We also
need
to
retain the logs for 18 months on tamper proof storage.
Please could anyone guide me ? Also pls share some good documentation
--
Abhilash.P.A
voice :+91 9663375151
" Known is a drop and unknown is an ocean"
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
