On Wed, 22 Jun 2016, warron.french wrote:
David Lang, you also have valid points obviously, but what I wonder is if
the server crashes where is the data going to be then. At least one
centralized on a NAS/SAN solution a massive hit to the server won't be as
destruction to the data on the NAS.
I mirror the logs to two datacenters
edge relays pick up the logs, do cleanup that sends them to the core relays.
core relays look to see if they got the log from the other datacenter core
relay, if not, it sends them there. It also sends them to the reporting
server(s) in the local datacenter
the relays all can spool to disk if a destination is down.
so logs are not lost, but may arrive out of order if some parts are down. I
build the relays as HA failover pairs, so they are unlikely to be the problem.
WAN links are the most common problem
Plus, did he mention if this is a brand new stand up project requiring the
ability to acquire a new hardware. I think I missed that part.
well, any logging project for a couple thousand systems is going to require
hardware and storage purchases (or at least a significant allocation out of the
reserve systems)
And even if you are a all vm/local cloud shop, you really should look at the
cost of local storage for Big Data systems compared to NetApp and similar. While
I was at Intuit they started with the "This is our standard", but after a few
projects showed 5x to 20x price differences (in the hundreds of thousands of
dollars per project) between following the "standard" and better performing
local storage, they added a new tier to the standard to accomodate Big Data
projects.
And logging qualifies as Big Data if you do much with it :-)
David Lang
\\Warron French from mobile
On Jun 22, 2016 7:33 PM, "David Lang" <[email protected]> wrote:
On Wed, 22 Jun 2016, Joe Blow wrote:
Our reqs are a bit beefier. I disagree on a few things. For the SAN/NAS
stuff, just buy beefy servers (36-72TB systems are dirt cheap now). Iops
won't be an issue that way either.
I agree, the impact on both the performance and on the netapp for this
much I/O can be significant, and while it can be done on central storage,
it's FAR cheaper to just do it on local storage
Also, instead of logrotate, I use a cron job which kicks off a threaded
compression script, using xz -e -z -9 for beast mode compression using
16/40 cores max. 1.2B logs is like 20GB / day. Do the math on a 20-30TB
mount and you're laughing.
A cron job can also prune logs beyond a given date, or rotate them off to
long-term storage (hello AWS Glacier, $0.01/GB/month with something like 14
9's of reliability)
check your logs. I did a bunch of tests of xz from -1 to -9e and found the
difference was far less than I expected:
10 min worth of logs, ~9.8GB
netapp compression ~2:1 or ~5G of storage
xz compression size and cores needed to compress 10 min of logs in 10 min
note that these numbers are only approximate as they will vary as the log
contents change, but this is probably within 20% or so of the cpu cost
level size cores
1 242MB 1
2 209MB 1
3 197MB 1
4 257MB 2
5 227MB 3
6 200MB 4
7 198MB 5
8 197MB 5
9 196MB 5
1e 184MB 14
2e 171MB 16
3e 177MB 12
4e 163MB 18
5e 171MB 12
6e 158MB 21
7e 154MB 25
8e 152MB 35
9e 150MB 31
David Lang
I'd love to see the audispatch work you've done.
Also, in case you were wondering, I have a patent on my log-to-jpg-to-sql
module, and the corresponding OCR extraction tool.
Cheers,
JB
Original Message
From:[email protected]
Sent:June 22, 2016 3:26 PM
To:[email protected]
Reply-to:[email protected]
Subject:Re: [rsyslog] Central logging solution
I worked on something similar.
Are your requirements as basic as what you have presented Abhilash?
Do your requirements require centralizing audit log data? That should be
handled through audispatch and I can advise you on how to do that. In
fact
I might even have a fully complete set of instructions for the audispatch
and another for rsyslog.
As for managing the volume of data (from 2000+ hosts) and data store, that
would require some reaching out to NAS/SAN consultants for optimizing the
storage configuration; and then if you set up the logrotation to be daily
you can then write a script and put it into crontab to maintain the
18month
auto-cropping of data from the volume based on a *find *command.
That's my input, since I have had to do something similar but not at the
same volume, or with the same storage requirements exactly.
Let me know if you need more input.
--------------------------
Warron French
On Mon, Jun 20, 2016 at 2:04 PM, Joe Blow <[email protected]> wrote:
I feel like this is a troll.......
1. Turn on imptcp
2. Turn on imudp
3. Point your systems to rsyslog
4. Profit
Some other pieces of glue you'll need:
Keepalived
Liblognormalize
Xz (for beast mode compression)
Drbd
Imagination
Iptables
I would suggest you convert the log lines to jpgs, so that you can store
the base64 encoded jpg inside a non-searchable relational database of
your
choice.
Then you can sell your new "customer" an OCR appliance to search your
jpg-log-storage-appliance.
Hope this helps with your homework. :)
/troll
Cheers,
JB
Original Message
From:[email protected]
Sent:June 20, 2016 11:55 AM
To:[email protected]
Reply-to:[email protected]
Subject:[rsyslog] Central logging solution
Hi Team,
I am working on a Central Logging solution for our environment using
Linux
native logging feature(rsyslog). The requirement is to store the logs
from
all servers(Windows +all UNIX) and retain for 18 months.Our account got
around 2000 server(1200 Windows+ 800 UNIX(HPUX/AIX/Linux/Soalris) and
appliances. The customer wants a solution based on rsyslog.We also need
to
retain the logs for 18 months on tamper proof storage.
Please could anyone guide me ? Also pls share some good documentation
--
Abhilash.P.A
voice :+91 9663375151
" Known is a drop and unknown is an ocean"
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
