It's not that one 'takes over' from another, it's that (unless you say stop),
every filter applies to every log entry, so if you have multiple filters that
match a given entry, the log will be written multiple times
This is a feature, not a bug :-)
David Lang
On Thu, 15 May 2014,
Josh Bitto wrote:
I may be premature in my conclusion here, but I think I might have solved my
issue. I just commented out the following from the original config.
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none /var/log/messages
Then the other config should take over and log to that host directory according
to severity.
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of David Lang
Sent: Wednesday, May 14, 2014 10:51 PM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server
I actually like having rsyslog send the logs directly to ES, but I agree that
ES should not be the only place the logs go to.
As Orangepeel Beef notes, there are many other destinations that you want your
logs to go to.
I like to keep a feed of the logs as a plain, combined feed (so I can easily
see what's going on across multiple machines without having to go into all the
different directories)
Sending a copy of the logs to Simple Event correlator is extremely handy
and then you really want the logs from your core server to exist elsewhere so
that when you have problems with it, you can figure out what's going wrong
so I like to have a rule
if $fromhost-ip == "127.0.0.1" then {
<set some vars>
@log-relay;JSON_format
} else {
send to local combined file
send to file split by server
send to file split by programname
send to SEC (split by programname)
send to ES (or other search tool)
etc
}
When you get to the point of making everything redundant, and/or splitting the workload
across servers for different functions, having all your logs, including from your log
servers, relayed through your relay boxes makes things "just work"
David Lang
On Wed, 14 May 2014, Orangepeel Beef wrote:
There are a ton of headaches associated with directly logging to
elasticsearch as well.
How do you reindex if an index crashes if you are not storing your
logs somewhere else as an intermediary? ES crashes indexes if it runs
out of memory, or disk space, and they crash hard. I've rebuilt
indexes many many times already.
What happens when you have a large burst of traffic and elasticsearch
can't handle it? rsyslog can handle a very large amount of
throughput, and writing to files it won't lose anything, but writing to es, it
can.
How do you pass data to Simple event correlator and then into
elasticsearch? pipe it out, and then back into rsyslog? no thanks.
How do you tag different file types if you are sending direct to ES?
each one of my different logtypes has patterns and filters setup to
parse data out of them that rely on the type being set appropriately.
How do you grok parse fields if you are going direct to ES? Logstash
does that bit, and you're bypassing it here.
I work in network security and can't lose pretty much *any* logs.
Logging to file bypasses all of these issues, and the logstash file
input maintains a sincedb state of file positioning and can index at
its leisure, even if logstash is stopped and restarted, it will pick up from
where it left off.
Plus we have requirements to maintain the logs for 6+ months, but we
do not need to maintain 6 month elasticsearch searchable data.
I keep 3 days of uncompressed raw logs for easy indexing / reindexing, and
everything older than that is bzipped, backed up, and stored. Sure you
could use elasticsearch-knapsack to export/backup your ES data, but
it's far easier to just maintain the raw logs.
But hey, to each their own.
On Wed, May 7, 2014 at 12:43 PM, David Lang <[email protected]> wrote:
at my old job we had ossec configured to send to rsyslog
personally I really dislike the 'write to a file and then scrape it
with another program' approach to logs
Yes, it handles cases where your logserver is down, but you should
have HA so that's a very rare case.
But it causes a bunch of headaches
1. a lot more disk I/O
2. polling to check if the file has changed
3. headaches if the files roll too fast
4. problems deciding when you can delete the files
It's just so much easier to pass the data directly to rsyslog and let
it deal with everything :-)
David Lang
On Wed, 7 May 2014, Josh Bitto wrote:
Date: Wed, 7 May 2014 09:44:43 -0700
From: Josh Bitto <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana
server
Hello Everyone and Good Morning!
I have a new question for you all. Does anyone have this current
setup with an OSSEC server as well? I'm wondering which would be the
better option to do. Just create an imfile for Rsyslog to monitor
the logs from OSSEC or forward them to rsyslog. I'm curious to find
out if anyone else has this implemented too!
Josh
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
