On Wed, 23 Jul 2014, Cristian Falcas wrote:
Unfortunately I have drop rules in omfile also, because I want to separate
the logs to different files.
get rid of the drop rules until after you have finished processing the log
With the new format you can do
if <condition> then {
multiple actions
}
multiple actions can be to do the write to a file, then the write to
elasticsearch, then drop
If your filters are mutually exclusive then the drop is just a performance
optimization, but if it causes grief, get rid of it.
Also, there are a bunch of other tricks that can be pulled to optimize writing
to files depending on your filters, but without seeing them we can't help you
David Lang
On Wed, Jul 23, 2014 at 3:16 PM, Rainer Gerhards <[email protected]>
wrote:
On Wed, Jul 23, 2014 at 2:12 PM, Cristian Falcas <[email protected]>
wrote:
I think I managed to solve this by calling the elasticsearch remote rule
from inside the remote rule:
# the remote rule is defined like:
$RuleSet remote
$RulesetCreateMainQueue on
call remote_elasticsearch
# and my elastic search config:
$ModLoad omelasticsearch
$RuleSet remote_elasticsearch
$RulesetCreateMainQueue on
From what I've read, this should send the messages asynchronously between
the rules.
I think this work, but don't remember the legacy syntax well enough to say
that 100% sure.
However, a simpler solution (and faster) is to simply move the omfile
actions in front of the omelasticsearch ones.
Rainer
Best regards,
Cristian Falcas
On Wed, Jul 23, 2014 at 2:16 PM, Cristian Falcas <
[email protected]>
wrote:
Sorry for being so vague.
What I'm trying to do is to send messages received via tcp module to 2
different output modules:
- one to write to local files
- second one to send messages to elasticsearch
Because in my elasticsearch configuration I have some drop rules,
nothing
reaches the omfile rules. How can I decouple those 2? I want everything
that comes from tcp to go to both output modules and not be tied to
each
other.
Is there a way to achive this?
Best regards,
Cristian Falcas
On Tue, Jul 22, 2014 at 9:10 PM, David Lang <[email protected]> wrote:
you have the right idea, I don't understand what you are asking for
help
on.
David Lang
On Tue, 22 Jul 2014, Cristian Falcas wrote:
Hi,
I have configured a rsyslog server where I want to send the logs from
the
other machines.
Here I want to keep local logs and also to send them to elastic
search.
For
elasticsearch I have multiple actions, because I want httpd (for ex.)
to
go
to searchType="httpd" and so on. After each action I would like to
drop
the previously catched lines, because I want a last action with the
default
search type. Because of my dop rule, nothing reaches past the first
action:
not the omfile writer and not the elasticsearch actions.
I'm using the same $RuleSet remote for both output modules.
Can anyone help me in setting this correctly?
Something like this (with multiple templates and actions):
$RuleSet remote
$ModLoad omelasticsearch
template(name="10-audit"
type="list"
option.json="on").
{
constant(value="{")
constant(value="\"@timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"timereported\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"timegenerated\":\"")
property(name="timegenerated" dateFormat="rfc3339")
constant(value="\",\"message\":\"")
property(name="msg")
constant(value="\",\"host\":\"")
property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"priority\":\"")
property(name="syslogpriority-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"tag\":\"")
property(name="syslogtag")
constant(value="\",\"program_name\":\"")
property(name="programname")
constant(value="\"}")
}
*.* action(type="omelasticsearch"
name="action_10-audit"
server="v-so-repo-02"
serverport="9200"
template="10-audit"
searchIndex="default-index"
searchType="audit"
bulkmode="on" # use the Bulk API
queue.dequeuebatchsize="5000" # ES bulk size
queue.size="100000" # capacity of the action queue
queue.workerthreads="15" # 5 workers for the action
queue.type="linkedlist"
queue.FileName="es_queue"
queue.MaxDiskSpace="1g"
queue.SaveOnShutdown="on"
action.resumeretrycount="-1"
errorFile="/srv/log/rsyslog_ES-error_default-index_audit.log"
)
& stop
Best regards,
Cristian Falcas
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
