Ok, that's a lot cleaner. Do you want the logs that you receive remotely to go
anywhere other than ES?
not that I want to discourage you from using rsyslog ;-)
But are you aware that syslog-ng can deliver logs directly to ES? If rsyslog
isn't providing any other value (and especially if you are already very
comforatable with syslog-ng), you may be better off going directly instead of
adding a hop. I don't know syslog-ng and what it can and can't do. I've hit
unpleasent surprises with both nxlog and logstash when I assumed that they would
do some of the things that I find trivial to do with rsyslog :-)
a quick google search found:
https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-elasticsearch.html
In reading the linux-help page, it looks like it's having you send logs to
logstash, so you would end up running syslog-ng, rsyslog, and logstash. I would
sure not want to do that if I could avoid it.
Assuming that you don't need the remote logs to go anywhere else, and you don't
need anything local to go to ES, then remove the UDP/TCP stuff at the top of
your current config and add the following
# load the pstats module and output data every 10 sec. change this after testing
# remove file (so the pstats data is processed like other log messages)
# change 10 to a value that is reasonable for your environment
# note that counters are not reset so most values are cumulative since startup
module(load="impstats" interval="10" resetCounters="off" format="legacy"
file="/var/log/pstats")
# load the modules to listen to the network and configure them to send their
# logs to the 'inbound' ruleset
module(load="imtcp")
module(load="imudp" timerequery="4" )
input(type="imtcp" port="514" ruleset="inbound" )
input(type="imudp" port="514" ruleset="inbound")
# define the inbound ruleset, all logs will go to elasticsearch
ruleset(name="inbound" queue.type="fixedarray"){
action(name = "send_es" type="omelasticsearch"
action.resumeretrycount="-1")
}
then do
rsyslogd -N2
to check the syntax of things.
If there are no errors, restart rsyslog and look at the /var/log/pstats logs,
specifically you are going to be looking for the imtcp, inblund, and send_es
lines in each 10 second batch.
now, how this differs from what you had before:
*.* action(type="omelasticsearch"
you don't need *.* (match everything)
server="localhost"
serverport="9200"
these are the default. validate that you do have ES running on the same box as
rsyslog, and it is listening on port 9200
template="logstash"
searchIndex="logstash-index"
dynSearchIndex="on"
searchType="syslog"
bulkmode="on"
These are more advanced features and interact with how you have elasticsearch
setup, so let's hold off on these for the moment.
The first one we will need to add is searchIndex. How do you have ES setup
currently? What index (or indexes) do you want the logs to go into?
queue.type="linkedlist"
queue.size="5000"
queue.dequeuebatchsize="300"
I moved the queue from the elasticsearch action to the ruleset. We may add one
back later if there are enough other actions in the rulset
action.resumeretrycount="-1")
without this, a failure won't be retried and you will have trouble figuring out
what's happening.
David Lang
On Fri, 29 Jan 2016, Brad Cox wrote:
Date: Fri, 29 Jan 2016 19:48:59 -0500
From: Brad Cox <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Logs from remote server not making it to Elastic Search
I only care about remote logs. I only need these to go to elastic search. I
don't care about the local ones except it would be nice to have them go to
files somewhere.
I started with the plain rsyslog.conf that came with the Xubuntu iso. I pasted
it at the end of this message.
I modified the security onion syslog-ng.conf file to copy everything it
collects to tcp port 10514. These are the key lines I added at the end of its
original conf.
destination d_net { tcp("192.168.48.145" port(10514)); };
log { source(s_syslog); destination(d_net); }
SO events come from various SO sensors that I don't yet fully understand
(snort, bro, elsa, others) with many more sensor types expected over time.
Right now I'm just trying to collect them all in ES. I expect to untangle the
mess once I know what I need.
Here's my backup of rsyslog.conf
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 10514
# Enable non-kernel facility klog messages
$KLogPermitNonKernelFacility on
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox
On Jan 29, 2016, at 7:33 PM, David Lang <[email protected]> wrote:
Ok, changing the default ruleset to remote is probably not what you want to do,
because then your local messages won't go to all the files defined in the local
ruleset.
Let's back up a bit here. Before you started with Elasticsearch, what did your
config look like? Let's go back to that first.
Then let's talk about what do you want to have go to elasticsearch. Do you want
everything there?, only some things there?, all in one index?, some logs in
one, some in another?, etc.
Then we can add an action to deliver the logs to elasticsearch to the right
place in your config file.
We will probably also want to add impstats so that you can see how many logs
are being processed and what are being queued. We may need to turn on debugging
if we see everything being queued to elasticsearch, but that's further down the
line.
For now, let's just get back to a state that you understand and is handling
your logs the way you want them handled (except for ES)
David Lang
On Fri, 29 Jan 2016, Brad Cox wrote:
Strange. I copied that directly from
https://linux-help.org/wiki/logging/rsyslog/advanced-rsyslog.
I've moved that action block into the remote section as shown below and
restarted rsyslogd. Also changed the DefaultRuleSet to remote.
ruleset(name="remote") {
$IncludeConfig /etc/rsyslog.d/*.remote
action(type="omfile" DynaFile="RemoteHost")
*.* action(type="omelasticsearch"
server="localhost"
serverport="9200"
template="logstash"
searchIndex="logstash-index"
dynSearchIndex="on"
searchType="syslog"
bulkmode="on"
queue.type="linkedlist"
queue.size="5000"
queue.dequeuebatchsize="300"
action.resumeretrycount="-1")
}
$DefaultRuleset remote
But still no sign of security onion messages in elastic search. I'm doing all
this with very little understanding at this point. Any insight into how to
diagnose such problems would be most helpful.
Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox
On Jan 29, 2016, at 6:53 PM, David Lang <[email protected]> wrote:
from a quick glance, the elasticsearch call is part of the local ruleset while
anything that arrives remotely only has the remote ruleset applied to it.
Am I missing something?
David Lang
On Fri, 29 Jan 2016, Brad Cox wrote:
Date: Fri, 29 Jan 2016 13:55:56 -0500
From: Brad Cox <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Logs from remote server not making it to Elastic Search
OK. Do you need the rsyslog.d/* files too?
# https://linux-help.org/wiki/logging/rsyslog/advanced-rsyslog
# rsyslog v7 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
# $DebugFile /var/log/rsyslog.debug.log
# $DebugLevel 1
#### MODULES ####
module(load="imuxsock") # local system logging (e.g. via logger command)
module(load="imklog") # provides kernel logging (previously done by rklogd)
#module(load"immark") # --MARK-- message capability
module(load="mmnormalize")
module(load="omelasticsearch")
# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")
module(load="imudp")
# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
# module(load="imtcp") # needs to be done just once
# input(type="imtcp" port="10514")
# module(load="imtcp" MaxSessions="500")
# module(load="imrelp" RuleSet="remote")
module(load="imtcp" MaxSessions="500")
#input(type="imtcp" port="10514")
input(type="imudp" port="10514" ruleset="remote")
input(type="imtcp" port="10514" ruleset="remote")
$AllowedSender TCP, 127.0.0.1, 192.168.48.137
#### GLOBAL DIRECTIVES ####
# Templates
template(name="RemoteHost" type="string"
string="/srv/log/%HOSTNAME%/%$YEAR%/%$MONTH%/syslog-%$DAY%.log")
$IncludeConfig /etc/rsyslog.d/*.template
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
#### RULES ####
ruleset(name="local") {
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
kern.* /var/log/kern.log
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
*.* action(type="omelasticsearch"
server="localhost"
serverport="9200"
template="logstash"
searchIndex="logstash-index"
dynSearchIndex="on"
searchType="syslog"
bulkmode="on"
queue.type="linkedlist"
queue.size="5000"
queue.dequeuebatchsize="300"
action.resumeretrycount="-1")
}
$DefaultRuleset local
ruleset(name="remote") {
$IncludeConfig /etc/rsyslog.d/*.remote
action(type="omfile" DynaFile="RemoteHost")
}
#input(type="imudp" port="10514" ruleset="remote")
#input(type="imtcp" port="10514" ruleset="remote")
# $InputRELPServerBindRuleset remote
# input(type="imrelp" port="20514")
Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox
On Jan 29, 2016, at 1:20 PM, David Lang <[email protected]> wrote:
please post your config (rsyslog.conf)
David Lang
On Fri, 29 Jan 2016, Brad Cox wrote:
Date: Fri, 29 Jan 2016 12:58:34 -0500
From: Brad Cox <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: [email protected]
Subject: [rsyslog] Logs from remote server not making it to Elastic Search
I'm trying to centralize logs that originate from syslog-ng on a Security
Onion server (Ubuntu 14.04.3-11) at an central server running rsyslog
latest. I've configured SO to copy all logs to the central server on TCP
port 10514 and confirmed that Tcpdump shows log traffic (on TCP port 10514)
is leaving SO and arriving at the central server.
My goal is to ingest the remote logs into Elastic Search so I've been
following the instructions at
https://linux-help.org/wiki/logging/rsyslog/advanced-rsyslog. I had to
tweak some settings to get past parsing errors from rsyslog but that seems
OK now (according to rsyslogd -f rsyslog.conf -N1 with and without
$DebugLevel=1). At least rsyslogd now launches and runs correctly.
Problem is, only local logs are appearing in the Elastic Search index
(according to kibana). There's no sign of remote logs from Security Onion
in the ES indices. The only trace of remote logs is in
/srv/log/SecurityOnionVM/...) which seems to contain the right contents,
although I can find no signs of how they wound up there in the debug logs.
I'm new at rsyslog and have ridden the documentation about as far as it
will take me. Can someone help me get this bus on its wheels?
Dr. Brad J. Cox Cell: 703-594-1883 Blog: http://bradjcox.blogspot.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
