Thanks!
Wasn't sure whether you meant to use your config instead of my default
rsyslog-config or to insert yours at the beginning of my module statements. I
did the former and patched a few errors to wind up with just this:
module(
load="impstats"
interval="10"
resetCounters="off"
format="legacy"
file="/var/log/pstats"
)
# load the modules to listen to the network and configure them to send their
# logs to the 'inbound' ruleset
module(load="imtcp")
module(load="imudp" timerequery="4" )
input(type="imtcp" port="10514" ruleset="inbound" )
input(type="imudp" port="10514" ruleset="inbound")
# define the inbound ruleset, all logs will go to elasticsearch
ruleset(
name="inbound"
queue.type="fixedarray"
){
action(
name = "send_es"
type="omelasticsearch"
action.resumeretrycount="-1"
)
What I'm seeing in /var/log/pstat is lots and lots of example at the end and
still still no sign of traffic from security onion in kibana. Ideas?
BTW: the goal is to keep SO clients and server completely intact and unmodified
(except for the two lines I showed earlier that copy all sensor -> server
events to a separate VM. This will grow into a spark/hadoop cluster that we'll
evolve to do advanced analytics that SO doesn't provide. The mixing of
syslog-ng and rsyslog is just because SO uses Ubuntu and bcoxVM uses Xubuntu.
According to the docs they're interchangable.
I've seen the recipe link you provided as well as a similar one that routes
traffic via kafka. I used a different one (link provided earlier) because it
didn't involve logstash/ruby to avoid unnecessary complexity. Once I get this
recipe working I'll probably try via kafka.
Sat Jan 30 12:13:09 2016: imtcp(10514): origin=imtcp submitted=0
Sat Jan 30 12:13:09 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:09 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:09 2016: resource-usage: origin=impstats utime=4000 stime=4000
maxrss=9724 minflt=184 majflt=0 inblock=8 oublock=176 nvcsw=124 nivcsw=4
Sat Jan 30 12:13:09 2016: inbound: origin=core.queue size=0 enqueued=0 full=0
discarded.full=0 discarded.nf=0 maxqsize=0
Sat Jan 30 12:13:09 2016: main Q: origin=core.queue size=7 enqueued=422 full=0
discarded.full=0 discarded.nf=0 maxqsize=9
Sat Jan 30 12:13:09 2016: imudp(w0): origin=imudp called.recvmmsg=0
called.recvmsg=0 msgs.received=0
Sat Jan 30 12:13:19 2016: omelasticsearch: origin=omelasticsearch submitted=0
failed.http=0 failed.httprequests=0 failed.es=0
Sat Jan 30 12:13:19 2016: send_es: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Sat Jan 30 12:13:19 2016: imtcp(10514): origin=imtcp submitted=0
Sat Jan 30 12:13:19 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:19 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:19 2016: resource-usage: origin=impstats utime=4000 stime=4000
maxrss=9724 minflt=184 majflt=0 inblock=8 oublock=192 nvcsw=126 nivcsw=4
Sat Jan 30 12:13:19 2016: inbound: origin=core.queue size=0 enqueued=0 full=0
discarded.full=0 discarded.nf=0 maxqsize=0
Sat Jan 30 12:13:19 2016: main Q: origin=core.queue size=7 enqueued=431 full=0
discarded.full=0 discarded.nf=0 maxqsize=9
Sat Jan 30 12:13:19 2016: imudp(w0): origin=imudp called.recvmmsg=0
called.recvmsg=0 msgs.received=0
Sat Jan 30 12:13:29 2016: omelasticsearch: origin=omelasticsearch submitted=0
failed.http=0 failed.httprequests=0 failed.es=0
Sat Jan 30 12:13:29 2016: send_es: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Sat Jan 30 12:13:29 2016: imtcp(10514): origin=imtcp submitted=0
Sat Jan 30 12:13:29 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:29 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:29 2016: resource-usage: origin=impstats utime=4000 stime=4000
maxrss=9724 minflt=184 majflt=0 inblock=8 oublock=192 nvcsw=128 nivcsw=4
Sat Jan 30 12:13:29 2016: inbound: origin=core.queue size=0 enqueued=0 full=0
discarded.full=0 discarded.nf=0 maxqsize=0
Sat Jan 30 12:13:29 2016: main Q: origin=core.queue size=7 enqueued=440 full=0
discarded.full=0 discarded.nf=0 maxqsize=9
Sat Jan 30 12:13:29 2016: imudp(w0): origin=imudp called.recvmmsg=0
called.recvmsg=0 msgs.received=0
Sat Jan 30 12:13:39 2016: omelasticsearch: origin=omelasticsearch submitted=0
failed.http=0 failed.httprequests=0 failed.es=0
Sat Jan 30 12:13:39 2016: send_es: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Sat Jan 30 12:13:39 2016: imtcp(10514): origin=imtcp submitted=0
Sat Jan 30 12:13:39 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:39 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:39 2016: resource-usage: origin=impstats utime=4000 stime=4000
maxrss=9724 minflt=184 majflt=0 inblock=8 oublock=192 nvcsw=130 nivcsw=4
Sat Jan 30 12:13:39 2016: inbound: origin=core.queue size=0 enqueued=0 full=0
discarded.full=0 discarded.nf=0 maxqsize=0
Sat Jan 30 12:13:39 2016: main Q: origin=core.queue size=7 enqueued=449 full=0
discarded.full=0 discarded.nf=0 maxqsize=9
Sat Jan 30 12:13:39 2016: imudp(w0): origin=imudp called.recvmmsg=0
called.recvmsg=0 msgs.received=0
Sat Jan 30 12:13:49 2016: omelasticsearch: origin=omelasticsearch submitted=0
failed.http=0 failed.httprequests=0 failed.es=0
Sat Jan 30 12:13:49 2016: send_es: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Sat Jan 30 12:13:49 2016: imtcp(10514): origin=imtcp submitted=0
Sat Jan 30 12:13:49 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:49 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:49 2016: resource-usage: origin=impstats utime=4000 stime=4000
maxrss=9724 minflt=184 majflt=0 inblock=8 oublock=192 nvcsw=132 nivcsw=4
Sat Jan 30 12:13:49 2016: inbound: origin=core.queue size=0 enqueued=0 full=0
discarded.full=0 discarded.nf=0 maxqsize=0
Sat Jan 30 12:13:49 2016: main Q: origin=core.queue size=7 enqueued=458 full=0
discarded.full=0 discarded.nf=0 maxqsize=9
Sat Jan 30 12:13:49 2016: imudp(w0): origin=imudp called.recvmmsg=0
called.recvmsg=0 msgs.received=0
Sat Jan 30 12:13:59 2016: omelasticsearch: origin=omelasticsearch submitted=0
failed.http=0 failed.httprequests=0 failed.es=0
Sat Jan 30 12:13:59 2016: send_es: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Sat Jan 30 12:13:59 2016: imtcp(10514): origin=imtcp submitted=0
Sat Jan 30 12:13:59 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:59 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:13:59 2016: resource-usage: origin=impstats utime=4000 stime=4000
maxrss=9724 minflt=184 majflt=0 inblock=8 oublock=208 nvcsw=134 nivcsw=4
Sat Jan 30 12:13:59 2016: inbound: origin=core.queue size=0 enqueued=0 full=0
discarded.full=0 discarded.nf=0 maxqsize=0
Sat Jan 30 12:13:59 2016: main Q: origin=core.queue size=7 enqueued=467 full=0
discarded.full=0 discarded.nf=0 maxqsize=9
Sat Jan 30 12:13:59 2016: imudp(w0): origin=imudp called.recvmmsg=0
called.recvmsg=0 msgs.received=0
Sat Jan 30 12:14:08 2016: omelasticsearch: origin=omelasticsearch submitted=0
failed.http=0 failed.httprequests=0 failed.es=0
Sat Jan 30 12:14:08 2016: send_es: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Sat Jan 30 12:14:08 2016: imtcp(10514): origin=imtcp submitted=0
Sat Jan 30 12:14:08 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:14:08 2016: imudp(*:10514): origin=imudp submitted=0
Sat Jan 30 12:14:08 2016: resource-usage: origin=impstats utime=4000 stime=4000
maxrss=9724 minflt=184 majflt=0 inblock=8 oublock=208 nvcsw=137 nivcsw=4
Sat Jan 30 12:14:08 2016: inbound: origin=core.queue size=0 enqueued=0 full=0
discarded.full=0 discarded.nf=0 maxqsize=0
Sat Jan 30 12:14:08 2016: main Q: origin=core.queue size=7 enqueued=476 full=0
discarded.full=0 discarded.nf=0 maxqsize=9
Sat Jan 30 12:14:08 2016: imudp(w0): origin=imudp called.recvmmsg=0
called.recvmsg=0 msgs.received=0
Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox
> On Jan 29, 2016, at 8:21 PM, David Lang <[email protected]> wrote:
>
> Ok, that's a lot cleaner. Do you want the logs that you receive remotely to
> go anywhere other than ES?
>
> not that I want to discourage you from using rsyslog ;-)
>
> But are you aware that syslog-ng can deliver logs directly to ES? If rsyslog
> isn't providing any other value (and especially if you are already very
> comforatable with syslog-ng), you may be better off going directly instead of
> adding a hop. I don't know syslog-ng and what it can and can't do. I've hit
> unpleasent surprises with both nxlog and logstash when I assumed that they
> would do some of the things that I find trivial to do with rsyslog :-)
>
> a quick google search found:
> https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-elasticsearch.html
>
> In reading the linux-help page, it looks like it's having you send logs to
> logstash, so you would end up running syslog-ng, rsyslog, and logstash. I
> would sure not want to do that if I could avoid it.
>
> Assuming that you don't need the remote logs to go anywhere else, and you
> don't need anything local to go to ES, then remove the UDP/TCP stuff at the
> top of your current config and add the following
>
> # load the pstats module and output data every 10 sec. change this after
> testing
> # remove file (so the pstats data is processed like other log messages)
> # change 10 to a value that is reasonable for your environment
> # note that counters are not reset so most values are cumulative since
> startup
>
> module(load="impstats" interval="10" resetCounters="off" format="legacy"
> file="/var/log/pstats")
>
> # load the modules to listen to the network and configure them to send their
> # logs to the 'inbound' ruleset
> module(load="imtcp")
> module(load="imudp" timerequery="4" )
> input(type="imtcp" port="514" ruleset="inbound" )
> input(type="imudp" port="514" ruleset="inbound")
>
> # define the inbound ruleset, all logs will go to elasticsearch
>
> ruleset(name="inbound" queue.type="fixedarray"){
> action(name = "send_es" type="omelasticsearch"
> action.resumeretrycount="-1")
> }
>
> then do
>
> rsyslogd -N2
>
> to check the syntax of things.
>
> If there are no errors, restart rsyslog and look at the /var/log/pstats logs,
> specifically you are going to be looking for the imtcp, inblund, and send_es
> lines in each 10 second batch.
>
> now, how this differs from what you had before:
>
>>>> *.* action(type="omelasticsearch"
>
> you don't need *.* (match everything)
>
>>>> server="localhost"
>>>> serverport="9200"
>
> these are the default. validate that you do have ES running on the same box
> as rsyslog, and it is listening on port 9200
>
>>>> template="logstash"
>>>> searchIndex="logstash-index"
>>>> dynSearchIndex="on"
>>>> searchType="syslog"
>>>> bulkmode="on"
>
> These are more advanced features and interact with how you have elasticsearch
> setup, so let's hold off on these for the moment.
>
> The first one we will need to add is searchIndex. How do you have ES setup
> currently? What index (or indexes) do you want the logs to go into?
>
>>>> queue.type="linkedlist"
>>>> queue.size="5000"
>>>> queue.dequeuebatchsize="300"
>
> I moved the queue from the elasticsearch action to the ruleset. We may add
> one back later if there are enough other actions in the rulset
>
>>>> action.resumeretrycount="-1")
>
> without this, a failure won't be retried and you will have trouble figuring
> out what's happening.
>
> David Lang
>
> On Fri, 29 Jan 2016, Brad Cox wrote:
>
>> Date: Fri, 29 Jan 2016 19:48:59 -0500
>> From: Brad Cox <[email protected]>
>> Reply-To: rsyslog-users <[email protected]>
>> To: rsyslog-users <[email protected]>
>> Subject: Re: [rsyslog] Logs from remote server not making it to Elastic
>> Search
>> I only care about remote logs. I only need these to go to elastic search. I
>> don't care about the local ones except it would be nice to have them go to
>> files somewhere.
>>
>> I started with the plain rsyslog.conf that came with the Xubuntu iso. I
>> pasted it at the end of this message.
>>
>> I modified the security onion syslog-ng.conf file to copy everything it
>> collects to tcp port 10514. These are the key lines I added at the end of
>> its original conf.
>>
>> destination d_net { tcp("192.168.48.145" port(10514)); };
>> log { source(s_syslog); destination(d_net); }
>>
>> SO events come from various SO sensors that I don't yet fully understand
>> (snort, bro, elsa, others) with many more sensor types expected over time.
>> Right now I'm just trying to collect them all in ES. I expect to untangle
>> the mess once I know what I need.
>>
>> Here's my backup of rsyslog.conf
>>
>> # /etc/rsyslog.conf Configuration file for rsyslog.
>> #
>> # For more information see
>> # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
>> #
>> # Default logging rules can be found in /etc/rsyslog.d/50-default.conf
>>
>>
>> #################
>> #### MODULES ####
>> #################
>>
>> $ModLoad imuxsock # provides support for local system logging
>> $ModLoad imklog # provides kernel logging support
>> #$ModLoad immark # provides --MARK-- message capability
>>
>> # provides UDP syslog reception
>> #$ModLoad imudp
>> #$UDPServerRun 514
>>
>> # provides TCP syslog reception
>> $ModLoad imtcp
>> $InputTCPServerRun 10514
>>
>> # Enable non-kernel facility klog messages
>> $KLogPermitNonKernelFacility on
>>
>> ###########################
>> #### GLOBAL DIRECTIVES ####
>> ###########################
>>
>> #
>> # Use traditional timestamp format.
>> # To enable high precision timestamps, comment out the following line.
>> #
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>
>> # Filter duplicated messages
>> $RepeatedMsgReduction on
>>
>> #
>> # Set the default permissions for all log files.
>> #
>> $FileOwner syslog
>> $FileGroup adm
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> #
>> # Where to place spool and state files
>> #
>> $WorkDirectory /var/spool/rsyslog
>>
>> #
>> # Include all config files in /etc/rsyslog.d/
>> #
>> $IncludeConfig /etc/rsyslog.d/*.conf
>>
>>
>> Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox
>>
>>
>>
>>
>>> On Jan 29, 2016, at 7:33 PM, David Lang <[email protected]> wrote:
>>>
>>> Ok, changing the default ruleset to remote is probably not what you want to
>>> do, because then your local messages won't go to all the files defined in
>>> the local ruleset.
>>>
>>> Let's back up a bit here. Before you started with Elasticsearch, what did
>>> your config look like? Let's go back to that first.
>>>
>>> Then let's talk about what do you want to have go to elasticsearch. Do you
>>> want everything there?, only some things there?, all in one index?, some
>>> logs in one, some in another?, etc.
>>>
>>> Then we can add an action to deliver the logs to elasticsearch to the right
>>> place in your config file.
>>>
>>> We will probably also want to add impstats so that you can see how many
>>> logs are being processed and what are being queued. We may need to turn on
>>> debugging if we see everything being queued to elasticsearch, but that's
>>> further down the line.
>>>
>>> For now, let's just get back to a state that you understand and is handling
>>> your logs the way you want them handled (except for ES)
>>>
>>> David Lang
>>>
>>> On Fri, 29 Jan 2016, Brad Cox wrote:
>>>
>>>> Strange. I copied that directly from
>>>> https://linux-help.org/wiki/logging/rsyslog/advanced-rsyslog.
>>>>
>>>> I've moved that action block into the remote section as shown below and
>>>> restarted rsyslogd. Also changed the DefaultRuleSet to remote.
>>>>
>>>> ruleset(name="remote") {
>>>> $IncludeConfig /etc/rsyslog.d/*.remote
>>>> action(type="omfile" DynaFile="RemoteHost")
>>>> *.* action(type="omelasticsearch"
>>>> server="localhost"
>>>> serverport="9200"
>>>> template="logstash"
>>>> searchIndex="logstash-index"
>>>> dynSearchIndex="on"
>>>> searchType="syslog"
>>>> bulkmode="on"
>>>> queue.type="linkedlist"
>>>> queue.size="5000"
>>>> queue.dequeuebatchsize="300"
>>>> action.resumeretrycount="-1")
>>>> }
>>>> $DefaultRuleset remote
>>>>
>>>> But still no sign of security onion messages in elastic search. I'm doing
>>>> all this with very little understanding at this point. Any insight into
>>>> how to diagnose such problems would be most helpful.
>>>>
>>>> Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox
>>>>
>>>>
>>>>
>>>>
>>>>> On Jan 29, 2016, at 6:53 PM, David Lang <[email protected]> wrote:
>>>>>
>>>>> from a quick glance, the elasticsearch call is part of the local ruleset
>>>>> while anything that arrives remotely only has the remote ruleset applied
>>>>> to it.
>>>>>
>>>>> Am I missing something?
>>>>>
>>>>> David Lang
>>>>>
>>>>>
>>>>> On Fri, 29 Jan 2016, Brad Cox wrote:
>>>>>
>>>>>> Date: Fri, 29 Jan 2016 13:55:56 -0500
>>>>>> From: Brad Cox <[email protected]>
>>>>>> Reply-To: rsyslog-users <[email protected]>
>>>>>> To: rsyslog-users <[email protected]>
>>>>>> Subject: Re: [rsyslog] Logs from remote server not making it to Elastic
>>>>>> Search
>>>>>> OK. Do you need the rsyslog.d/* files too?
>>>>>>
>>>>>> # https://linux-help.org/wiki/logging/rsyslog/advanced-rsyslog
>>>>>> # rsyslog v7 configuration file
>>>>>>
>>>>>> # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
>>>>>> # If you experience problems, see
>>>>>> http://www.rsyslog.com/doc/troubleshoot.html
>>>>>> # $DebugFile /var/log/rsyslog.debug.log
>>>>>> # $DebugLevel 1
>>>>>>
>>>>>> #### MODULES ####
>>>>>>
>>>>>> module(load="imuxsock") # local system logging (e.g. via logger command)
>>>>>> module(load="imklog") # provides kernel logging (previously done by
>>>>>> rklogd)
>>>>>> #module(load"immark") # --MARK-- message capability
>>>>>> module(load="mmnormalize")
>>>>>> module(load="omelasticsearch")
>>>>>>
>>>>>> # Provides UDP syslog reception
>>>>>> # for parameters see http://www.rsyslog.com/doc/imudp.html
>>>>>> #module(load="imudp") # needs to be done just once
>>>>>> #input(type="imudp" port="514")
>>>>>> module(load="imudp")
>>>>>>
>>>>>> # Provides TCP syslog reception
>>>>>> # for parameters see http://www.rsyslog.com/doc/imtcp.html
>>>>>> # module(load="imtcp") # needs to be done just once
>>>>>> # input(type="imtcp" port="10514")
>>>>>> # module(load="imtcp" MaxSessions="500")
>>>>>> # module(load="imrelp" RuleSet="remote")
>>>>>> module(load="imtcp" MaxSessions="500")
>>>>>> #input(type="imtcp" port="10514")
>>>>>> input(type="imudp" port="10514" ruleset="remote")
>>>>>> input(type="imtcp" port="10514" ruleset="remote")
>>>>>>
>>>>>> $AllowedSender TCP, 127.0.0.1, 192.168.48.137
>>>>>>
>>>>>> #### GLOBAL DIRECTIVES ####
>>>>>> # Templates
>>>>>> template(name="RemoteHost" type="string"
>>>>>> string="/srv/log/%HOSTNAME%/%$YEAR%/%$MONTH%/syslog-%$DAY%.log")
>>>>>> $IncludeConfig /etc/rsyslog.d/*.template
>>>>>>
>>>>>> # Use default timestamp format
>>>>>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>>>>>>
>>>>>> # Include all config files in /etc/rsyslog.d/
>>>>>> $IncludeConfig /etc/rsyslog.d/*.conf
>>>>>>
>>>>>>
>>>>>> #### RULES ####
>>>>>> ruleset(name="local") {
>>>>>> # Log all kernel messages to the console.
>>>>>> # Logging much else clutters up the screen.
>>>>>> kern.* /var/log/kern.log
>>>>>>
>>>>>> # Log anything (except mail) of level info or higher.
>>>>>> # Don't log private authentication messages!
>>>>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>>>>>
>>>>>> # The authpriv file has restricted access.
>>>>>> authpriv.* /var/log/secure
>>>>>>
>>>>>> # Log all the mail messages in one place.
>>>>>> mail.* -/var/log/maillog
>>>>>>
>>>>>>
>>>>>> # Log cron stuff
>>>>>> cron.* /var/log/cron
>>>>>>
>>>>>> # Everybody gets emergency messages
>>>>>> *.emerg :omusrmsg:*
>>>>>>
>>>>>> # Save news errors of level crit and higher in a special file.
>>>>>> uucp,news.crit /var/log/spooler
>>>>>>
>>>>>> # Save boot messages also to boot.log
>>>>>> local7.* /var/log/boot.log
>>>>>>
>>>>>> *.* action(type="omelasticsearch"
>>>>>> server="localhost"
>>>>>> serverport="9200"
>>>>>> template="logstash"
>>>>>> searchIndex="logstash-index"
>>>>>> dynSearchIndex="on"
>>>>>> searchType="syslog"
>>>>>> bulkmode="on"
>>>>>> queue.type="linkedlist"
>>>>>> queue.size="5000"
>>>>>> queue.dequeuebatchsize="300"
>>>>>> action.resumeretrycount="-1")
>>>>>> }
>>>>>> $DefaultRuleset local
>>>>>>
>>>>>> ruleset(name="remote") {
>>>>>> $IncludeConfig /etc/rsyslog.d/*.remote
>>>>>> action(type="omfile" DynaFile="RemoteHost")
>>>>>> }
>>>>>> #input(type="imudp" port="10514" ruleset="remote")
>>>>>> #input(type="imtcp" port="10514" ruleset="remote")
>>>>>>
>>>>>> # $InputRELPServerBindRuleset remote
>>>>>> # input(type="imrelp" port="20514")
>>>>>>
>>>>>> Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Jan 29, 2016, at 1:20 PM, David Lang <[email protected]> wrote:
>>>>>>>
>>>>>>> please post your config (rsyslog.conf)
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>> On Fri, 29 Jan 2016, Brad Cox wrote:
>>>>>>>
>>>>>>>> Date: Fri, 29 Jan 2016 12:58:34 -0500
>>>>>>>> From: Brad Cox <[email protected]>
>>>>>>>> Reply-To: rsyslog-users <[email protected]>
>>>>>>>> To: [email protected]
>>>>>>>> Subject: [rsyslog] Logs from remote server not making it to Elastic
>>>>>>>> Search
>>>>>>>> I'm trying to centralize logs that originate from syslog-ng on a
>>>>>>>> Security
>>>>>>>> Onion server (Ubuntu 14.04.3-11) at an central server running rsyslog
>>>>>>>> latest. I've configured SO to copy all logs to the central server on
>>>>>>>> TCP
>>>>>>>> port 10514 and confirmed that Tcpdump shows log traffic (on TCP port
>>>>>>>> 10514)
>>>>>>>> is leaving SO and arriving at the central server.
>>>>>>>>
>>>>>>>> My goal is to ingest the remote logs into Elastic Search so I've been
>>>>>>>> following the instructions at
>>>>>>>> https://linux-help.org/wiki/logging/rsyslog/advanced-rsyslog. I had to
>>>>>>>> tweak some settings to get past parsing errors from rsyslog but that
>>>>>>>> seems
>>>>>>>> OK now (according to rsyslogd -f rsyslog.conf -N1 with and without
>>>>>>>> $DebugLevel=1). At least rsyslogd now launches and runs correctly.
>>>>>>>>
>>>>>>>> Problem is, only local logs are appearing in the Elastic Search index
>>>>>>>> (according to kibana). There's no sign of remote logs from Security
>>>>>>>> Onion
>>>>>>>> in the ES indices. The only trace of remote logs is in
>>>>>>>> /srv/log/SecurityOnionVM/...) which seems to contain the right
>>>>>>>> contents,
>>>>>>>> although I can find no signs of how they wound up there in the debug
>>>>>>>> logs.
>>>>>>>>
>>>>>>>> I'm new at rsyslog and have ridden the documentation about as far as it
>>>>>>>> will take me. Can someone help me get this bus on its wheels?
>>>>>>>>
>>>>>>>> Dr. Brad J. Cox Cell: 703-594-1883 Blog:
>>>>>>>> http://bradjcox.blogspot.com
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>>>>>> if you DON'T LIKE THAT.
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>>>>> if you DON'T LIKE THAT.
>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>>> DON'T LIKE THAT.
>>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
