OK, I rebooted and ran your grep. No sign of anything being processed ( /=[1-9] in vim came up empty on the whole file so i omitted most duplicates:
grep -e imtcp -e inbount -e omelasticsearch -e send_es pstats ... Sat Jan 30 12:13:49 2016: imtcp(10514): origin=imtcp submitted=0 Sat Jan 30 12:13:59 2016: omelasticsearch: origin=omelasticsearch submitted=0 failed.http=0 failed.httprequests=0 failed.es=0 Sat Jan 30 12:13:59 2016: send_es: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Sat Jan 30 12:13:59 2016: imtcp(10514): origin=imtcp submitted=0 Sat Jan 30 12:14:08 2016: omelasticsearch: origin=omelasticsearch submitted=0 failed.http=0 failed.httprequests=0 failed.es=0 Sat Jan 30 12:14:08 2016: send_es: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Sat Jan 30 12:14:08 2016: imtcp(10514): origin=imtcp submitted=0 /= Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox > On Jan 30, 2016, at 9:22 PM, David Lang <[email protected]> wrote: > > On Sat, 30 Jan 2016, Brad Cox wrote: > >> Thanks! >> >> Wasn't sure whether you meant to use your config instead of my default >> rsyslog-config or to insert yours at the beginning of my module statements. >> I did the former and patched a few errors to wind up with just this: >> >> module( >> load="impstats" >> interval="10" >> resetCounters="off" >> format="legacy" >> file="/var/log/pstats" >> ) >> >> # load the modules to listen to the network and configure them to send their >> # logs to the 'inbound' ruleset >> module(load="imtcp") >> module(load="imudp" timerequery="4" ) >> input(type="imtcp" port="10514" ruleset="inbound" ) >> input(type="imudp" port="10514" ruleset="inbound") >> >> # define the inbound ruleset, all logs will go to elasticsearch >> ruleset( >> name="inbound" >> queue.type="fixedarray" >> ){ >> action( >> name = "send_es" >> type="omelasticsearch" >> action.resumeretrycount="-1" >> ) >> >> What I'm seeing in /var/log/pstat is lots and lots of example at the end and >> still still no sign of traffic from security onion in kibana. Ideas? >> >> BTW: the goal is to keep SO clients and server completely intact and >> unmodified (except for the two lines I showed earlier that copy all sensor >> -> server events to a separate VM. This will grow into a spark/hadoop >> cluster that we'll evolve to do advanced analytics that SO doesn't provide. >> The mixing of syslog-ng and rsyslog is just because SO uses Ubuntu and >> bcoxVM uses Xubuntu. According to the docs they're interchangable. >> >> I've seen the recipe link you provided as well as a similar one that routes >> traffic via kafka. I used a different one (link provided earlier) because it >> didn't involve logstash/ruby to avoid unnecessary complexity. Once I get >> this recipe working I'll probably try via kafka. >> >> Sat Jan 30 12:13:09 2016: imtcp(10514): origin=imtcp submitted=0 >> Sat Jan 30 12:13:09 2016: imudp(*:10514): origin=imudp submitted=0 >> Sat Jan 30 12:13:09 2016: imudp(*:10514): origin=imudp submitted=0 > > these tell you how many messages have been submitted via these inputs. > >> Sat Jan 30 12:13:09 2016: resource-usage: origin=impstats utime=4000 >> stime=4000 maxrss=9724 minflt=184 majflt=0 inblock=8 oublock=176 nvcsw=124 >> nivcsw=4 > > ignore this for now, it is how many resources rsyslog has consumed in the > last 10 sec > >> Sat Jan 30 12:13:09 2016: inbound: origin=core.queue size=0 enqueued=0 >> full=0 discarded.full=0 discarded.nf=0 maxqsize=0 > > how many messages have gone to the inbound ruleset (enqueued=0 means none) > >> Sat Jan 30 12:13:09 2016: main Q: origin=core.queue size=7 enqueued=422 >> full=0 discarded.full=0 discarded.nf=0 maxqsize=9 > > ignore this for now, it shows how many messages went to the main queue. > >> Sat Jan 30 12:13:09 2016: imudp(w0): origin=imudp called.recvmmsg=0 >> called.recvmsg=0 msgs.received=0 > > more stats on inbound udp, but you whould be getting things via tcp, so > ignore it > >> Sat Jan 30 12:13:19 2016: omelasticsearch: origin=omelasticsearch >> submitted=0 failed.http=0 failed.httprequests=0 failed.es=0 > > stats from the elasticsearch module > >> Sat Jan 30 12:13:19 2016: send_es: origin=core.action processed=0 failed=0 >> suspended=0 suspended.duration=0 resumed=0 > > stats from the action delivering logs to ES > >> Sat Jan 30 12:14:08 2016: inbound: origin=core.queue size=0 enqueued=0 >> full=0 discarded.full=0 discarded.nf=0 maxqsize=0 > > so this shows that no logs have been deliverd to rsylog yet. you may need to > kick syslog-ng on the senders to get it to try again. > > do a grep -e imtcp -e inbount -e omelasticsearch -e send_es on the pstats > file, we need the end of it when it shows some data arriving. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
