Yes. Here's a sample. 137 is the security onion server, 145 is the proto-spark
cluster to be.
192.168.48.137.55850 > 192.168.48.145.10514: Flags [P.], cksum 0x6a3a
(correct), seq 374213:374307, ack 1, win 1024, options [nop,nop,TS val 84640133
ecr 13863448], length 94
04:51:01.599877 IP (tos 0x0, ttl 64, id 1678, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.48.145.10514 > 192.168.48.137.55850: Flags [.], cksum 0xe291
(incorrect -> 0x6ea1), seq 1, ack 374307, win 1452, options [nop,nop,TS val
13863460 ecr 84640133], length 0
04:51:08.056145 IP (tos 0x0, ttl 64, id 13371, offset 0, flags [DF], proto TCP
(6), length 146)
192.168.48.137.55850 > 192.168.48.145.10514: Flags [P.], cksum 0x5c81
(correct), seq 374307:374401, ack 1, win 1024, options [nop,nop,TS val 84641747
ecr 13863460], length 94
04:51:08.056164 IP (tos 0x0, ttl 64, id 1679, offset 0, flags [DF], proto TCP
(6), length 52)
192.168.48.145.10514 > 192.168.48.137.55850: Flags [.], cksum 0xe291
(incorrect -> 0x61a7), seq 1, ack 374401, win 1452, options [nop,nop,TS val
13865074 ecr 84641747], length 0
Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox
> On Jan 31, 2016, at 7:27 PM, David Lang <[email protected]> wrote:
>
> Ok, does a tcpdump show anything arriving on port 10514?
>
> Earlier, you were getting some logs in and writing them to a file, now it
> looks like you aren't getting any logs in. We need to get that working before
> we can worry about logs not getting out. :-)
>
> David Lang
>
> On Sun, 31 Jan 2016, Brad Cox wrote:
>
>> OK, I rebooted and ran your grep. No sign of anything being processed (
>> /=[1-9] in vim came up empty on the whole file so i omitted most duplicates:
>>
>> grep -e imtcp -e inbount -e omelasticsearch -e send_es pstats
>> ...
>> Sat Jan 30 12:13:49 2016: imtcp(10514): origin=imtcp submitted=0
>> Sat Jan 30 12:13:59 2016: omelasticsearch: origin=omelasticsearch
>> submitted=0 failed.http=0 failed.httprequests=0 failed.es=0
>> Sat Jan 30 12:13:59 2016: send_es: origin=core.action processed=0 failed=0
>> suspended=0 suspended.duration=0 resumed=0
>> Sat Jan 30 12:13:59 2016: imtcp(10514): origin=imtcp submitted=0
>> Sat Jan 30 12:14:08 2016: omelasticsearch: origin=omelasticsearch
>> submitted=0 failed.http=0 failed.httprequests=0 failed.es=0
>> Sat Jan 30 12:14:08 2016: send_es: origin=core.action processed=0 failed=0
>> suspended=0 suspended.duration=0 resumed=0
>> Sat Jan 30 12:14:08 2016: imtcp(10514): origin=imtcp submitted=0 /=
>>
>> Dr. Brad J. Cox Cell: 703-594-1883 Skype: dr.brad.cox
>>
>>
>>
>>
>>> On Jan 30, 2016, at 9:22 PM, David Lang <[email protected]> wrote:
>>>
>>> On Sat, 30 Jan 2016, Brad Cox wrote:
>>>
>>>> Thanks!
>>>>
>>>> Wasn't sure whether you meant to use your config instead of my default
>>>> rsyslog-config or to insert yours at the beginning of my module
>>>> statements. I did the former and patched a few errors to wind up with just
>>>> this:
>>>>
>>>> module(
>>>> load="impstats"
>>>> interval="10"
>>>> resetCounters="off"
>>>> format="legacy"
>>>> file="/var/log/pstats"
>>>> )
>>>>
>>>> # load the modules to listen to the network and configure them to send
>>>> their
>>>> # logs to the 'inbound' ruleset
>>>> module(load="imtcp")
>>>> module(load="imudp" timerequery="4" )
>>>> input(type="imtcp" port="10514" ruleset="inbound" )
>>>> input(type="imudp" port="10514" ruleset="inbound")
>>>>
>>>> # define the inbound ruleset, all logs will go to elasticsearch
>>>> ruleset(
>>>> name="inbound"
>>>> queue.type="fixedarray"
>>>> ){
>>>> action(
>>>> name = "send_es"
>>>> type="omelasticsearch"
>>>> action.resumeretrycount="-1"
>>>> )
>>>>
>>>> What I'm seeing in /var/log/pstat is lots and lots of example at the end
>>>> and still still no sign of traffic from security onion in kibana. Ideas?
>>>>
>>>> BTW: the goal is to keep SO clients and server completely intact and
>>>> unmodified (except for the two lines I showed earlier that copy all sensor
>>>> -> server events to a separate VM. This will grow into a spark/hadoop
>>>> cluster that we'll evolve to do advanced analytics that SO doesn't
>>>> provide. The mixing of syslog-ng and rsyslog is just because SO uses
>>>> Ubuntu and bcoxVM uses Xubuntu. According to the docs they're
>>>> interchangable.
>>>>
>>>> I've seen the recipe link you provided as well as a similar one that
>>>> routes traffic via kafka. I used a different one (link provided earlier)
>>>> because it didn't involve logstash/ruby to avoid unnecessary complexity.
>>>> Once I get this recipe working I'll probably try via kafka.
>>>>
>>>> Sat Jan 30 12:13:09 2016: imtcp(10514): origin=imtcp submitted=0
>>>> Sat Jan 30 12:13:09 2016: imudp(*:10514): origin=imudp submitted=0
>>>> Sat Jan 30 12:13:09 2016: imudp(*:10514): origin=imudp submitted=0
>>>
>>> these tell you how many messages have been submitted via these inputs.
>>>
>>>> Sat Jan 30 12:13:09 2016: resource-usage: origin=impstats utime=4000
>>>> stime=4000 maxrss=9724 minflt=184 majflt=0 inblock=8 oublock=176 nvcsw=124
>>>> nivcsw=4
>>>
>>> ignore this for now, it is how many resources rsyslog has consumed in the
>>> last 10 sec
>>>
>>>> Sat Jan 30 12:13:09 2016: inbound: origin=core.queue size=0 enqueued=0
>>>> full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>>>
>>> how many messages have gone to the inbound ruleset (enqueued=0 means none)
>>>
>>>> Sat Jan 30 12:13:09 2016: main Q: origin=core.queue size=7 enqueued=422
>>>> full=0 discarded.full=0 discarded.nf=0 maxqsize=9
>>>
>>> ignore this for now, it shows how many messages went to the main queue.
>>>
>>>> Sat Jan 30 12:13:09 2016: imudp(w0): origin=imudp called.recvmmsg=0
>>>> called.recvmsg=0 msgs.received=0
>>>
>>> more stats on inbound udp, but you whould be getting things via tcp, so
>>> ignore it
>>>
>>>> Sat Jan 30 12:13:19 2016: omelasticsearch: origin=omelasticsearch
>>>> submitted=0 failed.http=0 failed.httprequests=0 failed.es=0
>>>
>>> stats from the elasticsearch module
>>>
>>>> Sat Jan 30 12:13:19 2016: send_es: origin=core.action processed=0 failed=0
>>>> suspended=0 suspended.duration=0 resumed=0
>>>
>>> stats from the action delivering logs to ES
>>>
>>>> Sat Jan 30 12:14:08 2016: inbound: origin=core.queue size=0 enqueued=0
>>>> full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>>>
>>> so this shows that no logs have been deliverd to rsylog yet. you may need
>>> to kick syslog-ng on the senders to get it to try again.
>>>
>>> do a grep -e imtcp -e inbount -e omelasticsearch -e send_es on the pstats
>>> file, we need the end of it when it shows some data arriving.
>>>
>>> David Lang
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
