Re: Parsing Syslog and Acting on it, using other input too

[shawn wilson]

Ah it seems they do:
https://github.com/fail2ban/fail2ban/blob/master/config/action.d/iptables-ipset-proto6.conf

IDK enough about fail2ban to know whether I can assign a per proto or per
log type config (I assume I can). In which casethis does what my script
does and then some. I would probably dump out a ipset save on exit and try
to 'restore' on resume (which /I/ do) and I'm sure there's a way fail2ban
can check a store of addresses and check what network a host belongs to
(instead of just a host).

So, fail2ban is probably the way to go.


On Fri, Aug 30, 2013 at 10:00 AM, Christopher Morrow <
morrowc.li...@gmail.com> wrote:

> On Fri, Aug 30, 2013 at 8:55 AM, Shawn Wilson  wrote:
> >
> >
> > Christopher Morrow  wrote:
> >>On Thu, Aug 29, 2013 at 10:50 AM, Don Wilder 
> >>wrote:
> >>> I wrote a script in Linux that watches for unauthorized login
> >>attempts and
> >>> adds the ip address to the blocked list in my firewall. You might
> >>want to
> >>> search sourceforge for a DYN Firewall and modify it from there.
> >>>
> >>
> >>because fail2ban was too hard to install? or because you just wanted
> >>to test yourself?
> >
> > Actually I did the same. I use ipset lists (generally with a timeout)
> and take a regex or two and black / white list from a YAML file and just
> take (possibly multiple inputs) from piping tail -F. I also store addresses
> for future reference (by the script or otherwise).
> >
> > This is quite maintainable as I can look at a list of people who have
> attacked the mail server and compare it to web attacks. Each process is a
> different type of service (different config file) and probably a different
> ipset. Due to ipset not actually doing anything until I make an iptables
> rule for it, I can run my script in a test mode (by default) and just see
> what happens (check it's logs and the ipset list it generates). I haven't
> found the need for this yet but I can use cymru to look up how big their
> net is (see geocidr for an example of how to do this in perl) and use a
> hash:net ipset type and cover a whole net.
> >
> > Basically what I'm saying in doing it this way is quite expandable and
> isn't very hard and I can do tons of stuff that fail2ban can't (I don't
> think - it's been a while since I looked).
>
> you seem to be describing what fail2ban does... that and some grep of
> syslog for fail2ban messages. If your solution works then great! :)
>

Re: Parsing Syslog and Acting on it, using other input too

[Christopher Morrow]

On Fri, Aug 30, 2013 at 8:55 AM, Shawn Wilson  wrote:
>
>
> Christopher Morrow  wrote:
>>On Thu, Aug 29, 2013 at 10:50 AM, Don Wilder 
>>wrote:
>>> I wrote a script in Linux that watches for unauthorized login
>>attempts and
>>> adds the ip address to the blocked list in my firewall. You might
>>want to
>>> search sourceforge for a DYN Firewall and modify it from there.
>>>
>>
>>because fail2ban was too hard to install? or because you just wanted
>>to test yourself?
>
> Actually I did the same. I use ipset lists (generally with a timeout) and 
> take a regex or two and black / white list from a YAML file and just take 
> (possibly multiple inputs) from piping tail -F. I also store addresses for 
> future reference (by the script or otherwise).
>
> This is quite maintainable as I can look at a list of people who have 
> attacked the mail server and compare it to web attacks. Each process is a 
> different type of service (different config file) and probably a different 
> ipset. Due to ipset not actually doing anything until I make an iptables rule 
> for it, I can run my script in a test mode (by default) and just see what 
> happens (check it's logs and the ipset list it generates). I haven't found 
> the need for this yet but I can use cymru to look up how big their net is 
> (see geocidr for an example of how to do this in perl) and use a hash:net 
> ipset type and cover a whole net.
>
> Basically what I'm saying in doing it this way is quite expandable and isn't 
> very hard and I can do tons of stuff that fail2ban can't (I don't think - 
> it's been a while since I looked).

you seem to be describing what fail2ban does... that and some grep of
syslog for fail2ban messages. If your solution works then great! :)

Re: Parsing Syslog and Acting on it, using other input too

[Shawn Wilson]

Christopher Morrow  wrote:
>On Thu, Aug 29, 2013 at 10:50 AM, Don Wilder 
>wrote:
>> I wrote a script in Linux that watches for unauthorized login
>attempts and
>> adds the ip address to the blocked list in my firewall. You might
>want to
>> search sourceforge for a DYN Firewall and modify it from there.
>>
>
>because fail2ban was too hard to install? or because you just wanted
>to test yourself?

Actually I did the same. I use ipset lists (generally with a timeout) and take 
a regex or two and black / white list from a YAML file and just take (possibly 
multiple inputs) from piping tail -F. I also store addresses for future 
reference (by the script or otherwise). 

This is quite maintainable as I can look at a list of people who have attacked 
the mail server and compare it to web attacks. Each process is a different type 
of service (different config file) and probably a different ipset. Due to ipset 
not actually doing anything until I make an iptables rule for it, I can run my 
script in a test mode (by default) and just see what happens (check it's logs 
and the ipset list it generates). I haven't found the need for this yet but I 
can use cymru to look up how big their net is (see geocidr for an example of 
how to do this in perl) and use a hash:net ipset type and cover a whole net.

Basically what I'm saying in doing it this way is quite expandable and isn't 
very hard and I can do tons of stuff that fail2ban can't (I don't think - it's 
been a while since I looked). 

Re: Parsing Syslog and Acting on it, using other input too

[Christopher Morrow]

On Thu, Aug 29, 2013 at 10:50 AM, Don Wilder  wrote:
> I wrote a script in Linux that watches for unauthorized login attempts and
> adds the ip address to the blocked list in my firewall. You might want to
> search sourceforge for a DYN Firewall and modify it from there.
>

because fail2ban was too hard to install? or because you just wanted
to test yourself?

Re: Parsing Syslog and Acting on it, using other input too

[chip]

http://www.elasticsearch.com/blog/welcome-jordan-logstash/

So now Logstash and Elasticsearch will be even more integrated than before.
 With Kibana on top of that, this seems like the ultimate log data "do
stuff" stack.

--chip


On Thu, Aug 29, 2013 at 2:03 PM, Carlos Alcantar  wrote:

> +1 on Splunk or if you don't mind using a SAS service check out
> https://papertrailapp.com/
>
> Carlos Alcantar
> Race Communications / Race Team Member
> 1325 Howard Ave. #604, Burlingame, CA. 94010
> Phone: +1 415 376 3314 / car...@race.com / http://www.race.com
>
>
>
>
>
> -Original Message-
> From: Kasper Adel 
> Date: Thursday, August 29, 2013 6:03 AM
> To: "nanog@nanog.org" 
> Subject: Parsing Syslog and Acting on it, using other input too
>
> Hello.
>
> I am looking for a way to do proactive monitoring of my network, what I am
> specifically thinking about is receiving syslog msgs from the routers and
> the backend engine would correlate certain msgs with output/data that i am
> receiving through SSH/telnet sessions. What i am after is not exposed to
> SNMP so i need to do it on my own.
>
>
> I am sure there are many tools that can do parsing of syslog and acting
> upon it but i wonder if there is something more flexible out there that I
> can just re-use to do the above ? Please point me to known public or
> home-grown scripts in use to achieve this.
>
> Regards,
>
> Sam
>
>
>
>


-- 
Just my $.02, your mileage may vary,  batteries not included, etc

Re: Parsing Syslog and Acting on it, using other input too

[Carlos Alcantar]

+1 on Splunk or if you don't mind using a SAS service check out
https://papertrailapp.com/

Carlos Alcantar
Race Communications / Race Team Member
1325 Howard Ave. #604, Burlingame, CA. 94010
Phone: +1 415 376 3314 / car...@race.com / http://www.race.com





-Original Message-
From: Kasper Adel 
Date: Thursday, August 29, 2013 6:03 AM
To: "nanog@nanog.org" 
Subject: Parsing Syslog and Acting on it, using other input too

Hello.

I am looking for a way to do proactive monitoring of my network, what I am
specifically thinking about is receiving syslog msgs from the routers and
the backend engine would correlate certain msgs with output/data that i am
receiving through SSH/telnet sessions. What i am after is not exposed to
SNMP so i need to do it on my own.


I am sure there are many tools that can do parsing of syslog and acting
upon it but i wonder if there is something more flexible out there that I
can just re-use to do the above ? Please point me to known public or
home-grown scripts in use to achieve this.

Regards,

Sam

Re: Parsing Syslog and Acting on it, using other input too

[Charles N Wyble]

Yes. Logstash shipper on your syslog proxy, forward to elasticsearch. Graylog2 
is very cool. Tried kibana and didn't care for it.

Actually setting up graylog2 right now to do AD authentication.  

So workflow is

End device -> syslog-ng vm -> graylog2/elasticsearch vm and other destinations 
(it corp security cloud for stuff they want to track, observium for anything 
matching my network gear hostname pattern, etc).

I have the middle syslog-ng box so I can have great control over where certain 
hosts ultimately send data. However that system can be used in any template, if 
I don't filter it just gets dumped to graylog.

Kevin Stone  wrote:
>Look at Logstash, http://logstash.net.
>
>Rsyslog can do a bit, on Windows you could look at the Solarwinds Kiwi
>syslog server.
>
>
>On Thu, Aug 29, 2013 at 9:10 AM, Jason Biel 
>wrote:
>
>> You should look into SPLUNK (http://www.splunk.com/), it will
>> collect/store
>> your syslog data and you can run customized reports and then act on
>them.
>>
>>
>> On Thu, Aug 29, 2013 at 8:03 AM, Kasper Adel 
>wrote:
>>
>> > Hello.
>> >
>> > I am looking for a way to do proactive monitoring of my network,
>what I
>> am
>> > specifically thinking about is receiving syslog msgs from the
>routers and
>> > the backend engine would correlate certain msgs with output/data
>that i
>> am
>> > receiving through SSH/telnet sessions. What i am after is not
>exposed to
>> > SNMP so i need to do it on my own.
>> >
>> >
>> > I am sure there are many tools that can do parsing of syslog and
>acting
>> > upon it but i wonder if there is something more flexible out there
>that I
>> > can just re-use to do the above ? Please point me to known public
>or
>> > home-grown scripts in use to achieve this.
>> >
>> > Regards,
>> >
>> > Sam
>> >
>>
>>
>>
>> --
>> Jason
>>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Parsing Syslog and Acting on it, using other input too

[Gino O'Donnell]

Check out Sagan: http://sagan.quadrantsec.com/

On 8/29/13 6:03 AM, Kasper Adel wrote:
> Hello.
> 
> I am looking for a way to do proactive monitoring of my network, what I am
> specifically thinking about is receiving syslog msgs from the routers and
> the backend engine would correlate certain msgs with output/data that i am
> receiving through SSH/telnet sessions. What i am after is not exposed to
> SNMP so i need to do it on my own.
> 
> 
> I am sure there are many tools that can do parsing of syslog and acting
> upon it but i wonder if there is something more flexible out there that I
> can just re-use to do the above ? Please point me to known public or
> home-grown scripts in use to achieve this.
> 
> Regards,
> 
> Sam
> 

Re: Parsing Syslog and Acting on it, using other input too

[Don Wilder]

I wrote a script in Linux that watches for unauthorized login attempts and
adds the ip address to the blocked list in my firewall. You might want to
search sourceforge for a DYN Firewall and modify it from there.


On Thu, Aug 29, 2013 at 10:44 AM, Mike Tancsa  wrote:

> On 8/29/2013 9:03 AM, Kasper Adel wrote:
> > Hello.
> >
> > I am looking for a way to do proactive monitoring of my network, what I
> am
> > specifically thinking about is receiving syslog msgs from the routers and
>
> You might want to look at
>
> http://www.ossec.net/
>
> ---Mike
>
>
>
>
> --
> ---
> Mike Tancsa, tel +1 519 651 3400
> Sentex Communications, m...@sentex.net
> Providing Internet services since 1994 www.sentex.net
> Cambridge, Ontario Canada   http://www.tancsa.com/
>
>


-- 
-
Don Wilder
-

Programming today is a race between software engineers striving to build
bigger and better idiot-proof programs, and the Universe trying to produce
bigger and better idiots. So far, the Universe is winning.

Re: Parsing Syslog and Acting on it, using other input too

[Mike Tancsa]

On 8/29/2013 9:03 AM, Kasper Adel wrote:
> Hello.
> 
> I am looking for a way to do proactive monitoring of my network, what I am
> specifically thinking about is receiving syslog msgs from the routers and

You might want to look at

http://www.ossec.net/

---Mike




-- 
---
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, m...@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/

Re: Parsing Syslog and Acting on it, using other input too

[Blake Dunlap]

Since you said you are willing to entertain home grown as well. I would
recommend looking at simple event correlator which is a perl script
designed to do the kind of thing you are talking about. I've used it in the
past to trigger bgp black holing and mail blacklists for example.


On Thu, Aug 29, 2013 at 8:25 AM, Sam Moats  wrote:

> My view on splunk,
> +1 if you intend to have a human act on the reports, it does an excellent
> job of reducing huge amounts of audit data into the valuable bits.
> -1 Seemed to be a pita to integrate with my scripting enviroment. I ended
> up kludging wget,awk and telnet together in a totally undignified way to
> make it reach out and act on something.
>
> +2 Customizable ingestion/parsing, I'm feeding everything from linux audit
> data to weird proprietary serial output from a multiplexer into it.
> -1 Proprietary database I would have liked to see an sql plugin for data
> storage, I would like the data in Mysql/Oracle but no-joy from splunk so
> that I can use other tools on it easily.
>
> +1 Free demo. You can download an eval version that is rate limited and
> cripples itself after a fixed time.
> -1 because The license costs are a bit high if your moving lots of data
> through it
>
>
> Sam Moats
>
> On 2013-08-29 09:10, Jason Biel wrote:
>
>> You should look into SPLUNK (http://www.splunk.com/), it will
>> collect/store
>> your syslog data and you can run customized reports and then act on them.
>>
>>
>> On Thu, Aug 29, 2013 at 8:03 AM, Kasper Adel 
>> wrote:
>>
>>  Hello.
>>>
>>> I am looking for a way to do proactive monitoring of my network, what I
>>> am
>>> specifically thinking about is receiving syslog msgs from the routers and
>>> the backend engine would correlate certain msgs with output/data that i
>>> am
>>> receiving through SSH/telnet sessions. What i am after is not exposed to
>>> SNMP so i need to do it on my own.
>>>
>>>
>>> I am sure there are many tools that can do parsing of syslog and acting
>>> upon it but i wonder if there is something more flexible out there that I
>>> can just re-use to do the above ? Please point me to known public or
>>> home-grown scripts in use to achieve this.
>>>
>>> Regards,
>>>
>>> Sam
>>>
>>>
>
>

Re: Parsing Syslog and Acting on it, using other input too

[Sam Moats]

My view on splunk,
+1 if you intend to have a human act on the reports, it does an 
excellent job of reducing huge amounts of audit data into the valuable 
bits.
-1 Seemed to be a pita to integrate with my scripting enviroment. I 
ended up kludging wget,awk and telnet together in a totally undignified 
way to make it reach out and act on something.


+2 Customizable ingestion/parsing, I'm feeding everything from linux 
audit data to weird proprietary serial output from a multiplexer into 
it.
-1 Proprietary database I would have liked to see an sql plugin for 
data storage, I would like the data in Mysql/Oracle but no-joy from 
splunk so that I can use other tools on it easily.


+1 Free demo. You can download an eval version that is rate limited and 
cripples itself after a fixed time.
-1 because The license costs are a bit high if your moving lots of data 
through it



Sam Moats
On 2013-08-29 09:10, Jason Biel wrote:
You should look into SPLUNK (http://www.splunk.com/), it will 
collect/store
your syslog data and you can run customized reports and then act on 
them.



On Thu, Aug 29, 2013 at 8:03 AM, Kasper Adel  
wrote:



Hello.

I am looking for a way to do proactive monitoring of my network, 
what I am
specifically thinking about is receiving syslog msgs from the 
routers and
the backend engine would correlate certain msgs with output/data 
that i am
receiving through SSH/telnet sessions. What i am after is not 
exposed to

SNMP so i need to do it on my own.


I am sure there are many tools that can do parsing of syslog and 
acting
upon it but i wonder if there is something more flexible out there 
that I

can just re-use to do the above ? Please point me to known public or
home-grown scripts in use to achieve this.

Regards,

Sam

RE: Parsing Syslog and Acting on it, using other input too

[Thijs Stuurman]

For some straightforward things I have used Logdog 
(http://caspian.dotconf.net/menu/Software/LogDog/).


With kind regards,

Thijs Stuurman

> -Original Message-
> From: Kasper Adel [mailto:karim.a...@gmail.com]
> Sent: donderdag 29 augustus 2013 15:03
> To: NANOG list
> Subject: Parsing Syslog and Acting on it, using other input too
> 
> Hello.
> 
> I am looking for a way to do proactive monitoring of my network, what I am
> specifically thinking about is receiving syslog msgs from the routers and the
> backend engine would correlate certain msgs with output/data that i am
> receiving through SSH/telnet sessions. What i am after is not exposed to
> SNMP so i need to do it on my own.
> 
> 
> I am sure there are many tools that can do parsing of syslog and acting upon
> it but i wonder if there is something more flexible out there that I can just 
> re-
> use to do the above ? Please point me to known public or home-grown
> scripts in use to achieve this.
> 
> Regards,
> 
> Sam

Re: Parsing Syslog and Acting on it, using other input too

[Kevin Stone]

Look at Logstash, http://logstash.net.

Rsyslog can do a bit, on Windows you could look at the Solarwinds Kiwi
syslog server.


On Thu, Aug 29, 2013 at 9:10 AM, Jason Biel  wrote:

> You should look into SPLUNK (http://www.splunk.com/), it will
> collect/store
> your syslog data and you can run customized reports and then act on them.
>
>
> On Thu, Aug 29, 2013 at 8:03 AM, Kasper Adel  wrote:
>
> > Hello.
> >
> > I am looking for a way to do proactive monitoring of my network, what I
> am
> > specifically thinking about is receiving syslog msgs from the routers and
> > the backend engine would correlate certain msgs with output/data that i
> am
> > receiving through SSH/telnet sessions. What i am after is not exposed to
> > SNMP so i need to do it on my own.
> >
> >
> > I am sure there are many tools that can do parsing of syslog and acting
> > upon it but i wonder if there is something more flexible out there that I
> > can just re-use to do the above ? Please point me to known public or
> > home-grown scripts in use to achieve this.
> >
> > Regards,
> >
> > Sam
> >
>
>
>
> --
> Jason
>

Re: Parsing Syslog and Acting on it, using other input too

[Dobbins, Roland]

On Aug 29, 2013, at 8:03 PM, Kasper Adel wrote:

> I am sure there are many tools that can do parsing of syslog and acting upon 
> it but i wonder if there is something more flexible out there that I can just 
> re-use to do the above ?





If network traffic is of interest, don't forget about flow telemetry like 
NetFlow and/or IPFIX.

---
Roland Dobbins  // 

  Luck is the residue of opportunity and design.

   -- John Milton

Re: Parsing Syslog and Acting on it, using other input too

[Jason Biel]

You should look into SPLUNK (http://www.splunk.com/), it will collect/store
your syslog data and you can run customized reports and then act on them.


On Thu, Aug 29, 2013 at 8:03 AM, Kasper Adel  wrote:

> Hello.
>
> I am looking for a way to do proactive monitoring of my network, what I am
> specifically thinking about is receiving syslog msgs from the routers and
> the backend engine would correlate certain msgs with output/data that i am
> receiving through SSH/telnet sessions. What i am after is not exposed to
> SNMP so i need to do it on my own.
>
>
> I am sure there are many tools that can do parsing of syslog and acting
> upon it but i wonder if there is something more flexible out there that I
> can just re-use to do the above ? Please point me to known public or
> home-grown scripts in use to achieve this.
>
> Regards,
>
> Sam
>



-- 
Jason

Parsing Syslog and Acting on it, using other input too

[Kasper Adel]

Hello.

I am looking for a way to do proactive monitoring of my network, what I am
specifically thinking about is receiving syslog msgs from the routers and
the backend engine would correlate certain msgs with output/data that i am
receiving through SSH/telnet sessions. What i am after is not exposed to
SNMP so i need to do it on my own.


I am sure there are many tools that can do parsing of syslog and acting
upon it but i wonder if there is something more flexible out there that I
can just re-use to do the above ? Please point me to known public or
home-grown scripts in use to achieve this.

Regards,

Sam