I like to put a sample of the log in the config (manually anonimized)
so what I have for this particular log message is currently:
prefix=%timestamp:date-rfc3164% %hostname:word%
# %ASA-6-302014:Teardown TCP connection 42095195 for outside:1.2.3.4/5678 to
inside:192.168.1.1/54321 duration 0:00:30 bytes 0 SYN Timeout
# %ASA-6-302014: Teardown TCP connection 43363071 for
outside:192.168.1.1\/58949(LOCAL\username) to
outside:192.168.2.2\/3283(LOCAL\machineid) duration 0:00:00 bytes 0 TCP Reset-O
(username)
# %ASA-6-302014: Teardown TCP connection 51708532 for outside:10.1.5.5/54853 to
backup:192.168.2.1/4784(LOCAL\machine) duration 0:00:00 bytes 0 <snp_drop_none>
rule=cisco,ASA-6-302014: \x25ASA-6-302014\x3a Teardown %proto:word% connection
%connection-id:number% for
%srciface:char-to:\x3a%\x3a%srcip:ipv4%/%srcport:number%%connection-label:word%
to %dstiface:char-to:\x3a%\x3a%dstip:ipv4%/%dstport:number% duration
%duration:char-to: % bytes %bytes:number% %reason:rest%
rule=cisco,ASA-6-302014: \x25ASA-6-302014\x3a Teardown %proto:word% connection
%connection-id:number% for
%srciface:char-to:\x3a%\x3a%srcip:ipv4%/%srcport:number%%connection-label:word%
to %dstiface:char-to:\x3a%\x3a%dstip:ipv4%/%dstport:number%%machine:word%
duration %duration:char-to: % bytes %bytes:number% %reason:rest%
rule=cisco,ASA-6-302014: \x25ASA-6-302014\x3a Teardown %proto:word% connection
%connection-id:number% for
%srciface:char-to:\x3a%\x3a%srcip:ipv4%/%srcport:number% to
%dstiface:char-to:\x3a%\x3a%dstip:ipv4%/%dstport:number% duration
%duration:char-to: % bytes %bytes:number% %reason:rest%
rule=cisco,ASA-6-302014: \x25ASA-6-302014\x3a Teardown %proto:word% connection
%connection-id:number% for
%srciface:char-to:\x3a%\x3a%srcip:ipv4%/%srcport:number% to
%dstiface:char-to:\x3a%\x3a%dstip:ipv4%/%dstport:number%%machine:word% duration
%duration:char-to: % bytes %bytes:number% %reason:rest%
Using the new (in v8.8 descent capability, I expect to change this to something
like:
file cisco.endpoint
rule:%ip:ipv4%/%tail:rest%
rule:%ip:ipv4%/%port:number%%tail:rest%
rule:%ip:ipv4%/%port:number%(%label1:char-to:)%)%tail:rest%
rule:%ip:ipv4%/%port:number% (%label2:char-to:)%)%tail:rest%
rule:%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%)%tail:rest%
rule:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%%tail:rest%
rule:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%)%tail:rest%
rule:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%
(%label2:char-to:)%)%tail:rest%
rule:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%)
(%label2:char-to:)%)%tail:rest%
Then in the main rule file it would simplify down to:
rule=cisco,ASA-6-302014: \x25ASA-6-302014\x3a Teardown %proto:word% connection
%connection-id:number% for %source:descent:cisco.endpoint% to
%dest:descent:cisco.endpoint% duration %duration:char-to: % bytes
%bytes:number% %reason:rest%
to handle the problem of flags possibly listing multiple flags, I do flags
%flags:tokenized: :word%
I have a script that I run to prioritize what logs I need to write parsers for
(it reads from the most recently rotated file of unknown messages)
#!/bin/bash
#uncomment this when creating a file from scratch, otherwise this script
appaends to the rulebase f
ile
#echo "prefix=%timestamp:date-rfc5424% %hostname:word%" >rsyslog.rulebase
cut -c 17- /var/log/oldlogs/cisco-unknown-messages-*[0-9] |cut -f 2 -d ' ' |sort
|uniq -c |sort -rn |while read count code
do
grep $code /var/log/oldlogs/cisco-unknown-messages-*[0-9] |head -1
done | cut -c 17- cisco.log.examples |while read junk tag log
do
(
echo "# $tag $log"
echo "$tag $tag $log" | sed \
-e s/'\%'/'rule=cisco,'/ \
-e s/'\%ASA'/'\\x25ASA'/g \
-e s/':'/'\\x3a'/g \
-e s/'\\x3a'/':'/ \
-e s/'TCP'/'%proto:word%'/ \
-e s/'UDP'/'%proto:word%'/ \
-e s/'ICMP'/'%proto:word%'/ \
echo
) >>rsyslog.rulebase
done
in rsyslog.conf I have the following (my relay boxes package the messages into
JSON with some extra metadata, so I have to extract it to have 'normal stuff'
understand it. One the parsing settles, this mmnormalize work will be done on
the relays, not the central system)
$template stdmsg,"%timereported% %hostname% %syslogtag% %$!msg%"
set $.stdmsg = exec_template("stdmsg");
action(type="mmnormalize" path="$!extracted" variable="$.stdmsg"
ruleBase="rsyslog.rulebase")
if $!extracted!originalmsg == '' then {
set $!extracted!timestamp = $timestamp;
set $!extracted!hostname = $hostname;
}
$template unknown,"%$!extracted!originalmsg%\n"
# if we failed to parse a cisco message, log what we failed to parse
if $!extracted!originalmsg != '' and $programname startswith '%ASA-' then {
/var/log/cisco-unknown-messages;unknown
}
Improvements planned
1. update to current git rsyslog/liblognorm to be able to use the descent
function
2. create more parsers
3. go back through the rulebase and add additional tags to classify the log.
I've only just started thinking about this, but the classifications that I have
thought of so far are roughly
stats
connection teardowns where we parse it apart for statistical purposes
useless
connection setups where all the info is in the teardown
deny
any denied messages
attack
anything that is either an attack or a seriously misconfigured system
an example of a misconfigured internal system issue "no route to
destination" on an internal firewall that doesn't have a default route means
that a box tried to talk someplace that it was never supposed to, it could be a
missing route on the firewall, a misconfigured app on the box, or an
attacker/malware trying to phone home
I don't yet know a good way to be able to filter on tags in rsyslog, but getting
the info into the JSON is a good start :-)
David Lang
On Fri, 6 Feb 2015, Joe Blow wrote:
Date: Fri, 6 Feb 2015 15:14:13 -0500
From: Joe Blow <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] rulebase question
Sold. I'll do some digging on my end, but i'll be throwing all of my
configs/rules for the working ASA rules on github once i'm done.
Let me know how you make out, i'll be posting back on once i've played with
the lognormalizer binary.
Thanks tons for the prompt reply.
Cheers,
JB
On Fri, Feb 6, 2015 at 3:06 PM, David Lang <[email protected]> wrote:
On Fri, 6 Feb 2015, Joe Blow wrote:
Hey folks,
I'm trying to parse some cisco ASA logs using rulebase rules. Here is the
example rule:
rule=: %begin:char-to:"%"%\x25ASA-%ddd:char-to:-%-%eee:number%: Teardown
TCP connection %cnumber:number% for outside:%ohost:ipv4%/%oport:number%
to
inside:%ihost:ipv4%/%iport:number% duration %duration:word% bytes
%msgbytes:number% TCP %flags:word% (%uname:word%)
Which i'm hoping matches this (broken) rsyslog line:
06 2015 13:12:55: %ASA-6-302014: Teardown TCP connection 2127154348 for
outside:1.1.1.1/64053 to inside:2.2.2.2/443 duration 0:00:00 bytes 3267
TCP
FINs (myusername)
<snip>
Does anyone have any ideas for why none of my rulebase objects seem to
work? The only data I receive in ES is this, which I believe confirms the
leading space and the broken date at the beginning (which i'm happy to
just
strip off). Has anyone seen anything like this before? Throwing rsyslog
into debug mode doesn't give me great debug logging related to the
mmnormalize module (that I can tell).
rsyslog has no way to debug mmnormalize items, but liblognorm has the
tool: /usr/lib/lognorm/lognormalizer that you can use
echo "log line" | /usr/lib/lognorm/lognormalizer -r ruleset -T
will show you some output that will probably be originalmessage and
notparsed, the notparsed is the part of the log message left when the
parser gave up. If that doesn't help you figure out your problem, you can
add -v which will give you a character-by-character breakdown of the
parsing, look for the last thing before it starts backtracking.
In your case, you have a problem that the flags is not always one word,
it's one or more words. I posted what I had to do for this in an e-mail a
day or so ago (mmnormalize thoughts).
Also, to strip off the date and time at the beginning, you need to have a
prefix that will match them or have it as part of your rule, otherwise it
won't match. You can use a template to set a variable and have mmnormalize
parse that variable instead of parsing $msg or $rawmsg.
I've been doing a bunch of work on this in the last few days, I'll see
about posting my config later today.
We really do need to put together parse rulesets for the common log types,
Cisco being probably the most common one that people need to parse.
liblognorm has a spot on it's website for contributed rules, but nobody has
contributed any :-(
I'll see about posting what I have later today and let's see about sharing
the effort for a bit. sound reasonable?
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
