Greetings,
I'm trying to implement multiple rulesets with per-ruleset queues in
order to handle messages going to MySQL. Rsyslog is doing template/regex
based parsing for the MySQL inserts - it's not raw log messages, or the
Adiscon format. I seem to be running into some errors.
I tried creating a disk-assisted queue, stopping mysql for a minute or
so, grabbing some relatively unique strings from the text files
containing the raw log data, starting MySQL back up, and then searching
for the strings - which I presumed would eventually be flushed from the
queue to the database.
Not only did I never see any log data collected when MySQL is down end
up in the database, but after about 3 minutes with MySQL down, not only
did I not get any disk queue files, but I saw slightly garbled versions
of the log messages ending up in /var/log/messages - which isn't even
mentioned anywhere in the rulesets!
I'm binding imtcp and imudp to a ruleset ("remote") that first logs
everything to dynfiles based on hostname, and then (for a few specific
source IPs) passes messages on to omruleset for further processing.
Aside from some repetitive rules for string matching, my ruleset looks like:
$RuleSet DHCP-parsing
# create a Disk-Assisted LinkedList main queue specific to this ruleset
$WorkDirectory /var/rsyslog/work
$RulesetCreateMainQueue on
$MainMsgQueueFilename DHCPqueue
$MainMsgQueueType LinkedList
$MainMsgQueueMaxDiskSpace 500M
$MainMsgQueueSaveOnShutdown on # persist queue contents to disk on shutdown
# TESTING ONLY
*.* /var/log/dhcp-parsing
# END TESTING ONLY
$ActionResumeRetryCount -1 # retry infinitely
:msg, startswith, " DHCPREQUEST for"
:ommysql:localhost,test,syslogger,syslogger;DHCPREQUESTMAC
& :ommysql:localhost,test,syslog,foo;DHCPREQUESTIP
& ~ ### DISCARD
:msg, startswith, " DHCPACK on"
:ommysql:localhost,test,syslogger,syslogger;DHCPACKonIP
& :ommysql:localhost,test,syslog,foo;DHCPACKonMAC
& ~ ### DISCARD
:msg, startswith, " DHCPINFORM from"
:ommysql:localhost,test,syslog,foo;DHCPINFORM
& ~ ### DISCARD
if $msg startswith ' DHCPACK to' and ( not ( $msg contains 'no client
hardware address' ) ) \
then :ommysql:localhost,test,syslog,foo;DHCPACKtoMAC
& :ommysql:localhost,test,syslog,foo;DHCPACKtoIP
& ~ ### DISCARD
The garbage that I was seeing in /var/log/messages looked like (as you
can see, right after the timestamp, it's obviously not properly
formatted, though prior to killing mysql, it looked fine in the correct
log files):
Dec 22 15:20:41 : dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
(Courtney-PC) via eth0
Dec 22 15:20:41 dhcpd: DHCPREQUEST for 123.456.78.123 from
01:23:45:67:89:ab (Courtney-PC) via 123.456.78.123
Dec 22 15:20:41 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
(Courtney-PC) via 123.456.78.123
Dec 22 15:20:41 dhcpd: DHCPOFFER on 123.456.78.123 to
01:23:45:67:89:ab (LeVoyageur) via 123.456.78.123
Dec 22 15:20:41 dhcpd: DHCPREQUEST for 123.456.78.123 (123.456.78.123)
from 01:23:45:67:89:ab (LeVoyageur) via 123.456.78.123
Dec 22 15:20:41 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
(LeVoyageur) via 123.456.78.123
Dec 22 15:20:41 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via
123.456.78.123
Dec 22 15:20:41 dhcpd: DHCPREQUEST for 123.456.78.123 from
01:23:45:67:89:ab (F52F2867C1364CC) via eth0
Dec 22 15:20:41 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
(F52F2867C1364CC) via eth0
Dec 22 15:20:42 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via
123.456.78.123
Dec 22 15:20:42 l dhcpd: DHCPOFFER on 123.456.78.123 to
01:23:45:67:89:ab via 123.456.78.123
Dec 22 15:20:42 dhcpd: DHCPREQUEST for 123.456.78.123 from
01:23:45:67:89:ab via eth0
Dec 22 15:20:42 l dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
via eth0
Dec 22 15:20:42 dhcpd: DHCPREQUEST for 123.456.78.123 from
01:23:45:67:89:ab via 123.456.78.123
Dec 22 15:20:42 ast dhcpd: DHCPACK on 123.456.78.123 to
01:23:45:67:89:ab via 172.31.16
Dec 22 15:20:47 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
via 123.456.78.123
Dec 22 15:20:47 l dhcpd: DHCPREQUEST for 123.456.78.123 from
01:23:45:67:89:ab (Blue-PC) via eth0
Dec 22 15:20:47 eER from 01:23:45:67:89:ab via 172.dhcpd: dhcpd: DHCPACK
on 123.456.78.123 to 01:23:45:67:89:ab (Blue-PC) via eth0
Dec 22 15:20:47 l.25.114 dhcpd: DHCPREQUEST for 123.456.78.123 from
01:23:45:67:89:ab (Blue-PC) via 123.456.78.123
Dec 22 15:20:47 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
(Blue-PC) via 123.456.78.123
Any ideas? I chose rsyslog in the hopes that I'd be able to replace our
current cron-triggered log parsing scripts with rsyslog builtin
templates and regex functions, but it seems like I'm running into more
and more problems trying to develop complex rules that can also handle
decent message volume and things like restarts of mysql.
Thanks for any advice,
Jason
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
