Hi, I have rsyslogd up and running and logging to mysql with loganalyzer as a front end. Very cool.
However, I can't seem to get on-disk queuing working, which would be nice if mysql goes down or a table is locked because I'm purging records during a nightly maintenance cron. I never see any queue files created. I dropped the size of the memory queue to something ridiculously small, but still no queue files even after 20 minutes. I followed the instructions given here: http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along with a couple of other tutorials on rsyslogd and queuing. I'm quite sure I'm just completely missing something obvious. Details below... And thanks to anyone who might be able to point out what I'm doing wrong. -Steve MY CONFIG, please, no flames re the horrendously complex rules ;-) # if you experience problems, check # http://www.rsyslog.com/troubleshoot for assistance # rsyslog v3: load input modules # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # kernel logging (formerly provided by rklogd) $ModLoad ommysql # SC 2010.11.11: configure disk caching in case mysql is unavailable $MainMsgQueueSize 5 $WorkDirectory /var/log/rsyslogq # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure # send snmpd INFO messages to the dustbin if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' then ~ $template vtfw,"insert into vtfw (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if ($fromhost-ip contains '10.128.255') then :ommysql:localhost,Syslog,root,mysqldb44;vtfw & ~ #if ($fromhost-ip contains '10.128.255') then ~ $template vt1hs1_switches,"insert into vt1hs1_switches (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches & ~ #if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then ~ $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if $fromhost-ip contains '10.128.244' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi & ~ #if $fromhost-ip contains '10.128.244' then ~ $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'warning' or $syslogseverity-text == 'warn' or $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or $syslogseverity-text == 'emergency' or $syslogseverity-text == 'panic') then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere & ~ if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'notice') then ~ #*.* :ommysql:localhost,Syslog,root,mysqldb44 *.notice :ommysql:localhost,Syslog,root,mysqldb44 # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* -/var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $ModLoad imtcp.so # load module $InputTCPServerRun 1470 # start up TCP listener at port 514 # UDP Syslog Server: $ModLoad imudp.so # provides UDP syslog reception $UDPServerRun 514 # start a UDP syslog server at standard port 514 SOME DEBUG OUTPUT: (see attached) Steve Chupack | IT Systems Administrator V: 877.327.8422 x 1242 [email protected] | www.dealer.com
# if you experience problems, check # http://www.rsyslog.com/troubleshoot for assistance # rsyslog v3: load input modules # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark # provides --MARK-- message capability $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # kernel logging (formerly provided by rklogd) $ModLoad ommysql # SC 2010.11.11: configure disk caching in case mysql is unavailable $MainMsgQueueSize 5 $WorkDirectory /var/log/rsyslogq # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure # send snmpd INFO messages to the dustbin if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' then ~ $template vtfw,"insert into vtfw (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if ($fromhost-ip contains '10.128.255') then :ommysql:localhost,Syslog,root,mysqldb44;vtfw & ~ #if ($fromhost-ip contains '10.128.255') then ~ $template vt1hs1_switches,"insert into vt1hs1_switches (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches & ~ #if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then ~ $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if $fromhost-ip contains '10.128.244' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi & ~ #if $fromhost-ip contains '10.128.244' then ~ $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'warning' or $syslogseverity-text == 'warn' or $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or $syslogseverity-text == 'emergency' or $syslogseverity-text == 'panic') then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere & ~ if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'notice') then ~ #*.* :ommysql:localhost,Syslog,root,mysqldb44 *.notice :ommysql:localhost,Syslog,root,mysqldb44 # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* -/var/log/cron # Everybody gets emergency messages *.emerg 01-110-en_US.jpg 01-110-en_US.jpg.1 0213-0213dk_48simpletech.pdf 045901ef7f00000101ae9565c390cc1b.jpg 045901ef7f00000101ae9565c390cc1b.jpg.1 045901ef7f00000101ae9565c390cc1b.jpg.10 045901ef7f00000101ae9565c390cc1b.jpg.11 045901ef7f00000101ae9565c390cc1b.jpg.12 045901ef7f00000101ae9565c390cc1b.jpg.2 045901ef7f00000101ae9565c390cc1b.jpg.3 045901ef7f00000101ae9565c390cc1b.jpg.4 045901ef7f00000101ae9565c390cc1b.jpg.5 045901ef7f00000101ae9565c390cc1b.jpg.6 045901ef7f00000101ae9565c390cc1b.jpg.7 045901ef7f00000101ae9565c390cc1b.jpg.8 045901ef7f00000101ae9565c390cc1b.jpg.9 0618-0624st_15sandisk.pdf 090107_buchholz_1000.wmv:Zone.Identifier 1 20070530_gore28.mp3 2007-10-06-14-59-11.016-VirtualBox-4914.log 2007_self_review.doc 2008-06-18-02-42-42.087-VirtualBox-5536.log 20090813.png 2010TaxReturn.PDF 2010TaxReturn.PDF:Zone.Identifier 20_greenfield_loan_commit.jpg 20_greenfield_master_insurance.pdf 2kprobt.exe 50M 622C.IMG a Abadi_MT_Condensed_Light.ttf admins AGEN.zip An_Teallach_panorama.jpg ApplicationSettings.xml audiofile.wav audiorpms.txt Avocent Avocent\10.11.0.3910.dat Avocent\avctVideoGlobal.dat Avocent\avctVideoUser.dat Avocent\vtswitchview2.net.office.vt.dealer.ddc10.dat bash_aliases.bak bashhists Benchmarking Filesystems Part II LG #122.mht beryl.out beryl+xfce bigfile bin Bon Echo_wallpaper.png bookmarks.html bookmarks.html~ boot622 boot622.exe boot622.IMA boot98.exe bucky_love_hate.jpg c: cables-to-go_rebate-CTG-5115.pdf camels_hump_pano_large_cropped_fix_stiches.tif camels_hump_skydome_1.png cape coral ccsm-config-apr-5-08 cd060213.zip cd.zip ch6.tif clonezilla-live-1.2.1-17.iso clonezilla-live-1.2.6-38-amd64.iso compiz.log Conne crapper crystal_project.tar.gz cups-pdf D820_A06.EXE database.sqlite datalen.tar datavault ddcgirls.jpg DDC-IP-Scheme.xls ddclappybackups DDC.ovpn.bak dead.letter default-wide_2100.jpg desktop Desktop dlmgr_.pro DMS-form-2.pdf DMS-Form-2.ps DMS_Form.pdf DMS_Form.ps Documents dos622 dos622.exe Dos6.22.img dos-6.22-img.zip dot_xbindkeys_config.tgz Download drivermax.exe DSS_Trader_LOGO_STACK_4C.jpg dtp-aim-target.png dwhelper ebooks eclipse eclipse.pdt.bak eclipse-SDK-3.3.1.1-linux-gtk.tar.gz egw_bookmarks.html Employment Lead - Dealer.com Website.txt en eqlgroupmgr exit extension-list.xls f1lter Fast Quote - Dealer.com Website.txt Favorites fc6rpms.txt FGEN.zip file: filer switch_s trunking.html filer switch_s trunking.html_files firefox-plugin-oops.txt fit fixresolv fnl0203 FORW Foundry_FastIron_User_Guide.pdf fusion-icon.log g2mdlhlpx.exe glade_revised_preapp.pdf glpi.slk gpodder gpodder-downloads gps greysS04E04 greysS04E05 GTA3 User Files heidi_slips_up.jpg heywardallen_reports.JPG hostsfrombb.txt hp_usb_boot_tool_cp006049.exe huh idea-7.0.3.tar.gz iftop.jpeg image1.jpg index.html index.html.1 index.html.10 index.html.11 index.html.12 index.html.13 index.html.14 index.html.15 index.html.16 index.html.17 index.html.18 index.html.19 index.html.2 index.html.20 index.html.21 index.html.22 index.html.23 index.html.24 index.html.3 index.html.4 index.html.5 index.html.6 index.html.7 index.html.8 index.html.9 install_flash_player_7_linux install_flash_player_7_linux.tar.gz IT_Interrogatories__SC_TC_042511_.doc jac.log Jarrel_Perschuk_08-09_review.doc Jarrel_Perschuk_08-09_self-review.doc julie.tmp jungledisk jungledisk_readonly4rickandpete jungledisk-readonly4rickandpete jungledisk_readwrite4all jungledisk-readwrite4all jungle_readonly4rickandpete jungle_readwrite4all katadin_it_staff katadin_public keepalivedUserGuide.pdf KeyboardMacros.xml kubuntu-10.10-desktop-i386.iso life_calendar.txt linux_journal_sysadmin_edition.pdf liquid-weather-14.8.skz lisezmoi.txt logoglogs lotus Mail Maps mc.rez.doc me MHT migs.txt minicom.log mnt MonumentValley01_jpg.wpa mounts.sh mozilla.pdf mulberry Music My Documents MySQL-client-5.0.27-0.glibc23.i386.rpm nautilus-debug-log.txt ncresume.pdf Network Advisor newbuilding-floorplan.jpg NewFolder nfshome nmapout.gnmap nmapout.nmap nmapout.xml nokia-backups Nokia N810 Tablet Wimax Disassembly - Dengel Share.pdf Nokia N810 Tablet Wimax Disassembly - Dengel Share.rar npatgpc.dl_ nvidia-settings-rc.bak ocs_teton.csv ocs_vt2k3.csv ocs_vt2k.csv ocs_vt.csv ocs_vt_katadin.csv ocs_vt+wc_servers.xls ocs_vt_xen.csv ocs_wc.csv ocs_wc.xls ocs_xen_vt.csv openfiredoc.html openofficelangpack.rpms OpenVPN-DDC-ConfigKeys.tar ophcrack Ophcrack_-_Collection_of_5_Tables.4559202.TPB.torrent Ophcrack Tables Orchard_Commons_Purchase___Sale_Contract_markedup.doc Orchard_Commons_Purchase___Sale_Contract_markedup_w_roger_round1.doc org.zend.php.debug_feature-I20071210.zip osx_jas_10.4.8 out out.1 out.2 overview.htm -p80.gnmap -p80.nmap -p80.xml pci_filerecovery.exe pdnsd-1.2.4-par_sl4.i386.rpm pdt-all-in-one-S20071213_M1-linux-gtk.tar.gz permtest personal -p.gnmap Pictures pingout.txt plots -p.nmap poop.tgz pto_request_feb_2009.doc pto_request_july13+15_2007.doc pto_request_sept_21_2007.doc public Public -p.xml Quick Quote - Dealer.com Website.txt rac4vm.xpi rainbowtables readme.txt redirect.htm?url=%2Fabout.htm redirect.htm?url=%2Fabout.htm.1 redirect.htm?url=%2Fabout.htm.2 redirect.htm?url=%2Fabout.htm.3 redirect.htm?url=%2Fabout.htm.4 redirect.htm?url=%2Fabout.htm.5 redirect.htm?url=%2Fabout.htm.6 redirect.htm?url=%2Fabout.htm.7 renee_corkboard.jpg Request More Info - Dealer.com Website.txt resetlog.txt rootkit.txt root.pem rstudio_rsd_en_4.exe rsynctest [email protected] Saillon_jpg.wpa sales_laptop_checkup.ods sanibel-hammond.gpx screenlog.0 Screenshot.png Screenshot-VNC: WEBPOSITION.png scripts sessions shame.txt shite.mp3 sig_spotlight.jpg skydome005-arbol-360.png skydome_An_Teallach_panorama_orig.png skydome_An_Teallach_panorama.png skydome_camels_hump.png skydome_kelso_1.png skydome_limontaur.png skydome_MonumentValley01.jpg skydome_MonumentValley01.png skydome.png skydome_Saillon_corrected.png skydome_Saillon.jpg skydome_Saillon.png skydome_serratusLarge.jpg skydome_wphr_monumentvalley01.zip skydome_wphr_saillon.zip smbhome software SP27608.exe speedtest SpinRite.exe sshfs sshkeys Start Menu stchupack stchupack.spamassassin.tgz stc_new stopfw_startvnc subdivision_ammend_report_042007.doc SW_CD_Visio_Pro_2007_W32_English_1_PA_BP_MLF_X12-19212.EXE tech-support Templates test test.com.pdf Test Drive - Dealer.com Website.txt testform.pdf testguide.doc testjabber.sh test.pdf testsmtp.sh test.test thinkingrock_data tmp traceroute.out track-01.inf track-01.wav track-01.xinf transset-df-5 transset-df-5.tar.gz tsclient ucky_glowing_eyes_1.jpg ucky_glowing_eyes.jpg untitled folder 2 untitled.ncd upgrade courier.html upgrade courier.html_files userdic.tlx V V_(2009)_-_Season_1_Complete_(S01)_--_S01E01_to_S01E12--ADreaNaL.5591118.TPB.torrent Videos viewpageattachments.action?pageId=14746263 virtual-drives vmscsi-1.2.0.4.flp vmware vtcas1.rdp vtfiler_admins vtfiler_public vtfiler_teams-nfs vtlvs1.etc vtlvs2.etc wallpaper_horsehead_1920x1200.jpg wallpaper_horsehead_orig.jpg wave_license.txt webposition_hog.png wgt624_4_2_11_1_0_1.chk win2k_boot_floppy win2k_boot_floppy.exe win98boot win98boot.zip workspace wtf xbindkeys_config-0.1.3 xbindkeys-config_0.1.3.orig.tar.gz xcompmgr-1.1.1 xcompmgr-1.1.1.tar.gz xfce-menu-edit-0.8 xfce-menu-edit-0.8.tar XGEN.zip xorg.conf.dualview_auto xorg.conf.nview xorg.conf.twinview yum.grouplist yumsearchbackground.txt ZGEN.zip Zimbra # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $ModLoad imtcp.so # load module $InputTCPServerRun 1470 # start up TCP listener at port 514 # UDP Syslog Server: $ModLoad imudp.so # provides UDP syslog reception $UDPServerRun 514 # start a UDP syslog server at standard port 514 EOF
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
