Hello all, I'm very new to rsyslog, so please bear with me...
I have managed to get something going, below is what I wanted. 1. High throughput (> 200K). Using multiple queues. 2. Memory-type disk-assisted multiple queues (multiple action queues?). How does $ActionQueue* fit in with multiple rulesets? Is it possible with my existing setup? I would like to spool to disk if any of the rulesets cannot forward by TCP for whatever reason. This reliability is quite important. I see there is $RulesetCreateMainQueue available since 5.3.5+ but I cannot use this for now as I cannot sacrifice stability on the RHEL/CentOS platform and 5.6.x packages won't be available for quite some time (CentOS 6 is only on rsyslog-4.4.2-3.el6.i686.rpm!). 3. Reliable forwarding using RELP. Is there something like $InputRELPServerRun? How, if possible can we do RELP with each ruleset? I tried using $InputRELPServerRun instead of $InputTCPServerRun after $RuleSet but it broke the ruleset match. Can we do something fancy like bind a TCP port to a ruleset and start a RELP listener and forward from the RELP port to the plain TCP port (just guessing here)? Possibly some trickery with Unix domain sockets? OpenSSH local port forwarding? 4. Single/multiple files (option to switch on/off to troubleshoot). In the long run, what seems to better? We have had a debate over this but maybe users can offer real world experience please? 5. Compression. Compressed RELP seemed to work in the forwarding part (no errors in rsyslog restart) but is this a desired config? Should we be doing compression elsewhere if possible like in an ASIC in our Juniper firewall to save precious bandwidth over our WAN link? Has anyone tried OpenSSH compression and local port forwarding? 6. Expression and property-based filters. Which is faster/preferred? 7. Store, filter and forward. What is better in terms of OS throughput, single or multiple files? We are looking at very high throughput like > 200K. If I have ten ruleset inputs all writing to the same file what happens? I'm interested in how concurrent writes are implemented. 8. Versions: Collector: CentOS-5.5 x86_64 on an IBM x3550 M2. kernel 2.6.18-194.32.1.el5 rsyslog4-relp-4.6.5-2.ius.el5.x86_64.rpm. rsyslog4-4.6.5-2.ius.el5.x86_64.rpm librelp-0.1.1-3.el5.kb.x86_64.rpm Relay: CentOS-5.5 x86_64 on an IBM BladeCenter HS22 kernel 2.6.18-194.32.1.el5 rsyslog4-relp-4.6.5-2.ius.el5.x86_64.rpm. rsyslog4-4.6.5-2.ius.el5.x86_64.rpm librelp-0.1.1-3.el5.kb.x86_64.rpm 9. Config: Client collector: For now, I cannot change anything on the client side due to contractual agreements hence the filter match and then forward which will become a ruleset with each source sending to it's own TCP port. At the moment everything is coming in on TCP 514. <open> $ModLoad imklog $ModLoad imuxsock $ModLoad omrelp $ModLoad imudp $UDPServerRun 514 $UDPServerTimeRequery 10 $ModLoad imtcp $InputTCPMaxSessions 500 $InputTCPServerRun 514 $MainMsgQueueSize 100000 $WorkDirectory /media/data/var/spool/ryslog $ActionQueueType LinkedList $ActionQueueFileName gdf $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on if $msg contains 'SymantecServer' then @@(z9)172.nnn.nnn.nnn:4515;RSYSLOG_ForwardFormat if $msg contains 'D75FADC001' then @@(z9)172.nnn.nnn.nnn:4516;RSYSLOG_ForwardFormat if $msg contains 'fw.log' then @@(z9)172.nnn.nnn.nnn:4517;RSYSLOG_ForwardFormat # The below I've got wrong somewhere. Corrections please? #if $pri-text isequal 'local0.info' then @@(z9)172.20.10.30:4517;RSYSLOG_ForwardFormat #:msg, contains, 'SymantecServer' @@(z9)172.20.10.30:4515;RSYSLOG_ForwardFormat #:msg, contains, 'D75FADC001' @@(z9)172.20.10.30:4516;RSYSLOG_ForwardFormat #:pri, isequal, '134' @@(z9)172.20.10.30:4517;RSYSLOG_ForwardFormat :fromhost-ip, isequal, '172.20.26.245' -/media/data/var/log/gdf & ~ local0.info -/media/data/var/log/gdf & ~ kern.* -/var/log/kernel *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* -/var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron *.emerg * uucp,news.crit -/var/log/spooler local7.* -/var/log/boot <close/> Relay: The relay will forward the separate TCP streams onto Splunk data inputs as TCP ports and indexed. We want to separate the streams into separate indexes to enable easier analysis of different sourcetypes. Normalisation could be done as far back as the log source, on the collector or in Splunk itself. The end point for the data will be in AlienVault to do SIEM and event correlation. <open> $ModLoad imklog $ModLoad imuxsock $ModLoad imrelp $InputRELPServerRun 4514 $ModLoad imudp $UDPServerRun 514 $ModLoad imtcp $InputTCPServerRun 514 # Is this right ordering? # How does it pertain to the below rulesets? $WorkDirectory /var/spool/rsyslog/ $ActionQueueFileName gdf $ActionQueueType LinkedList $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on kern.* -/var/log/kernel *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* -/var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron *.emerg * uucp,news.crit -/var/log/spooler local7.* -/var/log/boot $Ruleset FW1 #:pri, isequal, "134" then @@(z9)172.nnn.nnn.nnn:4517;RSYSLOG_ForwardFormat #*.* -/var/log/fw1 *.* @@172.nnn.nnn.nnn:44517;RSYSLOG_ForwardFormat $InputTCPServerBindRuleset FW1 $InputTCPServerRun 4517 $Ruleset D75FADC001 #:msg, contains, "D75FADC001" @@(z9)172.nnn.nnn.nnn:4516;RSYSLOG_ForwardFormat #*.* -/var/log/D75FADC001 # Will this work? $WorkDirectory /var/spool/rsyslog/ $ActionQueueFileName D75FADC001 $ActionQueueType LinkedList $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.* @@172.nnn.nnn.nnn:44516;RSYSLOG_ForwardFormat $InputTCPServerBindRuleset D75FADC001 $InputTCPServerRun 4516 $Ruleset SEP #:msg, contains, "SymantecServer" @@(z9)172.nnn.nnn.nnn:4515;RSYSLOG_ForwardFormat #*.* -/var/log/sep *.* @@nnn.nnn.nnn.nnn:44515;RSYSLOG_ForwardFormat $InputTCPServerBindRuleset SEP $InputTCPServerRun 4515 <close/> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
