Coincidentally I decided to scrap everything I thought right on Monday and just tried my luck with a little REGEX, sheets of paper and careful reverse-testing.
http://kb.monitorware.com/feedback-snare-parsing-t10768.html $template dbFormat,"insert into SystemEvents (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag, EventLogType, EventSource, EventId, EventUser) values ('%msg:F:10%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%msg:F:3%', '%msg:F:6%', '%msg:F:8%', '%msg:F:2%','%msg:F:4%')",sql :syslogtag, contains, "Win" /var/log/windows #Personal checker, don't need & >localhost,Syslog,rsyslog,password;dbFormat That does the trick for me for now :) Is the .snare parser documented at some point to see what it's doing? regards and thanks Stefan Von: [email protected] An: [email protected] Datum: 14.03.2011 21:12 Betreff: Re: [rsyslog] Parsing Snare input Gesendet von: [email protected] On Thu, 10 Mar 2011, [email protected] wrote: > After spending a few days(in relation to many others this is probably > childs play) I?m stuck with a problem and I?m out of ideas on how to fix > it. > > I?m having a Windows 2003 Server pump his Syslogs via Snare onto a RHEL 6.0 > with the 5.6.4 build of rsyslog(compiled with ./configure --enable-mysql > --enable-gnutls --enable-mail) running the latest LogAnalyzer Gui. > I'm also having a RHEL 5.2 pump data into it via rsyslog, default yum > package and settings. UDP, case it matters. > > I'm sending all the input to the DB and also logging it to the disk into > seperate files(see config later). > > This works fine for the "Syslog Fields" view, where only the ProcessID > field is empty(for either of them). > But, the EventLog fields are just in very bad shape. > EventLog Type, Event Source, EventID and Event User are all missing - for > both the RHEL and the Windows servers. > The whole info is dumped into the message property - like it is by default. > > My goal would be to get a properly filled view like > http://loganalyzer-demo.adiscon.com/ (eventlog fields) where especially the > different Event Users are working - but I can't seem to manage. > I?d like to have all the data, parsed or not, on the disk saved and > properly parsed in the MySQL/in the GUI. > > I tried template parsing via the rsyslog wiki and David Langs example from > http://lists.adiscon.net/pipermail/rsyslog/2009-October/010978.html This now needs to be updated since we have the pmsnare parser in the tree. it greatly simplifies things. instead of everything listed on that page, $ModLoad pmsnare $RulesetParser rsyslog.snare $RulesetPArser rsyslog.rfc3164 $template fixsnareFormat,"%timereported% %HOSTNAME%%msg:::drop-last-lf%\n" $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME%%msg:::drop-last-lf%\n" :syslogtag, isequal, "#001" /var/log/mesages;fixsnareFormat & @192.168.1.1;fixsnareForwardFormat & ~ this may not be needed in a properly setup environment, but I have some servers sending a hex 0x01 character instead of their name. This may be a snare config error but I haven't had time to track it down. > I added the Parser from rsyslog forums > http://kb.monitorware.com/snare-msg-parser-t10171.html#p18502 but I can't > see to apply this to a SQL DB. > > I tried direct SQL statements in the rsyslog.conf - to no avail. > > > Any ideas on how to parse this properly/easy? I suspect that the problem you are having is that rsyslog is escaping the tab characters (changing a tab into #011) this is getting to be a problem for me as well, but currently there is not a way to just disable tab escapign without disabling all control character escaping. When I can get a little time I need to create a patch to enable tabs to be left alone, but I'm on Jury duty for the next couple of weeks, plus have some work emergancies that I am having to work on (which is why I haven't been checking e-mail for the last week) so I don't know how soon I can get this done. David Lang > regards > > > # for Snare > $EscapeControlCharactersOnReceive off > # > # #load modules > $ModLoad immark # provides --MARK-- message capability > $ModLoad imudp # provides UDP syslog reception > $ModLoad imtcp # provides TCP syslog reception > $ModLoad imuxsock # provides support for local system logging (e.g. via > logger command) > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > $ModLoad ommail # direct email sending > $ModLoad ommysql > > # Provides UDP syslog reception > $UDPServerRun 514 > > # Provides TCP syslog reception > #$InputTCPServerRun 514 > > ##################################################### > ## Log everything to a per host daily logfile # > ###################################################### > $template > DailyPerHostLogs,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%.log" > *.* -?DailyPerHostLogs > > # Writes everything into DB > *.* :ommysql:localhost,Syslog,rsyslog,password > > #MAILTEST - functional 9.3.2011 > #$ActionMailSMTPServer xxxx > #$ActionMailFrom xxx > #$ActionMailTo xxx > #$template mailSubject,"%hostname% - %timereported%" > #$template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'" > #$ActionMailSubject mailSubject > #$ActionExecOnlyOnceEveryInterval 20 > #:hostname, contains, "xl" :ommail:;mailBody > > #### GLOBAL DIRECTIVES #### > > # Use default timestamp format > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > #### RULES #### > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none /var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/maillog > > # Log cron stuff > cron.* /var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit /var/log/spooler > > # Save boot messages also to boot.log > local7.* > > > > Windows Example Message: > Service 17978 Thu Mar 10 11:40:39 2011 1139 NTDS LDAP ANONYMOUS LOGON Well > Known Group Information 38866A01 LDAP Interface Internal event: Function > ldap_search completed with an elapsed time of 0 ms. 8585 > > ---------------------------------------- > Raiffeisen-Landesbank Steiermark AG, Graz, FN 264700 s, Landesgericht f?r > Zivilrechtssachen Graz, DVR 0040495 > > Der Austausch von Nachrichten mit oa. Absender via E-Mail dient > ausschliesslich Informationszwecken. Rechtsgeschaeftliche Erklaerungen > duerfen ueber dieses Medium nicht ausgetauscht werden. > Correspondence with a.m. sender via e-mail is only for information > purposes. This medium is not to be used for the exchange of legally-binding > communications. > ---------------------------------------- > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com ---------------------------------------- Raiffeisen-Landesbank Steiermark AG, Graz, FN 264700 s, Landesgericht für Zivilrechtssachen Graz, DVR 0040495 Der Austausch von Nachrichten mit oa. Absender via E-Mail dient ausschliesslich Informationszwecken. Rechtsgeschaeftliche Erklaerungen duerfen ueber dieses Medium nicht ausgetauscht werden. Correspondence with a.m. sender via e-mail is only for information purposes. This medium is not to be used for the exchange of legally-binding communications. ----------------------------------------
<<inline: graycol.gif>>
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
