hi, i have made set up for mmnormalize, my input file is file1 contaning sample logs mentioned in rainer's blog http://doc.liblognorm.com/samples/messages.log
and according to my understanding rulebase.rb is http://doc.liblognorm.com/samples/messages.sampdb in rainer's belog. now in below setting how input file will use this rulebase.rb input(type="imfile" File="/etc/file1" Tag="tag1" StateFile="statefile1" #Severity="error" Facility="local7") module(load="mmnormalize") action(type="mmnormalize" ruleBase="/etc/rulebase.rb") if i add following two lines , it shows $template cee,"msg is '%msg%', %$!all-json%\n" *.* /etc/normlog;cee msg is 'May 16 04:02:01 ns1 syslogd 1.4-0: restart.', { "originalmsg": "May 16 04:02:01 ns1 syslogd 1.4-0: restart.", "unparsed-data": "May 16 04:02:01 ns1 syslogd 1.4-0: restart." } msg is 'May 16 04:02:01 ns1 syslogd 1.4-0: restart.', { "originalmsg": "May 16 04:02:01 ns1 syslogd 1.4-0: restart.", "unparsed-data": "May 16 04:02:01 ns1 syslogd 1.4-0: restart." } msg is 'May 16 04:02:01 ns1 syslogd 1.4-0: restart.', { "originalmsg": "May 16 04:02:01 ns1 syslogd 1.4-0: restart.", "unparsed-data": "May 16 04:02:01 ns1 syslogd 1.4-0: restart." } On Fri, Jul 25, 2014 at 11:51 PM, David Lang <[email protected]> wrote: > On Fri, 25 Jul 2014, Zeshan wrote: > > Hi, >> >> we have the scenario in which we want to parse the logs received at our >> rsyslog server and store them in DB say MongoDB. e.g. its squid log >> "Jun 12 22:02:17 172.20.8.4 (squid): 1402544268.502 80990 172.20.9.78 >> TCP_MISS/200 54721 GET http://img.objectembed.info/intro.swf? - DIRECT/ >> 93.184.220.20 application/x-shockwave-flash" >> >> and i want to extract following fields from the above log >> >> time stamp : Jun 12 22:02:17 >> host : 172.20.8.4 >> type : squid >> device_event_id : TCP_MISS/200 >> url: http://img.objectembed.info/intro.swf? >> > > this is trivial, you already have timestamp host and programname parsed, > so all you need to extract is device_event_id and url, and those are fields > 4 and 6 with a space delimiter. > > > 2ndly i have another device apache with the following log >> >> 172.20.16.37 - - [11/Jun/2014:10:38:26 +0500] "GET /nagios HTTP/1.1" 404 >> 281 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, >> like Gecko) Chrome/35.0.1916.114 Safari/537.36" >> >> and the following fields can be extracted >> >> host : >> time stamp: >> OS: >> etc etc >> > > this is messier because there are a lot of optional spaces inside the > fields, and so you need to take into account quotes. I actually recommend > creating a custom format that is easier to parse rather than trying to use > the default log format. > > > fields to be extracted will be different based on different logs and the >> list goes on for log devices . >> >> I can write regex for all the devices and used them in logstash. >> >> My question is what would be the right approach to parse these fields in >> rsyslog level. will it be right to write python files parsing these logs >> and calling them through mmexternal or i should go through these docs >> http://www.rsyslog.com/doc/syslog_parsing.html >> and http://www.rsyslog.com/doc/messageparser.html >> >> i have to support at least 200 device logs , have to parse them and store >> them in database say MongoDB . >> > > There are many ways you can do this job. > > you can write a lot of regex expressions in rsyslog > > you can write a lot of regex expressions in python, have rsyslog serialize > the logs and send them to your python program, have your python program > parse them, serialize them to send back to rsyslog, and have rsyslog parse > them (mmexternal) > > but the best way to do this would be to write some rules using mmnormalize > http://www.rsyslog.com/doc/mmnormalize.html and have it parse the logs > > David Lang > > > >> >> >> >> On Tue, Jul 22, 2014 at 11:45 AM, Rainer Gerhards < >> [email protected]> >> wrote: >> >> As I said, omprog is for feeding output data to an external plugin. In >>> contrast, mmexternal is to call an external message modification plugin. >>> So >>> If you want to make modifications to the message, you need to call >>> mmexternal. I highly suggest eading the links I have posted. >>> >>> Rainer >>> >>> >>> On Tue, Jul 22, 2014 at 8:23 AM, Zeshan <[email protected]> wrote: >>> >>> David , can you tell me the difference bet omprog and mmexternal. both >>>> >>> can >>> >>>> be used to call the external plugin? my purpose is two feed raw log/msg >>>> to some external plugin and do the parsing? which one will do the >>>> >>> purpose >>> >>>> for me? >>>> >>>> >>>> On Tue, Jul 22, 2014 at 6:22 AM, David Lang <[email protected]> wrote: >>>> >>>> In any language, there is a huge variation in the performance of things >>>>> like this, depending on implementation details, you will have to try it >>>>> >>>> and >>>> >>>>> see. >>>>> >>>>> >>>>> David Lang >>>>> >>>>> On Mon, 21 Jul 2014, Zeshan wrote: >>>>> >>>>> we were using ruby files in case of logstash for normalization , now >>>>> >>>> we >>> >>>> want to reuse them in rsyslog , by calling them through omprog and >>>>>> >>>>> writing >>>> >>>>> the data to mongoDB. >>>>>> >>>>>> >>>>>> On Mon, Jul 21, 2014 at 4:35 PM, David Lang <[email protected]> wrote: >>>>>> >>>>>> It probably depends mostly on what code you are using for your >>>>>> >>>>>>> normalization. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> >>>>>>> On Mon, 21 Jul 2014, Rainer Gerhards wrote: >>>>>>> >>>>>>> On Mon, Jul 21, 2014 at 11:43 AM, masoom alam < >>>>>>> >>>>>> [email protected]> >>> >>>> >>>>>>> wrote: >>>>>>>> >>>>>>>> David: >>>>>>>> >>>>>>>> >>>>>>>>> How the performance will affected in the case where we want to pass >>>>>>>>> >>>>>>>> the >>>> >>>>> control to do some normalization - name,value pair conversion by >>>>>>>>> Rsyslog. >>>>>>>>> We want to get rid of Logstash. >>>>>>>>> >>>>>>>>> >>>>>>>>> depending on what you do it's "not too bad", but of course it >>>>>>>>> >>>>>>>> depends >>>> >>>>> on >>>>>>>>> >>>>>>>>> the circumstances. However, you do not want to use output modules >>>>>>>> >>>>>>> but >>> >>>> you >>>>>>>> want to use the message modification internface, implemented via >>>>>>>> mmexternal. See these links: >>>>>>>> >>>>>>>> https://github.com/rsyslog/rsyslog/blob/master/plugins/ >>>>>>>> external/INTERFACE.md >>>>>>>> >>>>>>>> >>>>>>>> http://www.rsyslog.com/doc/master/configuration/modules/ >>>> mmexternal.html >>>> >>>>> >>>>>>>> HTH >>>>>>>> Rainer >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jul 21, 2014 at 2:20 PM, David Lang <[email protected]> wrote: >>>>>>>>> >>>>>>>>> On Mon, 21 Jul 2014, Zeshan wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks David, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It worked now with the following lines in my rsyslog.conf >>>>>>>>>>> >>>>>>>>>>> $ModLoad omprog >>>>>>>>>>> $actionomprogbinary /var/log/test.sh >>>>>>>>>>> *.* :omprog:;RSYSLOG_TraditionalFileFormat >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> that's the legacy format, the new format I gave below does the >>>>>>>>>>> >>>>>>>>>> same >>>> >>>>> >>>>>>>>>> thing, >>>>>>>>>> >>>>>>>>> >>>>>>>>> but is clearer in many ways. It allos allows you to give >>>>>>>>> >>>>>>>> parameters >>> >>>> to >>>> >>>>> >>>>>>>>>> your >>>>>>>>>> >>>>>>>>> >>>>>>>>> program (the $actionomprogbinary value can't have any spaces in >>>>>>>>> >>>>>>>> it) >>> >>>> >>>>>>>>>> now that you have this working, you can change the format of data >>>>>>>>>> passed >>>>>>>>>> to your script by using a different template, and change ot >>>>>>>>>> >>>>>>>>> program >>> >>>> >>>>>>>>>> itself >>>>>>>>>> >>>>>>>>> >>>>>>>>> to any other language. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> David Lang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I was doing exactly the same except that the script was in >>>>>>>>>> /opt/test.sh >>>>>>>>>> >>>>>>>>>> and >>>>>>>>>> >>>>>>>>>>> now it is in /var/log/test.sh >>>>>>>>>>> >>>>>>>>>>> Thanks for all the kindness >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Jul 21, 2014 at 1:42 PM, David Lang <[email protected]> >>>>>>>>>>> >>>>>>>>>> wrote: >>> >>>> >>>>>>>>>>> Ok, looking at the documentation >>>>>>>>>>> >>>>>>>>>>> http://www.rsyslog.com/doc/omprog.html >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> try >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Module (load="omprog") >>>>>>>>>>>> action(type="omprog" binary="/pathto/test.sh" template="RSYSLOG_ >>>>>>>>>>>> TraditionalFileFormat") >>>>>>>>>>>> >>>>>>>>>>>> this will send the script every log entry that arrives on the >>>>>>>>>>>> rsyslog >>>>>>>>>>>> server >>>>>>>>>>>> >>>>>>>>>>>> if you run this with the -dn flags, we should see test.sh and >>>>>>>>>>>> >>>>>>>>>>> the >>> >>>> prog >>>>>>>>>>>> output module show up with the first log entry that it >>>>>>>>>>>> >>>>>>>>>>> processes. >>> >>>> >>>>>>>>>>>> David Lang >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, 21 Jul 2014, Zeshan wrote: >>>>>>>>>>>> >>>>>>>>>>>> Thanks David for your reply >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ok , i have script named test.sh having following code >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> #!/bin/bash >>>>>>>>>>>>> echo "$@" >/var/log/myoutput >>>>>>>>>>>>> >>>>>>>>>>>>> and it is is executable. >>>>>>>>>>>>> >>>>>>>>>>>>> Now what should me rsyslog.conf file. and how to test this >>>>>>>>>>>>> >>>>>>>>>>>> whole >>> >>>> procedure >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jul 21, 2014 at 1:09 PM, Muhammad Asif Ihsan < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> David, now I have its binary and what should I code in >>>>>>>>>>>>> rsyslog.conf >>>>>>>>>>>>> >>>>>>>>>>>>> for >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> loading and redirecting to my executable myfile. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>>>>>>>>>>>>> >>>>>>>>>>>>> by >>> >>>> a >>>> >>>>> myriad >>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>>>>>> >>>>>>>>>>>>> POST >>> >>>> if >>>> >>>>> >>>>>>>>>>>>>> you >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> >>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>>>>>>>>>>>> >>>>>>>>>>>> by a >>> >>>> >>>>>>>>>>>>> myriad >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>> POST if >>>>>>>>> >>>>>>>> you >>>> >>>>> >>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> >>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>> >>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >>>>>>>>>>>> >>>>>>>>>>> a >>> >>>> >>>>>>>>>>>> myriad >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>>>>>>> if >>>>>>>>> >>>>>>>> you >>>> >>>>> >>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> >>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>> >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>>>> myriad >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>>>>>>>>> >>>>>>>>>> if >>> >>>> you >>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> >>>>>>>>>>> rsyslog mailing list >>>>>>>>>>> >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>>> myriad >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>>> >>>>>>>>> you >>>> >>>>> DON'T LIKE THAT. >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> >>>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>> myriad >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>> >>>>>>>> you >>>> >>>>> DON'T LIKE THAT. >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> >>>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>> >>>>>>> myriad >>>> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>> >>>>>>> you >>> >>>> DON'T LIKE THAT. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> >>>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> >>>>>> myriad >>>> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>> >>>>>> you >>> >>>> DON'T LIKE THAT. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> >>>>> myriad >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> _______________________________________________ >>>>>> >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> >>>> myriad >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
