@ rainer , waiting for your hint
On Tue, Aug 5, 2014 at 5:14 PM, Zeshan <[email protected]> wrote: > hi, > > i have made set up for mmnormalize, my input file is file1 contaning > sample logs mentioned in rainer's blog > http://doc.liblognorm.com/samples/messages.log > > and according to my understanding rulebase.rb is > http://doc.liblognorm.com/samples/messages.sampdb in rainer's belog. > > now in below setting how input file will use this rulebase.rb > > input(type="imfile" File="/etc/file1" > Tag="tag1" > StateFile="statefile1" > #Severity="error" > Facility="local7") > > > module(load="mmnormalize") > action(type="mmnormalize" ruleBase="/etc/rulebase.rb") > > > if i add following two lines , it shows > > $template cee,"msg is '%msg%', %$!all-json%\n" > *.* /etc/normlog;cee > > msg is 'May 16 04:02:01 ns1 syslogd 1.4-0: restart.', { "originalmsg": > "May 16 04:02:01 ns1 syslogd 1.4-0: restart.", "unparsed-data": "May 16 > 04:02:01 ns1 syslogd 1.4-0: restart." } > msg is 'May 16 04:02:01 ns1 syslogd 1.4-0: restart.', { "originalmsg": > "May 16 04:02:01 ns1 syslogd 1.4-0: restart.", "unparsed-data": "May 16 > 04:02:01 ns1 syslogd 1.4-0: restart." } > msg is 'May 16 04:02:01 ns1 syslogd 1.4-0: restart.', { "originalmsg": > "May 16 04:02:01 ns1 syslogd 1.4-0: restart.", "unparsed-data": "May 16 > 04:02:01 ns1 syslogd 1.4-0: restart." } > > > > > On Fri, Jul 25, 2014 at 11:51 PM, David Lang <[email protected]> wrote: > >> On Fri, 25 Jul 2014, Zeshan wrote: >> >> Hi, >>> >>> we have the scenario in which we want to parse the logs received at our >>> rsyslog server and store them in DB say MongoDB. e.g. its squid log >>> "Jun 12 22:02:17 172.20.8.4 (squid): 1402544268.502 80990 172.20.9.78 >>> TCP_MISS/200 54721 GET http://img.objectembed.info/intro.swf? - DIRECT/ >>> 93.184.220.20 application/x-shockwave-flash" >>> >>> and i want to extract following fields from the above log >>> >>> time stamp : Jun 12 22:02:17 >>> host : 172.20.8.4 >>> type : squid >>> device_event_id : TCP_MISS/200 >>> url: http://img.objectembed.info/intro.swf? >>> >> >> this is trivial, you already have timestamp host and programname parsed, >> so all you need to extract is device_event_id and url, and those are fields >> 4 and 6 with a space delimiter. >> >> >> 2ndly i have another device apache with the following log >>> >>> 172.20.16.37 - - [11/Jun/2014:10:38:26 +0500] "GET /nagios HTTP/1.1" 404 >>> 281 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, >>> like Gecko) Chrome/35.0.1916.114 Safari/537.36" >>> >>> and the following fields can be extracted >>> >>> host : >>> time stamp: >>> OS: >>> etc etc >>> >> >> this is messier because there are a lot of optional spaces inside the >> fields, and so you need to take into account quotes. I actually recommend >> creating a custom format that is easier to parse rather than trying to use >> the default log format. >> >> >> fields to be extracted will be different based on different logs and the >>> list goes on for log devices . >>> >>> I can write regex for all the devices and used them in logstash. >>> >>> My question is what would be the right approach to parse these fields in >>> rsyslog level. will it be right to write python files parsing these logs >>> and calling them through mmexternal or i should go through these docs >>> http://www.rsyslog.com/doc/syslog_parsing.html >>> and http://www.rsyslog.com/doc/messageparser.html >>> >>> i have to support at least 200 device logs , have to parse them and store >>> them in database say MongoDB . >>> >> >> There are many ways you can do this job. >> >> you can write a lot of regex expressions in rsyslog >> >> you can write a lot of regex expressions in python, have rsyslog >> serialize the logs and send them to your python program, have your python >> program parse them, serialize them to send back to rsyslog, and have >> rsyslog parse them (mmexternal) >> >> but the best way to do this would be to write some rules using >> mmnormalize http://www.rsyslog.com/doc/mmnormalize.html and have it >> parse the logs >> >> David Lang >> >> >> >>> >>> >>> >>> On Tue, Jul 22, 2014 at 11:45 AM, Rainer Gerhards < >>> [email protected]> >>> wrote: >>> >>> As I said, omprog is for feeding output data to an external plugin. In >>>> contrast, mmexternal is to call an external message modification >>>> plugin. So >>>> If you want to make modifications to the message, you need to call >>>> mmexternal. I highly suggest eading the links I have posted. >>>> >>>> Rainer >>>> >>>> >>>> On Tue, Jul 22, 2014 at 8:23 AM, Zeshan <[email protected]> wrote: >>>> >>>> David , can you tell me the difference bet omprog and mmexternal. both >>>>> >>>> can >>>> >>>>> be used to call the external plugin? my purpose is two feed raw >>>>> log/msg >>>>> to some external plugin and do the parsing? which one will do the >>>>> >>>> purpose >>>> >>>>> for me? >>>>> >>>>> >>>>> On Tue, Jul 22, 2014 at 6:22 AM, David Lang <[email protected]> wrote: >>>>> >>>>> In any language, there is a huge variation in the performance of >>>>>> things >>>>>> like this, depending on implementation details, you will have to try >>>>>> it >>>>>> >>>>> and >>>>> >>>>>> see. >>>>>> >>>>>> >>>>>> David Lang >>>>>> >>>>>> On Mon, 21 Jul 2014, Zeshan wrote: >>>>>> >>>>>> we were using ruby files in case of logstash for normalization , now >>>>>> >>>>> we >>>> >>>>> want to reuse them in rsyslog , by calling them through omprog and >>>>>>> >>>>>> writing >>>>> >>>>>> the data to mongoDB. >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 21, 2014 at 4:35 PM, David Lang <[email protected]> wrote: >>>>>>> >>>>>>> It probably depends mostly on what code you are using for your >>>>>>> >>>>>>>> normalization. >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> >>>>>>>> On Mon, 21 Jul 2014, Rainer Gerhards wrote: >>>>>>>> >>>>>>>> On Mon, Jul 21, 2014 at 11:43 AM, masoom alam < >>>>>>>> >>>>>>> [email protected]> >>>> >>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> David: >>>>>>>>> >>>>>>>>> >>>>>>>>>> How the performance will affected in the case where we want to >>>>>>>>>> pass >>>>>>>>>> >>>>>>>>> the >>>>> >>>>>> control to do some normalization - name,value pair conversion by >>>>>>>>>> Rsyslog. >>>>>>>>>> We want to get rid of Logstash. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> depending on what you do it's "not too bad", but of course it >>>>>>>>>> >>>>>>>>> depends >>>>> >>>>>> on >>>>>>>>>> >>>>>>>>>> the circumstances. However, you do not want to use output modules >>>>>>>>> >>>>>>>> but >>>> >>>>> you >>>>>>>>> want to use the message modification internface, implemented via >>>>>>>>> mmexternal. See these links: >>>>>>>>> >>>>>>>>> https://github.com/rsyslog/rsyslog/blob/master/plugins/ >>>>>>>>> external/INTERFACE.md >>>>>>>>> >>>>>>>>> >>>>>>>>> http://www.rsyslog.com/doc/master/configuration/modules/ >>>>> mmexternal.html >>>>> >>>>>> >>>>>>>>> HTH >>>>>>>>> Rainer >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Jul 21, 2014 at 2:20 PM, David Lang <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> On Mon, 21 Jul 2014, Zeshan wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks David, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It worked now with the following lines in my rsyslog.conf >>>>>>>>>>>> >>>>>>>>>>>> $ModLoad omprog >>>>>>>>>>>> $actionomprogbinary /var/log/test.sh >>>>>>>>>>>> *.* :omprog:;RSYSLOG_TraditionalFileFormat >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> that's the legacy format, the new format I gave below does the >>>>>>>>>>>> >>>>>>>>>>> same >>>>> >>>>>> >>>>>>>>>>> thing, >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> but is clearer in many ways. It allos allows you to give >>>>>>>>>> >>>>>>>>> parameters >>>> >>>>> to >>>>> >>>>>> >>>>>>>>>>> your >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> program (the $actionomprogbinary value can't have any spaces in >>>>>>>>>> >>>>>>>>> it) >>>> >>>>> >>>>>>>>>>> now that you have this working, you can change the format of data >>>>>>>>>>> passed >>>>>>>>>>> to your script by using a different template, and change ot >>>>>>>>>>> >>>>>>>>>> program >>>> >>>>> >>>>>>>>>>> itself >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> to any other language. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> David Lang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I was doing exactly the same except that the script was in >>>>>>>>>>> /opt/test.sh >>>>>>>>>>> >>>>>>>>>>> and >>>>>>>>>>> >>>>>>>>>>>> now it is in /var/log/test.sh >>>>>>>>>>>> >>>>>>>>>>>> Thanks for all the kindness >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jul 21, 2014 at 1:42 PM, David Lang <[email protected]> >>>>>>>>>>>> >>>>>>>>>>> wrote: >>>> >>>>> >>>>>>>>>>>> Ok, looking at the documentation >>>>>>>>>>>> >>>>>>>>>>>> http://www.rsyslog.com/doc/omprog.html >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> try >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Module (load="omprog") >>>>>>>>>>>>> action(type="omprog" binary="/pathto/test.sh" >>>>>>>>>>>>> template="RSYSLOG_ >>>>>>>>>>>>> TraditionalFileFormat") >>>>>>>>>>>>> >>>>>>>>>>>>> this will send the script every log entry that arrives on the >>>>>>>>>>>>> rsyslog >>>>>>>>>>>>> server >>>>>>>>>>>>> >>>>>>>>>>>>> if you run this with the -dn flags, we should see test.sh and >>>>>>>>>>>>> >>>>>>>>>>>> the >>>> >>>>> prog >>>>>>>>>>>>> output module show up with the first log entry that it >>>>>>>>>>>>> >>>>>>>>>>>> processes. >>>> >>>>> >>>>>>>>>>>>> David Lang >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, 21 Jul 2014, Zeshan wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks David for your reply >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ok , i have script named test.sh having following code >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> #!/bin/bash >>>>>>>>>>>>>> echo "$@" >/var/log/myoutput >>>>>>>>>>>>>> >>>>>>>>>>>>>> and it is is executable. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Now what should me rsyslog.conf file. and how to test this >>>>>>>>>>>>>> >>>>>>>>>>>>> whole >>>> >>>>> procedure >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Jul 21, 2014 at 1:09 PM, Muhammad Asif Ihsan < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> David, now I have its binary and what should I code in >>>>>>>>>>>>>> rsyslog.conf >>>>>>>>>>>>>> >>>>>>>>>>>>>> for >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> loading and redirecting to my executable myfile. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>>>>>>>>>>>>>> >>>>>>>>>>>>>> by >>>> >>>>> a >>>>> >>>>>> myriad >>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>>>>>>> >>>>>>>>>>>>>> POST >>>> >>>>> if >>>>> >>>>>> >>>>>>>>>>>>>>> you >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>>>>>>>>>>>>> >>>>>>>>>>>>> by a >>>> >>>>> >>>>>>>>>>>>>> myriad >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>> POST if >>>>>>>>>> >>>>>>>>> you >>>>> >>>>>> >>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> >>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >>>>>>>>>>>>> >>>>>>>>>>>> a >>>> >>>>> >>>>>>>>>>>>> myriad >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>> POST if >>>>>>>>>> >>>>>>>>> you >>>>> >>>>>> >>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> >>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>> >>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >>>>>>>>>>>> a >>>>>>>>>>>> myriad >>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>>>>>>>>>> >>>>>>>>>>> if >>>> >>>>> you >>>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> >>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>> >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>>>> myriad >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>>>>>>>>> if >>>>>>>>>>> >>>>>>>>>> you >>>>> >>>>>> DON'T LIKE THAT. >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> >>>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>>> myriad >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>>> >>>>>>>>> you >>>>> >>>>>> DON'T LIKE THAT. >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> >>>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>> >>>>>>>> myriad >>>>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>> >>>>>>>> you >>>> >>>>> DON'T LIKE THAT. >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> >>>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>> >>>>>>> myriad >>>>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>> >>>>>>> you >>>> >>>>> DON'T LIKE THAT. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> >>>>>> myriad >>>> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>>> DON'T LIKE THAT. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> >>>>> myriad >>>> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
