Re: [rsyslog] Want to run python code using omprog on rsyslog 7.6.3

David Lang Wed, 06 Aug 2014 10:59:25 -0700

Rainer is on vacation for two weeks. There are several other people on the listwho have experience with mmnormalize.

One thing that jumps out at me is that if you are reading the messages from thatfile, the timestamp in the file is being treated as part of the message andrsyslog will add it's own timestamp


output a message using RSYSLOG_DebugFormat and take a look at it.

If you are reading messages via imfile, you need to strip off the timestamp,hostname, and programname (programname is set by the tag that you set)


David Lang

 On Wed, 6 Aug 2014, Zeshan wrote:

Date: Wed, 6 Aug 2014 15:11:25 +0500
From: Zeshan <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Want to run python code using omprog on rsyslog 7.6.3

@ rainer ,

waiting for your hint


On Tue, Aug 5, 2014 at 5:14 PM, Zeshan <[email protected]> wrote:

hi,

i have made set up for mmnormalize, my input file is file1 contaning
sample logs mentioned in rainer's blog
http://doc.liblognorm.com/samples/messages.log

and according to my understanding rulebase.rb is
http://doc.liblognorm.com/samples/messages.sampdb  in rainer's belog.

now in below setting how input file will use this rulebase.rb

input(type="imfile" File="/etc/file1"
     Tag="tag1"
     StateFile="statefile1"
     #Severity="error"
     Facility="local7")


module(load="mmnormalize")
action(type="mmnormalize" ruleBase="/etc/rulebase.rb")


if i add following two lines , it shows

$template cee,"msg is '%msg%', %$!all-json%\n"
*.*     /etc/normlog;cee

msg is 'May 16 04:02:01 ns1 syslogd 1.4-0: restart.', { "originalmsg":
"May 16 04:02:01 ns1 syslogd 1.4-0: restart.", "unparsed-data": "May 16
04:02:01 ns1 syslogd 1.4-0: restart." }
msg is 'May 16 04:02:01 ns1 syslogd 1.4-0: restart.', { "originalmsg":
"May 16 04:02:01 ns1 syslogd 1.4-0: restart.", "unparsed-data": "May 16
04:02:01 ns1 syslogd 1.4-0: restart." }
msg is 'May 16 04:02:01 ns1 syslogd 1.4-0: restart.', { "originalmsg":
"May 16 04:02:01 ns1 syslogd 1.4-0: restart.", "unparsed-data": "May 16
04:02:01 ns1 syslogd 1.4-0: restart." }




On Fri, Jul 25, 2014 at 11:51 PM, David Lang <[email protected]> wrote:

On Fri, 25 Jul 2014, Zeshan wrote:

 Hi,


we have the scenario in which we want to parse the logs received at our
rsyslog server and store them in DB say MongoDB. e.g. its squid log
"Jun 12 22:02:17 172.20.8.4 (squid): 1402544268.502  80990 172.20.9.78
TCP_MISS/200 54721 GET http://img.objectembed.info/intro.swf? - DIRECT/
93.184.220.20 application/x-shockwave-flash"

and i want to extract following fields from the above log

time stamp :  Jun 12 22:02:17
host : 172.20.8.4
type : squid
device_event_id : TCP_MISS/200
url: http://img.objectembed.info/intro.swf?


this is trivial, you already have timestamp host and programname parsed,
so all you need to extract is device_event_id and url, and those are fields
4 and 6 with a space delimiter.


 2ndly i have another device apache with the following log


172.20.16.37 - - [11/Jun/2014:10:38:26 +0500] "GET /nagios HTTP/1.1" 404
281 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/35.0.1916.114 Safari/537.36"

and the following fields can be extracted

host :
time stamp:
OS:
etc etc


this is messier because there are a lot of optional spaces inside the
fields, and so you need to take into account quotes. I actually recommend
creating a custom format that is easier to parse rather than trying to use
the default log format.


 fields to be extracted will be different based on different logs and the

list goes on for log devices .

I can write regex for all the devices and used them in logstash.

My question is  what would be the right approach to parse these fields in
rsyslog level.  will it be right to write python files parsing these logs
and calling them through mmexternal or i should go through these docs
http://www.rsyslog.com/doc/syslog_parsing.html
and http://www.rsyslog.com/doc/messageparser.html

i have to support at least 200 device logs , have to parse them and store
them in database say MongoDB .


There are many ways you can do this job.

you can write a lot of regex expressions in rsyslog

you can write a lot of regex expressions in python, have rsyslog
serialize the logs and send them to your python program, have your python
program parse them, serialize them to send back to rsyslog, and have
rsyslog parse them (mmexternal)

but the best way to do this would be to write some rules using
mmnormalize http://www.rsyslog.com/doc/mmnormalize.html and have it
parse the logs

David Lang




On Tue, Jul 22, 2014 at 11:45 AM, Rainer Gerhards <
[email protected]>
wrote:

 As I said, omprog is for feeding output data to an external plugin. In

contrast, mmexternal is to call an external message modification
plugin. So
If you want to make modifications to the message, you need to call
mmexternal. I highly suggest eading the links I have posted.

Rainer


On Tue, Jul 22, 2014 at 8:23 AM, Zeshan <[email protected]> wrote:

 David , can you tell me the difference bet omprog and mmexternal. both

can

be used to call the external plugin?  my purpose is two feed raw
log/msg
 to some external plugin and do the parsing? which one will do the

 purpose

for me?


On Tue, Jul 22, 2014 at 6:22 AM, David Lang <[email protected]> wrote:

 In any language, there is a huge variation in the performance of

things
like this, depending on implementation details, you will have to try
it

and

see.


David Lang

On Mon, 21 Jul 2014, Zeshan wrote:

 we were using ruby files in case of logstash for normalization , now

we

want to reuse them in rsyslog , by calling them through omprog and

writing

the data to mongoDB.



On Mon, Jul 21, 2014 at 4:35 PM, David Lang <[email protected]> wrote:

 It probably depends mostly on what code you are using for your

normalization.

David Lang


On Mon, 21 Jul 2014, Rainer Gerhards wrote:

 On Mon, Jul 21, 2014 at 11:43 AM, masoom alam <

[email protected]>

 wrote:


 David:

How the performance will affected in the case where we want to
pass

the

 control to do some normalization - name,value pair conversion by

Rsyslog.
We want to get rid of Logstash.


 depending on what you do it's "not too bad", but of course it

depends

on


 the circumstances. However, you do not want to use output modules

but

you

want to use the message modification internface, implemented via
mmexternal. See these links:

https://github.com/rsyslog/rsyslog/blob/master/plugins/
external/INTERFACE.md


 http://www.rsyslog.com/doc/master/configuration/modules/

mmexternal.html

HTH
Rainer


 Thanks


On Mon, Jul 21, 2014 at 2:20 PM, David Lang <[email protected]>
wrote:

 On Mon, 21 Jul 2014, Zeshan wrote:

 Thanks David,


 It worked now with the following lines in my rsyslog.conf


$ModLoad omprog
$actionomprogbinary /var/log/test.sh
*.* :omprog:;RSYSLOG_TraditionalFileFormat


 that's the legacy format, the new format I gave below does the

same

 thing,


 but is clearer in many ways. It allos allows you to give

parameters

to

 your


 program (the $actionomprogbinary value can't have any spaces in

it)

now that you have this working, you can change the format of data
passed
to your script by using a different template, and change ot

program

 itself


 to any other language.


David Lang


 I was doing exactly the same except that the script was in
/opt/test.sh

 and

now it is in /var/log/test.sh

Thanks for all the kindness


On Mon, Jul 21, 2014 at 1:42 PM, David Lang <[email protected]>

wrote:

 Ok, looking at the documentation

 http://www.rsyslog.com/doc/omprog.html

try

Module (load="omprog")
action(type="omprog" binary="/pathto/test.sh"
template="RSYSLOG_
TraditionalFileFormat")

this will send the script every log entry that arrives on the
rsyslog
server

if you run this with the -dn flags, we should see test.sh and

the

 prog

output module show up with the first log entry that it

processes.

David Lang



On Mon, 21 Jul 2014, Zeshan wrote:

 Thanks David for your reply


 ok , i have script named test.sh having following code


#!/bin/bash
echo "$@" >/var/log/myoutput

and it is is executable.

Now what should me rsyslog.conf file. and how to test this

whole

  procedure



On Mon, Jul 21, 2014 at 1:09 PM, Muhammad Asif Ihsan <
[email protected]> wrote:



  David, now I have its binary and what should I code in
rsyslog.conf

 for



  loading and redirecting to my executable myfile.



_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED

by

  myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT

POST

if

you

  DON'T LIKE THAT.

  _______________________________________________


 rsyslog mailing list

 http://lists.adiscon.net/mailman/listinfo/rsyslog

http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED

by a

 myriad

  of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT

POST if

you

 DON'T LIKE THAT.

 _______________________________________________

 rsyslog mailing list

 http://lists.adiscon.net/mailman/listinfo/rsyslog

http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by

 myriad

  of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT

POST if

you

 DON'T LIKE THAT.


 _______________________________________________

 rsyslog mailing list

http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST

if

you

DON'T LIKE THAT.

 _______________________________________________

 rsyslog mailing list

http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if

you

 DON'T LIKE THAT.


 _______________________________________________

 rsyslog mailing list

http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if

you

 DON'T LIKE THAT.


 _______________________________________________

 rsyslog mailing list

http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if

you

 DON'T LIKE THAT.


 _______________________________________________

 rsyslog mailing list

http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if

you

 DON'T LIKE THAT.


 _______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you

DON'T LIKE THAT.

 _______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you

DON'T LIKE THAT.

 _______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

 _______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

 _______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

 _______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Want to run python code using omprog on rsyslog 7.6.3

Reply via email to