On Fri, 25 Jul 2014, Zeshan wrote:
Hi,
we have the scenario in which we want to parse the logs received at our
rsyslog server and store them in DB say MongoDB. e.g. its squid log
"Jun 12 22:02:17 172.20.8.4 (squid): 1402544268.502 80990 172.20.9.78
TCP_MISS/200 54721 GET http://img.objectembed.info/intro.swf? - DIRECT/
93.184.220.20 application/x-shockwave-flash"
and i want to extract following fields from the above log
time stamp : Jun 12 22:02:17
host : 172.20.8.4
type : squid
device_event_id : TCP_MISS/200
url: http://img.objectembed.info/intro.swf?
this is trivial, you already have timestamp host and programname parsed, so all
you need to extract is device_event_id and url, and those are fields 4 and 6
with a space delimiter.
2ndly i have another device apache with the following log
172.20.16.37 - - [11/Jun/2014:10:38:26 +0500] "GET /nagios HTTP/1.1" 404
281 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/35.0.1916.114 Safari/537.36"
and the following fields can be extracted
host :
time stamp:
OS:
etc etc
this is messier because there are a lot of optional spaces inside the fields,
and so you need to take into account quotes. I actually recommend creating a
custom format that is easier to parse rather than trying to use the default log
format.
fields to be extracted will be different based on different logs and the
list goes on for log devices .
I can write regex for all the devices and used them in logstash.
My question is what would be the right approach to parse these fields in
rsyslog level. will it be right to write python files parsing these logs
and calling them through mmexternal or i should go through these docs
http://www.rsyslog.com/doc/syslog_parsing.html
and http://www.rsyslog.com/doc/messageparser.html
i have to support at least 200 device logs , have to parse them and store
them in database say MongoDB .
There are many ways you can do this job.
you can write a lot of regex expressions in rsyslog
you can write a lot of regex expressions in python, have rsyslog serialize the
logs and send them to your python program, have your python program parse them,
serialize them to send back to rsyslog, and have rsyslog parse them (mmexternal)
but the best way to do this would be to write some rules using mmnormalize
http://www.rsyslog.com/doc/mmnormalize.html and have it parse the logs
David Lang
On Tue, Jul 22, 2014 at 11:45 AM, Rainer Gerhards <[email protected]>
wrote:
As I said, omprog is for feeding output data to an external plugin. In
contrast, mmexternal is to call an external message modification plugin. So
If you want to make modifications to the message, you need to call
mmexternal. I highly suggest eading the links I have posted.
Rainer
On Tue, Jul 22, 2014 at 8:23 AM, Zeshan <[email protected]> wrote:
David , can you tell me the difference bet omprog and mmexternal. both
can
be used to call the external plugin? my purpose is two feed raw log/msg
to some external plugin and do the parsing? which one will do the
purpose
for me?
On Tue, Jul 22, 2014 at 6:22 AM, David Lang <[email protected]> wrote:
In any language, there is a huge variation in the performance of things
like this, depending on implementation details, you will have to try it
and
see.
David Lang
On Mon, 21 Jul 2014, Zeshan wrote:
we were using ruby files in case of logstash for normalization , now
we
want to reuse them in rsyslog , by calling them through omprog and
writing
the data to mongoDB.
On Mon, Jul 21, 2014 at 4:35 PM, David Lang <[email protected]> wrote:
It probably depends mostly on what code you are using for your
normalization.
David Lang
On Mon, 21 Jul 2014, Rainer Gerhards wrote:
On Mon, Jul 21, 2014 at 11:43 AM, masoom alam <
[email protected]>
wrote:
David:
How the performance will affected in the case where we want to pass
the
control to do some normalization - name,value pair conversion by
Rsyslog.
We want to get rid of Logstash.
depending on what you do it's "not too bad", but of course it
depends
on
the circumstances. However, you do not want to use output modules
but
you
want to use the message modification internface, implemented via
mmexternal. See these links:
https://github.com/rsyslog/rsyslog/blob/master/plugins/
external/INTERFACE.md
http://www.rsyslog.com/doc/master/configuration/modules/mmexternal.html
HTH
Rainer
Thanks
On Mon, Jul 21, 2014 at 2:20 PM, David Lang <[email protected]> wrote:
On Mon, 21 Jul 2014, Zeshan wrote:
Thanks David,
It worked now with the following lines in my rsyslog.conf
$ModLoad omprog
$actionomprogbinary /var/log/test.sh
*.* :omprog:;RSYSLOG_TraditionalFileFormat
that's the legacy format, the new format I gave below does the
same
thing,
but is clearer in many ways. It allos allows you to give
parameters
to
your
program (the $actionomprogbinary value can't have any spaces in
it)
now that you have this working, you can change the format of data
passed
to your script by using a different template, and change ot
program
itself
to any other language.
David Lang
I was doing exactly the same except that the script was in
/opt/test.sh
and
now it is in /var/log/test.sh
Thanks for all the kindness
On Mon, Jul 21, 2014 at 1:42 PM, David Lang <[email protected]>
wrote:
Ok, looking at the documentation
http://www.rsyslog.com/doc/omprog.html
try
Module (load="omprog")
action(type="omprog" binary="/pathto/test.sh" template="RSYSLOG_
TraditionalFileFormat")
this will send the script every log entry that arrives on the
rsyslog
server
if you run this with the -dn flags, we should see test.sh and
the
prog
output module show up with the first log entry that it
processes.
David Lang
On Mon, 21 Jul 2014, Zeshan wrote:
Thanks David for your reply
ok , i have script named test.sh having following code
#!/bin/bash
echo "$@" >/var/log/myoutput
and it is is executable.
Now what should me rsyslog.conf file. and how to test this
whole
procedure
On Mon, Jul 21, 2014 at 1:09 PM, Muhammad Asif Ihsan <
[email protected]> wrote:
David, now I have its binary and what should I code in
rsyslog.conf
for
loading and redirecting to my executable myfile.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by
a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
