I want to use elasticsearch. I have already taken your so much time.
Apology for that. Actually my goal is just to send the following log in ES.
Actually whole message is saved in msg field but I want it different fields.
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP","dns":{"type":"query","id":19589,"rrname":"
daisy.ubuntu.com","rrtype":"A"}}
I followed many tutorials but get more confuse. Please elaborate these
templates.
template(name="CEETemplate" type="string" string="%TIMESTAMP% %HOSTNAME%
%syslogtag% @cee: %$!all-json%\n")
This template is converting simple log into json format. Adding these
three things "%TIMESTAMP% %HOSTNAME% %syslogtag% and convert it in json.
This is not my requirement. I just want my log fields. Why it add slashes
in message like this.
log
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP"}
output
Jun 11 11:52:31 new-sr @cee: {
"msg":"{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\","
}
--------------------------------------------------------------------------------------------------------------------------
What is happening in below template. I used it and take output in file.
It is writing {}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}.
template(name="messageToES" type="list") { property(name="$!all-json") }
------------------------------------------------------------------------------------------------------------------------
Why we use constant. Is it field name.
template(name="customTemplate"
type="list") {
constant(value="{\"timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"syslogtag\":\"")
property(name="syslogtag" format="json")
constant(value="\",")
property(name="$!all-json" position.from="2")
}
$template JSONDefault,
"{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}"
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP"}
template(name="custom"
type="list") {
constant(value="{\"timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"syslogtag\":\"")
property(name="syslogtag" format="json")
constant(value="\",")
property(name="$!all-json" position.from="2")
}
All these templates are add some rsyslog properties like %msg:::json%
%syslogtag% etc. how can I add my own fields which are in log. Which
template is near to my requirement. I want below form of output in ES.
log:
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP"}
Jun 11 11:52:31 new-sr @cee: {
"msg":"{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\","
}
output
{
"message":
"{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP"}
Jun 11 11:52:31 new-sr @cee: {
"msg":"{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\","
}",
"fromhost": "test-host",
"facility": "user",
"priority": "info",
"timereported": "2013-03-12T18:05:01.344864+02:00",
"timegenerated": "2013-03-12T18:05:01.344864+02:00"
"src_ip":"172.20.16.93"
"src_port":"49112"
"dest_ip":"4.2.2.2"
"dest_port":"53"
"proto":"UDP"
}
action(
type="omelasticsearch"
template="mytemplate"
server="127.0.0.1"
serverport="9200"
searchType="syslogapp"
searchIndex="myindex"
bulkmode="on"
queue.dequeuebatchsize="1000"
action.resumeretrycount="-1"
)
On Thu, Jun 11, 2015 at 12:08 PM, David Lang <[email protected]> wrote:
> On Thu, 11 Jun 2015, Muhammad Asif wrote:
>
> 1- I have installed mmjasonparse, mmnormalize, omelasticsearch by
>> following
>> command.
>> sudo apt-get install rsyslog-mmjsonparse,
>> rsyslog-omelasticsearch,rsyslog-mmnormalize
>> Still I need to compile rsyslog as ./configure --prefix=/usr
>> --enable-imtcp --enable-mmjsonparse --enable-ommongodb ? or not.
>>
>
> you shouldn't mix a self-compiled version with distro packages, do one or
> the other.
>
> do you need mongodb? you don't mention it anywhere else? If you need
> mmnormalize, omelasticsearch, and other things, you would need to include
> them as well
>
>
> 2- You mean the following set of instructions help me. Things on left side
>> are items in log and things on right side are fields in elasticsearch
>> against which these values will be saved. Am I on right track?
>>
>
> the variable name is what will show up as the fieldname in elasticsearch,
> you are setting the variable equal to the value on the right.
>
> But you are saying elasticsearch, but invoking mongodb, which are you
> trying to do?
>
> you also don't show us the definition of the template that you are using.
>
> David Lang
>
> ruleset(name="mongodb") {
>> action(type="mmjsonparse")
>> if $parsesuccess == "OK" then { # How this condition will
>> be OK, mean what will be checked.
>> set $!time = $timestamp;
>> set $!sys = $hostname;
>> set $!procid = $syslogtag;
>> set $!syslog_fac = $syslogfacility;
>> set $!syslog_sever = $syslogpriority;
>> set $!pid = $procid;
>> action(type="ommongodb" server="127.0.0.1" db="logs"
>> collection="syslog" template="mongodball")
>> }
>> }
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
