On Thu, 11 Jun 2015, Muhammad Asif wrote:
I want to use elasticsearch. I have already taken your so much time.
Apology for that. Actually my goal is just to send the following log in ES.
Actually whole message is saved in msg field but I want it different fields.
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP","dns":{"type":"query","id":19589,"rrname":"
daisy.ubuntu.com","rrtype":"A"}}
I followed many tutorials but get more confuse. Please elaborate these
templates.
template(name="CEETemplate" type="string" string="%TIMESTAMP% %HOSTNAME%
%syslogtag% @cee: %$!all-json%\n")
This template is converting simple log into json format. Adding these
three things "%TIMESTAMP% %HOSTNAME% %syslogtag% and convert it in json.
This is not my requirement. I just want my log fields. Why it add slashes
in message like this.
log
{"timestamp":"2014-12-29T21:01:13.600536","event_type":"dns","src_ip":"172.20.16.93","src_port":49112,"dest_ip":"4.2.2.2","dest_port":53,"proto":"UDP"}
output
Jun 11 11:52:31 new-sr @cee: {
"msg":"{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\","
}
well, in this case, you are adding an extra space between the : of @cee: and the
{ that will start the beginning of the json data. That will cause parsing
errors. If you write the log entry out with the format RSYSLOG_DebugFormat, you
can see the raw log, and the result of parsing. Eliminate that space and look at
the result again.
with the space there, you will see that $! includes one property, "msg" that
containst a bunch of text that would be JSON if it wasn't escaped.
Afterwords, you will see not only the msg property, but also the various things
that are parsed from it.
--------------------------------------------------------------------------------------------------------------------------
What is happening in below template. I used it and take output in file.
It is writing {}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}.
template(name="messageToES" type="list") { property(name="$!all-json") }
try doing
template(name="messageToES" type="subtree" subtree="$!all-json")
I think it will e a lot closer to what you want.
------------------------------------------------------------------------------------------------------------------------
Why we use constant. Is it field name.
template(name="customTemplate"
type="list") {
constant(value="{\"timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"syslogtag\":\"")
property(name="syslogtag" format="json")
constant(value="\",")
property(name="$!all-json" position.from="2")
}
try
template(name="customTemplate" type="list") {
property(name="timereported" dateFormat="rfc3339" format="jsonf"
outname="timestamp")
property(name="syslogtag" format="jsonf" outname="syslogtag")
property(name="$!all-json" format=jsonf)
}
or just
set $!timestamp = $timereported;
set $!syslgtag = $syslogtag
$template customTemplate,"$!"
<snip>
Ok, if this is what you want to output
{
"message":
{
"timestamp":"2014-12-29T21:01:13.600536",
"event_type":"dns",
"src_ip":"172.20.16.93",
"src_port":49112,
"dest_ip":"4.2.2.2",
"dest_port":53,
"proto":"UDP"
}
"msg":"{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\"}",
"fromhost": "test-host",
"facility": "user",
"priority": "info",
"timereported": "2013-03-12T18:05:01.344864+02:00",
"timegenerated": "2013-03-12T18:05:01.344864+02:00"
"src_ip":"172.20.16.93"
"src_port":"49112"
"dest_ip":"4.2.2.2"
"dest_port":"53"
"proto":"UDP"
}
whem you receive the log message
<pri>Jun 11 11:52:31 new-sr something[pid]:
@cee:{"timestamp":"2014-12-29T21:01:13.586962","event_type":"dns","src_ip":"8.8.8.8","src_port":53,"dest_ip":"172.20.16.93","dest_port":49112,"proto":"UDP"}
then what I would do is:
action(type="mmjsonparse" name="jsonparse")
(note 'pri', 'something', 'pid' are placeholders for this example)
This will populate variables (as shown by RSYSLOG_DebugFormat
$FROMHOST: 'new-sr',
$fromhost-ip: 'ipaddress',
$HOSTNAME: 'new-sr',
$PRI: pri,
$syslogtag 'something[pid]:',
$programname: 'something',
$APP-NAME: 'something',
$PROCID: 'pid',
$MSGID: '-',
$TIMESTAMP: 'Jun 11 11:52:31'
$STRUCTURED-DATA: '-',
$msg: '
@cee:{"timestamp":"2014-12-29T21:01:13.586962","event_type":"dns","src_ip":"8.8.8.8","src_port":53,"dest_ip":"172.20.16.93","dest_port":49112,"proto":"UDP"}',
escaped msg: '
@cee:{\"timestamp\":\"2014-12-29T21:01:13.586962\",\"event_type\":\"dns\",\"src_ip\":\"8.8.8.8\",\"src_port\":53,\"dest_ip\":\"172.20.16.93\",\"dest_port\":49112,\"proto\":\"UDP\"}',
$inputname: imtcp,
$rawmsg: '<pri>Jun 11 11:52:31 new-sr something[pid]:
@cee:{"timestamp":"2014-12-29T21:01:13.586962","event_type":"dns","src_ip":"8.8.8.8","src_port":53,"dest_ip":"172.20.16.93","dest_port":49112,"proto":"UDP"}',
$!:{
"timestamp":"2014-12-29T21:01:13.586962",
"event_type":"dns",
"src_ip":"8.8.8.8",
"src_port":53,
"dest_ip":"172.20.16.93",
"dest_port":49112,
"proto":"UDP"
}
$.:
$\:
(note, this is not quite all properties, see
http://www.rsyslog.com/doc/v8-stable/configuration/properties.html for the full
list)
to get from here to the format that you want you will need to do some
manipulation.
The brute force way is to use a set for every line, but let's use a template to
make it a bit easier
templete(name="test" type="list")
{
property(outname="message" name="$!" format="jsonf"),
property(name="msg: format=jsonf),
property(name="fromhost" format=jsonf"),
property(outname="facility" name="syslogfacility-text" format="jsonf"),
property(outname="priority" name="syslogseverity-text" format="jsonf"),
property(name="timereported" format="jsonf"),
property(name="timegenerated" format="jsonf"),
property(name="$!" format="json")
}
unfortunantly there is a bug that prevents you from just using $! in a set or I
would do
set $!messaage = $!;
set $!fromhost = $fromhost;
set $!facility = $syslogfacility-text;
set $!priority = $syslogseverity-text;
set $!timereported = $timereported;
set $!timegenerated = $timegenerated;
and then my template would be
$template test,"$!"
because all the variables would be set properly as a subset of $!
David Lang
action(
type="omelasticsearch"
template="mytemplate"
server="127.0.0.1"
serverport="9200"
searchType="syslogapp"
searchIndex="myindex"
bulkmode="on"
queue.dequeuebatchsize="1000"
action.resumeretrycount="-1"
)
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
