Thanks a lot Radu for helping me with this and for being patient with me.
I created the following template in rsyslog.conf
$template apsimTemplate,"\n{\n\"message\":\"%msg:::json%\",\n
\"ip\":\"%msg:F,44:1%\",\n \"name\":\"%msg:F,44:2%\"\n}"
*.* action(type="omelasticsearch" template="apsimTemplate"
server="10.16.131.8" serverport="9200")
and with a syslog entries created as
void main()
{
int i = 0;
char * string = "{\n \"name\":\"joys of programming\" \n }";
setlogmask (LOG_UPTO (LOG_NOTICE));
openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON);
perror("openlog");
for(i = 0; i < 5; i++) {
syslog (LOG_NOTICE, "2.2.0.%d,apsim_00,this is a test %d",
i);
}
closelog ();
}
I do get some filtered documents however, I am not able to filter it out.
(I get all the records with the query)
Is the hierarchy a mandatory thing to get filters correctly?
[root@localhost rsyslog]# curl -XGET
10.16.131.8:9200/system/events/_search?pretty=true,q="ip":"2.2.0.1"
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 5,
"max_score" : 1.0,
"hits" : [ {
"_index" : "system",
"_type" : "events",
"_id" : "D43JsNcTRkyxjoCSRkKbxQ",
"_score" : 1.0, "_source" :
{
"message":" 2.2.0.4,apsim_00,this is a test 764997288",
"ip":" 2.2.0.4",
"name":"apsim_00"
}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "0GP7SgtPQ0mxokvhwWgC4g",
"_score" : 1.0, "_source" :
{
"message":" 2.2.0.1,apsim_00,this is a test 764997288",
"ip":" 2.2.0.1",
"name":"apsim_00"
}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "3I8kFSCVQD-Kh2ow-uu4Dw",
"_score" : 1.0, "_source" :
{
"message":" 2.2.0.3,apsim_00,this is a test 764997288",
"ip":" 2.2.0.3",
"name":"apsim_00"
}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "yLs8_GLPQsWiYKlZP9n3fw",
"_score" : 1.0, "_source" :
{
"message":" 2.2.0.0,apsim_00,this is a test 0",
"ip":" 2.2.0.0",
"name":"apsim_00"
}
}, {
"_index" : "system",
"_type" : "events",
"_id" : "LXbayzZaQGuwCuPMZ2ZiAQ",
"_score" : 1.0, "_source" :
{
"message":" 2.2.0.2,apsim_00,this is a test 764997288",
"ip":" 2.2.0.2",
"name":"apsim_00"
}
} ]
}
}
On Wed, Jun 19, 2013 at 6:25 PM, Radu Gheorghe <[email protected]>wrote:
> 2013/6/19 Mahesh V <[email protected]>
>
> > Hi ,
> >
> > I now understand what you are saying.
> >
> > My requirement is something like this
> >
> > 1) I will have as many as 16 processes and each process has 23 threads
> > sending syslog.
> > 2) My current architecture for logs is file based where I separate each
> > process log to a different file.
> > 3) I did the same thing with standalone mysql /sqlite where I had a
> single
> > database and multiple tables for each process.
> > 4) I start all processes one by one they run for a few hours and stop
> them
> > all at once and collect/examine the logs.
> > 5) I wont need to store the logs permanently, so I can go ahead and
> delete
> > them once the analysis is over.
> >
> > Is it possible in rsyslog to
> > 1) create dynamic indexes for each of the process based on name or time
> (I
> > think you mentioned that it is possible using time)
> >
>
> Yes, it's up to you to make a template that works. You can put time in
> there, program name, or anything from your log.
>
>
> > 2) Query indexes separately for values (e.g. 1 process may have logs from
> > ip addresses 1.1.1.1 to 1.1.1.10 and other process may have logs from
> > 1.1.1.11 to 1.1.1.20 etc)
> >
>
> Yes, when querying you can specify one index or multiple indices (even use
> wildcards). But in your case it might be better to just throw everything in
> a single index and just add filters by IP addresses, processes, etc.)
>
>
> >
> > So my index should look something like this
> >
> > "process1"
> > "ip" : "x.x.x.x"
> > "name": "abcd"
> > "log": "test log"
> >
>
> You can start by putting all that info in your log. No need to be
> hierarchical about it, unless you find later that you need to. Then you can
> filter by any of those fields.
>
>
> >
> > I have been trying out too many things in very little time and hence a
> bit
> > confused. Sorry If I am asking stupid questions.
> >
>
> Heh, I'm admiring you for ingesting all this info in this short time :)
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
