Hi ,
I now understand what you are saying.
My requirement is something like this
1) I will have as many as 16 processes and each process has 23 threads
sending syslog.
2) My current architecture for logs is file based where I separate each
process log to a different file.
3) I did the same thing with standalone mysql /sqlite where I had a single
database and multiple tables for each process.
4) I start all processes one by one they run for a few hours and stop them
all at once and collect/examine the logs.
5) I wont need to store the logs permanently, so I can go ahead and delete
them once the analysis is over.
Is it possible in rsyslog to
1) create dynamic indexes for each of the process based on name or time (I
think you mentioned that it is possible using time)
2) Query indexes separately for values (e.g. 1 process may have logs from
ip addresses 1.1.1.1 to 1.1.1.10 and other process may have logs from
1.1.1.11 to 1.1.1.20 etc)
So my index should look something like this
"process1"
"ip" : "x.x.x.x"
"name": "abcd"
"log": "test log"
I have been trying out too many things in very little time and hence a bit
confused. Sorry If I am asking stupid questions.
Thanks
Mahesh
On Wed, Jun 19, 2013 at 12:26 PM, Radu Gheorghe <[email protected]>wrote:
> 2013/6/19 Mahesh V <[email protected]>
>
> > Thanks a lot Radu.
> >
>
> You're welcome :)
>
>
> >
> > Here is what I did using curl command line in linux. (below).
> > However, If I were to do this with syslog, how is it possible?
> >
>
> The curl snippets you pasted are searches. You can't do searches with
> rsyslog. Maybe I'm missing something or there was a copy-paste error.
>
>
> > I mean, I need to create the index (apsim in my case) and type (some ip
> > address index) using rsyslog.
> >
>
> rsyslog can't create indices and types. It just submits new docs, and the
> indices and types are created automatically. You can provide some settings
> for automatically created indices via
> templates<
> http://www.elasticsearch.org/guide/reference/api/admin-indices-templates/>
> .
>
> Another thing you can do is to create your index before starting to index
> data. For example:
>
> curl -XPOST localhost:9200/apsim/ -d '{
> "settings": {
> [your settings go here in this
> format<
> http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings/
> >,
> which can include types and mappings]
> }
> }'
>
> But I'd recommend the template way, because you can use wildcards to apply
> those settings to multiple indices. And it's better to use multiple indices
> (eg: one index per day), and have the same settings. It will make your
> indexing speed better (indexing in a smaller index will be faster because
> there's less merging), and also most of your searches will be faster:
> assuming you often search only in recent data, you can restrict your search
> to only the most recent index (or indices). For example, with daily indices
> you can do something like this to search in the last two days:
>
> curl localhost:9200/2013-06-18,2013-06-19/_search -d '{[your query goes
> here]}'
>
> To make rsyslog send your logs into their respective index (by date), you
> can use dynamic indices. Something like the following should send logs to
> YYYY-MM-DD format indices (excuse the legacy template syntax):
>
> $template srchidx,"%timereported:1:10:date-rfc3339%"
> *.* action(type="omelasticsearch"
> searchIndex="srchidx"
> dynSearchIndex="on"
> [rest of your settings])
>
> If you want more details about index patterns, I'd recommend this
> talk<http://www.elasticsearch.org/videos/big-data-search-and-analytics/>
> .
>
>
> > Will the schema in rsyslog.conf help me create this?
> >
>
> I'm not sure what you mean by "schema in rsyslog.conf".
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
