Hello , My advice would be to keep all the logs in a single index , but apply index tailing. That is write logs of a day or hour ( depending upon traffic) to each index like logstash does. So name of the index would be of format logs-`yyyy-MM-dd` This way , you wont be stuck with the fixed shard problem and dynamic horizontal scaling can be achieved. Also , it would be a wise idea to remove old logs using TTL facility OR closing old index or even take a snapshot and remove the index.
TTL - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-ttl Index Close - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-open-close.html#indices-open-close Thanks Vineeth On Thu, Sep 11, 2014 at 7:39 AM, Konstantin Erman <[email protected]> wrote: > We use Elasticsearch to aggregate several types of logs - web server logs, > application logs, windows event logs, statistics, etc. > > As far as I understand I can do one of the following: > 1, Send each log to its own index and when I need to combine them in query > - specify several indices in Kibana settings; > 2. Send all logs to the same index (we turn them over every day) and give > logs from different sources different document types; > 3. Do more or less nothing, push all documents together without > distinguishing them explicitly; > > My question is - what are advantages and disadvantages of each approach? > We have substantial amount of logs going in every second, but querying is > rather rare, at least so far. > > Thank you! > Konstantin > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kmDEJ%2BmhfX8RtGm9KAiBKEK%3DT1-1r3kj7pCnNNwMY-PA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
