I'm setting up an Elasticsearch-based log cluster and I'm having some doubts about how I should choose the number of indices and shards. By default, Logstash and Kibana use per-day indices and Elasticsearch defaults to five shards per index. I'm worried that this will create an excessive number of shards with a log retension of, say, 100 days. With one replica per shard I'd be facing 1000 shards cluster-wide. With three or four data nodes that's at least 250 shards per node.
Whether this is too much obviously depends on the node and perhaps on the size of the daily indices, but regardless it doesn't seem particularly advantageous with such a number of shards. Would it make more sense to use week-based indices or reduce the number of (primary) shards per index to two or three to get the number of shards per node down towards or below 100? Or should I stop worrying? -- Magnus Bäck | Software Engineer, Development Tools [email protected] | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20140708071656.GA26839%40seldlx20533.corpusers.net. For more options, visit https://groups.google.com/d/optout.
