Adrien, Thanks so much for the response. It was very helpful. I will check out those links on capacity planning for sure.
One followup question. You mention that tens of shards per node would be ok. Are you meaning tens of shards from tens of indexes? Or tens of shards for a single index? Right now I have two servers configured with the index getting 2 shards (one per server), and 1 replica (per server). Chris On Fri, Aug 22, 2014 at 5:58 PM, Adrien Grand < [email protected]> wrote: > Hi Chris, > > Usually, the problem is not that much in terms of indices but shards, > which are the physical units of data storage (an index being a logical view > over several shards). > > Something to beware of is that shards typically have some constant > overhead (disk space, file descriptors, memory usage) that does not depend > on the amount of data that they store. Although it would be ok to have up > to a few tens of shards per nodes, you should avoid to have eg. thousands > of shards per node. > > if you plan on always adding a filter for a specific application in your > search requests, then splitting by application makes sense since this will > make the filter useless at search time, you will just need to query the > application-specific index. On the other hand if you don't filter by > application, then splitting data by yourself into smaller indices would be > pretty equivalent to storing everything in a single index with a higher > number of shards. > > You might want to check out the following resources that talk about > capacity planning: > - http://www.elasticsearch.org/videos/big-data-search-and-analytics/ > - > http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/capacity-planning.html > > > > On Fri, Aug 22, 2014 at 9:08 PM, Chris Neal <[email protected]> > wrote: > >> Hi all, >> >> As the subject says, I'm wondering about index size vs. number of indexes. >> >> I'm indexing many application log files, currently with an index by day >> for all logs, which will make a very large index. For just a few >> applications in Development, the index is 55GB a day (across 2 servers). >> In prod with all applications, it will be "much more than that". 1TB a >> day maybe? >> >> I'm wondering if there is value in splitting the indexes by day and by >> application, which would produce more indexes per day, but they would be >> smaller, vs. value in having a single, mammoth index by day alone. >> >> Is it just a resource question? If I have enough RAM/disk/CPU to support >> a "mammoth" index, then I'm fine? Or are there other reasons to (or to >> not) split up indexes? >> >> Very much appreciate your time. >> Chris >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Adrien Grand > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAND3Dph9Z1My%2B2%2BQ-NM-sWNn2vT1qktDi6%2BmR-b9rFN-Xc-_pw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
