First, you should really read this: http://aphyr.com/posts/317-call-me-maybe-elasticsearch regarding using ES as a single source of truth
Millions of indexes is not advisable, unless you plan on having millions of servers. Depending on index size and write frequency to them, you don't want to have more than a few dozen indexes per machine (including replicas). This is because of concerns of memory, CPU, I/O and file descriptors. One big single index may present its own problems due to the different schemas, although it may be solvable using dynamic index templates <http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html#dynamic-templates>. I will still expect you to have issues with number of shards (basically, running out of shards at some point). Therefore I will try and find a middle way here, using probably some sort of a mapping mechanism. Even also time based if its applicable. Re your questions: * are there production deployments out there that have a million active indexes? what do they look like? I'm not aware of such * how many different fields does it make sense to host in a single index? would it scale to millions of fields in a single index? You mean in a single document. I recall seeing Shay suggesting not to go over the 100 threshold or so. Lucene really isn't optimized for scaling vertically, especially in the document level. * are there other ways to go about this that we have overlooked? Maybe look at your data model and try to re-arrange it. -- Itamar Syn-Hershko http://code972.com | @synhershko <https://twitter.com/synhershko> Freelance Developer & Consultant Author of RavenDB in Action <http://manning.com/synhershko/> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsNarKqML9q4E%3DvdrDp80mC_rS5VXfJWkT2%3D7WghzwACg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
