Hello We are using some other search engine and consider moving to use Elasticsearch. After done quite a lot reading, I am still not quite sure what the optimized way should be in our case, especially after I read that the number of shards can NOT be changed once the index is created.
In our situation, our product is hosted in cloud environment and has rapid growing number of users, and each user is given various disk space(several gigabytes to hundreds gigabytes) to import their datasets. We index these datasets with fixed number of fields and the fields are all the same for some purpose. Each user can only search in their own imported datasets for security reason (segregated). So there is no need to query against the entire index and query time is much more important than indexing time. Our current query time is about 10 to 40 ms. It's very crucial for us how to scale out horizontally smoothly. If everything is added into one index with one type, I worried the index/search will be getting slower and slower with growing of the size of the indices. So I plan to split the indices to speed up query, and here are some options 1. Use one index and create a type for each user such that the query from one user is directly against his own type. But since the number of users can be over million, can elasticsearch be able to handle million types in one index? 2. Group users into different indices such that the index/query can be dispatched to different indices, so a smaller index to query from. But this means our application has to handle the complexity of horizontal scale out. Is any option doable? Any option would you recommend? Besides, could you please tell me how many shards one index should have in best practice? Does too many shards also have performance hit? Many thanks, Cindy -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a2adcd16-1c7b-4e78-a131-d9ae4d61379b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
