I had a project with the same context. We decided to increase the # of shards as it was impossible to have one index for each customer. Another approach is to have only some customers (hardcoded) separated from the rest. If you can, in advance, detect this users it might be a good idea and then have a "Rest of the world" index for non important ones.
Also when we increased the # of shards, we incremented the amount of servers but with smaller ones, that improved a lot our failure resiliency. Hope that helps. On Friday, March 20, 2015 at 5:28:55 PM UTC+1, Vladi Feigin wrote: > > Hello, > > Please share your thoughts > We have one big ES index and 18 shards (9 primary and 9 replicas) > We have thousands of customers and each customer could have millions or as > opposite very small number of documents > We never search across all customers but within a specific customer. In > other words all our queries have a customer id filter. > The big disadvantage of having one big index is we always search the data > of all customers rather than looking in one customer > Obviously it hurts our queries performance. > We're thinking to create multiple indexes : an index per customer. But in > our case it means having hundreds or maybe thousands indexes > In terms of the maintenance is a big overhead > Other approach is create many shards > Could you, please share your experience and thoughts? > What would you recommend in this scenario > Thank you in advance, > Vladi Feigin > > > > > > This message may contain confidential and/or privileged information. > If you are not the addressee or authorized to receive this on behalf of > the addressee you must not use, copy, disclose or take action based on this > message or any information herein. > If you have received this message in error, please advise the sender > immediately by reply email and delete this message. Thank you. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/974e95e4-8c25-4500-8823-853806bd5cbb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
