Thanks for the reply, yes I know I exaggerated a bit there :) but to be honest finding a starting point for my cluster is driving me nuts. And how many shards/index is just not clear. Now I know each shard is a Lucene instance and having many instances running on a single node is a bad practice. But the question that remain is "How many is too many?"
So lets assume that my first index will only support 100 users. Given the same size of data how many shards should I allocate for that index? Is there a way to estimate or calculate that? On Thursday, April 23, 2015 at 2:22:23 PM UTC+3, christian...@elasticsearch.com wrote: > > Hi, > > If I have calculated correctly, that corresponds to about 238TB of raw > data. If this is the size of JSON documents being indexed in Elasticsearch, > you will definitely need more than 2 nodes. > > The good thing about using aliases the way David describes is that you > will not need to put all users in the same index as the aliases hides the > underlying index and makes that transparent. You can therefore e.g. put > your first 100 customers in one index and then add new indices as the > number of customers grow. This makes it easier to incrementally handle > growth. > > Best regards, > > Christian > > On Thursday, April 23, 2015 at 8:30:52 AM UTC+1, Zaid Amir wrote: >> >> Hi, >> >> I am trying to figure out the best way to design my ES cluster. Currently >> my search service is subscription based and each user can only search his >> own data. >> >> So looking around I found several examples about users data flow and the >> way of using aliases and its all straight forward. >> >> One thing that I am struggling to understand is the routing setup. Now >> lets assume that I started an index named "accounts" with 100 primary >> shards and 1 replica. Now users started subscribing so I start creating an >> alias per user and routing each alias to a specific shard (1, 2, 3, ..., >> 100). >> >> No if 100 users have already subscribed and a new user comes along, can I >> route the new user to the first shard? Or should I start another index for >> the next 100 users? >> >> My concern with this also is performance. Lets say that both nodes are >> running on a Quad core CPU with 32 GB of ram? Is there a good indicator of >> how many shards I should allocate per index assuming that each document is >> around 512KB in size? >> >> >> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/86b979ac-1e3f-4e3a-991e-32c8f63444cf%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
