With ES, you can go up to the bandwidth limit the OS allows for writing I/O (if you disable throttling etc.)
This means, if you write to one shard, it can be as fast as writing to thousands of shards in parallel in summary. There is an OS limit for file system buffers so the more shards, the more RAM is recommended. If OS restricts file descriptor limits and you want to write to shards, you can estimate you need a peak of 100-200 file descriptors for active merges etc. (this varies from version to version and from setting to setting). ES does not impose a limit here. These resource demands and custom configurations are not related to the choice SSD or HDD. Jörg On Wed, Oct 8, 2014 at 1:30 AM, Kevin Burton <[email protected]> wrote: > Perhaps. I just assumed that ES would keep each shard in its own lucene > index on disk. These files aren't free and mean that you have to fsync > them (eventually) and those requires seeks to write the data. > > So more shards , should, in theory mean more seeks. > > Else you would just create 1B shards. :-P > > On Tuesday, October 7, 2014 12:32:16 PM UTC-7, Jörg Prante wrote: >> >> The number of shards is in not related to HDD or SSD. >> >> SSD reduces vastly the latency of I/O operations, ES must no longer wait >> for I/O seeks, reads, or writes, so the overall throughput is significantly >> faster, which is noticeable while indexing bulks of data. >> >> Jörg >> >> On Tue, Oct 7, 2014 at 9:07 PM, Kevin Burton <[email protected]> wrote: >> >>> I'm curious how many shards would be too many shards per server... and >>> wanted feedback based on community experience. >>> >>> On HDD you probably want to be a big conservative.... but on SSD I think >>> you could probably get away with a higher number. >>> >>> 10 seems to be somewhat reasonable I imagine. Not perfect but not bad >>> either. This way you could grow 10x... >>> >>> PS. Really a shame that ES can't do shard splitting yet. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/8c22ef66-5794-4c74-9410-5d83d3a1ee5d% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/8c22ef66-5794-4c74-9410-5d83d3a1ee5d%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/b7160306-0ce1-4e13-8523-93f6735a1fad%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/b7160306-0ce1-4e13-8523-93f6735a1fad%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFAgKVgd%2BgZsmudCBZGk9nY9g_0XZy%2BnhRL%2BsVzxP_-pg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
