More thoughts in addition to Nik: - the default setting is refresh by every second. Refresh works very fast when segments are small. If you have more than one shard and use bulk indexing, the segments are small enough for refresh for a longer time. So you will observe a faster bulk indexing, but only for the first 20 minutes or so (longer runs will also show increasing response times). Disabling refresh at bulk indexing time is strongly recommended.
- the default, ES settings are selected for more than one shard (default is 5). To utilize the server resources (CPU, RAM) by a single shard, a little optimization of thread pools and buffer sizes may be required, especially the bulk thread pool and the index buffer size. Jörg On Mon, Apr 14, 2014 at 1:30 AM, Nik Everett <[email protected]> wrote: > Sorry, you can't reduce it. I imagine the performance increase you get is > because the merge logic is per shard so it does less when there are more > shards for the same data. You can likely get similar numbers if you set the > refresh interval to -1 and play with the merge policy before the bulk load. > You'd want to reset it afterwords and then run an optimize. This amounts to > the same thing as starting with more shards and merging them. Mostly. I > think. > > Sent from my iPhone > > On Apr 12, 2014, at 4:05 PM, [email protected] wrote: > > Hi, > > I'm testing on a single node. > > I find I can get better bulk indexing performance when the index has more > shards. Does that make sense ? > > My own theory is that when I have multiple bulk clients, then by > increasing shards the server achieves better concurrency (?) > > So if I increase the shards to say 30, and get a good indexing run... is > it possible to reduce the number of shards subsequently.. or does it matter > if the number remains at say 30? > > Thanks, > > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/d6006698-0ced-43f4-959f-52def820013f%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/d6006698-0ced-43f4-959f-52def820013f%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/0CCF99CF-B24F-4C39-9C12-703039EB5BB1%40gmail.com<https://groups.google.com/d/msgid/elasticsearch/0CCF99CF-B24F-4C39-9C12-703039EB5BB1%40gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFEsXc5BOuC%2B85TwpSz%3DV_9RACUHmUnA269uW-194uPjA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
