Hmm maybe your nodes are IO bound? What IO system are you using? You should not need to increase the default bulk thread pool size, and if you are using default 5 shards then each single bulk request to one index is done concurrently 5X so you only need enough concurrent bulk requests to saturate the number of CPUs, e.g. 40 / 5 = 8 concurrent bulk indexing clients.
Mike McCandless On Mon, Apr 13, 2015 at 7:41 AM, Hajime <[email protected]> wrote: > Hi Mike, > > I guess you are right.Cpu usage was about 10~20% (of 40 cpu cores) and > load average was about 10.I might well as check out with the condition > using about 50 ~70% of Cpu while maintaining the lower load average. > > However,do you have any idea how can I use more Cpu when indexing?I'm > increasing the input amount but Cpu usage remains almost same (and the > processing speed seems almost same). > Should I configure something like "*bulk.thread_pool*" size or > "indices.memory.max_shard_index_buffer_size" > ( > https://github.com/elastic/elasticsearch/blob/97559c0614d900a682d01afc241615cf5627fb4c/src/main/java/org/elasticsearch/indices/memory/IndexingMemoryController.java#L96 > )? > > > On Mon, Apr 13, 2015 at 6:00 PM, Michael McCandless <[email protected]> > wrote: > >> You should see better performance with -1 refresh_interval, because >> Lucene will flush larger, single segments, causing less merging pressure. >> >> Are both of your tests (-1 vs 10s) fully saturating CPU and/or IO on your >> nodes? >> >> If not, then that can explain it: when you have 10s refresh_interval, a >> separate thread (refresh thread) bears the cost of moving the new segments >> to disk, but with -1, the bulk index threads themselves bear the cost. >> >> But if you test with enough client-side concurrency to saturate your >> resources you should see the opposite (-1 refresh_interval is faster >> indexing throughput). >> >> Mike McCandless >> >> On Mon, Apr 13, 2015 at 4:05 AM, Hajime <[email protected]> >> wrote: >> >>> Hi, >>> >>> I'm trying to improve the indexing performances.I follow these >>> instructions >>> >>> https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing >>> >>> http://www.elastic.co/guide/en/elasticsearch/guide/master/indexing-performance.html >>> >>> and created 20 different indexes by changing >>> translog,merge,refresh...etc while "*number_of_shards:1,number_of_replica:0" >>> part is common.* >>> I found that the best performed index was* actually **default settings >>> + refresh_interval:"10s"(not **refresh_interval:"-1") *in terms of doc >>> count accuracy or load average or lesser bulk queue*.* >>> >>> Can anyone explain why this settings is the best? >>> >>> In addition,I can observe when refresh thread is active,the segments >>> count decrease.What is exactly the refresh thread doing? >>> >>> thanks >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsrZNVJtOj__YO6enFqKDt4T1Hxi_pT94W9YQx7bNe%3Dg1g%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsrZNVJtOj__YO6enFqKDt4T1Hxi_pT94W9YQx7bNe%3Dg1g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAKHUQPiEsvcVdYdgYLK2PqkA7L-VGngTh-7kMDBXccz%3DkvD%2BjQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAKHUQPiEsvcVdYdgYLK2PqkA7L-VGngTh-7kMDBXccz%3DkvD%2BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsqnykKeMnwotNVkJL_R8XMe88t6sYMc0yM0c744RpDbsQ%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsqnykKeMnwotNVkJL_R8XMe88t6sYMc0yM0c744RpDbsQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKHUQPiQQmhVEyuyJOyE2oc37V%2BARVHi9aFM7f-uqct4VKC92w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
