Disabling refresh (-1) is a good choice if you are fully maximizing your cluster's CPU/IO resources (using enough bulk client threads or async requests). In that case it should give faster indexing throughput than 30s refresh.
But if you are not saturating the cluster's resources, then a refresh interval of 30s may in fact get you faster indexing throughput because refreshes are done with a background thread in ES, so you effectively get one more thread working for you than if you disable refresh which causes the bulk indexing threads to do the flushing. Try both and see and then report back! Mike McCandless http://blog.mikemccandless.com On Tue, Jul 29, 2014 at 3:11 AM, [email protected] < [email protected]> wrote: > There is no more a massive hit when opening an index for read once than at > every 30 seconds. > > The only explanation I can think of is that users perform searches while > indexing and somehow want up-to-date results while they search along. > > This is not the case when I do bulk indexing, search is disabled > completely. So I still recommend disabling refresh_interval in the case > when there is no search activity while bulk indexing. And before search is > opened again, the index is flushed, optimized, and extended to replica > levels as well. > > Jörg > > > On Tue, Jul 29, 2014 at 8:23 AM, Mark Walkom <[email protected]> > wrote: > >> I'd say because if you are inserting a lot of data, you will have a >> massive hit at the end when you need to index, as opposed to smaller ones >> along the way. >> >> Regards, >> Mark Walkom >> >> Infrastructure Engineer >> Campaign Monitor >> email: [email protected] >> web: www.campaignmonitor.com >> >> >> On 29 July 2014 16:20, shikhar <[email protected]> wrote: >> >>> The 1.3.0 release notes state: >>> >>> >>>> - Increase the refresh_interval >>>> >>>> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html#bulk> >>>> if >>>> you are doing heavy bulk indexing, or you are happy with your search >>>> results being refreshed less frequently than once every second. *It >>>> is better to use a refresh interval like “30s” rather than disabling it >>>> completely with “-1″.* >>>> >>>> I'd love to get an explanation on why 30s is better than -1, which is >>> the setting we are using right now when reindexing. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAHWG4DOaGphdmA%3DbQqV-0ic8HXxLM3ZmBzbW9YFDtZ_zWG8BHA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAHWG4DOaGphdmA%3DbQqV-0ic8HXxLM3ZmBzbW9YFDtZ_zWG8BHA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEM624Y6qPWwG5rtqh7V2WaqTTqcWJoRP6%2Bct5CYhGR2xUXPgQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAEM624Y6qPWwG5rtqh7V2WaqTTqcWJoRP6%2Bct5CYhGR2xUXPgQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFEKO7pkn97TY9CbNn3JNb%2B18qVyzaFiXw_rDU8fDg-YQ%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFEKO7pkn97TY9CbNn3JNb%2B18qVyzaFiXw_rDU8fDg-YQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smRfRmRbSYxixHxZBfQ-VKEOwPACc4xFC1tFmNhr7HaRxsQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
