I had changed the bulk size and the concurrent bulks, but I had changed the flush interval in the other way round..(/facepalm) I will try it. thank you!! El 19/03/2014 02:23, "[email protected]" <[email protected]> escribió:
> You can change the parameters bulk_size from 100 to a higher value, > max_concurrent_bulk from 1 to a higher value, and disable flush_interval > from 5s to -1 to increase wikipedia river bulk performance. Also make sure > the bzip2 file is downloading quick enough over the wire, which is often > not the case (some KB/sec here). Otherwise download it to the local file > system and change url parameter to this file. > > The cluster stats show the Lucene indexing time. If you want overall > indexing time, you should take measurements at client side. > > Jörg > > > On Tue, Mar 18, 2014 at 10:59 PM, jorge canellas < > [email protected]> wrote: > >> Hi! >> >> That was something that I was thinking since the indexing time shown in >> the indexing statistic is much lower than the real time the process is >> running. Each 5 seconds that the process is running, only a bit more than a >> second is spent indexing. >> I was thinking on parsing the wikipedia dumps manually and then use the >> bulk api to index them. >> Only one question: the indexing time shown in the stats of the cluster >> takes into account the time used to choose the shard, send the document >> through the network, analyze the document and write the terms? or only the >> time spent writting the terms in the index? I am asking this because I am >> interested on the indexing performance of Elasricsearch, and I could use >> this time and forget the time spent by the river parsing the pages of the >> wikipedia. >> El 18/03/2014 20:32, "Ivan Brusic" <[email protected]> escribió: >> >>> It all depends on where the bottleneck is. A river, by design, runs on >>> only one node. Perhaps Elasticsearch is quickly indexing the content, but >>> the Wikipedia ingestion is the slowdown. Is the Wikipedia indexer threaded >>> or are all the bulks being executed on one thread? The BulkProcessor >>> supports multithreading. >>> >>> Have you modified any of the default settings? In particular the merge >>> settings and throttling. Elasticsearch has low defaults for throttling out >>> of the box. If you have fast disks, you can easily raise the values. Are >>> you searching on the index while it is being built? If you are just >>> building the index, you can increase the number of segments and/or reduce >>> the amount of merging done at once. But it all depends on where the >>> bottleneck is. If you disk performance is fine, you might need to look >>> elsewhere. >>> >>> Cheers, >>> >>> Ivan >>> >>> >>> On Tue, Mar 18, 2014 at 10:52 AM, jorge canellas < >>> [email protected]> wrote: >>> >>>> Hi! >>>> >>>> I am trying to index the wikipedia dumps (downloaded and uncompressed) >>>> using the wikipedia river, but it takes about 6 hours in the cluster. >>>> I have increased the number of nodes and primary shards from 5 to 8, >>>> but the performance does not increase. >>>> >>>> - I have set the refresh time to -1, the number of replicas to 0, >>>> increased the amount of memory from 256MB/1GB to 3GB/4GB, allowed the >>>> mlockall, and set the buffer to 30% from 10%. >>>> - I have changed the river settings increasing the bulk_size from >>>> 100 to 10k, refresh interval is set to 1s >>>> - I have changed the mapping to only index title and text. And set >>>> _source to false. >>>> >>>> >>>> I do not know what else can I modify to increase the indexing ratio, at >>>> this moment it is indexing about 120 docs/sec. >>>> >>>> Any ideas? >>>> >>>> Kind regards, >>>> >>>> Jorge >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/c9bd6437-3241-46a4-838e-5327067bc3cc%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/c9bd6437-3241-46a4-838e-5327067bc3cc%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/-DxbnAqNlZU/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQD_ZWNYmqxu-7qNuv%3D6%3Dfm9Yska_vCNReYa8VNG_i1p8Q%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQD_ZWNYmqxu-7qNuv%3D6%3Dfm9Yska_vCNReYa8VNG_i1p8Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEXx9%2BWdibgD8NLt1NUCZWSLo2eLxDxbSeJgOsJgi%3DuRmYOf_w%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAEXx9%2BWdibgD8NLt1NUCZWSLo2eLxDxbSeJgOsJgi%3DuRmYOf_w%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/-DxbnAqNlZU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHFpQG1Lm8LzYT2WahisQG7uh1ke2K%2BtEfQVgeqJNcN0g%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHFpQG1Lm8LzYT2WahisQG7uh1ke2K%2BtEfQVgeqJNcN0g%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEXx9%2BUGwsiVvRQBbRK%3DRV6jF3sfAKkw0GpCbAS-2rXUZR7_Fw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
