I think you should send less requests within the same bulk. You could try to send 10000 requests and do that 20 times. I bet you’ll get better results.
Not sure that will fix your issue though. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr <https://twitter.com/elasticsearchfr> | @scrutmydocs <https://twitter.com/scrutmydocs> > Le 14 janv. 2015 à 14:32, Marek Dabrowski <[email protected]> a écrit > : > > Hello > > My configuration it's 2 master nodes + 6 data nodes. Master nodes have 128GB > RAM + 2 CPU/8Core, data nodes 32GB RAM + 2CPU/6Core. I'm testing migrate data > from Oracle to ES. It is about 12TB. Now I have been migrated ~4TB (> 20mld. > records, > 1200 indices, > 15000 shards). Queries are working OK, but after > indeksing about 3 TB ES cluster problems ocured during indexing new data. > Data to indexing are prepared like txt file in form: > { "index" : { } } > { > "gcr":"val1","a_m":"val1","b_m":"2","x_m":"233","tor_id":"3232","start":"2014-01-17 > > 07:31:44","on":"474","ci":"003","im":"22","ei":"01172037","service":"126","class":"22","sc":"","rc":"498","for":"4" > } > { "index" : { } } > .... > and is send to cluster by curl (curl --silent --show-error --request POST > --data-binary @"$file_name" $host:9200/$idx_name/schema/_bulk) > Each file have 200000 records (after test best value). > Problem appear when I run 2 and more indeksing stream. Some operation has > been finished with timeout error, some didn't indexing all records. > And third problem is that after restart, cluster need couple hours to > allocate shards and setting status to green > I don't know have knowledge what I have to change to solve this problems. > > Any body have suggestion? > > Regards > Marek > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/59354a68-1ff6-46ac-8d77-01328b353f1b%40googlegroups.com > > <https://groups.google.com/d/msgid/elasticsearch/59354a68-1ff6-46ac-8d77-01328b353f1b%40googlegroups.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/96C066A0-BA5E-4BF5-A292-1E9D9709877E%40pilato.fr. For more options, visit https://groups.google.com/d/optout.
