Hello
My configuration it's 2 master nodes + 6 data nodes. Master nodes have
128GB RAM + 2 CPU/8Core, data nodes 32GB RAM + 2CPU/6Core. I'm testing
migrate data from Oracle to ES. It is about 12TB. Now I have been migrated
~4TB (> 20mld. records, > 1200 indices, > 15000 shards). Queries are
working OK, but after indeksing about 3 TB ES cluster problems ocured
during indexing new data. Data to indexing are prepared like txt file in
form:
{ "index" : { } }
{
"gcr":"val1","a_m":"val1","b_m":"2","x_m":"233","tor_id":"3232","start":"2014-01-17
07:31:44","on":"474","ci":"003","im":"22","ei":"01172037","service":"126","class":"22","sc":"","rc":"498","for":"4"
}
{ "index" : { } }
....
and is send to cluster by curl (curl --silent --show-error --request POST
--data-binary @"$file_name" $host:9200/$idx_name/schema/_bulk)
Each file have 200000 records (after test best value).
Problem appear when I run 2 and more indeksing stream. Some operation has
been finished with timeout error, some didn't indexing all records.
And third problem is that after restart, cluster need couple hours to
allocate shards and setting status to green
I don't know have knowledge what I have to change to solve this problems.
Any body have suggestion?
Regards
Marek
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/59354a68-1ff6-46ac-8d77-01328b353f1b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
