Hi J On Wednesday, April 23, 2014 12:04:05 PM UTC-4, Jörg Prante wrote: > > High sustainable bulk indexing is very stable here. > > I have 3x HP DL165 G7 32 core machines and can index for hours at same > speed with this settings > > https://gist.github.com/jprante/10666960 >
At what rate, though? Segment merges will eventually slow you down if you have but a single index, won't they? Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ > > On Wed, Apr 23, 2014 at 5:52 PM, John Smith <[email protected]<javascript:> > > wrote: > >> Ok so I decided to skip in-memory for now just to test bassic >> functionality. >> >> I'm running elastic search with defaults as >> >> ./elasticsearch -Xms32g -Xmx32g >> >> I also got bigdesk installed. >> >> Either I'm not getting something... But why as I write more documents to >> the index the >> >> Indexing requests per second (Δ) >> >> goes down and the >> >> Indexing time per second (Δ) >> >> Is going up >> >> So basically it's getting slower and slower. >> >> Is their any sensible tuning parameters. I would expect that the >> insertion should be stable and not getting slower. >> >> It's a 32 core machine and enough ram, standard drives though. There has >> to be a way to setup buffers and queues to alliviate the issue of disks >> "being slow" >> >> >> On Wednesday, April 23, 2014 10:42:37 AM UTC-4, John Smith wrote: >>> >>> On a 32 core machine? Plus I think 1.7_51 uses G1 >>> >>> I have tested another "indexing" api up to 190GB or so with 30,000,000 >>> objects and my latency was 3ms overall including network and app logic. >>> >>> And I haven't tested that many records with elastic search yet ;) >>> >>> >>> On Wednesday, 23 April 2014 10:20:20 UTC-4, Jörg Prante wrote: >>>> >>>> The ES "memory" or "ram" store (Lucene RAMDirectory) puts enormous >>>> pressure on JVM garbage collection. >>>> >>>> You can not expect that standard JVM with CMS GC can give the best >>>> performance. >>>> >>>> More info in this great article by Mike McCandless >>>> >>>> http://blog.mikemccandless.com/2012/07/lucene-index-in- >>>> ram-with-azuls-zing-jvm.html >>>> >>>> Maybe Java 8 with G1 GC is giving slightly better numbers. But do not >>>> expect too much. >>>> >>>> Jörg >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Apr 23, 2014 at 3:19 PM, John Smith <[email protected]>wrote: >>>> >>>>> 1.7_51 but i dont see how their could be a limitation. >>>>> >>>>> I used java up to 200GB easily and with no issues either... >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/elasticsearch/cf8950c9-923e-4cea-90f5-c51a09bf1b16% >>>>> 40googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/c6e486c7-6076-4c9e-9073-0f0b808a36e8%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/c6e486c7-6076-4c9e-9073-0f0b808a36e8%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b2cbfa21-a87d-4036-aac6-e1e0c69f2409%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
