High sustainable bulk indexing is very stable here. I have 3x HP DL165 G7 32 core machines and can index for hours at same speed with this settings
https://gist.github.com/jprante/10666960 Jörg On Wed, Apr 23, 2014 at 5:52 PM, John Smith <[email protected]> wrote: > Ok so I decided to skip in-memory for now just to test bassic > functionality. > > I'm running elastic search with defaults as > > ./elasticsearch -Xms32g -Xmx32g > > I also got bigdesk installed. > > Either I'm not getting something... But why as I write more documents to > the index the > > Indexing requests per second (Δ) > > goes down and the > > Indexing time per second (Δ) > > Is going up > > So basically it's getting slower and slower. > > Is their any sensible tuning parameters. I would expect that the insertion > should be stable and not getting slower. > > It's a 32 core machine and enough ram, standard drives though. There has > to be a way to setup buffers and queues to alliviate the issue of disks > "being slow" > > > On Wednesday, April 23, 2014 10:42:37 AM UTC-4, John Smith wrote: >> >> On a 32 core machine? Plus I think 1.7_51 uses G1 >> >> I have tested another "indexing" api up to 190GB or so with 30,000,000 >> objects and my latency was 3ms overall including network and app logic. >> >> And I haven't tested that many records with elastic search yet ;) >> >> >> On Wednesday, 23 April 2014 10:20:20 UTC-4, Jörg Prante wrote: >>> >>> The ES "memory" or "ram" store (Lucene RAMDirectory) puts enormous >>> pressure on JVM garbage collection. >>> >>> You can not expect that standard JVM with CMS GC can give the best >>> performance. >>> >>> More info in this great article by Mike McCandless >>> >>> http://blog.mikemccandless.com/2012/07/lucene-index-in- >>> ram-with-azuls-zing-jvm.html >>> >>> Maybe Java 8 with G1 GC is giving slightly better numbers. But do not >>> expect too much. >>> >>> Jörg >>> >>> >>> >>> >>> >>> On Wed, Apr 23, 2014 at 3:19 PM, John Smith <[email protected]> wrote: >>> >>>> 1.7_51 but i dont see how their could be a limitation. >>>> >>>> I used java up to 200GB easily and with no issues either... >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/cf8950c9-923e-4cea-90f5-c51a09bf1b16% >>>> 40googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/c6e486c7-6076-4c9e-9073-0f0b808a36e8%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/c6e486c7-6076-4c9e-9073-0f0b808a36e8%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGQVzxFMK7TQuNxUAGte4S61kYgeN%2BmcBxTUcEqraQXQA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
