I've had loads of issues with hard to reproduce out of memory errors using es.index.store.type=memory. It seems like a nice idea but it's rather fragile. We were using this for tests. In the end after some feedback in this mailinglist I made our tests create a random directory and used a file based approach with that.
In any case, you should probably be using bulkindexing instead of feeding elasticsearch single document updates. When bulkindexing, elasticsearch scales a lot better. Alternatively, you might want to use something like logstash in combination with redis to do this for you. You then simply write your new documents to redis and have logstash index what gets written there. IMHO, support for es.index.store.type=memory should probably be removed or at least strongly discouraged unless it can be properly stabilized. Part of the problem is that the memory for in memory indices lives outside the normal java heap (direct memory) and things get quite hairy when you run out of that. Jilles On Tuesday, April 22, 2014 11:09:16 PM UTC+2, John Smith wrote: > > Hi I downloaded the latest ES 1.1.1 > > I have a 200GB RAM with 2 x 8 cores hyper threaded. "32" cores total > machine 1.6T of disk space. > > I start elastic search as follows... > > ./elasticsearch -Xms100g -Xmx100g -Des.index.store.type=memory > Using Java 1.7_51 > > I then create my index as follows... > > $ curl -XPUT http://localhost:9200/myindex/ -d \ > ' > index : > store: > type: memory > ' > > And my Java web app (Using vertx.io) > > // On app startup... Ensure we have one instance of client. Regardless how > many app threads may write to the index. > synchronized(clientCreated) { > if(clientCreated.compareAndSet(false, true)) { > node = nodeBuilder().clusterName("elasticsearch").client(true).node(); > client = node.client(); > } > } > > > // Per request coming into my web application. Using vertx for the web > framework. > // For each request we use the one client instance. > client.prepareIndex("myindex", "doc", request.getString("id")) > .setSource(bodyStr) // Already sending Json so no need to convert > it! > .execute(new ActionListener<IndexResponse>(){ > > @Override > public void onFailure(Throwable t) { > req.response().end("Error: " + t.toString()); > } > > @Override > public void onResponse(IndexResponse res) { > req.response().end(res.getIndex()); > }}); > > > Both the webapp and ES running on same server. So all write/read requests > are localhost. > > Testing as follows > > JMeter (100 users, running on my desktop) ------ Remote ----> WebApp ----- > localhost ----> ES > > I get about 6000 writes/sec and it seems to get lower as the number of > docs that get indexed increases. > Average request/response latency is about 15-20ms. > Network Time/Jmeter data generation( Each document is about 1000 > bytes)/web app is about 5 ms. I know this because I also have a simple > hello world response to test the average latency of those 3 "parameters". > So it seems that in-memory takes average 15ms I would think ES can do much > better then that? > > Is there any tuning settings I can try for strictly in-memory index? > > Thanks > > > > > > > > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8df90b9a-ebe0-422e-a7a9-f046833fd18c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
