If you are doing indexing intensively, maybe merge throttling[1] is set to a too low value and background merging cannot keep up with the new segments that are created? Can you check how many segments you have in your index on the moment when you stop indexing?
[1] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html#store-throttling On Thu, Feb 13, 2014 at 6:13 PM, Ankush Jhalani <[email protected]>wrote: > If it helps, following is jmap memory summary and object heap histogram ( > https://gist.github.com/ajhalani/8977548) > > On Thursday, February 13, 2014 10:19:28 AM UTC-5, Ankush Jhalani wrote: > >> We have a single node, 12GB, 16 core ES instance to which we are 12 >> threads bulk indexing into a 12shard index. Each thread sends a request of >> size kb to couple megabytes. The thread bulk queue_size is increased from >> default 50 to 100. >> >> With v0.90.11, we are noticing that the jvm memory usage keeps growing >> slowly and doesn't go down, gc runs frequently but doesn't free up much >> memory. From debug logs, it seems the segment merges are happening. However >> even after we stop indexing, for many hours the instance is busy doing >> segment merges. Sample gist from hot threads I ran couple minutes apart - ( >> https://gist.github.com/ajhalani/8976792). Even after 16 hours and >> little use on the machine, the jvm memory usage is about 80% (CMS should >> run at 75%) and nodes stats show is running very frequently. >> >> If we don't stop indexing, eventually after 60-70GB indexing the instance >> goes out of memory. This seems like a memory leak, we didn't face this >> issue with 0.90.7 (though we were probably using a 6 thread process for >> bulk indexing). >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/3d8c3388-9932-4fbd-a430-e9ee73e613bb%40googlegroups.com > . > > For more options, visit https://groups.google.com/groups/opt_out. > -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j54Nk%2BvT4xGnRy6yFFKzne2vyyazwcJznN%2B9w-yXKkfyA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
