Cool, curious to see what happens. As an aside, I would recommend
downgrading to Java 1.7.0_u25. There are known bugs in the most recent
Oracle JVM versions which have not been resolved yet. u25 is the most
recent safe version. I don't think that's your problem, but it's a good
general consideration anyway.
-Z
On Thursday, March 13, 2014 8:23:34 PM UTC-4, Jos Kraaijeveld wrote:
>
> @Mark:
> The heap is set to 2GB, using mlockall. The problem occurs with both
> OpenJDK7 and OracleJDK7, both the latest versions. I have one index, which
> is very small:
> index:
> {
> primary_size_in_bytes: 37710681
> size_in_bytes: 37710681
> }
>
> @Zachary Our systems are set up to alert when memory is about to run out.
> We use Ganglia to monitor our systems and that represents the memory as
> 'used', rather than 'cached'. I will try to just let it run until memory
> runs out and report back after that though.
>
>
>
> On Thursday, March 13, 2014 5:17:20 PM UTC-7, Zachary Tong wrote:
>>
>> I believe you are just witnessing the OS caching files in memory. Lucene
>> (and therefore by extension Elasticsearch) uses a large number of files to
>> represent segments. TTL + updates will cause even higher file turnover
>> than usual.
>>
>> The OS manages all of this caching and will reclaim it for other
>> processes when needed. Are you experiencing problems, or just witnessing
>> memory usage? I wouldn't be concerned unless there is an actual problem
>> that you are seeing.
>>
>>
>>
>> On Thursday, March 13, 2014 8:07:13 PM UTC-4, Jos Kraaijeveld wrote:
>>>
>>> Hey,
>>>
>>> I've run into an issue which is preventing me from moving forwards with
>>> ES. I've got an application where I keep 'live' documents in ElasticSearch.
>>> Each document is a combination from data from multiple sources, which are
>>> merged together using doc_as_upsert. Each document has a TTL which is
>>> updated whenever new data comes in for a document, so documents die
>>> whenever no data source has given information about it for a while. The
>>> amount of documents generally doesn't exceed 15.000 so it's a fairly small
>>> data set.
>>>
>>> Whenever I leave this running, slowly but surely memory usage on the box
>>> creeps up, seemingly unbounded until there is no more resident memory left.
>>> The Java process nicely keeps within its set ES_MAX_HEAP bounds, but it
>>> seems the mapping from storage on disk to memory is every-increasing, even
>>> when the amount of 'live' documents goes to 0.
>>>
>>> I was wondering if anyone has seen such a memory problem before and
>>> whether there are ways to debug memory usage which is unaccounted for by
>>> processes in 'top'.
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a492cb9a-abeb-4b0e-ae97-5db7df843565%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
