At the moment i can provide only the jvm stats ... i will capture the other
stats as soon as possible.
We use 5-20 threads which will proccess bulks with a max size of 100
entries.
We only use one node/maschine for development so we have no cluster for
development...
The maschine has 64gb RAM and we increase the heap from 16gb to 32gb...
Am Montag, 17. März 2014 12:21:09 UTC+1 schrieb Zachary Tong:
>
> Can you attach the full Node Stats and Node Info output? There were other
> stats/metrics that I wanted to check (such as field data, bulk queue/size,
> etc).
>
> - How large (physically, in kb/mb) are your bulk indexing requests?
> Bulks should be 5-15mb in size
> - How many concurrent bulks are you performing? Given you cluster
> size, a good number should probably be around 20-30
> - Are you distributing bulks evenly across the cluster?
> - I see that your heap is 32gb. How big are these machines?
>
>
> -Zach
>
>
>
> On Monday, March 17, 2014 5:33:30 AM UTC-4, Alexander Ott wrote:
>>
>> Hi,
>>
>> attatched you can find the es_log and the captured node jvm stats.
>> We are only indexing at this time and we use bulk requests.
>>
>> As you can see at log entry "2014-03-14 21:18:59,873" in es_log... at
>> this time our indexing process finished and afterwards the OOM occurs...
>>
>>
>> Am Freitag, 14. März 2014 14:47:14 UTC+1 schrieb Zachary Tong:
>>>
>>> Are you running searches at the same time, or only indexing? Are you
>>> bulk indexing? How big (in physical kb/mb) are your bulk requests?
>>>
>>> Can you attach the output of these APIs (preferably during memory
>>> buildup but before the OOM):
>>>
>>> - curl -XGET 'localhost:9200/_nodes/'
>>> - curl -XGET 'localhost:9200/_nodes/stats'
>>>
>>> I would recommend downgrading your JVM to Java 1.7.0_u25. There are
>>> known sigsegv bugs in the most recent versions of the JVM which have not
>>> been fixed yet. It should be unrelated to your problem, but best to rule
>>> the JVM out.
>>>
>>> I would not touch any of those configs. In general, when debugging
>>> problems it is best to restore as many of the configs to their default
>>> settings as possible.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Friday, March 14, 2014 5:46:12 AM UTC-4, Alexander Ott wrote:
>>>>
>>>> Hi,
>>>>
>>>> we always run in an OutOfMemoryError while indexing documents or
>>>> shortly afterwards.
>>>> We only have one instance of elasticsearch version 1.0.1 (no cluster)
>>>>
>>>> Index informations:
>>>> size: 203G (203G)
>>>> docs: 237.354.313 (237.354.313)
>>>>
>>>> Our JVM settings as following:
>>>>
>>>> /usr/lib/jvm/java-7-oracle/bin/java -Xms16g -Xmx16g -Xss256k -Djava.awt
>>>> .headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:
>>>> CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -
>>>> XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/
>>>> elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/
>>>> share/elasticsearch/lib/elasticsearch-1.0.1.jar:/usr/share/
>>>> elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>>>> -Des.default.path.home=/usr/share/elasticsearch
>>>> -Des.default.path.logs=/var/log/elasticsearch
>>>> -Des.default.path.data=/var/lib/elasticsearch
>>>> -Des.default.path.work=/tmp/elasticsearch
>>>> -Des.default.path.conf=/etc/elasticsearch
>>>> org.elasticsearch.bootstrap.Elasticsearch
>>>>
>>>>
>>>> OutOfMemoryError:
>>>> [2014-03-12 01:27:27,964][INFO ][monitor.jvm ] [Stiletto]
>>>> [gc][old][32451][309] duration [5.1s], collections [1]/[5.9s], total
>>>> [5.1s]/[3.1m], memory [15.8gb]->[15.7gb]/[15.9gb], all_pools {[young]
>>>> [665.6mb]->[583.7mb]/[665.6mb]}{[survivor] [32.9mb]->[0b]/[83.1mb]}{[old]
>>>> [15.1gb]->[15.1gb]/[15.1gb]}
>>>> [2014-03-12 01:28:23,822][INFO ][monitor.jvm ] [Stiletto]
>>>> [gc][old][32466][322] duration [5s], collections [1]/[5.9s], total
>>>> [5s]/[3.8m], memory [15.8gb]->[15.8gb]/[15.9gb], all_pools {[young]
>>>> [652.5mb]->[663.8mb]/[665.6mb]}{[survivor] [0b]->[0b]/[83.1mb]}{[old]
>>>> [15.1gb]->[15.1gb]/[15.1gb]}
>>>> [2014-03-12 01:33:29,814][WARN ][index.merge.scheduler ] [Stiletto]
>>>> [myIndex][0] failed to merge
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at
>>>> org.apache.lucene.util.fst.BytesStore.writeByte(BytesStore.java:83)
>>>> at org.apache.lucene.util.fst.FST.<init>(FST.java:282)
>>>> at org.apache.lucene.util.fst.Builder.<init>(Builder.java:163)
>>>> at
>>>> org.apache.lucene.codecs.BlockTreeTermsWriter$PendingBlock.compileIndex(BlockTreeTermsWriter.java:420)
>>>> at
>>>> org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:569)
>>>> at
>>>> org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter$FindBlocks.freeze(BlockTreeTermsWriter.java:544)
>>>> at
>>>> org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:214)
>>>> at org.apache.lucene.util.fst.Builder.add(Builder.java:394)
>>>> at
>>>> org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.finishTerm(BlockTreeTermsWriter.java:1000)
>>>> at
>>>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:166)
>>>> at
>>>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
>>>> at
>>>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
>>>> at
>>>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
>>>> at
>>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4071)
>>>> at
>>>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3668)
>>>> at
>>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
>>>> at
>>>> org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
>>>> at
>>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
>>>>
>>>> We also increased heap to 32g but with the same result
>>>> [2014-03-12 22:39:53,817][INFO ][monitor.jvm ] [Charcoal]
>>>> [gc][old][32895][86] duration [6.9s], collections [1]/[7.3s], total
>>>> [6.9s]/[19.6s], memory [20.5gb]->[12.7gb]/[31.9gb], all_pools {[youn
>>>> g] [654.9mb]->[1.9mb]/[665.6mb]}{[survivor]
>>>> [83.1mb]->[0b]/[83.1mb]}{[old] [19.8gb]->[12.7gb]/[31.1gb]}
>>>> [2014-03-12 23:11:07,015][INFO ][monitor.jvm ] [Charcoal]
>>>> [gc][old][34750][166] duration [8s], collections [1]/[8.6s], total
>>>> [8s]/[29.1s], memory [30.9gb]->[30.9gb]/[31.9gb], all_pools {[young]
>>>> [660.6mb]->[1mb]/[665.6mb]}{[survivor] [83.1mb]->[0b]/[83.1mb]}{[old]
>>>> [30.2gb]->[30.9gb]/[31.1gb]}
>>>> [2014-03-12 23:12:18,117][INFO ][monitor.jvm ] [Charcoal]
>>>> [gc][old][34812][182] duration [7.1s], collections [1]/[8.1s], total
>>>> [7.1s]/[36.6s], memory [31.5gb]->[31.5gb]/[31.9gb], all_pools {[you
>>>> ng] [655.6mb]->[410.3mb]/[665.6mb]}{[survivor]
>>>> [0b]->[0b]/[83.1mb]}{[old] [30.9gb]->[31.1gb]/[31.1gb]}
>>>> [2014-03-12 23:12:56,294][INFO ][monitor.jvm ] [Charcoal]
>>>> [gc][old][34844][193] duration [7.1s], collections [1]/[7.1s], total
>>>> [7.1s]/[43.9s], memory [31.9gb]->[31.9gb]/[31.9gb], all_pools {[you
>>>> ng] [665.6mb]->[665.2mb]/[665.6mb]}{[survivor]
>>>> [81.9mb]->[82.8mb]/[83.1mb]}{[old] [31.1gb]->[31.1gb]/[31.1gb]}
>>>> [2014-03-12 23:13:11,836][WARN ][index.merge.scheduler ] [Charcoal]
>>>> [myIndex][3] failed to merge
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at
>>>> org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:228)
>>>> at
>>>> org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:188)
>>>> at
>>>> org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:159)
>>>> at
>>>> org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:516)
>>>> at
>>>> org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:232)
>>>> at
>>>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:127)
>>>> at
>>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4071)
>>>> at
>>>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3668)
>>>> at
>>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
>>>> at
>>>> org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
>>>> at
>>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
>>>>
>>>> *java version: *
>>>> java version "1.7.0_51"
>>>> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
>>>>
>>>> *Elasticsearch.yml*: Settings which may should enabled?
>>>> #indices.memory.index_buffer_size: 40%
>>>> #indices.store.throttle.type: merge
>>>> #indices.store.throttle.max_bytes_per_sec: 50mb
>>>> #index.refresh_interval: 2s
>>>> #index.fielddata.cache: soft
>>>> #index.store.type: mmapfs
>>>> #index.fielddata.cache.size: 20%
>>>>
>>>>
>>>>
>>>> Any ideas how to solve this problem? Why old gen won't be clean up?
>>>> shouldn't it?
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9b9d5e07-5a47-4f1f-ab6e-543a809c9bf4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
