Hi,
attatched you can find the es_log and the captured node jvm stats.
We are only indexing at this time and we use bulk requests.
As you can see at log entry "2014-03-14 21:18:59,873" in es_log... at this
time our indexing process finished and afterwards the OOM occurs...
Am Freitag, 14. März 2014 14:47:14 UTC+1 schrieb Zachary Tong:
>
> Are you running searches at the same time, or only indexing? Are you bulk
> indexing? How big (in physical kb/mb) are your bulk requests?
>
> Can you attach the output of these APIs (preferably during memory buildup
> but before the OOM):
>
> - curl -XGET 'localhost:9200/_nodes/'
> - curl -XGET 'localhost:9200/_nodes/stats'
>
> I would recommend downgrading your JVM to Java 1.7.0_u25. There are known
> sigsegv bugs in the most recent versions of the JVM which have not been
> fixed yet. It should be unrelated to your problem, but best to rule the
> JVM out.
>
> I would not touch any of those configs. In general, when debugging
> problems it is best to restore as many of the configs to their default
> settings as possible.
>
>
>
>
>
>
> On Friday, March 14, 2014 5:46:12 AM UTC-4, Alexander Ott wrote:
>>
>> Hi,
>>
>> we always run in an OutOfMemoryError while indexing documents or shortly
>> afterwards.
>> We only have one instance of elasticsearch version 1.0.1 (no cluster)
>>
>> Index informations:
>> size: 203G (203G)
>> docs: 237.354.313 (237.354.313)
>>
>> Our JVM settings as following:
>>
>> /usr/lib/jvm/java-7-oracle/bin/java -Xms16g -Xmx16g -Xss256k -Djava.awt.
>> headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:
>> CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX
>> :+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/
>> elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share
>> /elasticsearch/lib/elasticsearch-1.0.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>
>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>> -Des.default.path.home=/usr/share/elasticsearch
>> -Des.default.path.logs=/var/log/elasticsearch
>> -Des.default.path.data=/var/lib/elasticsearch
>> -Des.default.path.work=/tmp/elasticsearch
>> -Des.default.path.conf=/etc/elasticsearch
>> org.elasticsearch.bootstrap.Elasticsearch
>>
>>
>> OutOfMemoryError:
>> [2014-03-12 01:27:27,964][INFO ][monitor.jvm ] [Stiletto]
>> [gc][old][32451][309] duration [5.1s], collections [1]/[5.9s], total
>> [5.1s]/[3.1m], memory [15.8gb]->[15.7gb]/[15.9gb], all_pools {[young]
>> [665.6mb]->[583.7mb]/[665.6mb]}{[survivor] [32.9mb]->[0b]/[83.1mb]}{[old]
>> [15.1gb]->[15.1gb]/[15.1gb]}
>> [2014-03-12 01:28:23,822][INFO ][monitor.jvm ] [Stiletto]
>> [gc][old][32466][322] duration [5s], collections [1]/[5.9s], total
>> [5s]/[3.8m], memory [15.8gb]->[15.8gb]/[15.9gb], all_pools {[young]
>> [652.5mb]->[663.8mb]/[665.6mb]}{[survivor] [0b]->[0b]/[83.1mb]}{[old]
>> [15.1gb]->[15.1gb]/[15.1gb]}
>> [2014-03-12 01:33:29,814][WARN ][index.merge.scheduler ] [Stiletto]
>> [myIndex][0] failed to merge
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.lucene.util.fst.BytesStore.writeByte(BytesStore.java:83)
>> at org.apache.lucene.util.fst.FST.<init>(FST.java:282)
>> at org.apache.lucene.util.fst.Builder.<init>(Builder.java:163)
>> at
>> org.apache.lucene.codecs.BlockTreeTermsWriter$PendingBlock.compileIndex(BlockTreeTermsWriter.java:420)
>> at
>> org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:569)
>> at
>> org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter$FindBlocks.freeze(BlockTreeTermsWriter.java:544)
>> at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:214)
>> at org.apache.lucene.util.fst.Builder.add(Builder.java:394)
>> at
>> org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.finishTerm(BlockTreeTermsWriter.java:1000)
>> at
>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:166)
>> at
>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
>> at
>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
>> at
>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
>> at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4071)
>> at
>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3668)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
>> at
>> org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
>>
>> We also increased heap to 32g but with the same result
>> [2014-03-12 22:39:53,817][INFO ][monitor.jvm ] [Charcoal]
>> [gc][old][32895][86] duration [6.9s], collections [1]/[7.3s], total
>> [6.9s]/[19.6s], memory [20.5gb]->[12.7gb]/[31.9gb], all_pools {[youn
>> g] [654.9mb]->[1.9mb]/[665.6mb]}{[survivor]
>> [83.1mb]->[0b]/[83.1mb]}{[old] [19.8gb]->[12.7gb]/[31.1gb]}
>> [2014-03-12 23:11:07,015][INFO ][monitor.jvm ] [Charcoal]
>> [gc][old][34750][166] duration [8s], collections [1]/[8.6s], total
>> [8s]/[29.1s], memory [30.9gb]->[30.9gb]/[31.9gb], all_pools {[young]
>> [660.6mb]->[1mb]/[665.6mb]}{[survivor] [83.1mb]->[0b]/[83.1mb]}{[old]
>> [30.2gb]->[30.9gb]/[31.1gb]}
>> [2014-03-12 23:12:18,117][INFO ][monitor.jvm ] [Charcoal]
>> [gc][old][34812][182] duration [7.1s], collections [1]/[8.1s], total
>> [7.1s]/[36.6s], memory [31.5gb]->[31.5gb]/[31.9gb], all_pools {[you
>> ng] [655.6mb]->[410.3mb]/[665.6mb]}{[survivor] [0b]->[0b]/[83.1mb]}{[old]
>> [30.9gb]->[31.1gb]/[31.1gb]}
>> [2014-03-12 23:12:56,294][INFO ][monitor.jvm ] [Charcoal]
>> [gc][old][34844][193] duration [7.1s], collections [1]/[7.1s], total
>> [7.1s]/[43.9s], memory [31.9gb]->[31.9gb]/[31.9gb], all_pools {[you
>> ng] [665.6mb]->[665.2mb]/[665.6mb]}{[survivor]
>> [81.9mb]->[82.8mb]/[83.1mb]}{[old] [31.1gb]->[31.1gb]/[31.1gb]}
>> [2014-03-12 23:13:11,836][WARN ][index.merge.scheduler ] [Charcoal]
>> [myIndex][3] failed to merge
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:228)
>> at
>> org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:188)
>> at
>> org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:159)
>> at
>> org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:516)
>> at
>> org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:232)
>> at
>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:127)
>> at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4071)
>> at
>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3668)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
>> at
>> org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
>>
>> *java version: *
>> java version "1.7.0_51"
>> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
>>
>> *Elasticsearch.yml*: Settings which may should enabled?
>> #indices.memory.index_buffer_size: 40%
>> #indices.store.throttle.type: merge
>> #indices.store.throttle.max_bytes_per_sec: 50mb
>> #index.refresh_interval: 2s
>> #index.fielddata.cache: soft
>> #index.store.type: mmapfs
>> #index.fielddata.cache.size: 20%
>>
>>
>>
>> Any ideas how to solve this problem? Why old gen won't be clean up?
>> shouldn't it?
>>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8341b9d6-9d9f-495b-9eec-323d1ecb150e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
2014-03-14 20:14:14,274 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394824665678,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394824665678,
"uptime_in_millis" : 97990516,
"mem" : {
"heap_used_in_bytes" : 24351266424,
"heap_used_percent" : 71,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58936968,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 172774992,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 85825248,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 24092670328,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 25604694416,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 194,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 27188,
"collection_time_in_millis" : 2218431
},
"old" : {
"collection_count" : 72,
"collection_time_in_millis" : 10126
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 160,
"used_in_bytes" : 53936312,
"total_capacity_in_bytes" : 53936312
},
"mapped" : {
"count" : 2714,
"used_in_bytes" : 231223917796,
"total_capacity_in_bytes" : 231223917796
}
}
}
}
}
}
2014-03-14 20:16:22,776 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394824794172,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394824794172,
"uptime_in_millis" : 98119010,
"mem" : {
"heap_used_in_bytes" : 16553984528,
"heap_used_percent" : 48,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58936968,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 187311944,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 87228416,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 16279440360,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 25604694416,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 195,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 27225,
"collection_time_in_millis" : 2221570
},
"old" : {
"collection_count" : 72,
"collection_time_in_millis" : 10227
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 160,
"used_in_bytes" : 53936312,
"total_capacity_in_bytes" : 53936312
},
"mapped" : {
"count" : 2698,
"used_in_bytes" : 231549730817,
"total_capacity_in_bytes" : 231549730817
}
}
}
}
}
}
2014-03-14 20:43:21,531 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394826412826,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394826412826,
"uptime_in_millis" : 99737664,
"mem" : {
"heap_used_in_bytes" : 16949846696,
"heap_used_percent" : 49,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58922056,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 3799296,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 87228416,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 16858818984,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 25768601424,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 195,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 28661,
"collection_time_in_millis" : 2350710
},
"old" : {
"collection_count" : 79,
"collection_time_in_millis" : 10940
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 157,
"used_in_bytes" : 53906735,
"total_capacity_in_bytes" : 53906735
},
"mapped" : {
"count" : 2774,
"used_in_bytes" : 242127969578,
"total_capacity_in_bytes" : 242127969578
}
}
}
}
}
}
2014-03-14 20:51:03,111 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394826874378,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394826874378,
"uptime_in_millis" : 100199216,
"mem" : {
"heap_used_in_bytes" : 15696979104,
"heap_used_percent" : 45,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58922056,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 641133640,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 72055544,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 14983789920,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 25768601424,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 193,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 28906,
"collection_time_in_millis" : 2367835
},
"old" : {
"collection_count" : 80,
"collection_time_in_millis" : 11020
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 158,
"used_in_bytes" : 53911892,
"total_capacity_in_bytes" : 53911892
},
"mapped" : {
"count" : 2831,
"used_in_bytes" : 244432003516,
"total_capacity_in_bytes" : 244432003516
}
}
}
}
}
}
2014-03-14 21:07:01,779 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394827832986,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394827832986,
"uptime_in_millis" : 101157824,
"mem" : {
"heap_used_in_bytes" : 12086671584,
"heap_used_percent" : 35,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58923856,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 5333880,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 55955152,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 12025382552,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 25768601424,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 191,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 29601,
"collection_time_in_millis" : 2416220
},
"old" : {
"collection_count" : 82,
"collection_time_in_millis" : 11539
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 159,
"used_in_bytes" : 53926969,
"total_capacity_in_bytes" : 53926969
},
"mapped" : {
"count" : 2751,
"used_in_bytes" : 253473157831,
"total_capacity_in_bytes" : 253473157831
}
}
}
}
}
}
2014-03-14 21:24:20,769 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394828871910,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394828871910,
"uptime_in_millis" : 102196748,
"mem" : {
"heap_used_in_bytes" : 8852378952,
"heap_used_percent" : 25,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58938016,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 647850656,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 58398120,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 8146130176,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 25768601424,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 170,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 30272,
"collection_time_in_millis" : 2461715
},
"old" : {
"collection_count" : 84,
"collection_time_in_millis" : 11986
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 153,
"used_in_bytes" : 53651385,
"total_capacity_in_bytes" : 53651385
},
"mapped" : {
"count" : 2875,
"used_in_bytes" : 260333574645,
"total_capacity_in_bytes" : 260333574645
}
}
}
}
}
}
2014-03-14 21:38:05,679 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394829696769,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394829696769,
"uptime_in_millis" : 103021607,
"mem" : {
"heap_used_in_bytes" : 13573348624,
"heap_used_percent" : 39,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58933664,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 582104064,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 87228416,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 12904016144,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 25768601424,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 167,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 30364,
"collection_time_in_millis" : 2465384
},
"old" : {
"collection_count" : 84,
"collection_time_in_millis" : 11986
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 153,
"used_in_bytes" : 53651385,
"total_capacity_in_bytes" : 53651385
},
"mapped" : {
"count" : 2888,
"used_in_bytes" : 262823147823,
"total_capacity_in_bytes" : 262823147823
}
}
}
}
}
}
2014-03-14 21:49:49,423 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394830400469,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394830400469,
"uptime_in_millis" : 103725307,
"mem" : {
"heap_used_in_bytes" : 21580401704,
"heap_used_percent" : 62,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58784488,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 129780312,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 87228416,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 21363392976,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 25768601424,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 167,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 30405,
"collection_time_in_millis" : 2467873
},
"old" : {
"collection_count" : 85,
"collection_time_in_millis" : 19045
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 153,
"used_in_bytes" : 53651385,
"total_capacity_in_bytes" : 53651385
},
"mapped" : {
"count" : 2467,
"used_in_bytes" : 259430469584,
"total_capacity_in_bytes" : 259430469584
}
}
}
}
}
}
2014-03-14 21:58:14,745 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394830905759,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394830905759,
"uptime_in_millis" : 104230597,
"mem" : {
"heap_used_in_bytes" : 23595898944,
"heap_used_percent" : 68,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58784488,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 478112696,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 87228416,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 23030557832,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 25768601424,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 166,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 30415,
"collection_time_in_millis" : 2468773
},
"old" : {
"collection_count" : 86,
"collection_time_in_millis" : 19062
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 153,
"used_in_bytes" : 53651385,
"total_capacity_in_bytes" : 53651385
},
"mapped" : {
"count" : 2467,
"used_in_bytes" : 259430469584,
"total_capacity_in_bytes" : 259430469584
}
}
}
}
}
}
2014-03-14 22:07:45,970 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394831476948,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394831476948,
"uptime_in_millis" : 104801786,
"mem" : {
"heap_used_in_bytes" : 30764430024,
"heap_used_percent" : 89,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58786280,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 317443784,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 87228416,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 30359757824,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 30359781440,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 165,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 30426,
"collection_time_in_millis" : 2469735
},
"old" : {
"collection_count" : 135,
"collection_time_in_millis" : 20460
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 153,
"used_in_bytes" : 53651385,
"total_capacity_in_bytes" : 53651385
},
"mapped" : {
"count" : 2477,
"used_in_bytes" : 265823838279,
"total_capacity_in_bytes" : 265823838279
}
}
}
}
}
}
2014-03-14 22:14:02,360 INFO [de.ott.test.Elasticsearch] {
"cluster_name" : "myCluster",
"nodes" : {
"xMLnl4-2QVe90BZZgabM4Q" : {
"timestamp" : 1394831853315,
"name" : "Authority",
"transport_address" : "inet[/192.168.100.12:9300]",
"host" : "PC10149",
"ip" : [ "inet[/192.168.100.12:9300]", "NONE" ],
"attributes" : {
"rack" : "ENTWICKLUNG",
"max_local_storage_nodes" : "1"
},
"jvm" : {
"timestamp" : 1394831853315,
"uptime_in_millis" : 105178153,
"mem" : {
"heap_used_in_bytes" : 34216687080,
"heap_used_percent" : 99,
"heap_committed_in_bytes" : 34272509952,
"heap_max_in_bytes" : 34272509952,
"non_heap_used_in_bytes" : 58785144,
"non_heap_committed_in_bytes" : 91357184,
"pools" : {
"young" : {
"used_in_bytes" : 697958400,
"max_in_bytes" : 697958400,
"peak_used_in_bytes" : 697958400,
"peak_max_in_bytes" : 697958400
},
"survivor" : {
"used_in_bytes" : 32014720,
"max_in_bytes" : 87228416,
"peak_used_in_bytes" : 87228416,
"peak_max_in_bytes" : 87228416
},
"old" : {
"used_in_bytes" : 33486715392,
"max_in_bytes" : 33487323136,
"peak_used_in_bytes" : 33486715392,
"peak_max_in_bytes" : 33487323136
}
}
},
"threads" : {
"count" : 164,
"peak_count" : 201
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 30432,
"collection_time_in_millis" : 2470062
},
"old" : {
"collection_count" : 201,
"collection_time_in_millis" : 46007
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 153,
"used_in_bytes" : 53651385,
"total_capacity_in_bytes" : 53651385
},
"mapped" : {
"count" : 2487,
"used_in_bytes" : 272706256881,
"total_capacity_in_bytes" : 272706256881
}
}
}
}
}
}[2014-03-14 21:18:44,576][INFO ][cluster.metadata ] [Authority]
[myIndex] update_mapping [searchdoc] (dynamic)
[2014-03-14 21:18:59,872][INFO ][index.shard.service ] [Authority]
[myIndex][0] updating refresh_interval from [1s] to [1s]
[2014-03-14 21:18:59,872][INFO ][index.shard.service ] [Authority]
[myIndex][3] updating refresh_interval from [1s] to [1s]
[2014-03-14 21:18:59,872][INFO ][index.shard.service ] [Authority]
[myIndex][1] updating refresh_interval from [1s] to [1s]
[2014-03-14 21:18:59,872][INFO ][index.shard.service ] [Authority]
[myIndex][2] updating refresh_interval from [1s] to [1s]
[2014-03-14 21:18:59,873][INFO ][index.shard.service ] [Authority]
[myIndex][4] updating refresh_interval from [1s] to [1s]
[2014-03-14 21:42:20,202][INFO ][monitor.jvm ] [Authority]
[gc][old][102574][85] duration [7s], collections [1]/[7.5s], total [7s]/[19s],
memory [14.5gb]->[13.7gb]/[31.9gb], all_pools {[young]
[643.8mb]->[4.9mb]/[665.6mb]}{[survivor] [83.1mb]->[0b]/[83.1mb]}{[old]
[13.8gb]->[13.7gb]/[31.1gb]}
[2014-03-14 22:12:02,365][INFO ][monitor.jvm ] [Authority]
[gc][old][104335][162] duration [7.8s], collections [1]/[8.7s], total
[7.8s]/[29.1s], memory [30.8gb]->[30.8gb]/[31.9gb], all_pools {[young]
[658.2mb]->[2.8mb]/[665.6mb]}{[survivor] [83.1mb]->[0b]/[83.1mb]}{[old]
[30.1gb]->[30.8gb]/[31.1gb]}
[2014-03-14 22:14:41,199][INFO ][monitor.jvm ] [Authority]
[gc][old][104471][203] duration [5.7s], collections [1]/[5.8s], total
[5.7s]/[51.7s], memory [31.9gb]->[31.8gb]/[31.9gb], all_pools {[young]
[665.6mb]->[665.5mb]/[665.6mb]}{[survivor] [80.3mb]->[50.5mb]/[83.1mb]}{[old]
[31.1gb]->[31.1gb]/[31.1gb]}
[2014-03-14 22:14:51,777][INFO ][monitor.jvm ] [Authority]
[gc][old][104475][205] duration [7.4s], collections [1]/[7.5s], total
[7.4s]/[59.2s], memory [31.9gb]->[31.9gb]/[31.9gb], all_pools {[young]
[665.6mb]->[665.6mb]/[665.6mb]}{[survivor] [80.3mb]->[83mb]/[83.1mb]}{[old]
[31.1gb]->[31.1gb]/[31.1gb]}
[2014-03-14 22:15:06,151][WARN ][index.merge.scheduler ] [Authority]
[myIndex][2] failed to merge
java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:228)
at
org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:188)
at
org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:159)
at
org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:516)
at
org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:232)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:127)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4071)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3668)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
