Okay solved this, but it needs further investigationfor me. One manner could be making the indices smaller in the future they are now 3,6GB each I think.
After reading: http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html I have added this: indices.fielddata.cache.size: 40% to elasticsearch.yml and I am back on track again and could do a complete recalculation. On Wednesday, September 24, 2014 10:27:38 PM UTC+2, Arie wrote: > > Hi Lennart > > in the yml file there is 8192m configured for ES_HEAP_SIZE. at first i > have foregotten the "m", > but now it is there and I can recalculated 4 out of 14 indices. Stil it > goes out-of-memory tho. > > The server has 16GB, and no problem there. > > > > On Wednesday, September 24, 2014 9:53:36 PM UTC+2, lennart wrote: >> >> Hey Arie, >> >> loos like your ElasticSearch process is running out of memory. How >> much heap space did you allocate to it? >> >> Thanks, >> Lennart >> >> On Wed, Sep 24, 2014 at 9:34 PM, Arie <[email protected]> wrote: >> > And ther is this error in ther server logfile for some indices: >> > >> > 2014-09-24T21:07:15.736+02:00 INFO [RebuildIndexRangesJob] Could not >> > calculate range of index [graylog2_12]. Skipping. >> > org.elasticsearch.action.search.SearchPhaseExecutionException: Failed >> to >> > execute phase [query_fetch], all shards failed; shardFailures >> > {[eUTBQqseT1mfR9rhU3OSRg][graylog2_12][0]: >> > >> RemoteTransportException[[Graylog2-test][inet[/10.64.91.14:9300]][search/phase/query+fetch]]; >> >> >> > nested: QueryPhaseExecutionException[[graylog2_12][0]: >> > query[ConstantScore(*:*)],from[0],size[1],sort[<custom:"timestamp": >> > >> org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@46977d73>!]: >> >> >> > Query Failed [Failed to execute main query]]; nested: >> > ElasticSearchException[java.lang.OutOfMemoryError: Java heap space]; >> nested: >> > ExecutionError[java.lang.OutOfMemoryError: Java heap space]; nested: >> > OutOfMemoryError[Java heap space]; } >> > at >> > >> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:272) >> >> >> > at >> > >> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:224) >> >> >> > at >> > >> org.elasticsearch.search.action.SearchServiceTransportAction$7.handleException(SearchServiceTransportAction.java:324) >> >> >> > at >> > >> org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:181) >> >> >> > at >> > >> org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:171) >> >> >> > at >> > >> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) >> >> >> > at >> > >> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) >> >> >> > at >> > >> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) >> >> >> > at >> > >> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) >> >> >> > at >> > >> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) >> >> >> > at >> > >> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >> >> >> > at >> > >> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >> >> >> > at >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >> >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >> >> > at java.lang.Thread.run(Thread.java:745) >> > 2014-09-24T21:07:15.739+02:00 INFO [RebuildIndexRangesJob] Done >> calculating >> > index ranges for 15 indices. Took 132244ms. >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "graylog2" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to [email protected]. >> > For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
