Hello,
Running Graylog V1.2.2 using the VM appliance from graylog.org.
Been having performance issues. When I first start Graylog, everything is
snappy. By the next day, things have gotten more sluggish. Sometimes it
takes 5-10 attempts to login to the web interface.
One problem I have is that two of the indices have dropped off the list on
the Systems->Indices page.
After some googling, I decided to try Maintenance->Recalculate index ranges.
The job completes but neither of the two indices reappear in the list.
I found these errors in /var/log/graylog/server/current:
2015-11-09_23:28:53.06554 INFO [RebuildIndexRangesJob] Re-calculating
index ranges.
2015-11-09_23:28:53.06590 INFO [SystemJobManager] Submitted SystemJob
<a3802c80-8739-11e5-8dd3-005056b859d5>
[org.graylog2.indexer.ranges.RebuildIndexRangesJob]
2015-11-09_23:28:53.12839 INFO [MongoIndexRangeService] Calculated range
of [graylog_47] in [56ms].
...
2015-11-09_23:28:54.49844 INFO [MongoIndexRangeService] Calculated range
of [graylog_55] in [101ms].
2015-11-09_23:28:54.81895 INFO [MongoIndexRangeService] Calculated range
of [graylog_58] in [211ms].
2015-11-09_23:28:54.94361 INFO [MongoIndexRangeService] Calculated range
of [graylog_57] in [123ms].
2015-11-09_23:28:55.04214 ERROR [Indices] Error while calculating timestamp
stats in index <graylog_59>
2015-11-09_23:28:55.04216
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
execute phase [query], all shards failed; shardFailures
{[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][0]:
RemoteTransportException[[X-Cutioner][inet
[/172.20.39.61:9300]][indices:data/read/search[phase/query]]]; nested:
ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][1]:
RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
nested: ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][2]:
RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
nested: ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_
59][3]:
RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
nested: ClassCastException; }
2015-11-09_23:28:55.04217 at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237)
2015-11-09_23:28:55.04218 at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:183)
2015-11-09_23:28:55.04218 at
org.elasticsearch.search.action.SearchServiceTransportAction$6.handleException(SearchServiceTransportAction.java:249)
2015-11-09_23:28:55.04219 at
org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:190)
2015-11-09_23:28:55.04219 at
org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:180)
2015-11-09_23:28:55.04220 at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:130)
2015-11-09_23:28:55.04220 at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
2015-11-09_23:28:55.04220 at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
2015-11-09_23:28:55.04221 at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
2015-11-09_23:28:55.04221 at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
2015-11-09_23:28:55.04222 at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
2015-11-09_23:28:55.04222 at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
2015-11-09_23:28:55.04223 at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
2015-11-09_23:28:55.04223 at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
2015-11-09_23:28:55.04224 at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
2015-11-09_23:28:55.04225 at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
2015-11-09_23:28:55.04225 at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
2015-11-09_23:28:55.04226 at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
2015-11-09_23:28:55.04226 at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
2015-11-09_23:28:55.04226 at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
2015-11-09_23:28:55.04227 at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
2015-11-09_23:28:55.04228 at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
2015-11-09_23:28:55.04228 at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
2015-11-09_23:28:55.04228 at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
2015-11-09_23:28:55.04229 at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
2015-11-09_23:28:55.04229 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2015-11-09_23:28:55.04230 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2015-11-09_23:28:55.04230 at java.lang.Thread.run(Thread.java:745)
2015-11-09_23:28:55.04250 INFO [RebuildIndexRangesJob] Could not calculate
range of index [graylog_59]. Skipping.
2015-11-09_23:28:55.04252 org.elasticsearch.indices.IndexMissingException:
[graylog_59] missing
2015-11-09_23:28:55.04252 at
org.graylog2.indexer.indices.Indices.timestampStatsOfIndex(Indices.java:482)
2015-11-09_23:28:55.04253 at
org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:118)
2015-11-09_23:28:55.04253 at
org.graylog2.indexer.ranges.RebuildIndexRangesJob.execute(RebuildIndexRangesJob.java:96)
2015-11-09_23:28:55.04253 at
org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:88)
2015-11-09_23:28:55.04254 at
com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235)
2015-11-09_23:28:55.04254 at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2015-11-09_23:28:55.04254 at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
2015-11-09_23:28:55.04255 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2015-11-09_23:28:55.04255 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2015-11-09_23:28:55.04256 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2015-11-09_23:28:55.04256 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2015-11-09_23:28:55.04257 at java.lang.Thread.run(Thread.java:745)
2015-11-09_23:28:55.20408 INFO [MongoIndexRangeService] Calculated range
of [graylog_61] in [161ms].
2015-11-09_23:28:55.38073 INFO [MongoIndexRangeService] Calculated range
of [graylog_60] in [175ms].
graylog_59 is one of the two missing indices.
Is it possible to "fix" these indices and gain access to the data contained
within them?
I originally configured the system to keep 30 indices, each with 24 hours
of data.
Today I reconfigured that to 60 indices at 12 hours each. Not sure if that
will help with the performance issues.
If their a rule-of-thumb for index sizing?
Anything else I should be looking at to figure out the performance issues?
The performance graphs for the VM look OK in vSphere; no resources appear
to be overwhelmed.
Thanks for any guidance.
Steve.
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/89388152-988d-4814-a88c-e40211f99401%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
