Hi Steve,
I suggest to check the indexed data with:
curl -XGET 'http://127.0.0.1:9200/graylog_59/_search?pretty=1'
Your ES state it's yellow because you have replica unassigned, should be
green.
Edit the config file elasticsearch.yml and disable replication :
index.number_of_replicas: 0
or configure this properly, depends of your needs.
Restart ES service.
Now the first command it's to check if your indexed data it's present in
that indice, if yes, like I guess, You can restart a graylog and check
again.
However now in your graylog web UI the status is it yellow or green ?
Sometimes in early versions of graylog had same problems with refresh
between ES and web UI.
HTH
Alberto
On Tuesday, November 10, 2015 at 7:07:36 PM UTC+1, Steve Kirkpatrick wrote:
>
> Thanks for the reply Alberto.
>
> Here's the first part of the output on the first command:
> Note: I used the IP of the graylog-server because localhost gave me:
> Connection refused
>
> root@graylog-server:/var/log/graylog/web# curl -XGET
> 172.20.39.61:9200/graylog_59/_stats?pretty
> {
> "_shards" : {
> "total" : 8,
> "successful" : 4,
> "failed" : 0
> },
> "_all" : {
> "primaries" : {
> "docs" : {
> "count" : 1318398,
> "deleted" : 0
> },
> "store" : {
> "size_in_bytes" : 685371590,
> "throttle_time_in_millis" : 0
> },
>
> That looks promising. Not sure if the rest of the output would be helpful.
>
> The second command:
>
> root@graylog-server:/var/log/graylog/web# curl -XGET
> http://172.20.39.61:9200/_cat/shards
> ...
> graylog_59 0 p STARTED 326183 161.9mb 127.0.1.1 X-Cutioner
> graylog_59 0 r UNASSIGNED
> graylog_59 3 p STARTED 329200 163.4mb 127.0.1.1 X-Cutioner
> graylog_59 3 r UNASSIGNED
> graylog_59 1 p STARTED 330482 163.4mb 127.0.1.1 X-Cutioner
> graylog_59 1 r UNASSIGNED
> graylog_59 2 p STARTED 332533 164.6mb 127.0.1.1 X-Cutioner
> graylog_59 2 r UNASSIGNED
> ...
> graylog_67 0 p STARTED 295826 145.3mb 127.0.1.1 X-Cutioner
> graylog_67 0 r UNASSIGNED
> graylog_67 3 p STARTED 299222 146.8mb 127.0.1.1 X-Cutioner
> graylog_67 3 r UNASSIGNED
> graylog_67 1 p STARTED 298980 146.4mb 127.0.1.1 X-Cutioner
> graylog_67 1 r UNASSIGNED
> graylog_67 2 p STARTED 304105 148.4mb 127.0.1.1 X-Cutioner
> graylog_67 2 r UNASSIGNED
>
> Those messages seem to match the ones from the "working" indices.
>
> Third command:
>
> root@graylog-server:/var/log/graylog/web# curl -XGET
> http://172.20.39.61:9200/_cluster/health?pretty
> {
> "cluster_name" : "graylog2",
> "status" : "yellow",
> "timed_out" : false,
> "number_of_nodes" : 2,
> "number_of_data_nodes" : 1,
> "active_primary_shards" : 132,
> "active_shards" : 132,
> "relocating_shards" : 0,
> "initializing_shards" : 0,
> "unassigned_shards" : 132,
> "delayed_unassigned_shards" : 0,
> "number_of_pending_tasks" : 0,
> "number_of_in_flight_fetch" : 0
> }
>
> Seems OK.
>
> Any other commands I could try or logs I should look at to determine why
> those two indices are not available within the Graylog Web UI?
>
>
> Appreciate the help.
>
> Steve.
>
> On Tuesday, November 10, 2015 at 8:19:57 AM UTC-8, Alberto Frosi wrote:
>>
>> Hi Steve,
>> I suggest to check if these indices exist yet, querying ES directly:
>>
>> curl -XGET localhost:9200/graylog_59/_stats?pretty
>>
>> curl -XGET http://localhost:9200/_cat/shards
>>
>>
>> curl -XGET http:///localhost:9200/_cluster/health?pretty
>>
>> HTH
>> Ciao
>> Alberto
>>
>> On Tuesday, November 10, 2015 at 1:16:52 AM UTC+1, Steve Kirkpatrick
>> wrote:
>>>
>>> Hello,
>>>
>>> Running Graylog V1.2.2 using the VM appliance from graylog.org.
>>>
>>> Been having performance issues. When I first start Graylog, everything
>>> is snappy. By the next day, things have gotten more sluggish. Sometimes
>>> it takes 5-10 attempts to login to the web interface.
>>>
>>> One problem I have is that two of the indices have dropped off the list
>>> on the Systems->Indices page.
>>> After some googling, I decided to try Maintenance->Recalculate index
>>> ranges.
>>> The job completes but neither of the two indices reappear in the list.
>>>
>>> I found these errors in /var/log/graylog/server/current:
>>>
>>> 2015-11-09_23:28:53.06554 INFO [RebuildIndexRangesJob] Re-calculating
>>> index ranges.
>>> 2015-11-09_23:28:53.06590 INFO [SystemJobManager] Submitted SystemJob
>>> <a3802c80-8739-11e5-8dd3-005056b859d5>
>>> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
>>> 2015-11-09_23:28:53.12839 INFO [MongoIndexRangeService] Calculated
>>> range of [graylog_47] in [56ms].
>>> ...
>>> 2015-11-09_23:28:54.49844 INFO [MongoIndexRangeService] Calculated
>>> range of [graylog_55] in [101ms].
>>> 2015-11-09_23:28:54.81895 INFO [MongoIndexRangeService] Calculated
>>> range of [graylog_58] in [211ms].
>>> 2015-11-09_23:28:54.94361 INFO [MongoIndexRangeService] Calculated
>>> range of [graylog_57] in [123ms].
>>> 2015-11-09_23:28:55.04214 ERROR [Indices] Error while calculating
>>> timestamp stats in index <graylog_59>
>>> 2015-11-09_23:28:55.04216
>>> org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
>>> execute phase [query], all shards failed; shardFailures
>>> {[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][0]:
>>> RemoteTransportException[[X-Cutioner][inet
>>> [/172.20.39.61:9300]][indices:data/read/search[phase/query]]]; nested:
>>> ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][1]:
>>> RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
>>>
>>> nested: ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][2]:
>>> RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
>>>
>>> nested: ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_
>>> 59][3]:
>>> RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
>>>
>>> nested: ClassCastException; }
>>> 2015-11-09_23:28:55.04217 at
>>> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237)
>>> 2015-11-09_23:28:55.04218 at
>>> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:183)
>>> 2015-11-09_23:28:55.04218 at
>>> org.elasticsearch.search.action.SearchServiceTransportAction$6.handleException(SearchServiceTransportAction.java:249)
>>> 2015-11-09_23:28:55.04219 at
>>> org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:190)
>>> 2015-11-09_23:28:55.04219 at
>>> org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:180)
>>> 2015-11-09_23:28:55.04220 at
>>> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:130)
>>> 2015-11-09_23:28:55.04220 at
>>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>>> 2015-11-09_23:28:55.04220 at
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> 2015-11-09_23:28:55.04221 at
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>>> 2015-11-09_23:28:55.04221 at
>>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>> 2015-11-09_23:28:55.04222 at
>>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
>>> 2015-11-09_23:28:55.04222 at
>>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
>>> 2015-11-09_23:28:55.04223 at
>>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
>>> 2015-11-09_23:28:55.04223 at
>>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>>> 2015-11-09_23:28:55.04224 at
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> 2015-11-09_23:28:55.04225 at
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>> 2015-11-09_23:28:55.04225 at
>>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>> 2015-11-09_23:28:55.04226 at
>>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>> 2015-11-09_23:28:55.04226 at
>>> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
>>> 2015-11-09_23:28:55.04226 at
>>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
>>> 2015-11-09_23:28:55.04227 at
>>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
>>> 2015-11-09_23:28:55.04228 at
>>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
>>> 2015-11-09_23:28:55.04228 at
>>> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 2015-11-09_23:28:55.04228 at
>>> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 2015-11-09_23:28:55.04229 at
>>> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 2015-11-09_23:28:55.04229 at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 2015-11-09_23:28:55.04230 at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 2015-11-09_23:28:55.04230 at java.lang.Thread.run(Thread.java:745)
>>> 2015-11-09_23:28:55.04250 INFO [RebuildIndexRangesJob] Could not
>>> calculate range of index [graylog_59]. Skipping.
>>> 2015-11-09_23:28:55.04252
>>> org.elasticsearch.indices.IndexMissingException: [graylog_59] missing
>>> 2015-11-09_23:28:55.04252 at
>>> org.graylog2.indexer.indices.Indices.timestampStatsOfIndex(Indices.java:482)
>>> 2015-11-09_23:28:55.04253 at
>>> org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:118)
>>> 2015-11-09_23:28:55.04253 at
>>> org.graylog2.indexer.ranges.RebuildIndexRangesJob.execute(RebuildIndexRangesJob.java:96)
>>> 2015-11-09_23:28:55.04253 at
>>> org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:88)
>>> 2015-11-09_23:28:55.04254 at
>>> com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235)
>>> 2015-11-09_23:28:55.04254 at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> 2015-11-09_23:28:55.04254 at
>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> 2015-11-09_23:28:55.04255 at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> 2015-11-09_23:28:55.04255 at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> 2015-11-09_23:28:55.04256 at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 2015-11-09_23:28:55.04256 at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 2015-11-09_23:28:55.04257 at java.lang.Thread.run(Thread.java:745)
>>> 2015-11-09_23:28:55.20408 INFO [MongoIndexRangeService] Calculated
>>> range of [graylog_61] in [161ms].
>>> 2015-11-09_23:28:55.38073 INFO [MongoIndexRangeService] Calculated
>>> range of [graylog_60] in [175ms].
>>>
>>> graylog_59 is one of the two missing indices.
>>>
>>> Is it possible to "fix" these indices and gain access to the data
>>> contained within them?
>>> I originally configured the system to keep 30 indices, each with 24
>>> hours of data.
>>> Today I reconfigured that to 60 indices at 12 hours each. Not sure if
>>> that will help with the performance issues.
>>> If their a rule-of-thumb for index sizing?
>>>
>>> Anything else I should be looking at to figure out the performance
>>> issues?
>>> The performance graphs for the VM look OK in vSphere; no resources
>>> appear to be overwhelmed.
>>>
>>> Thanks for any guidance.
>>>
>>> Steve.
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/d81ede60-fda1-47f1-8159-8a0b8f274449%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
