Hi,
after a couple of days, the graylog2-server stopped working with the
following error in the elasticsearch cluster log:
[2015-02-04 06:12:43,294][INFO ][cluster.metadata ]
[mo-3e6eeef78.iaas.mo.sap.corp] [graylog2_196] creating index, cause [api],
shards [2]/[1], mappings []
[2015-02-04 06:13:13,991][DEBUG][action.admin.indices.mapping.put]
[mo-3e6eeef78.iaas.mo.sap.corp] failed to put mappings on indices
[[graylog2_196]], type [message]
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:
failed to process cluster event (put-mapping [message]) within 30s
at
org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:263)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2015-02-04 06:13:21,566][INFO ][cluster.metadata ]
[mo-3e6eeef78.iaas.mo.sap.corp] [graylog2_196] update_mapping [message]
(dynamic)
[2015-02-04 06:13:23,453][INFO ][cluster.metadata ]
[mo-3e6eeef78.iaas.mo.sap.corp] [graylog2_196] update_mapping [message]
(dynamic)
The graylog2-server node reported an index switch:
2015-02-04T06:11:36.247Z INFO [AbstractValidatingSessionManager] Finished
session validation. No sessions were stopped.
2015-02-04T06:12:43.249Z INFO [IndexRotationThread] Deflector index
<graylog2_195> should be rotated, Pointing deflector to new index now!
2015-02-04T06:12:43.249Z INFO [Deflector] Cycling deflector to next index
now.
2015-02-04T06:12:43.274Z INFO [Deflector] Cycling from <graylog2_195> to
<graylog2_196>
2015-02-04T06:12:43.274Z INFO [Deflector] Creating index target
<graylog2_196>...
2015-02-04T06:12:44.201Z WARN [BatchedElasticSearchOutput] Clearing buffer
(973 messages) because the Elasticsearch cluster is down.
2015-02-04T06:12:45.208Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1070 messages) because the Elasticsearch cluster is down.
2015-02-04T06:12:46.203Z WARN [BatchedElasticSearchOutput] Clearing buffer
(975 messages) because the Elasticsearch cluster is down
The graylog2 instance is running graylog2 92.4 and the 5 elasticsearch
cluster nodes are running on 1.4.2. The web interface was still reporting
incoming messages in the messages/second widget in the upper right corner,
but when doing a search an exception came up and there was no data during
that time.
Any idea why this timeout was reached? 30 seconds sounds a lot for any
garbage collection or network foo.
Thanks a lot for your help.
Regards,
Olaf
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
