Hi,
I am running a graylog2 instance 92.3 with 4 elasticsearch nodes on 1.3.4
all with java 1.7.0.71.
Last night I saw, that the graylog2 instance left the elasticsearch
cluster, but the cluster itself was still up and running and has no
problems. Cluster state is green.
The graylog2-server log reported. I went through all the elasticsearch node
logs, but none of them reported that the graylog2-server instance left the
cluster. After I have stopped
the graylog2-server node this morning, I saw in the logs, that the
graylog2-server left the cluster during shutdown, but not during the night,
where this error happend.
Any idea anyone?
2015-01-21T23:10:31.498Z INFO [IndexRotationThread] Deflector index
<graylog2_172> should be rotated, Pointing deflector to new index now!
2015-01-21T23:10:31.498Z INFO [Deflector] Cycling deflector to next index
now.
2015-01-21T23:10:31.539Z INFO [Deflector] Cycling from <graylog2_172> to
<graylog2_173>
2015-01-21T23:10:31.539Z INFO [Deflector] Creating index target
<graylog2_173>...
2015-01-21T23:10:32.286Z WARN [BatchedElasticSearchOutput] Clearing buffer
(607 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:33.275Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1299 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:34.276Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1540 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:35.281Z WARN [BatchedElasticSearchOutput] Clearing buffer
(970 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:36.277Z WARN [BatchedElasticSearchOutput] Clearing buffer
(945 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:37.273Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1251 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:38.280Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1293 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:39.281Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1109 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:40.288Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1127 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:41.394Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1215 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:42.362Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1037 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:43.280Z WARN [BatchedElasticSearchOutput] Clearing buffer
(985 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:44.276Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1155 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:45.284Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1196 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:46.276Z WARN [BatchedElasticSearchOutput] Clearing buffer
(839 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:47.285Z WARN [BatchedElasticSearchOutput] Clearing buffer
(966 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:48.305Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1145 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:49.279Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1071 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:50.279Z WARN [BatchedElasticSearchOutput] Clearing buffer
(864 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:51.277Z WARN [BatchedElasticSearchOutput] Clearing buffer
(967 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:52.278Z WARN [BatchedElasticSearchOutput] Clearing buffer
(896 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:53.362Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1072 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:54.280Z WARN [BatchedElasticSearchOutput] Clearing buffer
(452 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:55.279Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1443 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:56.292Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1132 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:57.275Z WARN [BatchedElasticSearchOutput] Clearing buffer
(925 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:58.283Z WARN [BatchedElasticSearchOutput] Clearing buffer
(895 messages) because the Elasticsearch cluster is down.
2015-01-21T23:10:59.274Z WARN [BatchedElasticSearchOutput] Clearing buffer
(957 messages) because the Elasticsearch cluster is down.
2015-01-21T23:11:00.276Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1035 messages) because the Elasticsearch cluster is down.
2015-01-21T23:11:01.303Z WARN [BatchedElasticSearchOutput] Clearing buffer
(1175 messages) because the Elasticsearch cluster is down.
2015-01-21T23:11:01.757Z ERROR [IndexRotationThread] Couldn't point
deflector to a new index
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException:
failed to process cluster event (put-mapping [message]) within 30s
at
org.elasticsearch.cluster.service.InternalClusterService$3$1.run(InternalClusterService.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-01-21T23:11:01.793Z WARN [IndexRotationThread] Deflector is pointing
to [graylog2_172], not the newest one: [graylog2_173]. Re-pointing.
2015-01-21T23:11:07.898Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
2015-01-21T23:11:07.900Z INFO [AbstractValidatingSessionManager] Finished
session validation. No sessions were stopped.
2015-01-21T23:29:43.733Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
2015-01-21T23:29:43.735Z INFO [AbstractValidatingSessionManager] Finished
session validation. No sessions were stopped.
2015-01-21T23:38:57.151Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
2015-01-21T23:38:57.154Z INFO [AbstractValidatingSessionManager] Finished
session validation. No sessions were stopped.
2015-01-21T23:41:21.711Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
2015-01-21T23:41:21.713Z INFO [AbstractValidatingSessionManager] Finished
session validation. No sessions were stopped.
2015-01-21T23:52:37.549Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
2015-01-21T23:52:37.551Z INFO [AbstractValidatingSessionManager] Finished
session validation. No sessions were stopped.
2015-01-21T23:53:22.009Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
2015-01-21T23:53:22.011Z INFO [AbstractValidatingSessionManager] Finished
session validation. No sessions were stopped.
The index graylog2_173 is available and has data, but the graylog2 web
interface is not showing any data between 23:10 and 07:45 this morning.
curl 'localhost:9200/_cat/indices?v'|grep 173
% Total % Received % Xferd Average Speed Time Time Time
Current
green graylog2_173 2 1 28624157 0 58.1gb
28.9gb
I have restarted the graylog2 instance and did an index switch, the first
attempt failed, because graylog2-server was reporting, that the indexes
couldn't be found:
2015-01-22T07:08:01.131Z WARN [SearchResource] Unable to execute search:
Failed to execute phase [query], all shards failed; shardFailures
{[80NMOY6gSnGA-tX2Wajx9g][graylo g2_171][0]:
NodeDisconnectedException[[mo-ef308bbd4.iaas.mo.sap.corp][inet[/10.67.76.66:9300]][search/phase/query]
disconnected]}{[jON8pI1iRNuFepecfpf8ng][graylog2_172][0]:
SendRequestTransportException[[mo-6a9cffac5.iaas.mo.sap.corp][inet[/10.67.76.63:9300]][search/phase/query]];
nested: NodeNotConnectedException[[mo-6a9cffac5.iaas.mo.sap.co
rp][inet[/10.67.76.63:9300]]
Node not connected]; }{[0Y05uVgNSmun7MwejXL2GA][graylog2_171][1]:
SendRequestTransportException[[mo-7693cb781.iaas.mo.sap.corp][inet[/10.67.74.
65:9300]][search/phase/query]]; nested:
NodeNotConnectedException[[mo-7693cb781.iaas.mo.sap.corp][inet[/10.67.74.65:9300]]
Node not connected]; }{[LpVhdtCxRHy2QlI4-bXqnw][g raylog2_172][1]:
NodeDisconnectedException[[mo-d232774ba.iaas.mo.sap.corp][inet[/10.67.74.41:9300]][search/phase/query]
disconnected]}
21843 2015-01-22T07:08:09.489Z WARN [SearchResource] Unhandled
ShardSearchFailure
21844 org.elasticsearch.action.search.SearchPhaseExecutionException:
Failed to execute phase [query], all shards failed; shardFailures
{[80NMOY6gSnGA-tX2Wajx9g][graylog2_171][0]:
NodeDisconnectedException[[mo-ef308bbd4.iaas.mo.sap.corp][inet[/10.67.76.66:9300]][search/phase/query]
disconnected]}{[jON8pI1iRNuFepecfpf8ng][graylog2_172][0]: SendReques
tTransportException[[mo-6a9cffac5.iaas.mo.sap.corp][inet[/10.67.76.63:9300]][search/phase/query]];
nested: NodeNotConnectedException[[mo-6a9cffac5.iaas.mo.sap.corp][inet[/1
0.67.76.63:9300]] Node not connected];
}{[0Y05uVgNSmun7MwejXL2GA][graylog2_171][1]:
SendRequestTransportException[[mo-7693cb781.iaas.mo.sap.corp][inet[/10.67.74.65:9300]][s
earch/phase/query]]; nested:
NodeNotConnectedException[[mo-7693cb781.iaas.mo.sap.corp][inet[/10.67.74.65:9300]]
Node not connected]; }{[LpVhdtCxRHy2QlI4-bXqnw][graylog2_172 ][1]:
NodeDisconnectedException[[mo-d232774ba.iaas.mo.sap.corp][inet[/10.67.74.41:9300]][search/phase/query]
disconnected]}
21845 at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233)
21846 at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:179)
21847 at
org.elasticsearch.search.action.SearchServiceTransportAction$6.handleException(SearchServiceTransportAction.java:224)
21848 at
org.elasticsearch.transport.TransportService$3.run(TransportService.java:216)
21849 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
21850 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
21851 at java.lang.Thread.run(Thread.java:745)
21852 2015-01-22T07:08:09.523Z ERROR [IndexRotationThread] Couldn't point
deflector to a new index
21853 org.elasticsearch.cluster.block.ClusterBlockException: blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered /
initialized];[SERVICE_UNAVAILABLE/2/no master];
21854 at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
21855 at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:84)
21856 at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:53)
21857 at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.<init>(TransportBroadcastOperationAction.java:114)
21858 at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:70)
21859 at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:46)
21860 at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:65)
21861 at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:51)
21862 at
org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:64)
21863 at
org.elasticsearch.client.support.AbstractIndicesAdminClient.stats(AbstractIndicesAdminClient.java:409)
21864 at
org.graylog2.indexer.indices.Indices.getAll(Indices.java:153)
21865 at
org.graylog2.indexer.Deflector.getNewestTargetNumber(Deflector.java:184)
21866 at
org.graylog2.indexer.Deflector.getNewestTargetName(Deflector.java:236)
21867 at
org.graylog2.periodical.IndexRotationThread.checkForRotation(IndexRotationThread.java:83)
21868 at
org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:66)
21869 at
org.graylog2.plugin.periodical.Periodical.run(Periodical.java:83)
21870 at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
21871 at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
21872 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
21873 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
21874 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
21875 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
21876 at java.lang.Thread.run(Thread.java:745)
21877 2015-01-22T07:08:01.132Z WARN [action] [graylog2-server] Failed to
send release search context
21878 org.elasticsearch.transport.SendRequestTransportException:
[mo-d232774ba.iaas.mo.sap.corp][inet[/10.67.74.41:9300]][search/freeContext]
21879 at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:212)
21880 at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:182)
21881 at
org.elasticsearch.search.action.SearchServiceTransportAction.sendFreeContext(SearchServiceTransportAction.java:125)
21882 at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.releaseIrrelevantSearchContexts(TransportSearchTypeAction.java:348)
21883 at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:158)
21884 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
21885 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
21886 at java.lang.Thread.run(Thread.java:745)
21887 Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[mo-d232774ba.iaas.mo.sap.corp][inet[/10.67.74.41:9300]] Node not connected
..........
org.elasticsearch.transport.SendRequestTransportException:
[mo-ef308bbd4.iaas.mo.sap.corp][inet[/10.67.76.66:9300]][search/freeContext]
21894 at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:212)
21895 at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:182)
21896 at
org.elasticsearch.search.action.SearchServiceTransportAction.sendFreeContext(SearchServiceTransportAction.java:125)
21897 at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.releaseIrrelevantSearchContexts(TransportSearchTypeAction.java:348)
21898 at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:158)
21899 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
21900 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
21901 at java.lang.Thread.run(Thread.java:745)
21902 Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[mo-ef308bbd4.iaas.mo.sap.corp][inet[/10.67.76.66:9300]] Node not connected
21903 at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:874)
21904 at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:556)
21905 at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:198)
21906 ... 7 more
21907 2015-01-22T07:08:09.537Z INFO [IndexerClusterCheckerThread]
Indexer not fully initialized yet. Skipping periodic cluster check.
21908 2015-01-22T07:08:09.497Z WARN [NodePingThread] Did not find meta
info of this node. Re-registering.
21909 2015-01-22T07:08:09.590Z INFO [IndexerClusterCheckerThread]
Indexer not fully initialized yet. Skipping periodic cluster check.
21910 2015-01-22T07:08:09.591Z INFO [IndexerClusterCheckerThread]
Indexer not fully initialized yet. Skipping periodic cluster check.
21911 2015-01-22T07:08:09.592Z INFO [IndexerClusterCheckerThread]
Indexer not fully initialized yet. Skipping periodic cluster check.
21912 2015-01-22T07:08:09.593Z INFO [IndexerClusterCheckerThread]
Indexer not fully initialized yet. Skipping periodic cluster check.
21913 2015-01-22T07:08:09.594Z INFO [IndexerClusterCheckerThread]
Indexer not fully initialized yet. Skipping periodic cluster check.
21914 2015-01-22T07:08:09.653Z ERROR [Messages] Failed to index [1100]
messages. Please check the index error log in your web interface for the
reason. Error: failure in bulk exe cution:
21915 [0]: index [graylog2_173], type [message], id
[fd657090-a204-11e4-ac0f-005056bb54f4], message
[UnavailableShardsException[[graylog2_173][1] [2] shardIt, [2] active :
Timeou t waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@5ecd6006]]
21916 [1]: index [graylog2_173], type [message], id
[fd65beb1-a204-11e4-ac0f-005056bb54f4], message
[UnavailableShardsException[[graylog2_173][1] [2] shardIt, [2] active :
Timeou t waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@5ecd6006]]
21917 [2]: index [graylog2_173], type [message], id
[fd65beb3-a204-11e4-ac0f-005056bb54f4], message
[UnavailableShardsException[[graylog2_173][1] [2] shardIt, [2] active :
Timeou t waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@5ecd6006]]
21918 [3]: index [graylog2_173], type [message], id
[fd65e5c1-a204-11e4-ac0f-005056bb54f4], message
[UnavailableShardsException[[graylog2_173][1] [2] shardIt, [2] active :
Timeou t waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@5ecd6006]]
21919 [4]: index [graylog2_173], type [message], id
[fd65e5c3-a204-11e4-ac0f-005056bb54f4], message
[UnavailableShardsException[[graylog2_173][1] [2] shardIt, [2] active :
Timeou t waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@5ecd6006]]
21920 [9]: index [graylog2_173], type [message], id
[fd665af1-a204-11e4-ac0f-005056bb54f4], message
[UnavailableShardsException[[graylog2_173][1] [2] shardIt, [2] active :
Timeou t waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@5ecd6006]]
21921 [10]: index [graylog2_173], type [message], id
[fd665af3-a204-11e4-ac0f-005056bb54f4], message
[UnavailableShardsException[[graylog2_173][1] [2] shardIt, [2] active :
Timeo ut waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@5ecd6006]]
21922 [11]: index [graylog2_173], type [message], id
[fd668201-a204-11e4-ac0f-005056bb54f4], message
[UnavailableShardsException[[graylog2_173][1] [2] shardIt, [2] active :
Timeo ut waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@5ecd6006]]
21923 [12]: index [graylog2_173], type [message], id
[fd668203-a204-11e4-ac0f-005056bb54f4], message
[UnavailableShardsException[[graylog2_173][1] [2] shardIt, [2] active : Tim
.................
[1097]: index [graylog2_173], type [message], id
[fdfbbd21-a204-11e4-ac0f-005056bb54f4], message
[UnavailableShardsException[[graylog2_173][1] [2] shardIt, [2] active : Tim
eout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@5ecd6006]]
22482 2015-01-22T07:08:18.927Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
22483 2015-01-22T07:08:19.057Z INFO [AbstractValidatingSessionManager]
Finished session validation. No sessions were stopped.
22484 2015-01-22T07:08:30.424Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
22485 2015-01-22T07:08:30.484Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
22486 2015-01-22T07:08:30.849Z INFO [AbstractValidatingSessionManager]
Finished session validation. No sessions were stopped.
22487 2015-01-22T07:08:30.880Z INFO [AbstractValidatingSessionManager]
Finished session validation. No sessions were stopped.
22488 2015-01-22T07:08:31.217Z INFO [IndexerClusterCheckerThread]
Indexer not fully initialized yet. Skipping periodic cluster check.
22489 2015-01-22T07:08:42.631Z ERROR [IndexRotationThread] Couldn't point
deflector to a new index
22490 org.elasticsearch.discovery.MasterNotDiscoveredException: waited
for [30s]
22491 at
org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$4.onTimeout(TransportMasterNodeOperationAction.java:166)
22492 at
org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:239)
22493 at
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:513)
22494 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
22495 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
22496 at java.lang.Thread.run(Thread.java:745)
22497 2015-01-22T07:08:42.631Z INFO [IndexRetentionThread] Elasticsearch
cluster not available, skipping index retention checks.
22498 2015-01-22T07:08:42.632Z WARN [BatchedElasticSearchOutput]
Clearing buffer (1964 messages) because the Elasticsearch cluster is down.
22499 2015-01-22T07:08:54.915Z WARN [BatchedElasticSearchOutput]
Clearing buffer (30 messages) because the Elasticsearch cluster is down.
22500 2015-01-22T07:08:54.916Z WARN [BatchedElasticSearchOutput]
Clearing buffer (2 messages) because the Elasticsearch cluster is down.
22501 2015-01-22T07:08:54.916Z WARN [BatchedElasticSearchOutput]
Clearing buffer (1 messages) because the Elasticsearch cluster is down.
22502 2015-01-22T07:08:54.916Z WARN [BatchedElasticSearchOutput]
Clearing buffer (1 messages) because the Elasticsearch cluster is down.
22503 2015-01-22T07:08:54.917Z WARN [BatchedElasticSearchOutput]
Clearing buffer (1 messages) because the Elasticsearch cluster is down.
22504 2015-01-22T07:08:54.930Z WARN [BatchedElasticSearchOutput]
Clearing buffer (33 messages) because the Elasticsearch cluster is down.
22505 2015-01-22T07:08:54.931Z WARN [BatchedElasticSearchOutput]
Clearing buffer (1 messages) because the Elasticsearch cluster is down.
22506 2015-01-22T07:08:54.931Z WARN [BatchedElasticSearchOutput]
Clearing buffer (2 messages) because the Elasticsearch cluster is down.
22507 2015-01-22T07:08:54.932Z WARN [BatchedElasticSearchOutput]
Clearing buffer (1 messages) because the Elasticsearch cluster is down.
22508 2015-01-22T07:08:54.933Z WARN [BatchedElasticSearchOutput]
Clearing buffer (4 messages) because the Elasticsearch cluster is down.
22509 2015-01-22T07:09:06.296Z WARN [BatchedElasticSearchOutput]
Clearing buffer (230 messages) because the Elasticsearch cluster is down.
22510 2015-01-22T07:09:06.320Z INFO [IndexerClusterCheckerThread]
Indexer not fully initialized yet. Skipping periodic cluster check.
22511 2015-01-22T07:09:19.248Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
22512 2015-01-22T07:09:19.368Z WARN [BatchedElasticSearchOutput]
Clearing buffer (811 messages) because the Elasticsearch cluster is down.
22513 2015-01-22T07:09:19.251Z INFO [AbstractValidatingSessionManager]
Validating all active sessions...
Any idea? Is the load to high for one graylog2-server? We get around
1500-2000 messages/sec. All nodes have 32 gb ram and 8 cpus virtualized on
Vmware :-(.
Regards,
Olaf
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
