Hi We have a infrastructure with 2 Graylog nodes behind a LB and 3 Elasticsearch nodes (2 Data&Master, 1 Master --> because of the split brain problem). I've attached a painting of our infrastructure (graylog-web excluded).
<https://lh3.googleusercontent.com/-beICyjvnCkQ/Vzw83Q7tK1I/AAAAAAAAAAg/4fqTPTC7xqY37p24cNyMx2WeUax6r3VWgCLcB/s1600/IMG_1127.jpg> The problem I have is, that if one of the data nodes goes down, graylog cannot write any data into the ES database. There is no error about failure to elect the master as it was with just two ES master nodes (fixed that by adding a master only node on one graylog server). If I shutdown the elasticsearch on esnode1 all other systems acknowledge this action with 2016-05-18 12:15:06,207][INFO ][cluster.service ] [elasticsearch- node2] removed {[elasticsearch-node1][w6iXmbtTRkq3571YA39vcQ][idlog-es-node1 .campus.unibe.ch][inet[/esnode1:9300]],}, reason: zen-disco-node_left([ elasticsearch-node1][w6iXmbtTRkq3571YA39vcQ][idlog-es-node1.campus.unibe.ch ][inet[/esnode1:9300]]) That's it. The cluster can restore it's stae only if I'm going to start the service on esnode1 again. I have no idea why the failover doesn't work and would be more than happy if someone has an idea. Thank you very much in advance. Kind regards, Stefan Following are the ES related configurations of each server: ## Graylog 1 - /etc/graylog/server/server.conf elasticsearch_shards = 1 elasticsearch_replicas = 1 elasticsearch_index_prefix = graylog2 allow_leading_wildcard_searches = true allow_highlighting = true elasticsearch_cluster_name = graylog-production elasticsearch_node_name = idgraylog-srv1 elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_discovery_zen_ping_unicast_hosts = esnode1:9300,esnode2:9300, graylog2:9300 elasticsearch_analyzer = standard ## Graylog 2 - /etc/graylog/server/server.conf elasticsearch_shards = 1 elasticsearch_replicas = 1 elasticsearch_index_prefix = graylog2 allow_leading_wildcard_searches = true allow_highlighting = true elasticsearch_cluster_name = graylog-production elasticsearch_node_name = idgraylog-srv2 elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_discovery_zen_ping_unicast_hosts = esnode1:9300,esnode2:9300, graylog2:9300 elasticsearch_analyzer = standard /etc/elasticsearch/elasticsearch.yml cluster.name: graylog-production node.master: true node.data: false path.conf: /etc/elasticsearch path.data: /opt/es-data network.bind_host: graylog2 discovery.zen.minimum_master_nodes: 2 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["esnode1:9300", "esnode2:9300", "graylog2:9300"] script.disable_dynamic: true indices.fielddata.cache.size: 40% ## ESNode1 - /etc/elasticsearch/elasticsearch.yml cluster.name: graylog-production node.name: "elasticsearch-node2" path.conf: /etc/elasticsearch path.data: /data node.master: true node.data: true network.bind_host: esnode1 discovery.zen.minimum_master_nodes: 2 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["esnode1:9300", "esnode2:9300", "graylog2:9300"] script.disable_dynamic: true indices.fielddata.cache.size: 40% ## ESNode2 - /etc/elasticsearch/elasticsearch.yml cluster.name: graylog-production node.name: "elasticsearch-node1" path.conf: /etc/elasticsearch path.data: /data node.master: true node.data: true network.bind_host: esnode2 discovery.zen.minimum_master_nodes: 2 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["esnode1:9300", "esnode2:9300", "graylog2:9300"] script.disable_dynamic: true indices.fielddata.cache.size: 40% -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/9a3fd585-3a15-4f4d-8ac5-a715d6c76260%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
