I'd be grateful if anyone could help point me in the right direction for this... Here's the issue:
I've got a 3-node setup sitting behind an F5 as a POC... After about 5-10 minutes of all three nodes up and running, I start to see occasional "blips" in the web UI and the following entries in the logs of all three systems: 2016-05-12T10:25:24.514-07:00 WARN [ProxiedResource] Node <3116ac6b-604f-4436-955c-1458cb489415> not found while trying to call org.graylog2.rest.resources.system.jobs.RemoteSystemJobResource on it. 2016-05-12T10:25:27.603-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:30.910-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:37.556-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:40.808-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:44.663-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:47.922-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:57.609-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:00.731-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:04.334-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:07.561-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:10.761-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:26.575-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:30.348-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:33.645-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:37.474-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:40.659-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:43.920-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:50.739-07:00 WARN [ProxiedResource] Node <3116ac6b-604f-4436-955c-1458cb489415> not found while trying to call org.graylog2.shared.rest.resources.system.RemoteMetricsResource on it. I did some googling, and the only thing I'm finding is that it's possible the nodes are out of sync time wise.. I checked the system times, and those all look correct: root@gray00 /data]# ntpdate -q ntp0 server 10.201.136.38, stratum 3, offset -0.003738, delay 0.02574 12 May 10:31:40 ntpdate[30786]: adjust time server 10.201.136.38 offset -0.003738 sec [root@gray01 graylog]# ntpdate -q ntp0 server 10.201.136.38, stratum 3, offset -0.000990, delay 0.02576 12 May 10:31:41 ntpdate[28966]: adjust time server 10.201.136.38 offset -0.000990 sec [root@gray02 /data]# ntpdate -q ntp0 server 10.201.136.38, stratum 3, offset 0.001198, delay 0.02582 12 May 10:31:41 ntpdate[30441]: adjust time server 10.201.136.38 offset 0.001198 sec Anyone have any ideas? Here's my setup: 3 nodes: gray00, gray01, gray02 - each node runs one instance of elastic search, standard ports. mongodb, standard ports graylog-server - elastic search ports are on 9350. F5 provides a frontend as "graylog.somewhere.com" ports 12900 & 9000 load balanced to each of the 3 nodes. Sticky sessions enabled on both... so what am I missing here? Graylog configuration settings (same for all 3 nodes, just the listen IPs are different and gray01/gray02 'is_master = false') s_master = true node_id_file = /etc/graylog/server/node-id password_secret = WQBdx6xgWTTykN9LHJhEGxfiSJbeYdaZhHhKEwbvAKQEWkVrl8lgTLvDDkfUtwhe7jgdFDFCBqpmVvY4aea1GyrbQ791UOCv root_password_sha2 = e3ed009797ada49a3fd38a04069b13d5a7f62001a153ed4d9a3da22fa7a75c7b plugin_dir = /usr/share/graylog-server/plugin rest_listen_uri = http://10.201.137.208:12900/ rest_transport_uri = http://graylog.somewhere.com:12900/ rest_enable_gzip = true web_listen_uri = http://10.201.137.208:9000/ web_enable_gzip = true rotation_strategy = count elasticsearch_max_docs_per_index = 20000000 elasticsearch_max_number_of_indices = 20 retention_strategy = delete elasticsearch_shards = 4 elasticsearch_replicas = 1 elasticsearch_index_prefix = graylog allow_leading_wildcard_searches = false allow_highlighting = false elasticsearch_cluster_name = graylog elasticsearch_node_name_prefix = graylog- elasticsearch_discovery_zen_ping_unicast_hosts = gray00:9300, gray01:9300, gray02:9300, gray00:9350, gray01:9350, gray02:9350 elasticsearch_transport_tcp_port = 9350 elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_network_host = gray00 elasticsearch_network_bind_host = gray00 elasticsearch_network_publish_host = gray00 elasticsearch_analyzer = standard output_batch_size = 500 output_flush_interval = 1 output_fault_count_threshold = 5 output_fault_penalty_seconds = 30 processbuffer_processors = 5 outputbuffer_processors = 3 processor_wait_strategy = blocking ring_size = 65536 inputbuffer_ring_size = 65536 inputbuffer_processors = 2 inputbuffer_wait_strategy = blocking message_journal_enabled = true message_journal_dir = /data/graylog/journal lb_recognition_period_seconds = 3 mongodb_uri = mongodb://gray00:27017,gray01:27017,gray02:2701/graylog mongodb_max_connections = 1000 mongodb_threads_allowed_to_block_multiplier = 5 transport_email_enabled = true transport_email_hostname = smtp transport_email_port =25 transport_email_subject_prefix = [graylog] transport_email_from_email = [email protected] transport_email_web_interface_url = https://graylog.somewhere.com content_packs_dir = /usr/share/graylog-server/contentpacks content_packs_auto_load = grok-patterns.json -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/6c2d6c23-1658-4c09-a179-06dc6273b6b4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
