Hello, its running on our own VM Host. Name resolution works fine, after all curl from the graylog node does work as well.
Graylog conf: is_master = true node_id_file = /etc/graylog/server/node-id password_secret = *** root_username = *** root_password_sha2 = *** root_email = root_timezone = UTC plugin_dir = /usr/share/graylog-server/plugin rest_listen_uri = http://127.0.0.1:12900/ rest_transport_uri = http://127.0.0.1:12900/ rest_enable_cors = false rest_enable_gzip = false rest_enable_tls = false rest_max_chunk_size = 8192 rest_max_header_size = 8192 rest_max_initial_line_length = 4096 rest_thread_pool_size = 16 rest_worker_threads_max_pool_size = 16 rotation_strategy = count elasticsearch_max_docs_per_index = 20000000 elasticsearch_max_size_per_index = 1073741824 elasticsearch_max_time_per_index = 1d elasticsearch_disable_version_check = false no_retention = false elasticsearch_max_number_of_indices = 20 retention_strategy = delete elasticsearch_shards = 4 elasticsearch_replicas = 0 elasticsearch_index_prefix = graylog2 allow_leading_wildcard_searches = false allow_highlighting = false elasticsearch_cluster_name = graylog2 elasticsearch_node_name = graylog2-server elasticsearch_node_master = false elasticsearch_node_data = false elasticsearch_transport_tcp_port = 9350 elasticsearch_http_enabled = false elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_discovery_zen_ping_unicast_hosts = fqdn:9300 elasticsearch_cluster_discovery_timeout = 5000 elasticsearch_analyzer = standard elasticsearch_request_timeout = 1m output_batch_size = 500 output_flush_interval = 1 processbuffer_processors = 5 outputbuffer_processors = 3 outputbuffer_processor_keep_alive_time = 5000 outputbuffer_processor_threads_core_pool_size = 3 outputbuffer_processor_threads_max_pool_size = 30 udp_recvbuffer_sizes = 1048576 processor_wait_strategy = blocking ring_size = 65536 inputbuffer_ring_size = 65536 inputbuffer_processors = 2 inputbuffer_wait_strategy = blocking message_journal_enabled = true message_journal_dir = /var/lib/graylog-server/journal message_journal_max_age = 12h message_journal_max_size = 5gb message_journal_flush_age = 1m message_journal_flush_interval = 1000000 message_journal_segment_age = 1h message_journal_segment_size = 100mb async_eventbus_processors = 2 dead_letters_enabled = false lb_recognition_period_seconds = 3 stream_processing_timeout = 2000 stream_processing_max_faults = 3 alert_check_interval = 60 output_module_timeout = 10000 stale_master_timeout = 2000 shutdown_timeout = 30000 mongodb_useauth = false mongodb_host = 127.0.0.1 mongodb_database = graylog2 mongodb_port = 27017 mongodb_max_connections = 100 mongodb_threads_allowed_to_block_multiplier = 5 transport_email_enabled = false transport_email_hostname = mail.example.com transport_email_port = 587 transport_email_use_auth = true transport_email_use_tls = false transport_email_use_ssl = true transport_email_auth_username = [email protected] transport_email_auth_password = secret transport_email_subject_prefix = [Graylog] transport_email_from_email = [email protected] http_connect_timeout = 5s http_read_timeout = 10s http_write_timeout = 10s disable_index_optimization = false index_optimization_max_num_segments = 1 disable_index_range_calculation = true gc_warning_threshold = 15s ldap_connection_timeout = 2000 versionchecks = true versionchecks_uri = https://versioncheck.graylog.com/check enable_metrics_collection = false disable_sigar = false collector_inactive_threshold = 1m collector_expiration_threshold = 14d dashboard_widget_default_cache_time = 10s Elasticsearch.yml: --- cluster: name: graylog2 discovery: zen: ping: multicast: enable: false unicast: hosts: fqdn:9300 http: cors: allow-origin: '/https?:\/\/(graylog|localhost)(\tld)?(:[0-9]+)?/' enabled: true port: 9200 network: host: 0.0.0.0 node: name: graylog2-server path: data: /usr/share/elasticsearch/data/graylog2 transport: tcp: port: 9300 Thanks, Mattis On Monday, March 14, 2016 at 5:51:53 PM UTC+1, Jochen Schalanda wrote: > > Hi Mattis, > > please post the actual Graylog server and Elasticsearch configuration from > that system. Are you running on some kind of cloud provider (e. g. AWS or > GCE)? Also double-check that "fqdn" (the part you've redacted) resolves to > the correct IP address. > > > Cheers, > Jochen > > On Monday, 14 March 2016 17:14:10 UTC+1, Mattis Haase wrote: >> >> I didn't think there was anymore interesting stuff after that, turns out >> I was wrong >> >> 2016-03-14T17:05:16.858+01:00 INFO [node] [graylog2-server] starting ... >> 2016-03-14T17:05:16.863+01:00 INFO [Periodicals] Starting >> [org.graylog2.periodical.VersionCheckThread] periodical in [0s], polling >> every [1800s]. >> 2016-03-14T17:05:16.869+01:00 INFO [Periodicals] Starting >> [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], >> polling every [1s]. >> 2016-03-14T17:05:17.038+01:00 INFO [transport] [graylog2-server] >> bound_address {inet[/0:0:0:0:0:0:0:0:9350]}, publish_address >> {inet[/a.b.c.d:9350]} >> 2016-03-14T17:05:17.059+01:00 INFO [discovery] [graylog2-server] >> graylog2/M9Pb4cf5Qqqz5e3_E_VNqA >> 2016-03-14T17:05:17.062+01:00 WARN [ClusterStateMonitor] No >> Elasticsearch data nodes in cluster, cluster is completely offline. >> 2016-03-14T17:05:17.285+01:00 INFO [Reflections] Reflections took 421 ms >> to scan 1 urls, producing 2 keys and 2 values >> 2016-03-14T17:05:20.061+01:00 WARN [discovery] [graylog2-server] waited >> for 3s and no initial state was set by the discovery >> 2016-03-14T17:05:20.062+01:00 INFO [node] [graylog2-server] started >> 2016-03-14T17:05:23.350+01:00 INFO [RestApiService] Adding security >> context factory: >> <org.graylog2.security.ShiroSecurityContextFactory@767cc927> >> 2016-03-14T17:05:23.367+01:00 INFO [RestApiService] Started REST API at < >> http://127.0.0.1:12900/> >> 2016-03-14T17:05:25.063+01:00 INFO [IndexerSetupService] Checking >> Elasticsearch HTTP API at http://fqdn:9200/ >> 2016-03-14T17:05:25.153+01:00 ERROR [UI] >> >> >> ################################################################################ >> >> ERROR: Could not successfully connect to Elasticsearch, if you use >> multicast check that it is working in your network and that Elasticsearch >> is running properly and is reachable. Also check that the cluster.name >> setting is correct. >> >> Need help? >> >> * Official documentation: https://www.graylog.org/documentation/intro/ >> * Community support: https://www.graylog.org/community-support/ >> * Commercial support: https://www.graylog.com/support/ >> >> But we also got some specific help pages that might help you in this case: >> >> * https://www.graylog.org/documentation/setup/elasticsearch/ >> >> Terminating. :( >> >> >> ################################################################################ >> >> 2016-03-14T17:05:25.156+01:00 ERROR [ServiceManager] Service >> IndexerSetupService [FAILED] has failed in the STARTING state. >> java.lang.IllegalStateException >> at org.graylog2.UI.exitHardWithWall(UI.java:36) >> at >> org.graylog2.initializers.IndexerSetupService.startUp(IndexerSetupService.java:171) >> at >> com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54) >> at >> com.google.common.util.concurrent.Callables$3.run(Callables.java:95) >> at java.lang.Thread.run(Thread.java:745) >> 2016-03-14T17:05:25.161+01:00 ERROR [InputSetupService] Not starting any >> inputs because lifecycle is: Uninitialized?[LB:DEAD] >> 2016-03-14T17:05:25.163+01:00 INFO [PeriodicalsService] Shutting down >> periodical [org.graylog2.periodical.AlertScannerThread]. >> 2016-03-14T17:05:25.163+01:00 INFO [RestApiService] Shutting down REST >> API at <http://127.0.0.1:12900/> >> 2016-03-14T17:05:25.165+01:00 INFO [PeriodicalsService] Shutdown of >> periodical [org.graylog2.periodical.AlertScannerThread] complete, took >> <0ms>. >> 2016-03-14T17:05:25.165+01:00 INFO [PeriodicalsService] Shutting down >> periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread]. >> 2016-03-14T17:05:25.165+01:00 WARN [BufferSynchronizerService] >> Elasticsearch is unavailable. Not waiting to clear buffers and caches, as >> we have no healthy cluster. >> 2016-03-14T17:05:25.165+01:00 INFO [PeriodicalsService] Shutdown of >> periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] >> complete, took <0ms>. >> 2016-03-14T17:05:25.165+01:00 INFO [PeriodicalsService] Shutting down >> periodical [org.graylog2.periodical.ClusterHealthCheckThread]. >> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutdown of >> periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, >> took <0ms>. >> 2016-03-14T17:05:25.165+01:00 INFO [node] [graylog2-server] stopping ... >> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutting down >> periodical [org.graylog2.periodical.IndexerClusterCheckerThread]. >> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutdown of >> periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, >> took <0ms>. >> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutting down >> periodical [org.graylog2.periodical.IndexRetentionThread]. >> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutdown of >> periodical [org.graylog2.periodical.IndexRetentionThread] complete, took >> <0ms>. >> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutting down >> periodical [org.graylog2.periodical.IndexRotationThread]. >> 2016-03-14T17:05:25.167+01:00 INFO [PeriodicalsService] Shutdown of >> periodical [org.graylog2.periodical.IndexRotationThread] complete, took >> <0ms>. >> 2016-03-14T17:05:25.167+01:00 INFO [PeriodicalsService] Shutting down >> periodical [org.graylog2.periodical.VersionCheckThread]. >> 2016-03-14T17:05:25.167+01:00 INFO [PeriodicalsService] Shutdown of >> periodical [org.graylog2.periodical.VersionCheckThread] complete, took >> <0ms>. >> 2016-03-14T17:05:25.167+01:00 INFO [PeriodicalsService] Shutting down >> periodical [org.graylog2.periodical.ThrottleStateUpdaterThread]. >> 2016-03-14T17:05:25.167+01:00 INFO [PeriodicalsService] Shutdown of >> periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, >> took <0ms>. >> 2016-03-14T17:05:25.166+01:00 INFO [OutputSetupService] Stopping output >> org.graylog2.outputs.BlockingBatchedESOutput >> 2016-03-14T17:05:25.177+01:00 INFO [IndexerClusterCheckerThread] Indexer >> not fully initialized yet. Skipping periodic cluster check. >> 2016-03-14T17:05:25.169+01:00 INFO [zen] [graylog2-server] failed to >> send join request to master >> [[graylog2-server][MBlVN4ugSfyL0iDA7_z6vg][fqdn][inet[/a.b.c.d:9300]]], >> reason [org.elasticsearch.ElasticsearchIllegalStateException: Future got >> interrupted; java.lang.InterruptedException] >> 2016-03-14T17:05:25.187+01:00 INFO [node] [graylog2-server] stopped >> 2016-03-14T17:05:25.187+01:00 INFO [node] [graylog2-server] closing ... >> 2016-03-14T17:05:25.192+01:00 INFO [node] [graylog2-server] closed >> 2016-03-14T17:05:25.260+01:00 INFO [JournalReader] Stopping. >> 2016-03-14T17:05:25.261+01:00 INFO [ServiceManagerListener] Services are >> now stopped. >> 2016-03-14T17:05:25.261+01:00 ERROR [ServerBootstrap] Graylog startup >> failed. Exiting. Exception was: >> java.lang.IllegalStateException: Expected to be healthy after starting. >> The following services are not running: {FAILED=[IndexerSetupService >> [FAILED]]} >> at >> com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:710) >> at >> com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:535) >> at >> com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:301) >> at >> org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:113) >> at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:196) >> at org.graylog2.bootstrap.Main.main(Main.java:58) >> 2016-03-14T17:05:25.262+01:00 INFO [Server] SIGNAL received. Shutting >> down. >> 2016-03-14T17:05:25.268+01:00 INFO [GracefulShutdown] Graceful shutdown >> initiated. >> 2016-03-14T17:05:25.269+01:00 INFO [GracefulShutdown] Node status: >> [Halting?[LB:DEAD]]. Waiting <3sec> for possible load balancers to >> recognize state change. >> 2016-03-14T17:05:29.271+01:00 INFO [GracefulShutdown] Goodbye. >> >> >> >> On Monday, March 14, 2016 at 4:54:06 PM UTC+1, Jochen Schalanda wrote: >>> >>> Hi Mattis, >>> >>> you shouldn't cut off the error message from the logs you've posted if >>> you expect anyone to help you. ;-) >>> >>> Cheers, >>> Jochen >>> >>> On Monday, 14 March 2016 16:51:54 UTC+1, Mattis Haase wrote: >>>> >>>> Two machines on the same VM host, one ES, on graylog, configured via >>>> puppet. >>>> >>>> Graylog Server error: >>>> >>>> 2016-03-14T16:40:10.280+01:00 INFO [transport] [graylog2-server] >>>> bound_address {inet[/0:0:0:0:0:0:0:0:9350]}, publish_address >>>> {inet[/a.b.c.d:9350]} >>>> 2016-03-14T16:40:10.296+01:00 INFO [discovery] [graylog2-server] >>>> graylog2/glh_l29gT9Sk1_nRRT3VNw >>>> 2016-03-14T16:40:10.300+01:00 WARN [ClusterStateMonitor] No >>>> Elasticsearch data nodes in cluster, cluster is completely offline. >>>> 2016-03-14T16:40:10.574+01:00 INFO [Reflections] Reflections took 448 >>>> ms to scan 1 urls, producing 2 keys and 2 values >>>> 2016-03-14T16:40:13.299+01:00 WARN [discovery] [graylog2-server] >>>> waited for 3s and no initial state was set by the discovery >>>> 2016-03-14T16:40:13.299+01:00 INFO [node] [graylog2-server] started >>>> 2016-03-14T16:40:17.099+01:00 INFO [RestApiService] Adding security >>>> context factory: >>>> <org.graylog2.security.ShiroSecurityContextFactory@4b7ee77a> >>>> 2016-03-14T16:40:17.115+01:00 INFO [RestApiService] Started REST API >>>> at <http://127.0.0.1:12900/> >>>> 2016-03-14T16:40:18.301+01:00 INFO [IndexerSetupService] Checking >>>> Elasticsearch HTTP API at http://fqdn:9200/ >>>> 2016-03-14T16:40:18.384+01:00 ERROR [UI] >>>> >>>> curl -X GET http://fqdn:9200 >>>> >>>> { >>>> "status" : 200, >>>> "name" : "graylog2-server", >>>> "cluster_name" : "graylog2", >>>> "version" : { >>>> "number" : "1.7.5", >>>> "build_hash" : "00f95f4ffca6de89d68b7ccaf80d148f1f70e4d4", >>>> "build_timestamp" : "2016-02-02T09:55:30Z", >>>> "build_snapshot" : false, >>>> "lucene_version" : "4.10.4" >>>> }, >>>> "tagline" : "You Know, for Search" >>>> } >>>> >>>> Config (using puppet): >>>> ES: >>>> class { 'elasticsearch': >>>> ensure => 'present', >>>> package_url => " >>>> https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${elasticversion}.deb >>>> ", >>>> config => { >>>> 'elasticsearch.discovery.zen.ping.multicast.enable' => false, >>>> 'elasticsearch.discovery.zen.ping.unicast.hosts' => >>>> "${::fqdn}:9300", >>>> }, >>>> }-> >>>> >>>> elasticsearch::instance { 'graylog2': >>>> config => { >>>> 'node.name' => >>>> 'graylog2-server', >>>> 'cluster.name' => >>>> 'graylog2', >>>> 'network.host' => >>>> $::ipaddress, >>>> 'transport.tcp.port' => '9300', >>>> 'http.port' => '9200', >>>> } >>>> >>>> Graylog: >>>> class {'graylog2::server': >>>> password_secret => >>>> root_password_sha2 => ', >>>> elasticsearch_cluster_name => 'graylog2', >>>> elasticsearch_node_name => >>>> 'graylog2-server', >>>> elasticsearch_discovery_zen_ping_multicast_enabled => false, >>>> elasticsearch_discovery_zen_ping_unicast_hosts => 'fqdn:9300', >>>> gc_warning_threshold => '15s', >>>> usage_statistics_enabled => false, >>>> } >>>> >>>> I tried pretty much every combination of using multicast, using unicast >>>> with different ports, manually setting elasticsearch_network_host >>>> and elasticsearch_transport_tcp_port. No other software that uses >>>> elasticsearch has any issues whatsoever, it just works. Using the same >>>> config but on one machine it also works. >>>> >>>> Any ideas? >>>> >>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/df1e9e23-5ca4-4768-871f-3af27caabc5b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
