Hi Mattis,
please post the actual Graylog server and Elasticsearch configuration from
that system. Are you running on some kind of cloud provider (e. g. AWS or
GCE)? Also double-check that "fqdn" (the part you've redacted) resolves to
the correct IP address.
Cheers,
Jochen
On Monday, 14 March 2016 17:14:10 UTC+1, Mattis Haase wrote:
>
> I didn't think there was anymore interesting stuff after that, turns out I
> was wrong
>
> 2016-03-14T17:05:16.858+01:00 INFO [node] [graylog2-server] starting ...
> 2016-03-14T17:05:16.863+01:00 INFO [Periodicals] Starting
> [org.graylog2.periodical.VersionCheckThread] periodical in [0s], polling
> every [1800s].
> 2016-03-14T17:05:16.869+01:00 INFO [Periodicals] Starting
> [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s],
> polling every [1s].
> 2016-03-14T17:05:17.038+01:00 INFO [transport] [graylog2-server]
> bound_address {inet[/0:0:0:0:0:0:0:0:9350]}, publish_address
> {inet[/a.b.c.d:9350]}
> 2016-03-14T17:05:17.059+01:00 INFO [discovery] [graylog2-server]
> graylog2/M9Pb4cf5Qqqz5e3_E_VNqA
> 2016-03-14T17:05:17.062+01:00 WARN [ClusterStateMonitor] No Elasticsearch
> data nodes in cluster, cluster is completely offline.
> 2016-03-14T17:05:17.285+01:00 INFO [Reflections] Reflections took 421 ms
> to scan 1 urls, producing 2 keys and 2 values
> 2016-03-14T17:05:20.061+01:00 WARN [discovery] [graylog2-server] waited
> for 3s and no initial state was set by the discovery
> 2016-03-14T17:05:20.062+01:00 INFO [node] [graylog2-server] started
> 2016-03-14T17:05:23.350+01:00 INFO [RestApiService] Adding security
> context factory:
> <org.graylog2.security.ShiroSecurityContextFactory@767cc927>
> 2016-03-14T17:05:23.367+01:00 INFO [RestApiService] Started REST API at <
> http://127.0.0.1:12900/>
> 2016-03-14T17:05:25.063+01:00 INFO [IndexerSetupService] Checking
> Elasticsearch HTTP API at http://fqdn:9200/
> 2016-03-14T17:05:25.153+01:00 ERROR [UI]
>
>
> ################################################################################
>
> ERROR: Could not successfully connect to Elasticsearch, if you use
> multicast check that it is working in your network and that Elasticsearch
> is running properly and is reachable. Also check that the cluster.name
> setting is correct.
>
> Need help?
>
> * Official documentation: https://www.graylog.org/documentation/intro/
> * Community support: https://www.graylog.org/community-support/
> * Commercial support: https://www.graylog.com/support/
>
> But we also got some specific help pages that might help you in this case:
>
> * https://www.graylog.org/documentation/setup/elasticsearch/
>
> Terminating. :(
>
>
> ################################################################################
>
> 2016-03-14T17:05:25.156+01:00 ERROR [ServiceManager] Service
> IndexerSetupService [FAILED] has failed in the STARTING state.
> java.lang.IllegalStateException
> at org.graylog2.UI.exitHardWithWall(UI.java:36)
> at
> org.graylog2.initializers.IndexerSetupService.startUp(IndexerSetupService.java:171)
> at
> com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54)
> at
> com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
> at java.lang.Thread.run(Thread.java:745)
> 2016-03-14T17:05:25.161+01:00 ERROR [InputSetupService] Not starting any
> inputs because lifecycle is: Uninitialized?[LB:DEAD]
> 2016-03-14T17:05:25.163+01:00 INFO [PeriodicalsService] Shutting down
> periodical [org.graylog2.periodical.AlertScannerThread].
> 2016-03-14T17:05:25.163+01:00 INFO [RestApiService] Shutting down REST
> API at <http://127.0.0.1:12900/>
> 2016-03-14T17:05:25.165+01:00 INFO [PeriodicalsService] Shutdown of
> periodical [org.graylog2.periodical.AlertScannerThread] complete, took
> <0ms>.
> 2016-03-14T17:05:25.165+01:00 INFO [PeriodicalsService] Shutting down
> periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
> 2016-03-14T17:05:25.165+01:00 WARN [BufferSynchronizerService]
> Elasticsearch is unavailable. Not waiting to clear buffers and caches, as
> we have no healthy cluster.
> 2016-03-14T17:05:25.165+01:00 INFO [PeriodicalsService] Shutdown of
> periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread]
> complete, took <0ms>.
> 2016-03-14T17:05:25.165+01:00 INFO [PeriodicalsService] Shutting down
> periodical [org.graylog2.periodical.ClusterHealthCheckThread].
> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutdown of
> periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete,
> took <0ms>.
> 2016-03-14T17:05:25.165+01:00 INFO [node] [graylog2-server] stopping ...
> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutting down
> periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutdown of
> periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete,
> took <0ms>.
> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutting down
> periodical [org.graylog2.periodical.IndexRetentionThread].
> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutdown of
> periodical [org.graylog2.periodical.IndexRetentionThread] complete, took
> <0ms>.
> 2016-03-14T17:05:25.166+01:00 INFO [PeriodicalsService] Shutting down
> periodical [org.graylog2.periodical.IndexRotationThread].
> 2016-03-14T17:05:25.167+01:00 INFO [PeriodicalsService] Shutdown of
> periodical [org.graylog2.periodical.IndexRotationThread] complete, took
> <0ms>.
> 2016-03-14T17:05:25.167+01:00 INFO [PeriodicalsService] Shutting down
> periodical [org.graylog2.periodical.VersionCheckThread].
> 2016-03-14T17:05:25.167+01:00 INFO [PeriodicalsService] Shutdown of
> periodical [org.graylog2.periodical.VersionCheckThread] complete, took
> <0ms>.
> 2016-03-14T17:05:25.167+01:00 INFO [PeriodicalsService] Shutting down
> periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
> 2016-03-14T17:05:25.167+01:00 INFO [PeriodicalsService] Shutdown of
> periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete,
> took <0ms>.
> 2016-03-14T17:05:25.166+01:00 INFO [OutputSetupService] Stopping output
> org.graylog2.outputs.BlockingBatchedESOutput
> 2016-03-14T17:05:25.177+01:00 INFO [IndexerClusterCheckerThread] Indexer
> not fully initialized yet. Skipping periodic cluster check.
> 2016-03-14T17:05:25.169+01:00 INFO [zen] [graylog2-server] failed to send
> join request to master
> [[graylog2-server][MBlVN4ugSfyL0iDA7_z6vg][fqdn][inet[/a.b.c.d:9300]]],
> reason [org.elasticsearch.ElasticsearchIllegalStateException: Future got
> interrupted; java.lang.InterruptedException]
> 2016-03-14T17:05:25.187+01:00 INFO [node] [graylog2-server] stopped
> 2016-03-14T17:05:25.187+01:00 INFO [node] [graylog2-server] closing ...
> 2016-03-14T17:05:25.192+01:00 INFO [node] [graylog2-server] closed
> 2016-03-14T17:05:25.260+01:00 INFO [JournalReader] Stopping.
> 2016-03-14T17:05:25.261+01:00 INFO [ServiceManagerListener] Services are
> now stopped.
> 2016-03-14T17:05:25.261+01:00 ERROR [ServerBootstrap] Graylog startup
> failed. Exiting. Exception was:
> java.lang.IllegalStateException: Expected to be healthy after starting.
> The following services are not running: {FAILED=[IndexerSetupService
> [FAILED]]}
> at
> com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:710)
> at
> com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:535)
> at
> com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:301)
> at
> org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:113)
> at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:196)
> at org.graylog2.bootstrap.Main.main(Main.java:58)
> 2016-03-14T17:05:25.262+01:00 INFO [Server] SIGNAL received. Shutting
> down.
> 2016-03-14T17:05:25.268+01:00 INFO [GracefulShutdown] Graceful shutdown
> initiated.
> 2016-03-14T17:05:25.269+01:00 INFO [GracefulShutdown] Node status:
> [Halting?[LB:DEAD]]. Waiting <3sec> for possible load balancers to
> recognize state change.
> 2016-03-14T17:05:29.271+01:00 INFO [GracefulShutdown] Goodbye.
>
>
>
> On Monday, March 14, 2016 at 4:54:06 PM UTC+1, Jochen Schalanda wrote:
>>
>> Hi Mattis,
>>
>> you shouldn't cut off the error message from the logs you've posted if
>> you expect anyone to help you. ;-)
>>
>> Cheers,
>> Jochen
>>
>> On Monday, 14 March 2016 16:51:54 UTC+1, Mattis Haase wrote:
>>>
>>> Two machines on the same VM host, one ES, on graylog, configured via
>>> puppet.
>>>
>>> Graylog Server error:
>>>
>>> 2016-03-14T16:40:10.280+01:00 INFO [transport] [graylog2-server]
>>> bound_address {inet[/0:0:0:0:0:0:0:0:9350]}, publish_address
>>> {inet[/a.b.c.d:9350]}
>>> 2016-03-14T16:40:10.296+01:00 INFO [discovery] [graylog2-server]
>>> graylog2/glh_l29gT9Sk1_nRRT3VNw
>>> 2016-03-14T16:40:10.300+01:00 WARN [ClusterStateMonitor] No
>>> Elasticsearch data nodes in cluster, cluster is completely offline.
>>> 2016-03-14T16:40:10.574+01:00 INFO [Reflections] Reflections took 448
>>> ms to scan 1 urls, producing 2 keys and 2 values
>>> 2016-03-14T16:40:13.299+01:00 WARN [discovery] [graylog2-server] waited
>>> for 3s and no initial state was set by the discovery
>>> 2016-03-14T16:40:13.299+01:00 INFO [node] [graylog2-server] started
>>> 2016-03-14T16:40:17.099+01:00 INFO [RestApiService] Adding security
>>> context factory:
>>> <org.graylog2.security.ShiroSecurityContextFactory@4b7ee77a>
>>> 2016-03-14T16:40:17.115+01:00 INFO [RestApiService] Started REST API at
>>> <http://127.0.0.1:12900/>
>>> 2016-03-14T16:40:18.301+01:00 INFO [IndexerSetupService] Checking
>>> Elasticsearch HTTP API at http://fqdn:9200/
>>> 2016-03-14T16:40:18.384+01:00 ERROR [UI]
>>>
>>> curl -X GET http://fqdn:9200
>>>
>>> {
>>> "status" : 200,
>>> "name" : "graylog2-server",
>>> "cluster_name" : "graylog2",
>>> "version" : {
>>> "number" : "1.7.5",
>>> "build_hash" : "00f95f4ffca6de89d68b7ccaf80d148f1f70e4d4",
>>> "build_timestamp" : "2016-02-02T09:55:30Z",
>>> "build_snapshot" : false,
>>> "lucene_version" : "4.10.4"
>>> },
>>> "tagline" : "You Know, for Search"
>>> }
>>>
>>> Config (using puppet):
>>> ES:
>>> class { 'elasticsearch':
>>> ensure => 'present',
>>> package_url => "
>>> https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${elasticversion}.deb
>>> ",
>>> config => {
>>> 'elasticsearch.discovery.zen.ping.multicast.enable' => false,
>>> 'elasticsearch.discovery.zen.ping.unicast.hosts' =>
>>> "${::fqdn}:9300",
>>> },
>>> }->
>>>
>>> elasticsearch::instance { 'graylog2':
>>> config => {
>>> 'node.name' =>
>>> 'graylog2-server',
>>> 'cluster.name' =>
>>> 'graylog2',
>>> 'network.host' =>
>>> $::ipaddress,
>>> 'transport.tcp.port' => '9300',
>>> 'http.port' => '9200',
>>> }
>>>
>>> Graylog:
>>> class {'graylog2::server':
>>> password_secret =>
>>> root_password_sha2 => ',
>>> elasticsearch_cluster_name => 'graylog2',
>>> elasticsearch_node_name =>
>>> 'graylog2-server',
>>> elasticsearch_discovery_zen_ping_multicast_enabled => false,
>>> elasticsearch_discovery_zen_ping_unicast_hosts => 'fqdn:9300',
>>> gc_warning_threshold => '15s',
>>> usage_statistics_enabled => false,
>>> }
>>>
>>> I tried pretty much every combination of using multicast, using unicast
>>> with different ports, manually setting elasticsearch_network_host
>>> and elasticsearch_transport_tcp_port. No other software that uses
>>> elasticsearch has any issues whatsoever, it just works. Using the same
>>> config but on one machine it also works.
>>>
>>> Any ideas?
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/8dc0e0e5-c43e-4f9a-a66b-4e28e81478d8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
