I am testing out graylog 2.0.2 with elasticsearch 2.3.3 (with 2 nodes for
now for elasticsearch, will become 3 to avoid split-brain ) and mongodb
2.4.14 (in 3-way replica set running on 2 graylog nodes and 1 mongo arbiter
on the loadbalancer in front of graylog ).
However, I'm constantly encountering an error where graylog-server is
unable to connect to Elasticsearch and now I'm not sure why it is not
working. I had tested with the 2.0-Beta before and that worked without
issues.
Details :
-------------------------------------------------------------------------------------------
Elasticsearch -
rpm version: elasticsearch-2.3.3-1.noarch
config:
cluster.name: graylognew
node.name: graylog-es01
path.data: /elasticsearch
network.host: 10.30.20.58
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.30.20.58:9300","10.30.20.59:9300"]
log:
[2016-06-21 09:33:08,599][WARN ][bootstrap ] unable to
install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled
into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2016-06-21 09:33:08,697][INFO ][node ] [graylog-es01]
version[2.3.3], pid[3850], build[218bdf1/2016-05-17T15:40:04Z]
[2016-06-21 09:33:08,697][INFO ][node ] [graylog-es01]
initializing ...
[2016-06-21 09:33:09,034][INFO ][plugins ] [graylog-es01]
modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-06-21 09:33:09,048][INFO ][env ] [graylog-es01]
using [1] data paths, mounts [[/elasticsearch (/dev/md2)]], net
usable_space [733.1gb], net total_space [733.2gb], spins? [possibly], types
[ext4]
[2016-06-21 09:33:09,048][INFO ][env ] [graylog-es01]
heap size [31.8gb], compressed ordinary object pointers [false]
[2016-06-21 09:33:09,048][WARN ][env ] [graylog-es01]
max file descriptors [65535] for elasticsearch process likely too low,
consider increasing to at least [65536]
[2016-06-21 09:33:09,999][INFO ][node ] [graylog-es01]
initialized
[2016-06-21 09:33:09,999][INFO ][node ] [graylog-es01]
starting ...
[2016-06-21 09:33:10,141][INFO ][transport ] [graylog-es01]
publish_address {10.30.20.58:9300}, bound_addresses {10.30.20.58:9300}
[2016-06-21 09:33:10,144][INFO ][discovery ] [graylog-es01]
graylognew/aFMNHpUWScWRtr6AmpMa0Q
[2016-06-21 09:33:13,193][INFO ][cluster.service ] [graylog-es01]
new_master
{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300},
reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-06-21 09:33:13,208][INFO ][http ] [graylog-es01]
publish_address {10.30.20.58:9200}, bound_addresses {10.30.20.58:9200}
[2016-06-21 09:33:13,208][INFO ][node ] [graylog-es01]
started
[2016-06-21 09:33:13,223][INFO ][gateway ] [graylog-es01]
recovered [0] indices into cluster_state
[2016-06-21 09:34:25,603][INFO ][cluster.service ] [graylog-es01]
added
{{graylog-es02}{2Ty5iLUTSbWe5QznunoHkA}{10.30.20.59}{10.30.20.59:9300},},
reason: zen-disco-join(join from
node[{graylog-es02}{2Ty5iLUTSbWe5QznunoHkA}{10.30.20.59}{10.30.20.59:9300}])
<-- nothing else seen after this, was expecting to see graylog2 server
connect to elasticsearch
{
"cluster_name" : "graylognew",
"nodes" : {
"2Ty5iLUTSbWe5QznunoHkA" : {
"name" : "graylog-es02",
"transport_address" : "10.30.20.59:9300",
"host" : "10.30.20.59",
"ip" : "10.30.20.59",
"version" : "2.3.3",
"build" : "218bdf1",
"http_address" : "10.30.20.59:9200",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 3267,
"mlockall" : false
}
},
"aFMNHpUWScWRtr6AmpMa0Q" : {
"name" : "graylog-es01",
"transport_address" : "10.30.20.58:9300",
"host" : "10.30.20.58",
"ip" : "10.30.20.58",
"version" : "2.3.3",
"build" : "218bdf1",
"http_address" : "10.30.20.58:9200",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 3850,
"mlockall" : false
}
}
}
}
graylog-es02:/var/log/elasticsearch# curl
http://10.30.20.58:9200/_cluster/health?pretty
{
"cluster_name" : "graylognew",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
-------------------------------------------------------------------------------------------
Graylog
rpmversion : graylog-server-2.0.2-1.noarch
config :
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = .......
root_password_sha2 = ........
root_email = "[email protected]"
root_timezone = US/Eastern
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://graylog-web01:12900/
web_listen_uri = http://10.30.20.60:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 50
retention_strategy = delete
elasticsearch_shards = 8
elasticsearch_replicas = 1
elasticsearch_index_prefix = graylognew
allow_leading_wildcard_searches = false
allow_highlighting = true
elasticsearch_cluster_name = graylognew
elasticsearch_node_name_prefix = graylog2-web01-
elasticsearch_discovery_zen_ping_multicast_enabled = false
elasticsearch_discovery_zen_ping_unicast_hosts = 10.30.20.58:9300,
10.30.20.59:9300
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri =
mongodb://mongouser:password@graylog-web01,graylog-web02/graylog2
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_hostname = mail.XYZ.com
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_subject_prefix = [graylog2]
transport_email_from_email = [email protected]
transport_email_web_interface_url = http://graylog-web01:9000
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
log :
2016-06-21T09:36:45.014-04:00 INFO [CmdLineTool] Loaded plugin: Collector
1.0.2 [org.graylog.plugins.collector.CollectorPlugin]
2016-06-21T09:36:45.016-04:00 INFO [CmdLineTool] Loaded plugin: Enterprise
Integration Plugin 1.0.2
[org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2016-06-21T09:36:45.016-04:00 INFO [CmdLineTool] Loaded plugin:
MapWidgetPlugin 1.0.2 [org.graylog.plugins.map.MapWidgetPlugin]
2016-06-21T09:36:45.016-04:00 INFO [CmdLineTool] Loaded plugin: Pipeline
Processor Plugin 1.0.0-beta.4
[org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2016-06-21T09:36:45.017-04:00 INFO [CmdLineTool] Loaded plugin: Anonymous
Usage Statistics 2.0.2
[org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2016-06-21T09:36:45.169-04:00 INFO [CmdLineTool] Running with JVM
arguments: -Xms4g -Xmx4g -XX:NewRatio=1 -XX:+ResizeTLAB
-XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled
-XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC
-XX:-OmitStackTraceInFastThrow
-Djava.library.path=/usr/share/graylog-server/lib/sigar
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
-Dgraylog2.installation_source=rpm
2016-06-21T09:36:48.826-04:00 INFO [InputBufferImpl] Message journal is
enabled.
2016-06-21T09:36:49.243-04:00 INFO [LogManager] Loading logs.
2016-06-21T09:36:49.525-04:00 INFO [LogManager] Logs loading complete.
2016-06-21T09:36:49.526-04:00 INFO [KafkaJournal] Initialized Kafka based
journal at /var/lib/graylog-server/journal
2016-06-21T09:36:49.566-04:00 INFO [InputBufferImpl] Initialized
InputBufferImpl with ring size <65536> and wait strategy
<BlockingWaitStrategy>, running 2 parallel message handlers.
2016-06-21T09:36:49.643-04:00 INFO [cluster] Cluster created with settings
{hosts=[graylog-web01:27017, graylog-web02:27017], mode=MULTIPLE,
requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms',
maxWaitQueueSize=5000}
2016-06-21T09:36:49.644-04:00 INFO [cluster] Adding discovered server
graylog-web01:27017 to client view of cluster
2016-06-21T09:36:49.674-04:00 INFO [cluster] Adding discovered server
graylog-web02:27017 to client view of cluster
2016-06-21T09:36:49.731-04:00 INFO [cluster] No server chosen by
ReadPreferenceServerSelector{readPreference=primary} from cluster
description ClusterDescription{type=UNKNOWN, connectionMode=MULTIPLE,
all=[ServerDescription{address=graylog-web01:27017, type=UNKNOWN,
state=CONNECTING}, ServerDescription{address=graylog-web02:27017,
type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2016-06-21T09:36:49.817-04:00 INFO [connection] Opened connection
[connectionId{localValue:1, serverValue:27913}] to graylog-web02:27017
2016-06-21T09:36:49.817-04:00 INFO [connection] Opened connection
[connectionId{localValue:2, serverValue:27987}] to graylog-web01:27017
2016-06-21T09:36:49.834-04:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=graylog-web02:27017, type=REPLICA_SET_SECONDARY,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 14]},
minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216,
roundTripTimeNanos=2315552, setName='graylog2Repl',
canonicalAddress=graylog-web02.:27017, hosts=[graylog-web02.:27017,
graylog-web01.:27017], passives=[], arbiters=[graylog.:27017],
primary='graylog-web01.:27017', tagSet=TagSet{[]}, electionId=null,
setVersion=null}
2016-06-21T09:36:49.834-04:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=graylog-web01:27017, type=REPLICA_SET_PRIMARY,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 14]},
minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216,
roundTripTimeNanos=1859299, setName='graylog2Repl',
canonicalAddress=graylog-web01.:27017, hosts=[graylog-web02.:27017,
graylog-web01.:27017], passives=[], arbiters=[graylog.:27017],
primary='graylog-web01.:27017', tagSet=TagSet{[]}, electionId=null,
setVersion=null}
2016-06-21T09:36:49.840-04:00 INFO [cluster] Discovered cluster type of
REPLICA_SET
2016-06-21T09:36:49.842-04:00 INFO [cluster] Adding discovered server
graylog-web02.:27017 to client view of cluster
2016-06-21T09:36:49.845-04:00 INFO [cluster] Adding discovered server
graylog-web01.:27017 to client view of cluster
2016-06-21T09:36:49.846-04:00 INFO [cluster] Adding discovered server
graylog.:27017 to client view of cluster
2016-06-21T09:36:49.846-04:00 INFO [cluster] Canonical address
graylog-web02.:27017 does not match server address. Removing
graylog-web02:27017 from client view of cluster
2016-06-21T09:36:49.854-04:00 INFO [cluster] Server graylog-web01:27017 is
no longer a member of the replica set. Removing from client view of
cluster.
2016-06-21T09:36:49.855-04:00 INFO [connection] Opened connection
[connectionId{localValue:5, serverValue:27856}] to graylog.:27017
2016-06-21T09:36:49.856-04:00 INFO [cluster] Canonical address
graylog-web01.:27017 does not match server address. Removing
graylog-web01:27017 from client view of cluster
2016-06-21T09:36:49.857-04:00 INFO [connection] Opened connection
[connectionId{localValue:3, serverValue:27914}] to graylog-web02.:27017
2016-06-21T09:36:49.858-04:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=graylog.:27017, type=REPLICA_SET_ARBITER,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 14]},
minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216,
roundTripTimeNanos=1921928, setName='graylog2Repl',
canonicalAddress=graylog.:27017, hosts=[graylog-web02.:27017,
graylog-web01.:27017], passives=[], arbiters=[graylog.:27017],
primary='graylog-web01.:27017', tagSet=TagSet{[]}, electionId=null,
setVersion=null}
2016-06-21T09:36:49.860-04:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=graylog-web02.:27017, type=REPLICA_SET_SECONDARY,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 14]},
minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216,
roundTripTimeNanos=1739874, setName='graylog2Repl',
canonicalAddress=graylog-web02.:27017, hosts=[graylog-web02.:27017,
graylog-web01.:27017], passives=[], arbiters=[graylog.:27017],
primary='graylog-web01.:27017', tagSet=TagSet{[]}, electionId=null,
setVersion=null}
2016-06-21T09:36:49.896-04:00 INFO [connection] Opened connection
[connectionId{localValue:4, serverValue:27988}] to graylog-web01.:27017
2016-06-21T09:36:49.899-04:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=graylog-web01.:27017, type=REPLICA_SET_PRIMARY,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 14]},
minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216,
roundTripTimeNanos=1006498, setName='graylog2Repl',
canonicalAddress=graylog-web01.:27017, hosts=[graylog-web02.:27017,
graylog-web01.:27017], passives=[], arbiters=[graylog.:27017],
primary='graylog-web01.:27017', tagSet=TagSet{[]}, electionId=null,
setVersion=null}
2016-06-21T09:36:49.899-04:00 INFO [cluster] Discovered replica set
primary graylog-web01.:27017
2016-06-21T09:36:49.909-04:00 INFO [connection] Opened connection
[connectionId{localValue:6, serverValue:27989}] to graylog-web01.:27017
2016-06-21T09:36:50.292-04:00 INFO [NodeId] Node ID:
90a4086e-d119-483f-953e-4f34f9524578
2016-06-21T09:36:50.397-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] version[2.3.2],
pid[29127], build[b9e4a6a/2016-04-21T16:03:47Z]
2016-06-21T09:36:50.398-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] initializing ...
2016-06-21T09:36:50.408-04:00 INFO [plugins]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] modules [], plugins
[graylog-monitor], sites []
2016-06-21T09:36:53.199-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] initialized
2016-06-21T09:36:53.309-04:00 INFO [Version] HV000001: Hibernate Validator
5.2.4.Final
2016-06-21T09:36:53.536-04:00 INFO [ProcessBuffer] Initialized
ProcessBuffer with ring size <65536> and wait strategy
<BlockingWaitStrategy>.
2016-06-21T09:36:56.162-04:00 INFO [RulesEngineProvider] No static rules
file loaded.
2016-06-21T09:36:56.386-04:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-21T09:36:56.397-04:00 INFO [OutputBuffer] Initialized OutputBuffer
with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-06-21T09:36:56.469-04:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-21T09:36:56.534-04:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-21T09:36:56.637-04:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-21T09:36:56.715-04:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-21T09:36:57.591-04:00 INFO [connection] Opened connection
[connectionId{localValue:7, serverValue:27991}] to graylog-web01.:27017
2016-06-21T09:36:58.238-04:00 INFO [ServerBootstrap] Graylog server 2.0.2
(4da1379) starting up
2016-06-21T09:36:58.239-04:00 INFO [ServerBootstrap] JRE: Oracle
Corporation 1.8.0_91 on Linux 3.10.36-el6.ia32e.limeprod.0
2016-06-21T09:36:58.239-04:00 INFO [ServerBootstrap] Deployment: rpm
2016-06-21T09:36:58.239-04:00 INFO [ServerBootstrap] OS: Red Hat
Enterprise Linux Server release 6.8 (Santiago)
2016-06-21T09:36:58.239-04:00 INFO [ServerBootstrap] Arch: amd64
2016-06-21T09:36:58.245-04:00 WARN [DeadEventLoggingListener] Received
unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from
event bus <AsyncEventBus{graylog-eventbus}>
2016-06-21T09:36:58.273-04:00 INFO [PeriodicalsService] Starting 24
periodicals ...
2016-06-21T09:36:58.274-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling
every [1s].
2016-06-21T09:36:58.280-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling
every [60s].
2016-06-21T09:36:58.282-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical
in [0s], polling every [1s].
2016-06-21T09:36:58.285-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s],
polling every [20s].
2016-06-21T09:36:58.598-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running
forever.
2016-06-21T09:36:58.592-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] starting ...
2016-06-21T09:36:58.600-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.GarbageCollectionWarningThread] periodical,
running forever.
2016-06-21T09:36:58.603-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s],
polling every [30s].
2016-06-21T09:36:58.611-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling
every [300s].
2016-06-21T09:36:58.617-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling
every [10s].
2016-06-21T09:36:58.617-04:00 INFO [IndexRetentionThread] Elasticsearch
cluster not available, skipping index retention checks.
2016-06-21T09:36:58.623-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.NodePingThread] periodical in [0s], polling every
[1s].
2016-06-21T09:36:58.280-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling
every [60s].
2016-06-21T09:36:58.282-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical
in [0s], polling every [1s].
2016-06-21T09:36:58.285-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s],
polling every [20s].
2016-06-21T09:36:58.598-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running
forever.
2016-06-21T09:36:58.592-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] starting ...
2016-06-21T09:36:58.600-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.GarbageCollectionWarningThread] periodical,
running forever.
2016-06-21T09:36:58.603-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s],
polling every [30s].
2016-06-21T09:36:58.611-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling
every [300s].
2016-06-21T09:36:58.617-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling
every [10s].
2016-06-21T09:36:58.617-04:00 INFO [IndexRetentionThread] Elasticsearch
cluster not available, skipping index retention checks.
2016-06-21T09:36:58.623-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.NodePingThread] periodical in [0s], polling every
[1s].
2016-06-21T09:36:58.628-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling
every [1800s].
2016-06-21T09:36:58.628-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s],
polling every [1s].
2016-06-21T09:36:58.629-04:00 INFO [Periodicals] Starting
[org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling
every [1s].
2016-06-21T09:36:58.630-04:00 INFO [Periodicals] Starting
[org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s],
polling every [300s].
2016-06-21T09:36:58.631-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running
forever.
2016-06-21T09:36:58.631-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical,
running forever.
2016-06-21T09:36:58.632-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s],
polling every [3600s].
2016-06-21T09:36:58.633-04:00 INFO [connection] Opened connection
[connectionId{localValue:8, serverValue:27992}] to graylog-web01.:27017
2016-06-21T09:36:58.640-04:00 INFO [connection] Opened connection
[connectionId{localValue:9, serverValue:27993}] to graylog-web01.:27017
2016-06-21T09:36:58.641-04:00 INFO [connection] Opened connection
[connectionId{localValue:10, serverValue:27994}] to graylog-web01.:27017
2016-06-21T09:36:58.652-04:00 INFO [connection] Opened connection
[connectionId{localValue:11, serverValue:27995}] to graylog-web01.:27017
2016-06-21T09:36:58.659-04:00 INFO [IndexerClusterCheckerThread] Indexer
not fully initialized yet. Skipping periodic cluster check.
2016-06-21T09:36:58.706-04:00 INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not
configured to run on this node.
2016-06-21T09:36:58.707-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical,
running forever.
2016-06-21T09:36:58.708-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ConfigurationManagementPeriodical] periodical,
running forever.
2016-06-21T09:36:58.713-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.LdapGroupMappingMigration] periodical, running
forever.
2016-06-21T09:36:58.715-04:00 INFO [Periodicals] Starting
[org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical
in [300s], polling every [21600s].
2016-06-21T09:36:58.716-04:00 INFO [Periodicals] Starting
[org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical]
periodical in [300s], polling every [21600s].
2016-06-21T09:36:58.717-04:00 INFO [Periodicals] Starting
[org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread]
periodical in [0s], polling every [3600s].
2016-06-21T09:36:58.859-04:00 INFO [transport]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] publish_address
{127.0.0.1:9350}, bound_addresses {127.0.0.1:9350}
2016-06-21T09:36:58.870-04:00 INFO [discovery]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578]
graylognew/YH3f_DKNRGuzMVmX87B3DQ
2016-06-21T09:36:59.121-04:00 INFO [AbstractJerseyService] Enabling CORS
for HTTP endpoint
2016-06-21T09:37:01.875-04:00 WARN [discovery]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] waited for 3s and no
initial state was set by the discovery
2016-06-21T09:37:01.875-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] started
2016-06-21T09:37:01.917-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:03.891-04:00 INFO [NetworkListener] Started listener
bound to [10.30.20.60:9000]
2016-06-21T09:37:03.894-04:00 INFO [HttpServer] [HttpServer] Started.
2016-06-21T09:37:03.896-04:00 INFO [WebInterfaceService] Started Web
Interface at <http://10.30.20.60:9000/>
2016-06-21T09:37:04.934-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:06.884-04:00 WARN [IndexerSetupService] Could not connect
to Elasticsearch
2016-06-21T09:37:06.884-04:00 INFO [IndexerSetupService] If you're using
multicast, check that it is working in your network and that Elasticsearch
is accessible. Also check that the cluster name setting is correct.
2016-06-21T09:37:06.885-04:00 INFO [IndexerSetupService] See
http://docs.graylog.org/en/2.0/pages/configuring_es.html for details.
2016-06-21T09:37:07.958-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:10.857-04:00 INFO [NetworkListener] Started listener
bound to [graylog-web01:12900]
2016-06-21T09:37:10.858-04:00 INFO [HttpServer] [HttpServer-1] Started.
2016-06-21T09:37:10.860-04:00 INFO [RestApiService] Started REST API at
<http://graylog-web01:12900/>
2016-06-21T09:37:10.864-04:00 INFO [ServiceManagerListener] Services are
healthy
2016-06-21T09:37:10.866-04:00 INFO [InputSetupService] Triggering
launching persisted inputs, node transitioned from Uninitialized [LB:DEAD]
to Running [LB:ALIVE]
2016-06-21T09:37:10.868-04:00 INFO [ServerBootstrap] Services started,
startup times in ms: {MetricsReporterService [RUNNING]=4,
BufferSynchronizerService [RUNNING]=4, InputSetupService [RUNNING]=6,
JournalReader [RUNNING]=7, KafkaJournal [RUNNING]=11, OutputSetupService
[RUNNING]=310, PeriodicalsService [RUNNING]=448, WebInterfaceService
[RUNNING]=5612, IndexerSetupService [RUNNING]=8606, RestApiService
[RUNNING]=12590}
2016-06-21T09:37:10.875-04:00 INFO [ServerBootstrap] Graylog server up and
running.
2016-06-21T09:37:10.975-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:13.632-04:00 INFO [IndexRangesCleanupPeriodical] Skipping
index range cleanup because the Elasticsearch cluster is unreachable or
unhealthy
2016-06-21T09:37:13.989-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:17.000-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:20.010-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:23.020-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:26.030-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:29.046-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:32.056-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
...
Why is graylog-server unable to connect to elasticsearch? I can connect to
elasticsearch from the graylog servers without issue via netcat and curl.
There's no firewalls blocking anything and everything is in the same vlan.
Thanks,
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/70d5a81a-ba6b-420c-8b04-0b7489f3a831%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
