Hi,
today we updated Graylog to 2.0.3 and scince that no more data were send to
elasticsearch.
We recieve data but dont write them to elasticsearch....
Any idea why this could happen?
Greetings!
Graylog:
2016-06-27T10:26:43.744+02:00 INFO [CmdLineTool] Loaded plugin: Elastic
Beats Input 1.0.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2016-06-27T10:26:43.745+02:00 INFO [CmdLineTool] Loaded plugin: Collector
1.0.2 [org.graylog.plugins.collector.CollectorPlugin]
2016-06-27T10:26:43.745+02:00 INFO [CmdLineTool] Loaded plugin: Enterprise
Integration Plugin 1.0.2
[org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2016-06-27T10:26:43.745+02:00 INFO [CmdLineTool] Loaded plugin:
MapWidgetPlugin 1.0.2 [org.graylog.plugins.map.MapWidgetPlugin]
2016-06-27T10:26:43.746+02:00 INFO [CmdLineTool] Loaded plugin: Pipeline
Processor Plugin 1.0.0-beta.4
[org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2016-06-27T10:26:43.746+02:00 INFO [CmdLineTool] Loaded plugin: Anonymous
Usage Statistics 2.0.2
[org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2016-06-27T10:26:43.856+02:00 INFO [CmdLineTool] Running with JVM
arguments: -Xms1g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB
-XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled
-XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC
-XX:-OmitStackTraceInFastThrow
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
-Djava.library.path=/usr/share/graylog-server/lib/sigar
-Dgraylog2.installation_source=rpm
2016-06-27T10:26:46.138+02:00 INFO [InputBufferImpl] Message journal is
enabled.
2016-06-27T10:26:46.348+02:00 INFO [LogManager] Loading logs.
2016-06-27T10:26:46.525+02:00 INFO [LogManager] Logs loading complete.
2016-06-27T10:26:46.526+02:00 INFO [KafkaJournal] Initialized Kafka based
journal at /var/lib/graylog-server/journal
2016-06-27T10:26:46.541+02:00 INFO [InputBufferImpl] Initialized
InputBufferImpl with ring size <65536> and wait strategy
<BlockingWaitStrategy>, running 2 parallel message handlers.
2016-06-27T10:26:46.581+02:00 INFO [cluster] Cluster created with settings
{hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN,
serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2016-06-27T10:26:46.615+02:00 INFO [cluster] No server chosen by
ReadPreferenceServerSelector{readPreference=primary} from cluster
description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE,
all=[ServerDescription{address=localhost:27017, type=UNKNOWN,
state=CONNECTING}]}. Waiting for 30000 ms before timing out
2016-06-27T10:26:46.653+02:00 INFO [connection] Opened connection
[connectionId{localValue:1, serverValue:36}] to localhost:27017
2016-06-27T10:26:46.655+02:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=localhost:27017, type=STANDALONE,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 7]},
minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216,
roundTripTimeNanos=736400}
2016-06-27T10:26:46.661+02:00 INFO [connection] Opened connection
[connectionId{localValue:2, serverValue:37}] to localhost:27017
2016-06-27T10:26:46.903+02:00 INFO [NodeId] Node ID:
006e89e5-73c8-46dd-ac86-7c1ddb26ed84
2016-06-27T10:26:47.015+02:00 INFO [node]
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] version[2.3.2], pid[3533],
build[b9e4a6a/2016-04-21T16:03:47Z]
2016-06-27T10:26:47.015+02:00 INFO [node]
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] initializing ...
2016-06-27T10:26:47.022+02:00 INFO [plugins]
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] modules [], plugins
[graylog-monitor], sites []
2016-06-27T10:26:48.829+02:00 INFO [node]
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] initialized
2016-06-27T10:26:48.904+02:00 INFO [Version] HV000001: Hibernate Validator
5.2.4.Final
2016-06-27T10:26:49.054+02:00 INFO [ProcessBuffer] Initialized
ProcessBuffer with ring size <65536> and wait strategy
<BlockingWaitStrategy>.
2016-06-27T10:26:51.042+02:00 INFO [RulesEngineProvider] No static rules
file loaded.
2016-06-27T10:26:51.085+02:00 INFO [connection] Opened connection
[connectionId{localValue:3, serverValue:38}] to localhost:27017
2016-06-27T10:26:51.192+02:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-27T10:26:51.198+02:00 INFO [OutputBuffer] Initialized OutputBuffer
with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-06-27T10:26:51.242+02:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-27T10:26:51.285+02:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-27T10:26:51.327+02:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-27T10:26:51.362+02:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-27T10:26:51.864+02:00 INFO [ServerBootstrap] Graylog server 2.0.2
(4da1379) starting up
2016-06-27T10:26:51.864+02:00 INFO [ServerBootstrap] JRE: Oracle
Corporation 1.8.0_77 on Linux 3.10.0-327.22.2.el7.x86_64
2016-06-27T10:26:51.865+02:00 INFO [ServerBootstrap] Deployment: rpm
2016-06-27T10:26:51.865+02:00 INFO [ServerBootstrap] OS: CentOS Linux 7
(Core) (centos)
2016-06-27T10:26:51.865+02:00 INFO [ServerBootstrap] Arch: amd64
2016-06-27T10:26:51.868+02:00 WARN [DeadEventLoggingListener] Received
unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from
event bus <AsyncEventBus{graylog-eventbus}>
2016-06-27T10:26:51.894+02:00 INFO [PeriodicalsService] Starting 24
periodicals ...
2016-06-27T10:26:51.894+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling
every [1s].
2016-06-27T10:26:51.907+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling
every [60s].
2016-06-27T10:26:51.908+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical
in [0s], polling every [1s].
2016-06-27T10:26:51.910+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s],
polling every [20s].
2016-06-27T10:26:51.912+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running
forever.
2016-06-27T10:26:51.913+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.GarbageCollectionWarningThread] periodical,
running forever.
2016-06-27T10:26:51.915+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s],
polling every [30s].
2016-06-27T10:26:51.916+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling
every [300s].
2016-06-27T10:26:51.916+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling
every [10s].
2016-06-27T10:26:51.916+02:00 INFO [IndexRetentionThread] Elasticsearch
cluster not available, skipping index retention checks.
2016-06-27T10:26:51.921+02:00 INFO [node]
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] starting ...
2016-06-27T10:26:51.923+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.NodePingThread] periodical in [0s], polling every
[1s].
2016-06-27T10:26:51.923+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling
every [1800s].
2016-06-27T10:26:51.933+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s],
polling every [1s].
2016-06-27T10:26:51.933+02:00 INFO [Periodicals] Starting
[org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling
every [1s].
2016-06-27T10:26:51.934+02:00 INFO [Periodicals] Starting
[org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s],
polling every [300s].
2016-06-27T10:26:51.934+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running
forever.
2016-06-27T10:26:51.934+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical,
running forever.
2016-06-27T10:26:51.935+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s],
polling every [3600s].
2016-06-27T10:26:51.942+02:00 INFO [connection] Opened connection
[connectionId{localValue:7, serverValue:42}] to localhost:27017
2016-06-27T10:26:51.944+02:00 INFO [connection] Opened connection
[connectionId{localValue:6, serverValue:41}] to localhost:27017
2016-06-27T10:26:51.947+02:00 INFO [connection] Opened connection
[connectionId{localValue:5, serverValue:40}] to localhost:27017
2016-06-27T10:26:51.948+02:00 INFO [connection] Opened connection
[connectionId{localValue:4, serverValue:39}] to localhost:27017
2016-06-27T10:26:51.954+02:00 INFO [IndexerClusterCheckerThread] Indexer
not fully initialized yet. Skipping periodic cluster check.
2016-06-27T10:26:51.977+02:00 INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not
configured to run on this node.
2016-06-27T10:26:51.978+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical,
running forever.
2016-06-27T10:26:51.978+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ConfigurationManagementPeriodical] periodical,
running forever.
2016-06-27T10:26:51.983+02:00 INFO [Periodicals] Starting
[org.graylog2.periodical.LdapGroupMappingMigration] periodical, running
forever.
2016-06-27T10:26:51.991+02:00 INFO [Periodicals] Starting
[org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical
in [300s], polling every [21600s].
2016-06-27T10:26:51.991+02:00 INFO [Periodicals] Starting
[org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical]
periodical in [300s], polling every [21600s].
2016-06-27T10:26:51.991+02:00 INFO [Periodicals] Starting
[org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread]
periodical in [0s], polling every [3600s].
2016-06-27T10:26:52.215+02:00 INFO [transport]
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] publish_address
{127.0.0.1:9350}, bound_addresses {[::1]:9350}, {127.0.0.1:9350}
2016-06-27T10:26:52.230+02:00 INFO [discovery]
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84]
graylog/nHdvfXVhTeCRPTcMlns-vQ
2016-06-27T10:26:52.398+02:00 INFO [AbstractJerseyService] Enabling CORS
for HTTP endpoint
2016-06-27T10:26:55.243+02:00 INFO [NetworkListener] Started listener
bound to [10.10.31.39:9000]
2016-06-27T10:26:55.245+02:00 WARN [discovery]
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] waited for 3s and no initial
state was set by the discovery
2016-06-27T10:26:55.245+02:00 INFO [node]
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] started
2016-06-27T10:26:55.246+02:00 INFO [HttpServer] [HttpServer] Started.
2016-06-27T10:26:55.246+02:00 INFO [WebInterfaceService] Started Web
Interface at <http://10.10.31.39:9000/>
2016-06-27T10:26:55.409+02:00 INFO [service]
[graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84] detected_master {Ororo
Munroe}{2CdooCmUR-2PjnE2Q9wS3Q}{10.10.31.39}{10.10.31.39:9300}, added
{{Ororo Munroe}{2CdooCmUR-2PjnE2Q9wS3Q}{10.10.31.39}{10.10.31.39:9300},},
reason: zen-disco-receive(from master [{Ororo
Munroe}{2CdooCmUR-2PjnE2Q9wS3Q}{10.10.31.39}{10.10.31.39:9300}])
2016-06-27T10:26:59.658+02:00 INFO [NetworkListener] Started listener
bound to [10.10.31.39:12900]
2016-06-27T10:26:59.658+02:00 INFO [HttpServer] [HttpServer-1] Started.
2016-06-27T10:26:59.659+02:00 INFO [RestApiService] Started REST API at
<http://10.10.31.39:12900/>
2016-06-27T10:26:59.660+02:00 INFO [ServiceManagerListener] Services are
healthy
2016-06-27T10:26:59.661+02:00 INFO [ServerBootstrap] Services started,
startup times in ms: {InputSetupService [RUNNING]=5, KafkaJournal
[RUNNING]=21, OutputSetupService [RUNNING]=39, BufferSynchronizerService
[RUNNING]=51, MetricsReporterService [RUNNING]=52, PeriodicalsService
[RUNNING]=111, JournalReader [RUNNING]=133, WebInterfaceService
[RUNNING]=3326, IndexerSetupService [RUNNING]=3542, RestApiService
[RUNNING]=7767}
2016-06-27T10:26:59.666+02:00 INFO [ServerBootstrap] Graylog server up and
running.
2016-06-27T10:26:59.666+02:00 INFO [InputSetupService] Triggering
launching persisted inputs, node transitioned from Uninitialized [LB:DEAD]
to Running [LB:ALIVE]
2016-06-27T10:26:59.706+02:00 INFO [InputStateListener] Input [Syslog
UDP/575ec33c67bdba09b6797184] is now STARTING
2016-06-27T10:26:59.712+02:00 INFO [InputStateListener] Input [Syslog
UDP/575ec49967bdba09b6797313] is now STARTING
2016-06-27T10:26:59.713+02:00 INFO [InputStateListener] Input [Syslog
UDP/575ec50a67bdba09b6797396] is now STARTING
2016-06-27T10:26:59.720+02:00 INFO [InputStateListener] Input
[Beats/575ec43a67bdba09b67972a9] is now STARTING
2016-06-27T10:26:59.749+02:00 INFO [InputStateListener] Input [Syslog
UDP/575ec49967bdba09b6797313] is now RUNNING
2016-06-27T10:26:59.751+02:00 INFO [InputStateListener] Input [Syslog
UDP/575ec33c67bdba09b6797184] is now RUNNING
2016-06-27T10:26:59.757+02:00 INFO [InputStateListener] Input [Syslog
UDP/575ec50a67bdba09b6797396] is now RUNNING
2016-06-27T10:26:59.763+02:00 INFO [InputStateListener] Input
[Beats/575ec43a67bdba09b67972a9] is now RUNNING
Elasticsearch:
[2016-06-27 10:14:31,845][INFO ][node ] [Ororo Munroe]
version[2.3.2], pid[2529], build[b9e4a6a/2016-04-21T16:03:47Z]
[2016-06-27 10:14:31,846][INFO ][node ] [Ororo Munroe]
initializing ...
[2016-06-27 10:14:32,483][INFO ][plugins ] [Ororo Munroe]
modules [reindex, lang-expression, lang-groovy], plugins [kopf], sites
[kopf]
[2016-06-27 10:14:32,509][INFO ][env ] [Ororo Munroe]
using [1] data paths, mounts [[/var/opt/graylog
(/dev/mapper/daten-graylog)]], net usable_space [107.3gb], net total_space
[295.1gb], spins? [possibly], types [ext4]
[2016-06-27 10:14:32,509][INFO ][env ] [Ororo Munroe]
heap size [990.7mb], compressed ordinary object pointers [true]
[2016-06-27 10:14:32,509][WARN ][env ] [Ororo Munroe]
max file descriptors [65535] for elasticsearch process likely too low,
consider increasing to at least [65536]
[2016-06-27 10:14:34,528][INFO ][node ] [Ororo Munroe]
initialized
[2016-06-27 10:14:34,529][INFO ][node ] [Ororo Munroe]
starting ...
[2016-06-27 10:14:34,639][INFO ][transport ] [Ororo Munroe]
publish_address {10.10.31.39:9300}, bound_addresses {10.10.31.39:9300}
[2016-06-27 10:14:34,644][INFO ][discovery ] [Ororo Munroe]
graylog/2CdooCmUR-2PjnE2Q9wS3Q
[2016-06-27 10:14:37,673][INFO ][cluster.service ] [Ororo Munroe]
new_master {Ororo
Munroe}{2CdooCmUR-2PjnE2Q9wS3Q}{10.10.31.39}{10.10.31.39:9300}, reason:
zen-disco-join(elected_as_master, [0] joins received)
[2016-06-27 10:14:37,693][INFO ][http ] [Ororo Munroe]
publish_address {10.10.31.39:9200}, bound_addresses {10.10.31.39:9200}
[2016-06-27 10:14:37,693][INFO ][node ] [Ororo Munroe]
started
[2016-06-27 10:14:38,034][INFO ][gateway ] [Ororo Munroe]
recovered [20] indices into cluster_state
[2016-06-27 10:14:41,106][INFO ][cluster.routing.allocation] [Ororo Munroe]
Cluster health status changed from [RED] to [YELLOW] (reason: [shards
started [[graylog_0][0]] ...]).
[2016-06-27 10:14:51,576][INFO ][cluster.service ] [Ororo Munroe]
added
{{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{Tby982zpQRaAHjdvOTnWwg}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false},}, reason: zen-disco-join(join from
node[{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{Tby982zpQRaAHjdvOTnWwg}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false}])
[2016-06-27 10:23:52,016][INFO ][cluster.service ] [Ororo Munroe]
removed
{{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{Tby982zpQRaAHjdvOTnWwg}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false},}, reason:
zen-disco-node_left({graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{Tby982zpQRaAHjdvOTnWwg}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false})
[2016-06-27 10:24:07,320][INFO ][cluster.service ] [Ororo Munroe]
added
{{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{pRU53I8JThWnjOo8pdybSA}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false},}, reason: zen-disco-join(join from
node[{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{pRU53I8JThWnjOo8pdybSA}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false}])
[2016-06-27 10:25:24,559][INFO ][cluster.service ] [Ororo Munroe]
removed
{{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{pRU53I8JThWnjOo8pdybSA}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false},}, reason:
zen-disco-node_left({graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{pRU53I8JThWnjOo8pdybSA}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false})
[2016-06-27 10:25:40,075][INFO ][cluster.service ] [Ororo Munroe]
added
{{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{333_uNIgQEaOUtJI1xh6sA}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false},}, reason: zen-disco-join(join from
node[{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{333_uNIgQEaOUtJI1xh6sA}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false}])
[2016-06-27 10:26:40,177][INFO ][cluster.service ] [Ororo Munroe]
removed
{{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{333_uNIgQEaOUtJI1xh6sA}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false},}, reason:
zen-disco-node_left({graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{333_uNIgQEaOUtJI1xh6sA}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false})
[2016-06-27 10:26:55,392][INFO ][cluster.service ] [Ororo Munroe]
added
{{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{nHdvfXVhTeCRPTcMlns-vQ}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false},}, reason: zen-disco-join(join from
node[{graylog-006e89e5-73c8-46dd-ac86-7c1ddb26ed84}{nHdvfXVhTeCRPTcMlns-vQ}{127.0.0.1}{127.0.0.1:9350}{client=true,
data=false, master=false}])
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/dcfcaa0d-1ba8-4120-9484-7ef9bdb4b31b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
