Hi All !!
Graylog server is throwing following error. Exception in thread
"elasticsearch[graylog2-server][generic][T#1]"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"elasticsearch[graylog2-server][generic][T#1]".
complete logs trace is as follows
2016-06-23 12:30:47,811 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded
plugins: [AWS input 0.5.0 [com.graylog2.plugin.AWSInputPlugin], Anonymous
Usage Statistics 1.1.1
[org.graylog.plugins.usagestatistics.UsageStatsPlugin], HttpMonitorInput
1.0.0 [org.graylog2.plugin.httpmonitor.HttpMonitorInputPlugin]]
2016-06-23 12:30:47,950 INFO : org.graylog2.bootstrap.CmdLineTool - Running
with JVM arguments:
2016-06-23 12:30:48,806 INFO :
org.graylog2.shared.system.stats.SigarService - Failed to load SIGAR.
Falling back to JMX implementations.
2016-06-23 12:30:50,809 INFO : org.graylog2.shared.buffers.InputBufferImpl
- Message journal is enabled.
2016-06-23 12:30:51,279 INFO : kafka.log.LogManager - Loading log
'messagejournal-0'
2016-06-23 12:30:51,756 INFO : org.graylog2.shared.journal.KafkaJournal -
Initialized Kafka based journal at data/journal
2016-06-23 12:30:51,765 INFO : org.graylog2.shared.buffers.InputBufferImpl
- Initialized InputBufferImpl with ring size <65536> and wait strategy
<BlockingWaitStrategy>, running 2 parallel message handlers.
2016-06-23 12:30:51,997 INFO : org.graylog2.plugin.system.NodeId - Node ID:
c942aae1-e8e7-4d6a-899e-bfc0e3f10703
2016-06-23 12:30:52,329 INFO : org.elasticsearch.node - [graylog2-server]
version[1.6.2], pid[3547], build[6220391/2015-07-29T09:24:47Z]
2016-06-23 12:30:52,329 INFO : org.elasticsearch.node - [graylog2-server]
initializing ...
2016-06-23 12:30:52,425 INFO : org.elasticsearch.plugins -
[graylog2-server] loaded [graylog2-monitor], sites []
2016-06-23 12:30:54,215 INFO : org.elasticsearch.node - [graylog2-server]
initialized
2016-06-23 12:30:54,224 INFO : org.graylog2.shared.buffers.ProcessBuffer -
Initialized ProcessBuffer with ring size <65536> and wait strategy
<BlockingWaitStrategy>.
2016-06-23 12:30:55,901 INFO :
org.graylog2.bindings.providers.RulesEngineProvider - No static rules file
loaded.
2016-06-23 12:30:55,983 INFO : org.graylog2.buffers.OutputBuffer -
Initialized OutputBuffer with ring size <65536> and wait strategy
<BlockingWaitStrategy>.
2016-06-23 12:30:56,250 INFO :
org.hibernate.validator.internal.util.Version - HV000001: Hibernate
Validator 5.1.3.Final
2016-06-23 12:30:56,650 INFO : org.graylog2.bootstrap.ServerBootstrap -
Graylog server 1.1.6 (${git.commit.id.abbrev}) starting up. (JRE: Oracle
Corporation 1.7.0_79 on Linux 2.6.32-573.18.1.el6.x86_64)
2016-06-23 12:30:56,742 INFO :
org.graylog2.shared.initializers.PeriodicalsService - Starting 21
periodicals ...
2016-06-23 12:30:56,771 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.ThroughputCounterManagerThread]
periodical in [0s], polling every [1s].
2016-06-23 12:30:56,774 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s],
polling every [1s].
2016-06-23 12:30:56,774 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s],
polling every [60s].
2016-06-23 12:30:56,774 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread]
periodical in [0s], polling every [1s].
2016-06-23 12:30:56,774 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in
[0s], polling every [20s].
2016-06-23 12:30:56,774 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical,
running forever.
2016-06-23 12:30:56,795 INFO : org.elasticsearch.node - [graylog2-server]
starting ...
2016-06-23 12:30:56,796 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.DeadLetterThread] periodical, running
forever.
2016-06-23 12:30:56,796 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.GarbageCollectionWarningThread]
periodical, running forever.
2016-06-23 12:30:56,797 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical
in [0s], polling every [30s].
2016-06-23 12:30:56,797 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s],
polling every [300s].
2016-06-23 12:30:56,797 INFO : org.graylog2.periodical.IndexRetentionThread
- Elasticsearch cluster not available, skipping index retention checks.
2016-06-23 12:30:56,798 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s],
polling every [10s].
2016-06-23 12:30:56,798 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.NodePingThread] periodical in [0s],
polling every [1s].
2016-06-23 12:30:56,798 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.StreamThroughputCounterManagerThread]
periodical in [0s], polling every [1s].
2016-06-23 12:30:56,803 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.VersionCheckThread] periodical in [0s],
polling every [1800s].
2016-06-23 12:30:56,803 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in
[1s], polling every [1s].
2016-06-23 12:30:56,803 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s],
polling every [1s].
2016-06-23 12:30:56,803 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in
[0s], polling every [300s].
2016-06-23 12:30:56,804 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical,
running forever.
2016-06-23 12:30:56,804 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog2.periodical.PurgeExpiredCollectorsThread] periodical
in [0s], polling every [3600s].
2016-06-23 12:30:56,805 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical]
periodical in [300s], polling every [21600s].
2016-06-23 12:30:56,805 INFO : org.graylog2.periodical.Periodicals -
Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical]
periodical in [300s], polling every [21600s].
2016-06-23 12:30:57,034 INFO :
org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully
initialized yet. Skipping periodic cluster check.
2016-06-23 12:30:57,547 INFO : org.elasticsearch.transport -
[graylog2-server] bound_address {inet[/0:0:0:0:0:0:0:0:9350]},
publish_address {inet[/172.16.0.242:9350]}
2016-06-23 12:30:57,575 INFO : org.elasticsearch.discovery -
[graylog2-server] accessbroker/4Yzxfb6IS0K27aqNS6iTQg
2016-06-23 12:30:57,746 INFO : org.reflections.Reflections - Reflections
took 729 ms to scan 1 urls, producing 2 keys and 2 values
2016-06-23 12:31:00,575 WARN : org.elasticsearch.discovery -
[graylog2-server] waited for 3s and no initial state was set by the
discovery
2016-06-23 12:31:00,576 INFO : org.elasticsearch.node - [graylog2-server]
started
2016-06-23 12:31:00,615 INFO : org.elasticsearch.cluster.service -
[graylog2-server] detected_master
[Chamber][b2sWpQS1TzK5lfy-z7-SMw][accessbroker.leosys.net][inet[/172.16.0.242:9300]],
added
{[Chamber][b2sWpQS1TzK5lfy-z7-SMw][accessbroker.leosys.net][inet[/172.16.0.242:9300]],},
reason: zen-disco-receive(from master
[[Chamber][b2sWpQS1TzK5lfy-z7-SMw][accessbroker.leosys.net][inet[/172.16.0.242:9300]]])
2016-06-23 12:31:02,619 INFO :
org.graylog2.shared.initializers.RestApiService - Adding security context
factory: <org.graylog2.security.ShiroSecurityContextFactory@38d98a9e>
2016-06-23 12:31:02,633 INFO :
org.graylog2.shared.initializers.RestApiService - Started REST API at
<http://127.0.0.1:12900/>
2016-06-23 12:31:02,634 INFO :
org.graylog2.shared.initializers.ServiceManagerListener - Services are
healthy
2016-06-23 12:31:02,634 INFO : org.graylog2.bootstrap.ServerBootstrap -
Services started, startup times in ms: {OutputSetupService [RUNNING]=0,
InputSetupService [RUNNING]=2, BufferSynchronizerService [RUNNING]=3,
JournalReader [RUNNING]=4, MetricsReporterService [RUNNING]=5, KafkaJournal
[RUNNING]=5, GroovyShellSetupService [RUNNING]=31, PeriodicalsService
[RUNNING]=65, DashboardRegistryService [RUNNING]=183, IndexerSetupService
[RUNNING]=3850, RestApiService [RUNNING]=5891}
2016-06-23 12:31:02,635 INFO :
org.graylog2.shared.initializers.InputSetupService - Triggering launching
persisted inputs, node transitioned from Uninitialized [LB:DEAD] to
Running [LB:ALIVE]
2016-06-23 12:31:02,636 INFO : org.graylog2.bootstrap.ServerBootstrap -
Graylog server up and running.
2016-06-23 12:31:03,008 ERROR: org.graylog2.shared.journal.KafkaJournal -
Read offset 0 before start of log at 192062226, starting to read from the
beginning of the journal.
2016-06-23 12:31:03,646 INFO :
org.graylog2.shared.inputs.InputStateListener - Input [Syslog
TCP/56fbd911e4b06a01ddefad51] is now STARTING
2016-06-23 12:31:03,649 INFO :
org.graylog2.shared.inputs.InputStateListener - Input [Syslog
UDP/56fbd92be4b06a01ddefad71] is now STARTING
2016-06-23 12:31:03,810 INFO :
org.graylog2.shared.inputs.InputStateListener - Input [HTTP
Monitor/56fe5769e4b0cf6e89937691] is now STARTING
2016-06-23 12:31:04,391 INFO :
org.graylog2.shared.inputs.InputStateListener - Input [Syslog
UDP/573afc62e4b0fa8a76afeedc] is now STARTING
Exception in thread "elasticsearch[graylog2-server][generic][T#1]"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"elasticsearch[graylog2-server][generic][T#1]"
Exception in thread "elasticsearch[graylog2-server][generic][T#4]"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"elasticsearch[graylog2-server][generic][T#4]"
Exception in thread "elasticsearch[graylog2-server][generic][T#6]"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"elasticsearch[graylog2-server][generic][T#6]"
Exception in thread "elasticsearch[graylog2-server][generic][T#7]"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"elasticsearch[graylog2-server][generic][T#7]"
Exception in thread "eventbus-handler-0" Exception in thread
"elasticsearch[graylog2-server][generic][T#8]"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-0"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"elasticsearch[graylog2-server][generic][T#8]"
Exception in thread "elasticsearch[graylog2-server][generic][T#9]"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"elasticsearch[graylog2-server][generic][T#9]"
Exception in thread
"elasticsearch[graylog2-server][transport_client_worker][T#8]"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"elasticsearch[graylog2-server][transport_client_worker][T#8]"
Exception in thread "eventbus-handler-1" Exception in thread
"eventbus-handler-2"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-1"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-2"
Exception in thread "eventbus-handler-3"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-3"
Exception in thread "eventbus-handler-4"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-4"
Exception in thread "eventbus-handler-5"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-5"
Exception in thread "eventbus-handler-6"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-6"
Exception in thread "eventbus-handler-7" Exception in thread
"elasticsearch[graylog2-server][generic][T#10]"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-7"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"elasticsearch[graylog2-server][generic][T#10]"
Exception in thread "eventbus-handler-8" Exception in thread
"restapi-boss-0" Exception in thread "eventbus-handler-9"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-8"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "restapi-boss-0"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-9"
2016-06-23 12:36:23,165 INFO : org.graylog2.periodical.IndexRetentionThread
- Elasticsearch cluster not available, skipping index retention checks.
Exception in thread "eventbus-handler-10"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-10"
Exception in thread "eventbus-handler-11"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "eventbus-handler-11"
But the elasticsearch is running well, following are the logs of
elasticsearch.
[2016-06-23 12:29:41,777][INFO ][node ] [Chamber]
version[1.7.2], pid[3454], build[e43676b/2015-09-14T09:49:53Z]
[2016-06-23 12:29:41,777][INFO ][node ] [Chamber]
initializing ...
[2016-06-23 12:29:41,845][INFO ][plugins ] [Chamber]
loaded [], sites []
[2016-06-23 12:29:41,875][INFO ][env ] [Chamber] using
[1] data paths, mounts [[/home (/dev/mapper/vg_accessbroker-lv_home)]], net
usable_space [380.5gb], net total_space [401gb], types [ext4]
[2016-06-23 12:29:43,713][INFO ][node ] [Chamber]
initialized
[2016-06-23 12:29:43,713][INFO ][node ] [Chamber]
starting ...
[2016-06-23 12:29:43,776][INFO ][transport ] [Chamber]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/172.16.0.242:9300]}
[2016-06-23 12:29:43,783][INFO ][discovery ] [Chamber]
accessbroker/b2sWpQS1TzK5lfy-z7-SMw
[2016-06-23 12:29:46,808][INFO ][cluster.service ] [Chamber]
new_master
[Chamber][b2sWpQS1TzK5lfy-z7-SMw][accessbroker.leosys.net][inet[/172.16.0.242:9300]],
reason: zen-disco-join (elected_as_master)
[2016-06-23 12:29:46,820][INFO ][http ] [Chamber]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/172.16.0.242:9200]}
[2016-06-23 12:29:46,820][INFO ][node ] [Chamber]
started
[2016-06-23 12:29:46,946][INFO ][gateway ] [Chamber]
recovered [0] indices into cluster_state
[2016-06-23 12:31:00,605][INFO ][cluster.service ] [Chamber] added
{[graylog2-server][4Yzxfb6IS0K27aqNS6iTQg][accessbroker.leosys.net][inet[/172.16.0.242:9350]]{client=true,
data=false, master=false},}, reason: zen-disco-receive(join from
node[[graylog2-server][4Yzxfb6IS0K27aqNS6iTQg][accessbroker.leosys.net][inet[/172.16.0.242:9350]]{client=true,
data=false, master=false}])
[2016-06-23 12:33:34,881][INFO ][cluster.service ] [Chamber]
removed
{[graylog2-server][4Yzxfb6IS0K27aqNS6iTQg][accessbroker.leosys.net][inet[/172.16.0.242:9350]]{client=true,
data=false, master=false},}, reason:
zen-disco-node_failed([graylog2-server][4Yzxfb6IS0K27aqNS6iTQg][accessbroker.leosys.net][inet[/172.16.0.242:9350]]{client=true,
data=false, master=false}), reason failed to ping, tried [3] times, each
with maximum [30s] timeout
also the elasticsearch status is as follows
{
"status" : 200,
"name" : "Chamber",
"cluster_name" : "accessbroker",
"version" : {
"number" : "1.7.2",
"build_hash" : "e43676b1385b8125d647f593f7202acbd816e8ec",
"build_timestamp" : "2015-09-14T09:49:53Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
Is the issue related to the Graylog server or elasticsearch??
Using Graylog 1.1.6 and elasticsearch 1.7.2 on CentOS release 6.7.
Thanks in advance!!
Anant.
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/43c9f583-893e-4914-bcb8-50d059e61229%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
