Dear, I have Graylog 1.1 and today I have to remove all the files under /var/lib/graylog-server/journal/.
I remove all the files without stopping any service (elasticsearch, graylog-web and graylog-server). After that, I reboot the server but the graylog-server doesn't start at all, and I can see this error log message.....can you help please ??? Thanks a lot. 2015-09-28T12:00:42.414-03:00 INFO [CmdLineTool] Loaded plugins: [Anonymous Usage Statistics 1.0.5 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]] 2015-09-28T12:00:42.469-03:00 INFO [MongoDbConfiguration] You're using deprecated configuration options for MongoDB. Please use mongodb_uri. 2015-09-28T12:00:42.504-03:00 INFO [MongoDbConfiguration] Suggested value for mongodb_uri = mongodb://graylog2:[email protected]:27017/graylog2 2015-09-28T12:00:42.533-03:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configuration=file:///etc/graylog/server/log4j.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar 2015-09-28T12:00:45.834-03:00 INFO [InputBufferImpl] Message journal is enabled. 2015-09-28T12:00:46.083-03:00 INFO [LogManager] Found clean shutdown file. Skipping recovery for all logs in data directory '/var/lib/graylog-server/journal' 2015-09-28T12:00:46.084-03:00 INFO [LogManager] Loading log 'messagejournal-0' 2015-09-28T12:00:46.113-03:00 INFO [Log] Completed load of log messagejournal-0 with log end offset 0 2015-09-28T12:00:46.125-03:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal 2015-09-28T12:00:46.138-03:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers. 2015-09-28T12:00:46.318-03:00 INFO [NodeId] Node ID: b7b62947-250e-473b-b8df-7083d6df9886 2015-09-28T12:00:46.486-03:00 INFO [node] [graylog2-server] version[1.5.2], pid[3720], build[62ff986/2015-04-27T09:21:06Z] 2015-09-28T12:00:46.487-03:00 INFO [node] [graylog2-server] initializing ... 2015-09-28T12:00:46.496-03:00 INFO [plugins] [graylog2-server] loaded [graylog2-monitor], sites [] 2015-09-28T12:00:48.786-03:00 INFO [node] [graylog2-server] initialized 2015-09-28T12:00:48.796-03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>. 2015-09-28T12:00:50.543-03:00 INFO [RulesEngineProvider] No static rules file loaded. 2015-09-28T12:00:50.741-03:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>. 2015-09-28T12:00:51.221-03:00 INFO [Version] HV000001: Hibernate Validator 5.1.3.Final 2015-09-28T12:00:51.630-03:00 INFO [ServerBootstrap] Graylog server 1.1.1 (893e8e7) starting up. (JRE: Oracle Corporation 1.7.0_79 on Linux 3.2.0-4-amd64) 2015-09-28T12:00:51.645-03:00 INFO [PeriodicalsService] Starting 21 periodicals ... 2015-09-28T12:00:51.729-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCounterManagerThread] periodical in [0s], polling every [1s]. 2015-09-28T12:00:51.711-03:00 INFO [node] [graylog2-server] starting ... 2015-09-28T12:00:51.780-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s]. 2015-09-28T12:00:51.787-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s]. 2015-09-28T12:00:51.793-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s]. 2015-09-28T12:00:51.796-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s], polling every [20s]. 2015-09-28T12:00:51.806-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever. 2015-09-28T12:00:51.817-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.DeadLetterThread] periodical, running forever. 2015-09-28T12:00:51.818-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever. 2015-09-28T12:00:51.825-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s]. 2015-09-28T12:00:51.845-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s]. 2015-09-28T12:00:51.846-03:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks. 2015-09-28T12:00:51.846-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s]. 2015-09-28T12:00:51.848-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s]. 2015-09-28T12:00:51.869-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.StreamThroughputCounterManagerThread] periodical in [0s], polling every [1s]. 2015-09-28T12:00:51.870-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [0s], polling every [1800s]. 2015-09-28T12:00:51.878-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s]. 2015-09-28T12:00:51.878-03:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check. 2015-09-28T12:00:51.883-03:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s]. 2015-09-28T12:00:51.898-03:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [300s]. 2015-09-28T12:00:51.898-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever. 2015-09-28T12:00:51.908-03:00 INFO [Periodicals] Starting [org.graylog2.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s]. 2015-09-28T12:00:51.908-03:00 INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s]. 2015-09-28T12:00:51.908-03:00 INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s]. 2015-09-28T12:00:52.192-03:00 INFO [transport] [graylog2-server] bound_address {inet[/0.0.0.0:9350]}, publish_address {inet[/172.31.24.42:9350]} 2015-09-28T12:00:52.224-03:00 INFO [discovery] [graylog2-server] graylog2/DczDyICMTXa1k8fMjwcDvQ 2015-09-28T12:00:52.371-03:00 INFO [Reflections] Reflections took 531 ms to scan 1 urls, producing 2 keys and 2 values 2015-09-28T12:00:55.225-03:00 WARN [discovery] [graylog2-server] waited for 3s and no initial state was set by the discovery 2015-09-28T12:00:55.225-03:00 INFO [node] [graylog2-server] started 2015-09-28T12:00:55.342-03:00 INFO [service] [graylog2-server] detected_master [Hargen the Measurer][RCYNypIWT4W8Sd8EIQsYbA][MITLPLOG01.provincianet.com.ar][inet[/172.31.24.42:9300]], added {[Hargen the Measurer][RCYNypIWT4W8Sd8EIQsYbA][MITLPLOG01.provincianet.com.ar][inet[/172.31.24.42:9300]],}, reason: zen-disco-receive(from master [[Hargen the Measurer][RCYNypIWT4W8Sd8EIQsYbA][MITLPLOG01.provincianet.com.ar][inet[/172.31.24.42:9300]]]) 2015-09-28T12:00:55.411-03:00 ERROR [UI] ################################################################################ ERROR: The Elasticsearch cluster state is RED which means shards are unassigned. This usually indicates a crashed and corrupt cluster and needs to be investigated. Graylog will shut down. Need help? * Official documentation: http://docs.graylog.org/ * Community support: https://www.graylog.org/community-support/ * Commercial support: https://www.graylog.com/support/ But we also got some specific help pages that might help you in this case: * http://docs.graylog.org/en/1.0/pages/configuring_es.html Terminating. :( ################################################################################ 2015-09-28T12:00:55.413-03:00 ERROR [ServiceManager] Service IndexerSetupService [FAILED] has failed in the STARTING state. java.lang.IllegalStateException at org.graylog2.UI.exitHardWithWall(UI.java:36) at org.graylog2.initializers.IndexerSetupService.startUp(IndexerSetupService.java:137) at com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54) at com.google.common.util.concurrent.Callables$3.run(Callables.java:95) at java.lang.Thread.run(Thread.java:745) 2015-09-28T12:00:55.416-03:00 ERROR [InputSetupService] Not starting any inputs because lifecycle is: Uninitialized [LB:DEAD] 2015-09-28T12:00:55.423-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.AlertScannerThread]. 2015-09-28T12:00:55.424-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.AlertScannerThread] complete, took <0ms>. 2015-09-28T12:00:55.424-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread]. 2015-09-28T12:00:55.424-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>. 2015-09-28T12:00:55.424-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread]. 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>. 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread]. 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>. 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread]. 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>. 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread]. 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>. 2015-09-28T12:00:55.425-03:00 WARN [BufferSynchronizerService] Elasticsearch is unavailable. Not waiting to clear buffers and caches, as we have no healthy cluster. 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread]. 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>. 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread]. 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>. 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical]. 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>. 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical]. 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>. 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.PurgeExpiredCollectorsThread]. 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>. 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical]. 2015-09-28T12:00:55.427-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] complete, took <0ms>. 2015-09-28T12:00:55.427-03:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical]. 2015-09-28T12:00:55.427-03:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] complete, took <0ms>. 2015-09-28T12:00:55.429-03:00 INFO [JournalReader] Stopping. 2015-09-28T12:00:55.435-03:00 INFO [node] [graylog2-server] stopping ... 2015-09-28T12:00:55.444-03:00 INFO [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput 2015-09-28T12:00:55.493-03:00 INFO [node] [graylog2-server] stopped 2015-09-28T12:00:55.493-03:00 INFO [node] [graylog2-server] closing ... 2015-09-28T12:00:55.499-03:00 INFO [node] [graylog2-server] closed 2015-09-28T12:00:59.264-03:00 INFO [RestApiService] Adding security context factory: <org.graylog2.security.ShiroSecurityContextFactory@6d404877> 2015-09-28T12:00:59.294-03:00 INFO [RestApiService] Started REST API at <http://127.0.0.1:12900/> 2015-09-28T12:00:59.296-03:00 INFO [RestApiService] Shutting down REST API at <http://127.0.0.1:12900/> 2015-09-28T12:00:59.304-03:00 INFO [ServiceManagerListener] Services are now stopped. 2015-09-28T12:00:59.304-03:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was: java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[RestApiService [STARTING]], FAILED=[IndexerSetupService [FAILED]]} at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:710) at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:535) at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:301) at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:114) at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:196) at org.graylog2.bootstrap.Main.main(Main.java:58) 2015-09-28T12:00:59.306-03:00 INFO [Server] SIGNAL received. Shutting down. 2015-09-28T12:00:59.320-03:00 INFO [GracefulShutdown] Graceful shutdown initiated. 2015-09-28T12:00:59.321-03:00 INFO [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change. 2015-09-28T12:01:03.323-03:00 INFO [GracefulShutdown] Goodbye. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/833cbcc0-088a-42fa-a2d3-88c74c494931%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
