Hi Roberto, your Elasticsearch cluster health status is RED, which means that ES can't index documents anymore. Check the logs of your Elasticsearch nodes for the reason.
Additionally I'd recommend upgrading to Graylog 1.2.1, which also starts when Elasticsearch is unhealthy or not accessible. Cheers, Jochen On Monday, 28 September 2015 17:21:59 UTC+2, [email protected] wrote: > > Dear, I have Graylog 1.1 and today I have to remove all the files under > /var/lib/graylog-server/journal/. > > I remove all the files without stopping any service (elasticsearch, > graylog-web and graylog-server). > > After that, I reboot the server but the graylog-server doesn't start at > all, and I can see this error log message.....can you help please ??? > Thanks a lot. > > 2015-09-28T12:00:42.414-03:00 INFO [CmdLineTool] Loaded plugins: > [Anonymous Usage Statistics 1.0.5 > [org.graylog.plugins.usagestatistics.UsageStatsPlugin]] > 2015-09-28T12:00:42.469-03:00 INFO [MongoDbConfiguration] You're using > deprecated configuration options for MongoDB. Please use mongodb_uri. > 2015-09-28T12:00:42.504-03:00 INFO [MongoDbConfiguration] Suggested value > for mongodb_uri = mongodb://graylog2:[email protected]:27017/graylog2 > 2015-09-28T12:00:42.533-03:00 INFO [CmdLineTool] Running with JVM > arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:PermSize=128m > -XX:MaxPermSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC > -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC > -XX:-OmitStackTraceInFastThrow > -Dlog4j.configuration=file:///etc/graylog/server/log4j.xml > -Djava.library.path=/usr/share/graylog-server/lib/sigar > 2015-09-28T12:00:45.834-03:00 INFO [InputBufferImpl] Message journal is > enabled. > 2015-09-28T12:00:46.083-03:00 INFO [LogManager] Found clean shutdown > file. Skipping recovery for all logs in data directory > '/var/lib/graylog-server/journal' > 2015-09-28T12:00:46.084-03:00 INFO [LogManager] Loading log > 'messagejournal-0' > 2015-09-28T12:00:46.113-03:00 INFO [Log] Completed load of log > messagejournal-0 with log end offset 0 > 2015-09-28T12:00:46.125-03:00 INFO [KafkaJournal] Initialized Kafka based > journal at /var/lib/graylog-server/journal > 2015-09-28T12:00:46.138-03:00 INFO [InputBufferImpl] Initialized > InputBufferImpl with ring size <65536> and wait strategy > <BlockingWaitStrategy>, running 2 parallel message handlers. > 2015-09-28T12:00:46.318-03:00 INFO [NodeId] Node ID: > b7b62947-250e-473b-b8df-7083d6df9886 > 2015-09-28T12:00:46.486-03:00 INFO [node] [graylog2-server] > version[1.5.2], pid[3720], build[62ff986/2015-04-27T09:21:06Z] > 2015-09-28T12:00:46.487-03:00 INFO [node] [graylog2-server] initializing > ... > 2015-09-28T12:00:46.496-03:00 INFO [plugins] [graylog2-server] loaded > [graylog2-monitor], sites [] > 2015-09-28T12:00:48.786-03:00 INFO [node] [graylog2-server] initialized > 2015-09-28T12:00:48.796-03:00 INFO [ProcessBuffer] Initialized > ProcessBuffer with ring size <65536> and wait strategy > <BlockingWaitStrategy>. > 2015-09-28T12:00:50.543-03:00 INFO [RulesEngineProvider] No static rules > file loaded. > 2015-09-28T12:00:50.741-03:00 INFO [OutputBuffer] Initialized > OutputBuffer with ring size <65536> and wait strategy > <BlockingWaitStrategy>. > 2015-09-28T12:00:51.221-03:00 INFO [Version] HV000001: Hibernate > Validator 5.1.3.Final > 2015-09-28T12:00:51.630-03:00 INFO [ServerBootstrap] Graylog server 1.1.1 > (893e8e7) starting up. (JRE: Oracle Corporation 1.7.0_79 on Linux > 3.2.0-4-amd64) > 2015-09-28T12:00:51.645-03:00 INFO [PeriodicalsService] Starting 21 > periodicals ... > 2015-09-28T12:00:51.729-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.ThroughputCounterManagerThread] periodical in > [0s], polling every [1s]. > 2015-09-28T12:00:51.711-03:00 INFO [node] [graylog2-server] starting ... > 2015-09-28T12:00:51.780-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling > every [1s]. > 2015-09-28T12:00:51.787-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling > every [60s]. > 2015-09-28T12:00:51.793-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical > in [0s], polling every [1s]. > 2015-09-28T12:00:51.796-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s], > polling every [20s]. > 2015-09-28T12:00:51.806-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running > forever. > 2015-09-28T12:00:51.817-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.DeadLetterThread] periodical, running forever. > 2015-09-28T12:00:51.818-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, > running forever. > 2015-09-28T12:00:51.825-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], > polling every [30s]. > 2015-09-28T12:00:51.845-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling > every [300s]. > 2015-09-28T12:00:51.846-03:00 INFO [IndexRetentionThread] Elasticsearch > cluster not available, skipping index retention checks. > 2015-09-28T12:00:51.846-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling > every [10s]. > 2015-09-28T12:00:51.848-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every > [1s]. > 2015-09-28T12:00:51.869-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.StreamThroughputCounterManagerThread] periodical > in [0s], polling every [1s]. > 2015-09-28T12:00:51.870-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.VersionCheckThread] periodical in [0s], polling > every [1800s]. > 2015-09-28T12:00:51.878-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], > polling every [1s]. > 2015-09-28T12:00:51.878-03:00 INFO [IndexerClusterCheckerThread] Indexer > not fully initialized yet. Skipping periodic cluster check. > 2015-09-28T12:00:51.883-03:00 INFO [Periodicals] Starting > [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling > every [1s]. > 2015-09-28T12:00:51.898-03:00 INFO [Periodicals] Starting > [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], > polling every [300s]. > 2015-09-28T12:00:51.898-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running > forever. > 2015-09-28T12:00:51.908-03:00 INFO [Periodicals] Starting > [org.graylog2.periodical.PurgeExpiredCollectorsThread] periodical in [0s], > polling every [3600s]. > 2015-09-28T12:00:51.908-03:00 INFO [Periodicals] Starting > [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical > in [300s], polling every [21600s]. > 2015-09-28T12:00:51.908-03:00 INFO [Periodicals] Starting > [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] > periodical in [300s], polling every [21600s]. > 2015-09-28T12:00:52.192-03:00 INFO [transport] [graylog2-server] > bound_address {inet[/0.0.0.0:9350]}, publish_address {inet[/ > 172.31.24.42:9350]} > 2015-09-28T12:00:52.224-03:00 INFO [discovery] [graylog2-server] > graylog2/DczDyICMTXa1k8fMjwcDvQ > 2015-09-28T12:00:52.371-03:00 INFO [Reflections] Reflections took 531 ms > to scan 1 urls, producing 2 keys and 2 values > 2015-09-28T12:00:55.225-03:00 WARN [discovery] [graylog2-server] waited > for 3s and no initial state was set by the discovery > 2015-09-28T12:00:55.225-03:00 INFO [node] [graylog2-server] started > 2015-09-28T12:00:55.342-03:00 INFO [service] [graylog2-server] > detected_master [Hargen the Measurer][RCYNypIWT4W8Sd8EIQsYbA][ > MITLPLOG01.provincianet.com.ar][inet[/172.31.24.42:9300]], added {[Hargen > the > Measurer][RCYNypIWT4W8Sd8EIQsYbA][MITLPLOG01.provincianet.com.ar][inet[/172.31.24.42:9300]],}, > > reason: zen-disco-receive(from master [[Hargen the > Measurer][RCYNypIWT4W8Sd8EIQsYbA][MITLPLOG01.provincianet.com.ar > ][inet[/172.31.24.42:9300]]]) > 2015-09-28T12:00:55.411-03:00 ERROR [UI] > > > ################################################################################ > > ERROR: The Elasticsearch cluster state is RED which means shards are > unassigned. This usually indicates a crashed and corrupt cluster and needs > to be investigated. Graylog will shut down. > > Need help? > > * Official documentation: http://docs.graylog.org/ > * Community support: https://www.graylog.org/community-support/ > * Commercial support: https://www.graylog.com/support/ > > But we also got some specific help pages that might help you in this case: > > * http://docs.graylog.org/en/1.0/pages/configuring_es.html > > Terminating. :( > > > ################################################################################ > > 2015-09-28T12:00:55.413-03:00 ERROR [ServiceManager] Service > IndexerSetupService [FAILED] has failed in the STARTING state. > java.lang.IllegalStateException > at org.graylog2.UI.exitHardWithWall(UI.java:36) > at > org.graylog2.initializers.IndexerSetupService.startUp(IndexerSetupService.java:137) > at > com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54) > at > com.google.common.util.concurrent.Callables$3.run(Callables.java:95) > at java.lang.Thread.run(Thread.java:745) > 2015-09-28T12:00:55.416-03:00 ERROR [InputSetupService] Not starting any > inputs because lifecycle is: Uninitialized [LB:DEAD] > 2015-09-28T12:00:55.423-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.periodical.AlertScannerThread]. > 2015-09-28T12:00:55.424-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.periodical.AlertScannerThread] complete, took > <0ms>. > 2015-09-28T12:00:55.424-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread]. > 2015-09-28T12:00:55.424-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] > complete, took <0ms>. > 2015-09-28T12:00:55.424-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.periodical.ClusterHealthCheckThread]. > 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, > took <0ms>. > 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.periodical.IndexerClusterCheckerThread]. > 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, > took <0ms>. > 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.periodical.IndexRetentionThread]. > 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.periodical.IndexRetentionThread] complete, took > <0ms>. > 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.periodical.IndexRotationThread]. > 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.periodical.IndexRotationThread] complete, took > <0ms>. > 2015-09-28T12:00:55.425-03:00 WARN [BufferSynchronizerService] > Elasticsearch is unavailable. Not waiting to clear buffers and caches, as > we have no healthy cluster. > 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.periodical.VersionCheckThread]. > 2015-09-28T12:00:55.425-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.periodical.VersionCheckThread] complete, took > <0ms>. > 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.periodical.ThrottleStateUpdaterThread]. > 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, > took <0ms>. > 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.events.ClusterEventPeriodical]. > 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.events.ClusterEventPeriodical] complete, took > <0ms>. > 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.events.ClusterEventCleanupPeriodical]. > 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, > took <0ms>. > 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog2.periodical.PurgeExpiredCollectorsThread]. > 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog2.periodical.PurgeExpiredCollectorsThread] complete, > took <0ms>. > 2015-09-28T12:00:55.426-03:00 INFO [PeriodicalsService] Shutting down > periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical]. > 2015-09-28T12:00:55.427-03:00 INFO [PeriodicalsService] Shutdown of > periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] > complete, took <0ms>. > 2015-09-28T12:00:55.427-03:00 INFO [PeriodicalsService] Shutting down > periodical > [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical]. > 2015-09-28T12:00:55.427-03:00 INFO [PeriodicalsService] Shutdown of > periodical > [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] complete, > took <0ms>. > 2015-09-28T12:00:55.429-03:00 INFO [JournalReader] Stopping. > 2015-09-28T12:00:55.435-03:00 INFO [node] [graylog2-server] stopping ... > 2015-09-28T12:00:55.444-03:00 INFO [OutputSetupService] Stopping output > org.graylog2.outputs.BlockingBatchedESOutput > 2015-09-28T12:00:55.493-03:00 INFO [node] [graylog2-server] stopped > 2015-09-28T12:00:55.493-03:00 INFO [node] [graylog2-server] closing ... > 2015-09-28T12:00:55.499-03:00 INFO [node] [graylog2-server] closed > 2015-09-28T12:00:59.264-03:00 INFO [RestApiService] Adding security > context factory: > <org.graylog2.security.ShiroSecurityContextFactory@6d404877> > 2015-09-28T12:00:59.294-03:00 INFO [RestApiService] Started REST API at < > http://127.0.0.1:12900/> > 2015-09-28T12:00:59.296-03:00 INFO [RestApiService] Shutting down REST > API at <http://127.0.0.1:12900/> > 2015-09-28T12:00:59.304-03:00 INFO [ServiceManagerListener] Services are > now stopped. > 2015-09-28T12:00:59.304-03:00 ERROR [ServerBootstrap] Graylog startup > failed. Exiting. Exception was: > java.lang.IllegalStateException: Expected to be healthy after starting. > The following services are not running: {STARTING=[RestApiService > [STARTING]], FAILED=[IndexerSetupService [FAILED]]} > at > com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:710) > at > com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:535) > at > com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:301) > at > org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:114) > at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:196) > at org.graylog2.bootstrap.Main.main(Main.java:58) > 2015-09-28T12:00:59.306-03:00 INFO [Server] SIGNAL received. Shutting > down. > 2015-09-28T12:00:59.320-03:00 INFO [GracefulShutdown] Graceful shutdown > initiated. > 2015-09-28T12:00:59.321-03:00 INFO [GracefulShutdown] Node status: > [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to > recognize state change. > 2015-09-28T12:01:03.323-03:00 INFO [GracefulShutdown] Goodbye. > > > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/be9715b7-9223-4a72-9704-25030cdcd59b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
