I've been trying to trouble shoot an issue with my single ES node. When I went to go look at it, it was at 100% diskspace usage. A lot of this issue was due to ES logs taking up space on the volume.
When I cleared out the logs and recovered a lot of the space and tried to restart then I saw a failed shard issue: [INFO ][node ] [Bloodshed] stopping ... [2014-03-20 19:39:26,957][WARN ][cluster.action.shard ] [Bloodshed] sending failed shard for [logstash-2014.02.06][4], node[ap2Lm43gRIyoN0FDh-ZO_A], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[[logstash-2014.02.06][4]: Recovery failed from [Forbush Man][x5RvlhLLRPOtz_hcgW-JgA][inet[/10.0.30.55:9300]] into [Bloodshed][ap2Lm43gRIyoN0FDh-ZO_A][inet[/10.0.30.50:9300]]]; nested: RemoteTransportException[[Forbush Man][inet[/10.0.30.55:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[logstash-2014.02.06][4] Phase[2] Execution failed]; nested: RemoteTransportException[[Bloodshed][inet[/10.0.30.50:9300]][index/shard/recovery/prepareTranslog]]; nested: EngineCreationFailureException[[logstash-2014.02.06][4] failed to create engine]; nested: IndexFormatTooNewException[Format version is not supported (resource: ChecksumIndexInput(NIOFSIndexInput(path="/var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2014.02.06/4/index/segments_3"))): 1 (needs to be between 0 and 0)]; ]] [2014-03-20 19:39:28,114][INFO ][node ] [Bloodshed] stopped [2014-03-20 19:39:28,114][INFO ][node ] [Bloodshed] closing ... [2014-03-20 19:39:28,119][INFO ][node ] [Bloodshed] closed I had changed the config file to discovery.zen.ping.multicast.enabled: false and tried restarting the server - no more entries in the log I also tried moving some of the data (especially that corrupted entry) from the data directory and restarting. The system message from init.d would say that ES had started, but then checking the status again it would stay that it was stopped. Again no more log entries. I tried renaming the log file and then touching a file called elasticsearch.log and chowing it to elasticsearch:elasticsearch | a restart of the service, it still wasn't running and no log entries At this point I decided to upgrade my 0.90 ES server to the new 1.0 ES with a .deb package I installed, rebooted the machine and after restarting the service - no entries My last try was to make a new data directory, a clean data directory, and then point to that in the config file and restart the ES server. It still would not start and nothing in the log. At this point I'm not sure what else to troubleshoot on how to get it back working again. The data I'm not as worried about, but I need the Elastic Search server up and running again. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3e438abe-6fed-4333-9155-f6c8e7e2c040%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
