A few days after the upgrade to 1.3.1 we experienced our first corrupted
shard in a 2 node cluster:
[2014-08-06 15:54:28,815][WARN ][indices.cluster ]
[FiveAces.Coffee.Web_IN_0] [streamentry5][4] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[streamentry5][4] failed to fetch index version after copying it over
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:152)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.lucene.index.CorruptIndexException: [streamentry5][4]
Corrupted index [corrupted_fuDt8NuqR_egGJK0fcjl6g] caused by:
CorruptIndexException[Invalid fieldsStream maxPointer (file truncated?):
maxPointer=6833538, length=524288]
at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:343)
at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:328)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:119)
... 4 more
How do we recover from this?
We've tried explicitly assigning via the reroute API:
{ "commands" : [ { "allocate" : { "index" : "streamentry5", "shard" : 4 ,
"node" : "FiveAces.Coffee.Web_IN_0", "allow_primary" : 1 }}]}
This puts the shard in INITIALIZING but quickly reverts back to UNALLOCATED
with a similar error in the logs.
I'm interested in theories on how this could have happened assuming no
significant changes on our end during this period and never having
experienced this on ES before but more importantly how to recover from it.
Thank you
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d225d1cd-79a6-455c-a4d0-6cf0dfd88314%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
