Created and populated a new index on a 1.3.1 cluster. Primary shards work fine. Updated the index to create several replicas, and three of the four shards replicated, but one shard fails to replicate on any node with the following error (abbreviated some of the hashes for readability):
[2014-10-22 20:31:54,549][WARN ][index.engine.internal ] [NODENAME] >> [INDEXNAME][2] failed engine [corrupted preexisting index] > > [2014-10-22 20:31:54,549][WARN ][indices.cluster ] [NODENAME] >> [INDEXNAME][2] failed to start shard > > org.apache.lucene.index.CorruptIndexException: [INDEXNAME][2] Corrupted >> index [CORRUPTED] caused by: CorruptIndexException[codec footer mismatch: >> actual footer=1161826848 vs expected footer=-1071082520 (resource: >> MMapIndexInput(path="DATAPATH/INDEXNAME/2/index/_7cp.fdt"))] > > at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:343) > > at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:328) > > at >> org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:723) > > at >> org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:576) > > at >> org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:183) > > at >> org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:444) > > at >> org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:153) > > at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > [2014-10-22 20:31:54,549][WARN ][cluster.action.shard ] [NODENAME] >> [INDEXNAME][2] sending failed shard for [INDEXNAME][2], node[NODEID], [R], >> s[INITIALIZING], indexUUID [INDEXID], reason [Failed to start shard, >> message [CorruptIndexException[[INDEXNAME][2] Corrupted index [CORRUPTED] >> caused by: CorruptIndexException[codec footer mismatch: actual >> footer=1161826848 vs expected footer=-1071082520 (resource: >> MMapIndexInput(path="DATAPATH/INDEXNAME/2/index/_7cp.fdt"))]]]] > > [2014-10-22 20:31:54,550][WARN ][cluster.action.shard ] [NODENAME] >> [INDEXNAME][2] sending failed shard for [INDEXNAME][2], node[NODEID], [R], >> s[INITIALIZING], indexUUID [INDEXID], reason [engine failure, message >> [corrupted preexisting index][CorruptIndexException[[INDEXNAME][2] >> Corrupted index [CORRUPTED] caused by: CorruptIndexException[codec footer >> mismatch: actual footer=1161826848 vs expected footer=-1071082520 >> (resource: MMapIndexInput(path="DATAPATH/INDEXNAME/2/index/_7cp.fdt"))]]]] > > The index is stuck now in a state where the shards try to replicate on one set of nodes, hit this failure, and then switch to try to replicate on a different set of nodes. Have been looking around to see if anyone's encountered a similar issue but haven't found anything useful yet. Anybody know if this is recoverable or if I should just scrap it and try building a new one? - Nate -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/51f1b345-a19d-4c70-873f-a88880d47e5a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
