I should mention that there is a primary shard 4 on the other node, just
need to understand why it's not auto recovering here what I can do to
manually remove the corrupted shard to have the primary replicated to this
node.
On Wednesday, August 6, 2014 12:44:41 PM UTC-4, Nariman Haghighi wrote:
>
> A few days after the upgrade to 1.3.1 we experienced our first corrupted
> shard in a 2 node cluster:
>
> [2014-08-06 15:54:28,815][WARN ][indices.cluster ]
> [FiveAces.Coffee.Web_IN_0] [streamentry5][4] failed to start shard
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
> [streamentry5][4] failed to fetch index version after copying it over
> at
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:152)
> at
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.lucene.index.CorruptIndexException:
> [streamentry5][4] Corrupted index [corrupted_fuDt8NuqR_egGJK0fcjl6g] caused
> by: CorruptIndexException[Invalid fieldsStream maxPointer (file
> truncated?): maxPointer=6833538, length=524288]
> at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:343)
> at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:328)
> at
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:119)
> ... 4 more
>
> How do we recover from this?
>
> We've tried explicitly assigning via the reroute API:
>
> { "commands" : [ { "allocate" : { "index" : "streamentry5", "shard" : 4 ,
> "node" : "FiveAces.Coffee.Web_IN_0", "allow_primary" : 1 }}]}
>
> This puts the shard in INITIALIZING but quickly reverts back to
> UNALLOCATED with a similar error in the logs.
>
> I'm interested in theories on how this could have happened assuming no
> significant changes on our end during this period and never having
> experienced this on ES before but more importantly how to recover from it.
>
> Thank you
>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/96692c31-c938-41dd-aeb4-d4e61a9a515d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
