Bumping up. Does anyone have insight into this?? Mario
On Tuesday, February 10, 2015 at 3:02:12 PM UTC-6, Mario Rodriguez wrote: > > > > On Tuesday, February 10, 2015 at 2:55:59 PM UTC-6, Mario Rodriguez wrote: >> >> I have 7 indices in our ES cluster, and one of the indices has an issue >> that is preventing recovery. >> >> Here is what I am seeing in the log: >> >> [2015-02-10 00:00:02,483][WARN ][indices.recovery ] [ES_Server1] >> [prodcustomer][1] recovery from >> [[ES_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES_Server2][inet[/5.5.5.5:9300]]] >> failed >> org.elasticsearch.transport.RemoteTransportException: >> [ES_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery] >> Caused by: org.elasticsearch.index.engine.RecoveryEngineException: >> [prodcustomer][1] Phase[1] Execution failed >> at >> org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072) >> at >> org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636) >> at >> org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135) >> at >> org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72) >> at >> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440) >> at >> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426) >> at >> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: >> org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: >> [prodcustomer][1] Failed to transfer [0] files with total size of [0b] >> at >> org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280) >> at >> org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068) >> ... 9 more >> Caused by: java.io.EOFException: read past EOF: >> MMapIndexInput(path="D:\ElasticSearchData\new_es_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019") >> at >> org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81) >> at org.apache.lucene.store.DataInput.readInt(DataInput.java:96) >> at >> org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132) >> at >> org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523) >> at >> org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438) >> at >> org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:433) >> at org.elasticsearch.index.store.Store.getMetadata(Store.java:144) >> at >> org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145) >> ... 10 more >> [2015-02-10 00:00:02,483][WARN ][indices.cluster ] [ES_Server1] >> [prodcustomer][1] failed to start shard >> org.elasticsearch.indices.recovery.RecoveryFailedException: >> [prodcustomer][1]: Recovery failed from >> [ES_Server2][QDf3ZP3tQ3Kgund8YX2BBQ][ES_Server2][inet[/5.5.5.5:9300]] >> into >> [ES_Server1][R3ArnIZsSVSsd-VLEaU_Ug][ES_Server1][inet[ES_Server1.verify.local/ >> 4.4.4.4:9300]] >> at >> org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:306) >> at >> org.elasticsearch.indices.recovery.RecoveryTarget.access$200(RecoveryTarget.java:65) >> at >> org.elasticsearch.indices.recovery.RecoveryTarget$3.run(RecoveryTarget.java:184) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: org.elasticsearch.transport.RemoteTransportException: >> [ES_Server2][inet[/5.5.5.5:9300]][index/shard/recovery/startRecovery] >> Caused by: org.elasticsearch.index.engine.RecoveryEngineException: >> [prodcustomer][1] Phase[1] Execution failed >> at >> org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1072) >> at >> org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:636) >> at >> org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:135) >> at >> org.elasticsearch.indices.recovery.RecoverySource.access$2500(RecoverySource.java:72) >> at >> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:440) >> at >> org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:426) >> at >> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: >> org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: >> [prodcustomer][1] Failed to transfer [0] files with total size of [0b] >> at >> org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:280) >> at >> org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1068) >> ... 9 more >> Caused by: java.io.EOFException: read past EOF: >> MMapIndexInput(path="D:\ElasticSearchData\new_es_cluster\nodes\0\indices\prodcustomer\1\index\_checksums-1418875637019") >> at >> org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81) >> at org.apache.lucene.store.DataInput.readInt(DataInput.java:96) >> at >> org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:132) >> at >> org.elasticsearch.index.store.Store$MetadataSnapshot.readLegacyChecksums(Store.java:523) >> at >> org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:438) >> at >> org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:433) >> at org.elasticsearch.index.store.Store.getMetadata(Store.java:144) >> at >> org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:145) >> ... 10 more >> >> >> Is it as simple as deleting the checksum files? >> Thank you for any insight anyone can provide. >> >> *EDIT We are on version 1.3.1 > > >> Mario >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7b527f9d-8d3a-4fce-a5b5-b030c7ab8994%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
