Hi, the full stack is like this (from outstanding tasks api). We are using ES
1.4.1
insert_order : 69862
priority : HIGH
source : shard-failed ([agora_v1][24],
node[SEIBtFznTtGpLFPgCLgW4w], [R], s[INITIALIZING]),
reason [Failed to start shard, message
[CorruptIndexException[[agora_v1][24] Preexisting
corrupted index [corrupted_LrKHKRF7Q2KuL15TT_hPvw]
caused by: CorruptIndexException[Read past EOF while
reading segment infos]
EOFException[read past EOF:
MMapIndexInput(path="D:\
app\ES.ElasticSearch_v010\elasticsearch-1.4.1\data\AP-el
asticsearch\nodes\0\indices\agora_v1\24\index\segments_6
w")]
org.apache.lucene.index.CorruptIndexException: Read
past EOF while reading segment infos
at
org.elasticsearch.index.store.Store.readSegmentsI
nfo(Store.java:127)
at
org.elasticsearch.index.store.Store.access$400(St
ore.java:80)
at
org.elasticsearch.index.store.Store$MetadataSnaps
hot.buildMetadata(Store.java:575)
at
org.elasticsearch.index.store.Store$MetadataSnaps
hot.<init>(Store.java:568)
at
org.elasticsearch.index.store.Store.getMetadata(S
tore.java:186)
at
org.elasticsearch.index.store.Store.getMetadataOr
Empty(Store.java:150)
at
org.elasticsearch.indices.store.TransportNodesLis
tShardStoreMetaData.listStoreMetaData(TransportNodesList
ShardStoreMetaData.java:152)
at
org.elasticsearch.indices.store.TransportNodesLis
tShardStoreMetaData.nodeOperation(TransportNodesListShar
dStoreMetaData.java:138)
at
org.elasticsearch.indices.store.TransportNodesLis
tShardStoreMetaData.nodeOperation(TransportNodesListShar
dStoreMetaData.java:59)
at
org.elasticsearch.action.support.nodes.TransportN
odesOperationAction$NodeTransportHandler.messageReceived
(TransportNodesOperationAction.java:278)
at
org.elasticsearch.action.support.nodes.TransportN
odesOperationAction$NodeTransportHandler.messageReceived
(TransportNodesOperationAction.java:269)
at
org.elasticsearch.transport.netty.MessageChannelH
andler$RequestHandler.run(MessageChannelHandler.java:275
)
at
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.ru
n(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: read past EOF:
MMapInde
xInput(path="D:\app\ES.ElasticSearch_v010\elasticsearch-
1.4.1\data\AP-elasticsearch\nodes\0\indices\agora_v1\24\
index\segments_6w")
at
org.apache.lucene.store.ByteBufferIndexInput.read
Byte(ByteBufferIndexInput.java:81)
at
org.apache.lucene.store.BufferedChecksumIndexInpu
t.readByte(BufferedChecksumIndexInput.java:41)
at
org.apache.lucene.store.DataInput.readInt(DataInp
ut.java:98)
at
org.apache.lucene.index.SegmentInfos.read(Segment
Infos.java:343)
at
org.apache.lucene.index.SegmentInfos$1.doBody(Seg
mentInfos.java:454)
at
org.apache.lucene.index.SegmentInfos$FindSegments
File.run(SegmentInfos.java:906)
at
org.apache.lucene.index.SegmentInfos$FindSegments
File.run(SegmentInfos.java:752)
at
org.apache.lucene.index.SegmentInfos.read(Segment
Infos.java:450)
at
org.elasticsearch.common.lucene.Lucene.readSegmen
tInfos(Lucene.java:85)
at
org.elasticsearch.index.store.Store.readSegmentsI
nfo(Store.java:124)
... 14 more
]]]
executing : True
time_in_queue_millis : 52865
time_in_queue : 52.8s
insert_order : 69863
priority : HIGH
source : shard-failed ([agora_v1][24],
node[SEIBtFznTtGpLFPgCLgW4w], [R], s[INITIALIZING]),
reason [engine failure, message [corrupted
preexisting
index][CorruptIndexException[[agora_v1][24]
Preexisting
corrupted index [corrupted_LrKHKRF7Q2KuL15TT_hPvw]
caused by: CorruptIndexException[Read past EOF while
reading segment infos]
EOFException[read past EOF:
MMapIndexInput(path="D:\
app\ES.ElasticSearch_v010\elasticsearch-1.4.1\data\AP-el
asticsearch\nodes\0\indices\agora_v1\24\index\segments_6
w")]
org.apache.lucene.index.CorruptIndexException: Read
past EOF while reading segment infos
at
org.elasticsearch.index.store.Store.readSegmentsI
nfo(Store.java:127)
at
org.elasticsearch.index.store.Store.access$400(St
ore.java:80)
at
org.elasticsearch.index.store.Store$MetadataSnaps
hot.buildMetadata(Store.java:575)
at
org.elasticsearch.index.store.Store$MetadataSnaps
hot.<init>(Store.java:568)
at
org.elasticsearch.index.store.Store.getMetadata(S
tore.java:186)
at
org.elasticsearch.index.store.Store.getMetadataOr
Empty(Store.java:150)
at
org.elasticsearch.indices.store.TransportNodesLis
tShardStoreMetaData.listStoreMetaData(TransportNodesList
ShardStoreMetaData.java:152)
at
org.elasticsearch.indices.store.TransportNodesLis
tShardStoreMetaData.nodeOperation(TransportNodesListShar
dStoreMetaData.java:138)
at
org.elasticsearch.indices.store.TransportNodesLis
tShardStoreMetaData.nodeOperation(TransportNodesListShar
dStoreMetaData.java:59)
at
org.elasticsearch.action.support.nodes.TransportN
odesOperationAction$NodeTransportHandler.messageReceived
(TransportNodesOperationAction.java:278)
at
org.elasticsearch.action.support.nodes.TransportN
odesOperationAction$NodeTransportHandler.messageReceived
(TransportNodesOperationAction.java:269)
at
org.elasticsearch.transport.netty.MessageChannelH
andler$RequestHandler.run(MessageChannelHandler.java:275
)
at
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.ru
n(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: read past EOF:
MMapInde
xInput(path="D:\app\ES.ElasticSearch_v010\elasticsearch-
1.4.1\data\AP-elasticsearch\nodes\0\indices\agora_v1\24\
index\segments_6w")
at
org.apache.lucene.store.ByteBufferIndexInput.read
Byte(ByteBufferIndexInput.java:81)
at
org.apache.lucene.store.BufferedChecksumIndexInpu
t.readByte(BufferedChecksumIndexInput.java:41)
at
org.apache.lucene.store.DataInput.readInt(DataInp
ut.java:98)
at
org.apache.lucene.index.SegmentInfos.read(Segment
Infos.java:343)
at
org.apache.lucene.index.SegmentInfos$1.doBody(Seg
mentInfos.java:454)
at
org.apache.lucene.index.SegmentInfos$FindSegments
File.run(SegmentInfos.java:906)
at
org.apache.lucene.index.SegmentInfos$FindSegments
File.run(SegmentInfos.java:752)
at
org.apache.lucene.index.SegmentInfos.read(Segment
Infos.java:450)
at
org.elasticsearch.common.lucene.Lucene.readSegmen
tInfos(Lucene.java:85)
at
org.elasticsearch.index.store.Store.readSegmentsI
nfo(Store.java:124)
... 14 more
]]]
executing : False
time_in_queue_millis : 52862
time_in_queue : 52.8s
insert_order : 69865
priority : HIGH
source : shard-failed ([kibana-int][88],
node[adjp-WHHSP6kWEiPd3HkeQ], [R], s[INITIALIZING]),
reason [Failed to start shard, message
[RecoveryFailedException[[kibana-int][88]: Recovery
failed from
[Quasimodo][spfLOfnjTeiGwrYPMIiRjg][CH1SCH06
0021734][inet[/10.46.208.169:9300]] into
[Hyperion][adjp
-WHHSP6kWEiPd3HkeQ][CH1SCH050051642][inet[/10.46.216.169
:9300]]]; nested:
RemoteTransportException[[Quasimodo][i
net[/10.46.208.169:9300]][internal:index/shard/recovery/
start_recovery]]; nested:
RecoveryEngineException[[kibana-int][88] Phase[1]
Execution failed]; nested:
RecoverFilesRecoveryException[[kibana-int][88] Failed
to transfer [0] files with total size of [0b]];
nested:
NoSuchFileException[D:\app\ES.ElasticSearch_v010\elastic
search-1.4.1\data\AP-elasticsearch\nodes\0\indices\kiban
a-int\88\index\segments_2]; ]]
executing : False
time_in_queue_millis : 52860
time_in_queue : 52.8s
--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Index-corruption-when-upload-large-number-of-documents-4billion-tp4068757p4068766.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1420806261438-4068766.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.
