Hari Sekhon created SOLR-7255: --------------------------------- Summary: Index Corruption on HDFS whenever online bulk indexing (from Hive) Key: SOLR-7255 URL: https://issues.apache.org/jira/browse/SOLR-7255 Project: Solr Issue Type: Bug Affects Versions: 4.10.3 Environment: HDP 2.2 / HDP Search + LucidWorks hadoop-lws-job.jar Reporter: Hari Sekhon Priority: Blocker
When running SolrCloud on HDFS and using the LucidWorks hadoop-lws-job.jar to index a Hive table (620M rows) to Solr it runs for about 1500 secs and then gets this exception: {code}Exception in thread "Lucene Merge Thread #2191" org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1494817490 vs expected header=1071082519 (resource: BufferedChecksumIndexInput(_r3.nvm)) at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:549) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:522) Caused by: org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1494817490 vs expected header=1071082519 (resource: BufferedChecksumIndexInput(_r3.nvm)) at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:136) at org.apache.lucene.codecs.lucene49.Lucene49NormsProducer.<init>(Lucene49NormsProducer.java:75) at org.apache.lucene.codecs.lucene49.Lucene49NormsFormat.normsProducer(Lucene49NormsFormat.java:112) at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:127) at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:108) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:282) at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3951) at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3913) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3766) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486) {code} So I deleted the whole index, re-create it and re-ran the job to send Hive table contents to Solr again and it returned exactly the same exception the first time after trying to send a lot of updates to Solr. I moved off HDFS to a normal dataDir backend and then re-indexed the full table in 2 hours successfully without index corruptions. This implies that this is some sort of stability issue on the HDFS DirectoryFactory implementation. Regards, Hari Sekhon http://www.linkedin.com/in/harisekhon -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org