OK sorry I meant your first index, and it seems to have only one
(broken) segments file.  Can you post the "ls -l" output of that first
index?  It looks like the file was (illegally) filled with 0s, or at
least the first 4 bytes were.

Lucene writes this file, fsyncs it, does an atomic rename, and fsyncs
the directory, so this should not happen, if your IO system honors
fsync.

What IO devices are used by the NFS server?

NFS is not well tested and has several known problems with Lucene so
this is already risky ground...

Mike McCandless

http://blog.mikemccandless.com

On Thu, Sep 22, 2016 at 11:33 AM, Ziming Dong <dzm1016397...@gmail.com> wrote:
> second index is recovered by checkIndex, I don't know what are in second
> index directory before recover.
> checkIndex can't read first index. index filenames are attached.
> I use lucene6.0.0 at the beginning, then I upgrade to lucene6.1.0 to
> continue index.
>
> On Thu, Sep 22, 2016 at 10:17 PM, Michael McCandless
> <luc...@mikemccandless.com> wrote:
>>
>> Do you have 2 separate segments files in that 2nd index?
>>
>> Which exact Lucene version is this?
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Thu, Sep 22, 2016 at 7:44 AM, Ziming Dong <dzm1016397...@gmail.com>
>> wrote:
>> > I used checkIndex to recover second index though I lost many docs in
>> > index,
>> > but first index can't be read by checkIndex, error is
>> >
>> >> java -cp lucene-core-6.1.0.jar -ea:org.apache.lucene...
>> >> org.apache.lucene.index.CheckIndex
>> >> /Volumes/HPT8_56T/infomall-index/index0
>> >> Opening index @ /Volumes/HPT8_56T/infomall-index/index0
>> >> ERROR: could not read any segments file in directory
>> >> org.apache.lucene.index.IndexFormatTooOldException: Format version is
>> >> not
>> >> supported (resource
>> >>
>> >> BufferedChecksumIndexInput(MMapIndexInput(path="/Volumes/HPT8_56T/infomall-index/index0/segments_5t3"))):
>> >> 0 (needs to be between 1071082519 and 1071082519). This version of
>> >> Lucene
>> >> only supports indexes created with release 5.0 and later.
>> >>         at
>> >> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:295)
>> >>         at
>> >> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:284)
>> >>         at
>> >> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:507)
>> >>         at
>> >> org.apache.lucene.index.CheckIndex.doCheck(CheckIndex.java:2595)
>> >>         at
>> >> org.apache.lucene.index.CheckIndex.doMain(CheckIndex.java:2497)
>> >>         at
>> >> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2423)
>> >
>> >
>> >  I use NFS, but I set mount option as  mount -t nfs -o
>> > tcp,sync,retrans=10
>> > The index program has run 1 month without any problem before power
>> > failure.
>> >
>> > On Thu, Sep 22, 2016 at 6:06 PM, Michael McCandless
>> > <luc...@mikemccandless.com> wrote:
>> >>
>> >> Hmm I'm no longer so sure this is an IW bug: on commit we fsync the
>> >> pending_segments_N and then do an atomic rename to segments_N.
>> >>
>> >> Can you describe your IO system?  Is it possible it does not implement
>> >> fsync or atomic renames correctly?
>> >>
>> >> Also, your 2nd exception indices the segments_N file was intact but
>> >> the .cfs file was corrupt, which is also hard to explain unless fsync
>> >> isn't working on your IO system.
>> >>
>> >> Mike McCandless
>> >>
>> >> http://blog.mikemccandless.com
>> >>
>> >> On Thu, Sep 22, 2016 at 5:10 AM, Michael McCandless
>> >> <luc...@mikemccandless.com> wrote:
>> >> > Sorry for the slow reply here.  Curious that both of these exceptions
>> >> > are from IW.init.  I think this may be a real bug, caused by this:
>> >> >
>> >> >
>> >> > https://github.com/apache/lucene-solr/commit/981bfba841144d08df1d1a183d39fcd6f195ad56
>> >> >
>> >> > I'll see if I can make a standalone test case showing this.
>> >> >
>> >> > If you open those indices with an IndexReader instead, does it
>> >> > succeed?
>> >> >
>> >> > If you run CheckIndex, what does it report?
>> >> >
>> >> > Mike McCandless
>> >> >
>> >> > http://blog.mikemccandless.com
>> >> >
>> >> > On Wed, Sep 14, 2016 at 1:22 AM, Ziming Dong
>> >> > <dzm1016397...@gmail.com>
>> >> > wrote:
>> >> >> I have 6 machine and 6 index directories, each machine builds index
>> >> >> into
>> >> >> one index directory. After power failure last night, two of those
>> >> >> machine
>> >> >> can't start index program.
>> >> >>
>> >> >> one error is
>> >> >>
>> >> >>> INFO: 2016-09-14 12:31:38 [main]
>> >> >>>
>> >> >>> sewm.bdbox.search.InfomallIndexer$Builder:ignoreCollectionsFile(227):
>> >> >>> Loaded 2146 ignored collections from
>> >> >>> /mnt/HPT8_56T/infomall-index/index0/ignored_collections.txt
>> >> >>> ERROR: 2016-09-14 12:31:39 [main]
>> >> >>> sewm.bdbox.util.LogUtil:error(71):
>> >> >>> org.apache.lucene.index.IndexFormatTooOldException: Format version
>> >> >>> is
>> >> >>> not
>> >> >>> supported (resource
>> >> >>>
>> >> >>>
>> >> >>> BufferedChecksumIndexInput(MMapIndexInput(path="/mnt/HPT8_56T/infomall-index/index0/segments_5t3"))):
>> >> >>> 0 (needs to be between 1071082519 and 1071082519). This version of
>> >> >>> Lucene
>> >> >>> only supports indexes created with release 5.0 and later.
>> >> >>>         at
>> >> >>>
>> >> >>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:295)
>> >> >>>         at
>> >> >>>
>> >> >>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:284)
>> >> >>>         at
>> >> >>> org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:910)
>> >> >>>         at
>> >> >>> sewm.bdbox.search.InfomallIndexer.<init>(InfomallIndexer.java:60)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> sewm.bdbox.search.ThreadedInfomallIndexer.<init>(ThreadedInfomallIndexer.java:28)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> sewm.bdbox.search.ThreadedInfomallIndexer.<init>(ThreadedInfomallIndexer.java:21)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> sewm.bdbox.search.ThreadedInfomallIndexer$Builder.build(ThreadedInfomallIndexer.java:72)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> sewm.bdbox.search.ThreadedInfomallIndexer.main(ThreadedInfomallIndexer.java:129)
>> >> >>
>> >> >>
>> >> >> another is
>> >> >>
>> >> >> INFO: 2016-09-14 01:11:06 [main]
>> >> >>>
>> >> >>> sewm.bdbox.search.InfomallIndexer$Builder:ignoreCollectionsFile(227):
>> >> >>> Loaded 8575 ignored collections from
>> >> >>> /mnt/HPT8/infomall-index/index5/ignored_collections.txt
>> >> >>> ERROR: 2016-09-14 01:11:09 [main]
>> >> >>> sewm.bdbox.util.LogUtil:error(71):
>> >> >>> org.apache.lucene.index.CorruptIndexException: codec footer
>> >> >>> mismatch
>> >> >>> (file
>> >> >>> truncated?): actual footer=0 vs expected footer=-1071082520
>> >> >>>
>> >> >>>
>> >> >>> (resource=MMapIndexInput(path="/mnt/HPT8/infomall-index/index5/_1kqn.cfs"))
>> >> >>>         at
>> >> >>>
>> >> >>> org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:448)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:433)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:86)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1016)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1033)
>> >> >>>         at
>> >> >>> org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:938)
>> >> >>>         at
>> >> >>> sewm.bdbox.search.InfomallIndexer.<init>(InfomallIndexer.java:60)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> sewm.bdbox.search.ThreadedInfomallIndexer.<init>(ThreadedInfomallIndexer.java:28)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> sewm.bdbox.search.ThreadedInfomallIndexer.<init>(ThreadedInfomallIndexer.java:21)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> sewm.bdbox.search.ThreadedInfomallIndexer$Builder.build(ThreadedInfomallIndexer.java:72)
>> >> >>>         at
>> >> >>>
>> >> >>>
>> >> >>> sewm.bdbox.search.ThreadedInfomallIndexer.main(ThreadedInfomallIndexer.java:129)
>> >> >>>
>> >> >>
>> >> >>
>> >> >> it seems 1071082519 is a special number.
>> >> >>
>> >> >> - -
>> >> >>
>> >> >> Ziming Dong
>> >> >> *http://suiyuan2009.github.io/ <http://suiyuan2009.github.io/>*
>> >
>> >
>> >
>> >
>> > --
>> >
>> > Ziming Dong
>> > http://suiyuan2009.github.io/
>> >
>
>
>
>
> --
>
> Ziming Dong
> http://suiyuan2009.github.io/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to