Looks like an int overflow to me, but I'd have to take a look at the code to see where it can originate. Please file a Jira issue, attach the full stack trace and name it "int overflow in Lucene50SkipWriter.writeSkipData" because that's where the problem is. We'll definitely take a look.
D. On Thu, Dec 27, 2018 at 10:34 AM John Wilson <sami.hailu...@gmail.com> wrote: > > The data set I'm experimenting with is ~100G. If I break it up into four 25G > data sets and index is individually, it works just fine. But if I try to > index the entire 100G it breaks. I'm not sure if there is a size limitation > or sth (FYI, I have a really large heap of size 256G). > > Thanks, > > On Thu, Dec 27, 2018 at 12:27 AM Dawid Weiss <dawid.we...@gmail.com> wrote: >> >> This looks like a bug, the underlying cause is: >> >> Caused by: java.lang.IllegalArgumentException: cannot write negative >> vLong (got: -4294878395) >> at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) >> at >> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) >> >> Can you reproduce it on a smaller example and file a Jira issue? >> >> Dawid >> >> On Thu, Dec 27, 2018 at 7:18 AM John Wilson <sami.hailu...@gmail.com> wrote: >> > >> > Hi, >> > >> > I'm getting the below error message while running a simple multi-threaded >> > indexing using ByteBuffersDirectory. Any suggestions? >> > >> > Exception in thread "Lucene Merge Thread #879" >> > org.apache.lucene.index.MergePolicy$MergeException: >> > org.apache.lucene.store.AlreadyClosedException: refusing to delete any >> > files: this IndexWriter hit an unrecoverable exception >> > at >> > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705) >> > at >> > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685) >> > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to >> > delete any files: this IndexWriter hit an unrecoverable exception >> > at >> > org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349) >> > at >> > org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669) >> > at >> > org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664) >> > at >> > org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024) >> > at >> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539) >> > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075) >> > at >> > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626) >> > at >> > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663) >> > Caused by: java.lang.IllegalArgumentException: cannot write negative vLong >> > (got: -4294878395) >> > at >> > org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) >> > at >> > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) >> > at >> > org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143) >> > at >> > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162) >> > at >> > org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228) >> > at >> > org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148) >> > at >> > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865) >> > at >> > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344) >> > at >> > org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105) >> > at >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169) >> > at >> > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244) >> > at >> > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139) >> > at >> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453) >> > ... 3 more >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
