Re: ByteBuffersDirectory throws Exception
Looks like an int overflow to me, but I'd have to take a look at the code to see where it can originate. Please file a Jira issue, attach the full stack trace and name it "int overflow in Lucene50SkipWriter.writeSkipData" because that's where the problem is. We'll definitely take a look. D. On Thu, Dec 27, 2018 at 10:34 AM John Wilson wrote: > > The data set I'm experimenting with is ~100G. If I break it up into four 25G > data sets and index is individually, it works just fine. But if I try to > index the entire 100G it breaks. I'm not sure if there is a size limitation > or sth (FYI, I have a really large heap of size 256G). > > Thanks, > > On Thu, Dec 27, 2018 at 12:27 AM Dawid Weiss wrote: >> >> This looks like a bug, the underlying cause is: >> >> Caused by: java.lang.IllegalArgumentException: cannot write negative >> vLong (got: -4294878395) >> at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) >> at >> org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) >> >> Can you reproduce it on a smaller example and file a Jira issue? >> >> Dawid >> >> On Thu, Dec 27, 2018 at 7:18 AM John Wilson wrote: >> > >> > Hi, >> > >> > I'm getting the below error message while running a simple multi-threaded >> > indexing using ByteBuffersDirectory. Any suggestions? >> > >> > Exception in thread "Lucene Merge Thread #879" >> > org.apache.lucene.index.MergePolicy$MergeException: >> > org.apache.lucene.store.AlreadyClosedException: refusing to delete any >> > files: this IndexWriter hit an unrecoverable exception >> > at >> > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705) >> > at >> > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685) >> > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to >> > delete any files: this IndexWriter hit an unrecoverable exception >> > at >> > org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349) >> > at >> > org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669) >> > at >> > org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664) >> > at >> > org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024) >> > at >> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539) >> > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075) >> > at >> > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626) >> > at >> > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663) >> > Caused by: java.lang.IllegalArgumentException: cannot write negative vLong >> > (got: -4294878395) >> > at >> > org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) >> > at >> > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) >> > at >> > org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143) >> > at >> > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162) >> > at >> > org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228) >> > at >> > org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148) >> > at >> > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865) >> > at >> > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344) >> > at >> > org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105) >> > at >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169) >> > at >> > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244) >> > at >> > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139) >> > at >> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453) >> > ... 3 more >> > >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: ByteBuffersDirectory throws Exception
The data set I'm experimenting with is ~100G. If I break it up into four 25G data sets and index is individually, it works just fine. But if I try to index the entire 100G it breaks. I'm not sure if there is a size limitation or sth (FYI, I have a really large heap of size 256G). Thanks, On Thu, Dec 27, 2018 at 12:27 AM Dawid Weiss wrote: > This looks like a bug, the underlying cause is: > > Caused by: java.lang.IllegalArgumentException: cannot write negative > vLong (got: -4294878395) > at > org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) > at > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) > > Can you reproduce it on a smaller example and file a Jira issue? > > Dawid > > On Thu, Dec 27, 2018 at 7:18 AM John Wilson > wrote: > > > > Hi, > > > > I'm getting the below error message while running a simple > multi-threaded indexing using ByteBuffersDirectory. Any suggestions? > > > > Exception in thread "Lucene Merge Thread #879" > org.apache.lucene.index.MergePolicy$MergeException: > org.apache.lucene.store.AlreadyClosedException: refusing to delete any > files: this IndexWriter hit an unrecoverable exception > > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685) > > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to > delete any files: this IndexWriter hit an unrecoverable exception > > at > org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349) > > at > org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669) > > at > org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664) > > at > org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024) > > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539) > > at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075) > > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663) > > Caused by: java.lang.IllegalArgumentException: cannot write negative > vLong (got: -4294878395) > > at > org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) > > at > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) > > at > org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143) > > at > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162) > > at > org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228) > > at > org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148) > > at > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865) > > at > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344) > > at > org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105) > > at > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169) > > at > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244) > > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139) > > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453) > > ... 3 more > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: ByteBuffersDirectory throws Exception
This looks like a bug, the underlying cause is: Caused by: java.lang.IllegalArgumentException: cannot write negative vLong (got: -4294878395) at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) Can you reproduce it on a smaller example and file a Jira issue? Dawid On Thu, Dec 27, 2018 at 7:18 AM John Wilson wrote: > > Hi, > > I'm getting the below error message while running a simple multi-threaded > indexing using ByteBuffersDirectory. Any suggestions? > > Exception in thread "Lucene Merge Thread #879" > org.apache.lucene.index.MergePolicy$MergeException: > org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: > this IndexWriter hit an unrecoverable exception > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685) > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete > any files: this IndexWriter hit an unrecoverable exception > at > org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349) > at > org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669) > at > org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664) > at > org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663) > Caused by: java.lang.IllegalArgumentException: cannot write negative vLong > (got: -4294878395) > at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) > at > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) > at > org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143) > at > org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162) > at > org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228) > at > org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148) > at > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865) > at > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344) > at > org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105) > at > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169) > at > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244) > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453) > ... 3 more > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
ByteBuffersDirectory throws Exception
Hi, I'm getting the below error message while running a simple multi-threaded indexing using ByteBuffersDirectory. Any suggestions? Exception in thread "Lucene Merge Thread #879" org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685) Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349) at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669) at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664) at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663) Caused by: java.lang.IllegalArgumentException: cannot write negative vLong (got: -4294878395) at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225) at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180) at org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143) at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162) at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228) at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148) at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865) at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344) at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169) at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453) ... 3 more
