[jira] [Updated] (LUCENE-8624) ByteBuffersDataOutput Integer Overflow

2018-12-27 Thread Mulugeta Mammo (JIRA) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-8624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mulugeta Mammo updated LUCENE-8624:
---
Description: 
Hi,

When indexing large data sets with ByteBuffersDirectory, an exception like the 
below is thrown:

Caused by: java.lang.IllegalArgumentException: cannot write negative vLong 
(got: -4294888321)
 at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:182)
 at 
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
 at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
 at 
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
 at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
 at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
 at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
 at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)

{{The exception is caused by an integer overflow while calling getFilePointer() 
in Lucene50PostingsWriter, which eventually calls the size() method in 
ByteBuffersDataOutput.}}
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
public long size() {
 long size = 0;
 int blockCount = blocks.size();
 if (blockCount >= 1) {
 int fullBlockSize = (blockCount - 1) * blockSize(); // throws integer overflow
 int lastBlockSize = blocks.getLast().position();
 size = fullBlockSize + lastBlockSize;
 }
 return size;
}

{code}
In my case, I had a blockCount = 65 and a blockSize() = 33554432 which 
overflows fullBlockSize. The fix:
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
public long size() {
long size = 0;
int blockCount = blocks.size();
if (blockCount >= 1) {
long fullBlockSize = 1L * (blockCount - 1) * blockSize();
int lastBlockSize = blocks.getLast().position();
size = fullBlockSize + lastBlockSize;
}
return size;
}

{code}
 

Thanks

  was:
Hi,

When indexing large data sets with ByteBuffersDirectory, an exception like the 
below is thrown:

Caused by: java.lang.IllegalArgumentException: cannot write negative vLong 
(got: -4294888321)
 at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:182)
 at 
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
 at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
 at 
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
 at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
 at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
 at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
 at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)

{{The exception is caused by an integer overflow while calling getFilePointer() 
in Lucene50PostingsWriter, which eventually calls the size() method in 
ByteBuffersDataOutput.}}
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
public long size() {
 long size = 0;
 int blockCount = blocks.size();
 if (blockCount >= 1) {
 int fullBlockSize = (blockCount - 1) * blockSize();
 int lastBlockSize = blocks.getLast().position();
 size = fullBlockSize + lastBlockSize;
 }
 return size;
}

{code}
In my case, I had a blockCount = 65 and a blockSize() = 33554432 which 
overflows fullBlockSize. The fix:
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
public long size() {
long size = 0;
int blockCount = blocks.size();
if (blockCount >= 1) {
long fullBlockSize = 1L * (blockCount - 1) * blockSize();
int lastBlockSize = blocks.getLast().position();
size =

[jira] [Updated] (LUCENE-8625) Integer Overflow in ByteBuffersDataInput

2018-12-27 Thread Mulugeta Mammo (JIRA) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mulugeta Mammo updated LUCENE-8625:
---
Description: 
Hi,

Once I fixed the bug here, https://issues.apache.org/jira/browse/LUCENE-8624, I 
encountered another Integer Overflow error in ByteBuffersDataInput:

Exception in thread "Lucene Merge Thread #1540" Exception in thread "main" 
org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.ArithmeticException: integer overflow
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
 Caused by: java.lang.ArithmeticException: integer overflow
 at java.lang.Math.toIntExact(Math.java:1011)
 at 
org.apache.lucene.store.ByteBuffersDataInput.sliceBufferList(ByteBuffersDataInput.java:299)
 at 
org.apache.lucene.store.ByteBuffersDataInput.slice(ByteBuffersDataInput.java:223)
 at 
org.apache.lucene.store.ByteBuffersIndexInput.clone(ByteBuffersIndexInput.java:186)
 at 
org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:254)
 at 
org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:223)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
 at 
org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
 at 
org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5004)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4517)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)

The exception is caused by a Math.toIntExact in sliceBufferList in 
ByteBuffersDataInput.
{code:java|title=ByteBuffersDataInput.java|borderStyle=solid}
private static List sliceBufferList(List buffers, long 
offset, long length) {
 ensureAssumptions(buffers);

 if (buffers.size() == 1) {
 ByteBuffer cloned = buffers.get(0).asReadOnlyBuffer();
 cloned.position(Math.toIntExact(cloned.position() + offset));
 cloned.limit(Math.toIntExact(length + cloned.position()));
 return Arrays.asList(cloned);
 } else {
 long absStart = buffers.get(0).position() + offset;
 long absEnd = Math.toIntExact(absStart + length);  // throws integer overflow 

...

 {code}
Removing the Math.toIntExact works but I'm not sure if the logic will still be 
right since absEnd is used to calculate endOffset after a few lines:
{code:java}
int endOffset = (int) absEnd & blockMask;
{code}
 

Thanks,

  was:
Hi,

Once I fixed the bug here, https://issues.apache.org/jira/browse/LUCENE-8624, I 
encountered another Integer Overflow error in ByteBuffersDataInput:

Exception in thread "Lucene Merge Thread #1540" Exception in thread "main" 
org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.ArithmeticException: integer overflow
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
 Caused by: java.lang.ArithmeticException: integer overflow
 at java.lang.Math.toIntExact(Math.java:1011)
 at 
org.apache.lucene.store.ByteBuffersDataInput.sliceBufferList(ByteBuffersDataInput.java:299)
 at 
org.apache.lucene.store.ByteBuffersDataInput.slice(ByteBuffersDataInput.java:223)
 at 
org.apache.lucene.store.ByteBuffersIndexInput.clone(ByteBuffersIndexInput.java:186)
 at 
org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:254)
 at 
org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:223)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
 at 
org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
 at 
org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5004)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4517)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
 at

[jira] [Updated] (LUCENE-8625) Integer Overflow in ByteBuffersDataInput

2018-12-27 Thread Mulugeta Mammo (JIRA) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mulugeta Mammo updated LUCENE-8625:
---
Description: 
Hi,

Once I fixed the bug here, https://issues.apache.org/jira/browse/LUCENE-8624, I 
encountered another Integer Overflow error in ByteBuffersDataInput:

Exception in thread "Lucene Merge Thread #1540" Exception in thread "main" 
org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.ArithmeticException: integer overflow
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
 Caused by: java.lang.ArithmeticException: integer overflow
 at java.lang.Math.toIntExact(Math.java:1011)
 at 
org.apache.lucene.store.ByteBuffersDataInput.sliceBufferList(ByteBuffersDataInput.java:299)
 at 
org.apache.lucene.store.ByteBuffersDataInput.slice(ByteBuffersDataInput.java:223)
 at 
org.apache.lucene.store.ByteBuffersIndexInput.clone(ByteBuffersIndexInput.java:186)
 at 
org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:254)
 at 
org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:223)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
 at 
org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
 at 
org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5004)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4517)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)

The exception is caused by a Math.toIntExact made in sliceBufferList in 
ByteBuffersDataInput.
{code:java|title=ByteBuffersDataInput.java|borderStyle=solid}
private static List sliceBufferList(List buffers, long 
offset, long length) {
 ensureAssumptions(buffers);

 if (buffers.size() == 1) {
 ByteBuffer cloned = buffers.get(0).asReadOnlyBuffer();
 cloned.position(Math.toIntExact(cloned.position() + offset));
 cloned.limit(Math.toIntExact(length + cloned.position()));
 return Arrays.asList(cloned);
 } else {
 long absStart = buffers.get(0).position() + offset;
 long absEnd = Math.toIntExact(absStart + length);  // throws integer overflow 

...

 {code}
Removing the Math.toIntExact works but I'm not sure if the logic will still be 
right since absEnd is used to calculate endOffset after a few lines:
{code:java}
int endOffset = (int) absEnd & blockMask;
{code}
 

Thanks,

  was:
Hi,

Once I fixed the bug here, https://issues.apache.org/jira/browse/LUCENE-8624, I 
encountered another Integer Overflow error in ByteBuffersDataInput:

xception in thread "Lucene Merge Thread #1540" Exception in thread "main" 
org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.ArithmeticException: integer overflow
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
 Caused by: java.lang.ArithmeticException: integer overflow
 at java.lang.Math.toIntExact(Math.java:1011)
 at 
org.apache.lucene.store.ByteBuffersDataInput.sliceBufferList(ByteBuffersDataInput.java:299)
 at 
org.apache.lucene.store.ByteBuffersDataInput.slice(ByteBuffersDataInput.java:223)
 at 
org.apache.lucene.store.ByteBuffersIndexInput.clone(ByteBuffersIndexInput.java:186)
 at 
org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:254)
 at 
org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:223)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
 at 
org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
 at 
org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5004)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4517)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
 at

[jira] [Updated] (LUCENE-8625) Integer Overflow in ByteBuffersDataInput

2018-12-27 Thread Mulugeta Mammo (JIRA) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mulugeta Mammo updated LUCENE-8625:
---
Summary: Integer Overflow in ByteBuffersDataInput  (was: Integer Overflow 
Exception in ByteBuffersDataInpu)

> Integer Overflow in ByteBuffersDataInput
> 
>
> Key: LUCENE-8625
> URL: https://issues.apache.org/jira/browse/LUCENE-8625
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/store
>Affects Versions: 7.5
>Reporter: Mulugeta Mammo
>Priority: Major
> Fix For: 7.5
>
>
> Hi,
> Once I fixed the bug here, https://issues.apache.org/jira/browse/LUCENE-8624, 
> I encountered another Integer Overflow error in ByteBuffersDataInput:
> xception in thread "Lucene Merge Thread #1540" Exception in thread "main" 
> org.apache.lucene.index.MergePolicy$MergeException: 
> java.lang.ArithmeticException: integer overflow
>  at 
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
>  at 
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
>  Caused by: java.lang.ArithmeticException: integer overflow
>  at java.lang.Math.toIntExact(Math.java:1011)
>  at 
> org.apache.lucene.store.ByteBuffersDataInput.sliceBufferList(ByteBuffersDataInput.java:299)
>  at 
> org.apache.lucene.store.ByteBuffersDataInput.slice(ByteBuffersDataInput.java:223)
>  at 
> org.apache.lucene.store.ByteBuffersIndexInput.clone(ByteBuffersIndexInput.java:186)
>  at 
> org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:254)
>  at 
> org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:223)
>  at 
> org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
>  at 
> org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
>  at 
> org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
>  at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
>  at 
> org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
>  at 
> org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5004)
>  at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4517)
>  at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
>  at 
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
>  at 
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)
> The exception is caused by a Math.toIntExact made in sliceBufferList in 
> ByteBuffersDataInput.
> {code:java|title=ByteBuffersDataInput.java|borderStyle=solid}
> private static List sliceBufferList(List buffers, 
> long offset, long length) {
>  ensureAssumptions(buffers);
>  if (buffers.size() == 1) {
>  ByteBuffer cloned = buffers.get(0).asReadOnlyBuffer();
>  cloned.position(Math.toIntExact(cloned.position() + offset));
>  cloned.limit(Math.toIntExact(length + cloned.position()));
>  return Arrays.asList(cloned);
>  } else {
>  long absStart = buffers.get(0).position() + offset;
>  long absEnd = Math.toIntExact(absStart + length);  // throws integer 
> overflow 
> ...
>  {code}
> Removing the Math.toIntExact works but I'm not sure if the logic will still 
> be right since absEnd is used to calculate endOffset after a few lines:
> {code:java}
> int endOffset = (int) absEnd & blockMask;
> {code}
>  
> Thanks,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8625) Integer Overflow Exception in ByteBuffersDataInpu

2018-12-27 Thread Mulugeta Mammo (JIRA) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mulugeta Mammo updated LUCENE-8625:
---
Description: 
Hi,

Once I fixed the bug here, https://issues.apache.org/jira/browse/LUCENE-8624, I 
encountered another Integer Overflow error in ByteBuffersDataInput:

xception in thread "Lucene Merge Thread #1540" Exception in thread "main" 
org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.ArithmeticException: integer overflow
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
 Caused by: java.lang.ArithmeticException: integer overflow
 at java.lang.Math.toIntExact(Math.java:1011)
 at 
org.apache.lucene.store.ByteBuffersDataInput.sliceBufferList(ByteBuffersDataInput.java:299)
 at 
org.apache.lucene.store.ByteBuffersDataInput.slice(ByteBuffersDataInput.java:223)
 at 
org.apache.lucene.store.ByteBuffersIndexInput.clone(ByteBuffersIndexInput.java:186)
 at 
org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:254)
 at 
org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:223)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
 at 
org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
 at 
org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5004)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4517)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)

The exception is caused by a Math.toIntExact made in sliceBufferList in 
ByteBuffersDataInput.
{code:java|title=ByteBuffersDataInput.java|borderStyle=solid}

private static List sliceBufferList(List buffers, long 
offset, long length) {
 ensureAssumptions(buffers);

 if (buffers.size() == 1) {
 ByteBuffer cloned = buffers.get(0).asReadOnlyBuffer();
 cloned.position(Math.toIntExact(cloned.position() + offset));
 cloned.limit(Math.toIntExact(length + cloned.position()));
 return Arrays.asList(cloned);
 } else {
 long absStart = buffers.get(0).position() + offset;
 long absEnd = Math.toIntExact(absStart + length);  // throws integer overflow 

...

 {code}
Removing the Math.toIntExact works but I'm not sure if the logic will still be 
right since absEnd is used to calculate endOffset after a few lines:
{code:java}
int endOffset = (int) absEnd & blockMask;
{code}
 

Thanks,

  was:
Hi,

Once I fixed the bug here, https://issues.apache.org/jira/browse/LUCENE-8624, I 
encountered another Integer Overflow error in ByteBuffersDataInput:

xception in thread "Lucene Merge Thread #1540" Exception in thread "main" 
org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.ArithmeticException: integer overflow
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
Caused by: java.lang.ArithmeticException: integer overflow
 at java.lang.Math.toIntExact(Math.java:1011)
 at 
org.apache.lucene.store.ByteBuffersDataInput.sliceBufferList(ByteBuffersDataInput.java:299)
 at 
org.apache.lucene.store.ByteBuffersDataInput.slice(ByteBuffersDataInput.java:223)
 at 
org.apache.lucene.store.ByteBuffersIndexInput.clone(ByteBuffersIndexInput.java:186)
 at 
org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:254)
 at 
org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:223)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
 at 
org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
 at 
org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5004)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4517)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
 at

[jira] [Created] (LUCENE-8625) Integer Overflow Exception in ByteBuffersDataInpu

2018-12-27 Thread Mulugeta Mammo (JIRA) 
Mulugeta Mammo created LUCENE-8625:
--

 Summary: Integer Overflow Exception in ByteBuffersDataInpu
 Key: LUCENE-8625
 URL: https://issues.apache.org/jira/browse/LUCENE-8625
 Project: Lucene - Core
  Issue Type: Bug
  Components: core/store
Affects Versions: 7.5
Reporter: Mulugeta Mammo
 Fix For: 7.5


Hi,

Once I fixed the bug here, https://issues.apache.org/jira/browse/LUCENE-8624, I 
encountered another Integer Overflow error in ByteBuffersDataInput:

xception in thread "Lucene Merge Thread #1540" Exception in thread "main" 
org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.ArithmeticException: integer overflow
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)
Caused by: java.lang.ArithmeticException: integer overflow
 at java.lang.Math.toIntExact(Math.java:1011)
 at 
org.apache.lucene.store.ByteBuffersDataInput.sliceBufferList(ByteBuffersDataInput.java:299)
 at 
org.apache.lucene.store.ByteBuffersDataInput.slice(ByteBuffersDataInput.java:223)
 at 
org.apache.lucene.store.ByteBuffersIndexInput.clone(ByteBuffersIndexInput.java:186)
 at 
org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:254)
 at 
org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:223)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
 at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
 at 
org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
 at 
org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5004)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4517)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
 at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)

The exception is caused by a Math.toIntExact made in sliceBufferList in 
ByteBuffersDataInput.

{{{code:title=ByteBuffersDataInput.java|borderStyle=solid}}}

private static List sliceBufferList(List buffers, long 
offset, long length) {
 ensureAssumptions(buffers);

 if (buffers.size() == 1) {
 ByteBuffer cloned = buffers.get(0).asReadOnlyBuffer();
 cloned.position(Math.toIntExact(cloned.position() + offset));
 cloned.limit(Math.toIntExact(length + cloned.position()));
 return Arrays.asList(cloned);
 } else {
 long absStart = buffers.get(0).position() + offset;
 long absEnd = Math.toIntExact(absStart + length);  // throws integer overflow 

...

{{{code}}} 

 

Removing the Math.toIntExact works but I'm not sure if the logic will still be 
right since absEnd is used to calculate endOffset:
{code:java}
int endOffset = (int) absEnd & blockMask;
{code}
 

Thanks,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8624) ByteBuffersDataOutput Integer Overflow

2018-12-27 Thread Mulugeta Mammo (JIRA) 
Mulugeta Mammo created LUCENE-8624:
--

 Summary: ByteBuffersDataOutput Integer Overflow
 Key: LUCENE-8624
 URL: https://issues.apache.org/jira/browse/LUCENE-8624
 Project: Lucene - Core
  Issue Type: Bug
  Components: core/store
Affects Versions: 7.5
Reporter: Mulugeta Mammo
 Fix For: 7.5


Hi,

When indexing large data sets with ByteBuffersDirectory, an exception like the 
below is thrown:

{{}}Caused by: java.lang.IllegalArgumentException: cannot write negative vLong 
(got: -4294888321)
 at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:182)
 at 
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
 at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
 at 
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
 at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
 at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
 at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
 at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)

{{The exception is caused by an integer overflow while calling getFilePointer() 
in Lucene50PostingsWriter, which eventually calls the size() method in 
ByteBuffersDataOutput.}}

{{{code:title=ByteBuffersDataOutput.java|borderStyle=solid} }}

public long size() {
 long size = 0;
 int blockCount = blocks.size();
 if (blockCount >= 1) {
{color:#FF}int fullBlockSize = (blockCount - 1) * blockSize();{color}
 int lastBlockSize = blocks.getLast().position();
 size = fullBlockSize + lastBlockSize;
 }
 return size;
}

{code}

In my case, I had a blockCount = 65 and a blockSize() = 33554432 which 
overflows fullBlockSize. The fix:

{{{code:title=ByteBuffersDataOutput.java|borderStyle=solid}}}

public long size() {
long size = 0;
int blockCount = blocks.size();
if (blockCount >= 1) {
{color:#FF}long fullBlockSize = 1L * (blockCount - 1) * blockSize();{color}
int lastBlockSize = blocks.getLast().position();
size = fullBlockSize + lastBlockSize;
}
return size;
}

{code}

 

Thanks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8624) ByteBuffersDataOutput Integer Overflow

2018-12-27 Thread Mulugeta Mammo (JIRA) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-8624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mulugeta Mammo updated LUCENE-8624:
---
Description: 
Hi,

When indexing large data sets with ByteBuffersDirectory, an exception like the 
below is thrown:

{{}}Caused by: java.lang.IllegalArgumentException: cannot write negative vLong 
(got: -4294888321)
 at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:182)
 at 
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
 at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
 at 
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
 at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
 at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
 at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
 at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)

{{The exception is caused by an integer overflow while calling getFilePointer() 
in Lucene50PostingsWriter, which eventually calls the size() method in 
ByteBuffersDataOutput.}}
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
public long size() {
 long size = 0;
 int blockCount = blocks.size();
 if (blockCount >= 1) {
 int fullBlockSize = (blockCount - 1) * blockSize();
 int lastBlockSize = blocks.getLast().position();
 size = fullBlockSize + lastBlockSize;
 }
 return size;
}

{code}
In my case, I had a blockCount = 65 and a blockSize() = 33554432 which 
overflows fullBlockSize. The fix:
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
public long size() {
long size = 0;
int blockCount = blocks.size();
if (blockCount >= 1) {
long fullBlockSize = 1L * (blockCount - 1) * blockSize();
int lastBlockSize = blocks.getLast().position();
size = fullBlockSize + lastBlockSize;
}
return size;
}

{code}
 

Thanks

  was:
Hi,

When indexing large data sets with ByteBuffersDirectory, an exception like the 
below is thrown:

{{}}Caused by: java.lang.IllegalArgumentException: cannot write negative vLong 
(got: -4294888321)
 at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:182)
 at 
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
 at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
 at 
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
 at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
 at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
 at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
 at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)

{{The exception is caused by an integer overflow while calling getFilePointer() 
in Lucene50PostingsWriter, which eventually calls the size() method in 
ByteBuffersDataOutput.}}
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
 }}

public long size() {
 long size = 0;
 int blockCount = blocks.size();
 if (blockCount >= 1) {
{color:#FF}int fullBlockSize = (blockCount - 1) * blockSize();{color}
 int lastBlockSize = blocks.getLast().position();
 size = fullBlockSize + lastBlockSize;
 }
 return size;
}

{code}
In my case, I had a blockCount = 65 and a blockSize() = 33554432 which 
overflows fullBlockSize. The fix:
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
}}

public long size() {
long size = 0;
int blockCount = blocks.size();
if (blockCount >= 1) {
{color:#FF}long fullBlockSize = 1L * (blockCount - 1) * blockSize();{color}
int lastBlockSize =

[jira] [Updated] (LUCENE-8624) ByteBuffersDataOutput Integer Overflow

2018-12-27 Thread Mulugeta Mammo (JIRA) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-8624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mulugeta Mammo updated LUCENE-8624:
---
Description: 
Hi,

When indexing large data sets with ByteBuffersDirectory, an exception like the 
below is thrown:

Caused by: java.lang.IllegalArgumentException: cannot write negative vLong 
(got: -4294888321)
 at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:182)
 at 
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
 at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
 at 
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
 at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
 at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
 at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
 at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)

{{The exception is caused by an integer overflow while calling getFilePointer() 
in Lucene50PostingsWriter, which eventually calls the size() method in 
ByteBuffersDataOutput.}}
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
public long size() {
 long size = 0;
 int blockCount = blocks.size();
 if (blockCount >= 1) {
 int fullBlockSize = (blockCount - 1) * blockSize();
 int lastBlockSize = blocks.getLast().position();
 size = fullBlockSize + lastBlockSize;
 }
 return size;
}

{code}
In my case, I had a blockCount = 65 and a blockSize() = 33554432 which 
overflows fullBlockSize. The fix:
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
public long size() {
long size = 0;
int blockCount = blocks.size();
if (blockCount >= 1) {
long fullBlockSize = 1L * (blockCount - 1) * blockSize();
int lastBlockSize = blocks.getLast().position();
size = fullBlockSize + lastBlockSize;
}
return size;
}

{code}
 

Thanks

  was:
Hi,

When indexing large data sets with ByteBuffersDirectory, an exception like the 
below is thrown:

{{}}Caused by: java.lang.IllegalArgumentException: cannot write negative vLong 
(got: -4294888321)
 at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:182)
 at 
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
 at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
 at 
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
 at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
 at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
 at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
 at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)

{{The exception is caused by an integer overflow while calling getFilePointer() 
in Lucene50PostingsWriter, which eventually calls the size() method in 
ByteBuffersDataOutput.}}
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
public long size() {
 long size = 0;
 int blockCount = blocks.size();
 if (blockCount >= 1) {
 int fullBlockSize = (blockCount - 1) * blockSize();
 int lastBlockSize = blocks.getLast().position();
 size = fullBlockSize + lastBlockSize;
 }
 return size;
}

{code}
In my case, I had a blockCount = 65 and a blockSize() = 33554432 which 
overflows fullBlockSize. The fix:
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
public long size() {
long size = 0;
int blockCount = blocks.size();
if (blockCount >= 1) {
long fullBlockSize = 1L * (blockCount - 1) * blockSize();
int lastBlockSize = blocks.getLast().position();
size = fullBlockSize +

[jira] [Updated] (LUCENE-8624) ByteBuffersDataOutput Integer Overflow

2018-12-27 Thread Mulugeta Mammo (JIRA) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-8624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mulugeta Mammo updated LUCENE-8624:
---
Description: 
Hi,

When indexing large data sets with ByteBuffersDirectory, an exception like the 
below is thrown:

{{}}Caused by: java.lang.IllegalArgumentException: cannot write negative vLong 
(got: -4294888321)
 at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:182)
 at 
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
 at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
 at 
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
 at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
 at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
 at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
 at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)

{{The exception is caused by an integer overflow while calling getFilePointer() 
in Lucene50PostingsWriter, which eventually calls the size() method in 
ByteBuffersDataOutput.}}
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
 }}

public long size() {
 long size = 0;
 int blockCount = blocks.size();
 if (blockCount >= 1) {
{color:#FF}int fullBlockSize = (blockCount - 1) * blockSize();{color}
 int lastBlockSize = blocks.getLast().position();
 size = fullBlockSize + lastBlockSize;
 }
 return size;
}

{code}
In my case, I had a blockCount = 65 and a blockSize() = 33554432 which 
overflows fullBlockSize. The fix:
{code:java|title=ByteBuffersDataOutput.java|borderStyle=solid}
}}

public long size() {
long size = 0;
int blockCount = blocks.size();
if (blockCount >= 1) {
{color:#FF}long fullBlockSize = 1L * (blockCount - 1) * blockSize();{color}
int lastBlockSize = blocks.getLast().position();
size = fullBlockSize + lastBlockSize;
}
return size;
}

{code}
 

Thanks

  was:
Hi,

When indexing large data sets with ByteBuffersDirectory, an exception like the 
below is thrown:

{{}}Caused by: java.lang.IllegalArgumentException: cannot write negative vLong 
(got: -4294888321)
 at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:182)
 at 
org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)
 at 
org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)
 at 
org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)
 at 
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
 at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
 at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
 at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
 at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
 at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)
 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)

{{The exception is caused by an integer overflow while calling getFilePointer() 
in Lucene50PostingsWriter, which eventually calls the size() method in 
ByteBuffersDataOutput.}}

{{{code:title=ByteBuffersDataOutput.java|borderStyle=solid} }}

public long size() {
 long size = 0;
 int blockCount = blocks.size();
 if (blockCount >= 1) {
{color:#FF}int fullBlockSize = (blockCount - 1) * blockSize();{color}
 int lastBlockSize = blocks.getLast().position();
 size = fullBlockSize + lastBlockSize;
 }
 return size;
}

{code}

In my case, I had a blockCount = 65 and a blockSize() = 33554432 which 
overflows fullBlockSize. The fix:

{{{code:title=ByteBuffersDataOutput.java|borderStyle=solid}}}

public long size() {
long size = 0;
int blockCount = blocks.size();
if (blockCount >= 1) {
{color:#FF}long fullBlockSize = 1L * (blockCount - 1)