[
https://issues.apache.org/jira/browse/LUCENE-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498754#comment-13498754
]
Adrien Grand commented on LUCENE-4547:
--------------------------------------
About the current SimpleDVConsumer API: {{NumericDocValuesConsumer
addNumericField(FieldInfo field, long minValue, long maxValue)}} only allows
for compression based on the width of the range but there are many other ways
to compress values (maybe there are very few unique values, maybe the last 3
bits are always the same, etc...). Now that we buffer values into memory, what
would you think of changing the API to pass an iterable of longs instead so
that the doc values format can make better decisions?
> DocValues field broken on large indexes
> ---------------------------------------
>
> Key: LUCENE-4547
> URL: https://issues.apache.org/jira/browse/LUCENE-4547
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Robert Muir
> Priority: Blocker
> Fix For: 4.1
>
> Attachments: test.patch
>
>
> I tried to write a test to sanity check LUCENE-4536 (first running against
> svn revision 1406416, before the change).
> But i found docvalues is already broken here for large indexes that have a
> PackedLongDocValues field:
> {code}
> final int numDocs = 500000000;
> for (int i = 0; i < numDocs; ++i) {
> if (i == 0) {
> field.setLongValue(0L); // force > 32bit deltas
> } else {
> field.setLongValue(1<<33L);
> }
> w.addDocument(doc);
> }
> w.forceMerge(1);
> w.close();
> dir.close(); // checkindex
> {code}
> {noformat}
> [junit4:junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene
> Merge Thread #0,6,TGRP-Test2GBDocValues]
> [junit4:junit4] 2> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.ArrayIndexOutOfBoundsException: -65536
> [junit4:junit4] 2> at
> __randomizedtesting.SeedInfo.seed([5DC54DB14FA5979]:0)
> [junit4:junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:535)
> [junit4:junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:508)
> [junit4:junit4] 2> Caused by: java.lang.ArrayIndexOutOfBoundsException:
> -65536
> [junit4:junit4] 2> at
> org.apache.lucene.util.ByteBlockPool.deref(ByteBlockPool.java:305)
> [junit4:junit4] 2> at
> org.apache.lucene.codecs.lucene40.values.FixedStraightBytesImpl$FixedBytesWriterBase.set(FixedStraightBytesImpl.java:115)
> [junit4:junit4] 2> at
> org.apache.lucene.codecs.lucene40.values.PackedIntValues$PackedIntsWriter.writePackedInts(PackedIntValues.java:109)
> [junit4:junit4] 2> at
> org.apache.lucene.codecs.lucene40.values.PackedIntValues$PackedIntsWriter.finish(PackedIntValues.java:80)
> [junit4:junit4] 2> at
> org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:130)
> [junit4:junit4] 2> at
> org.apache.lucene.codecs.PerDocConsumer.merge(PerDocConsumer.java:65)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
