[
https://issues.apache.org/jira/browse/LUCENE-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14213752#comment-14213752
]
Shai Erera commented on LUCENE-6062:
------------------------------------
To prove this is the problem, I added this to PerFieldDVF.FieldsReader ctor
(line 256):
{code}
if (readState.segmentInfo.name.equals("_1") &&
fieldName.equals("ndv") && fi.getDocValuesGen() == 1 &&
readState.fieldInfos.size() > 1) {
continue;
}
{code}
And the test passes. So e.g. if we tracked startDVGen for each field, we'd know
in segment _1 that 'ndv' only appeared in gen=1, and therefore not pass it at
all to baseProducer. But that causes a format change, which I hope to avoid if
possible (especially as it doesn't solve the issue for existing indexes, though
I think this is an extreme case for somebody to have run into).
> Index corruption from numeric DV updates
> ----------------------------------------
>
> Key: LUCENE-6062
> URL: https://issues.apache.org/jira/browse/LUCENE-6062
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Michael McCandless
> Fix For: 4.10.3, 5.0, Trunk
>
>
> I hit this while working on on LUCENE-6005: when cutting over
> TestNumericDocValuesUpdates to the new Document2 API, I accidentally enabled
> additional docValues in the test, and this this:
> {noformat}
> There was 1 failure:
> 1)
> testUpdateSegmentWithNoDocValues(org.apache.lucene.index.TestNumericDocValuesUpdates)
> java.io.FileNotFoundException: _1_Asserting_0.dvm in
> dir=RAMDirectory@259847e5
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@30981eab
> at __randomizedtesting.SeedInfo.seed([0:7C88A439A551C47D]:0)
> at
> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:645)
> at
> org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
> at
> org.apache.lucene.codecs.lucene50.Lucene50DocValuesProducer.<init>(Lucene50DocValuesProducer.java:130)
> at
> org.apache.lucene.codecs.lucene50.Lucene50DocValuesFormat.fieldsProducer(Lucene50DocValuesFormat.java:182)
> at
> org.apache.lucene.codecs.asserting.AssertingDocValuesFormat.fieldsProducer(AssertingDocValuesFormat.java:66)
> at
> org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:267)
> at
> org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:357)
> at
> org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
> at
> org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:68)
> at
> org.apache.lucene.index.SegmentDocValuesProducer.<init>(SegmentDocValuesProducer.java:63)
> at
> org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:167)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:109)
> at
> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
> at
> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:556)
> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:50)
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
> at
> org.apache.lucene.index.TestNumericDocValuesUpdates.testUpdateSegmentWithNoDocValues(TestNumericDocValuesUpdates.java:769)
> {noformat}
> A one-line change to the existing test (on trunk) causes this corruption:
> {noformat}
> Index:
> lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java
> ===================================================================
> ---
> lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java
> (revision 1639580)
> +++
> lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java
> (working copy)
> @@ -750,6 +750,7 @@
> // second segment with no NDV
> doc = new Document();
> doc.add(new StringField("id", "doc1", Store.NO));
> + doc.add(new NumericDocValuesField("foo", 3));
> writer.addDocument(doc);
> doc = new Document();
> doc.add(new StringField("id", "doc2", Store.NO)); // document that isn't
> updated
> {noformat}
> For some reason, the base doc values for the 2nd segment is not being
> written, but clearly should have (to hold field "foo")... I'm not sure why.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
