Revision: 16582
http://sourceforge.net/p/gate/code/16582
Author: valyt
Date: 2013-03-12 12:58:32 +0000 (Tue, 12 Mar 2013)
Log Message:
-----------
Fixed serious bug: when constructing the direct index, all term IDs originating
in the second or later batches were wrong.
Modified Paths:
--------------
mimir/trunk/mimir-core/src/gate/mimir/index/mg4j/MimirDirectIndexBuilder.java
Modified:
mimir/trunk/mimir-core/src/gate/mimir/index/mg4j/MimirDirectIndexBuilder.java
===================================================================
---
mimir/trunk/mimir-core/src/gate/mimir/index/mg4j/MimirDirectIndexBuilder.java
2013-03-12 12:56:41 UTC (rev 16581)
+++
mimir/trunk/mimir-core/src/gate/mimir/index/mg4j/MimirDirectIndexBuilder.java
2013-03-12 12:58:32 UTC (rev 16582)
@@ -125,17 +125,11 @@
// we are iterating over the input terms (output 'documents')
NumberFormat percentNF = NumberFormat.getPercentInstance();
IndexIterator inputTermIterator = inputIndexReader.nextIterator();
+ // document pointer is actually the current input term ID
+ documentPointer = 0;
long termsProcessed = 0;
while(inputTermIterator != null) {
try { // Start: process output document
- // the current input term ID is an output document ID.
- // BUG? This value seems to be wrong on large indexes. Maybe we
should
- // just count them instead?
- long outputDocId = inputTermIterator.termNumber();
- if(outputDocId != termsProcessed) {
- logger.warn("Mismatch between term ID from index " + outputDocId +
- ", counted ID: " + termsProcessed + ".");
- }
//zero document related counters
tokenPosition = -1;
// for each input term, we iterate over its documents
@@ -153,7 +147,7 @@
termMap.put(currentTerm.copy(), termPostings = new PostingsList(
new byte[ 32 ], true, Completeness.COUNTS));
}
- termPostings.setDocumentPointer(outputDocId);
+ termPostings.setDocumentPointer(documentPointer);
termPostings.setCount(count);
occurrencesInTheCurrentBatch += count;
if(termPostings.outOfMemoryError) {
@@ -162,7 +156,7 @@
}
// and move to the next output term (input document)
outputTermId = inputTermIterator.nextDocument();
- }
+ }
} finally {
//write the size of the current document to the sizes stream
try {
This was sent by the SourceForge.net collaborative development platform, the
world's largest Open Source development site.
------------------------------------------------------------------------------
Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester
Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the
endpoint security space. For insight on selecting the right partner to
tackle endpoint security challenges, access the full report.
http://p.sf.net/sfu/symantec-dev2dev
_______________________________________________
GATE-cvs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/gate-cvs
