As expected it did not make much difference. Here is the code:
FieldsReader.java
final int docs(byte[] buffer,int[] lengths,int start,int max)
throws IOException {
final IndexInput indexStream = (IndexInput) indexLocal.get();
indexStream.seek(start * 8L);
long startoffset = indexStream.readLong();
long lastoffset = startoffset;
int count=0;
while(count<max) {
long offset = indexStream.readLong();
if(offset-startoffset>buffer.length)
break;
lengths[count]=(int) (offset-lastoffset);
lastoffset = offset;
count++;
}
if(count==0)
throw new IllegalStateException("buffer too small for a single
document");
final IndexInput fieldsStream = (IndexInput) fieldsLocal.get();
fieldsStream.seek(startoffset);
fieldsStream.readBytes(buffer,0,(int) (lastoffset-startoffset));
return count;
}
FieldsWriter.java
final void addDocuments(byte[] buffer, int[] lengths, int ndocs)
throws IOException {
long position = fieldsStream.getFilePointer();
long start = position;
for(int i=0;i<ndocs;i++) {
indexStream.writeLong(position);
position+=lengths[i];
}
fieldsStream.writeBytes(buffer,(int) (position-start));
}
SegmentReader.java
public int documents(byte[] buffer,int[] lengths,int start,int
max) throws IOException {
return fieldsReader.docs(buffer,lengths,start,max);
}
SegmentMerger.java
private final int mergeFields() throws IOException {
fieldInfos = new FieldInfos(); // merge field names
int docCount = 0;
for (int i = 0; i < readers.size(); i++) {
IndexReader reader = (IndexReader) readers.elementAt(i);
if (reader instanceof SegmentReader) {
SegmentReader sreader = (SegmentReader) reader;
for (int j = 0; j < sreader.getFieldInfos().size(); j++) {
FieldInfo fi = sreader.getFieldInfos().fieldInfo(j);
fieldInfos.add(fi.name, fi.isIndexed, fi.storeTermVector,
fi.storePositionWithTermVector, fi.storeOffsetWithTermVector, !
reader.hasNorms(fi.name));
}
} else {
addIndexed(reader, fieldInfos, reader.getFieldNames
(IndexReader.FieldOption.TERMVECTOR_WITH_POSITION_OFFSET), true,
true, true);
addIndexed(reader, fieldInfos, reader.getFieldNames
(IndexReader.FieldOption.TERMVECTOR_WITH_POSITION), true, true, false);
addIndexed(reader, fieldInfos, reader.getFieldNames
(IndexReader.FieldOption.TERMVECTOR_WITH_OFFSET), true, false, true);
addIndexed(reader, fieldInfos, reader.getFieldNames
(IndexReader.FieldOption.TERMVECTOR), true, false, false);
addIndexed(reader, fieldInfos, reader.getFieldNames
(IndexReader.FieldOption.INDEXED), false, false, false);
fieldInfos.add(reader.getFieldNames
(IndexReader.FieldOption.UNINDEXED), false);
}
}
fieldInfos.write(directory, segment + ".fnm");
SegmentReader[] sreaders = new SegmentReader[readers.size()];
for (int i = 0; i < readers.size(); i++) {
IndexReader reader = (IndexReader) readers.elementAt(i);
boolean same = reader.getFieldNames().size() == fieldInfos.size
() && reader instanceof SegmentReader;
if(same) {
SegmentReader sreader = (SegmentReader) reader;
for (int j = 0; same && j < fieldInfos.size(); j++) {
same = fieldInfos.fieldName(j).equals(sreader.getFieldInfos
().fieldName(j));
}
if(same)
sreaders[i] = sreader;
}
}
byte[] buffer = new byte[512*1024];
int[] lengths = new int[10000];
// merge field values
FieldsWriter fieldsWriter = new FieldsWriter(directory, segment,
fieldInfos);
try {
for (int i = 0; i < readers.size(); i++) {
IndexReader reader = (IndexReader) readers.elementAt(i);
SegmentReader sreader = sreaders[i];
int maxDoc = reader.maxDoc();
for (int j = 0; j < maxDoc;)
if (!reader.isDeleted(j)) { // skip deleted docs
if (sreader!=null) {
int start=j;
int ndocs=1;
for(j++;j<maxDoc && !reader.isDeleted(j) && ndocs<10000;j+
+,ndocs++);
ndocs = sreader.documents(buffer,lengths,start,j);
fieldsWriter.addDocuments(buffer,lengths,ndocs);
} else {
fieldsWriter.addDocument(reader.document(j));
j++;
}
docCount++;
}
}
} finally {
fieldsWriter.close();
}
return docCount;
}
On Nov 2, 2007, at 3:49 PM, robert engels wrote:
I am working on a bulk document copy right now - I will let you
know if it improves things much.
I doubt it, because I already configure the streams to use fairly
large input and output buffers during a merge - but the memory
index merge may see additional benefits due to lower CPU calls.
On Nov 2, 2007, at 3:39 PM, robert engels (JIRA) wrote:
[ https://issues.apache.org/jira/browse/LUCENE-1043?
page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
tabpanel#action_12539696 ]
robert engels commented on LUCENE-1043:
---------------------------------------
When bulk copying the documents, I think you need to:
read array of long from index (8 * (ndocs+1)) in long[ndocs+1]
offsets;
calculate length = offset[ndocs]-offset[0];
read bytes of length from document file
startoffset = current output document stream position
write bytes to output document
modify offset[] adding startoffset-offset[0] to each entry
write offset[] in bulk to index output
Speedup merging of stored fields when field mapping "matches"
-------------------------------------------------------------
Key: LUCENE-1043
URL: https://issues.apache.org/jira/browse/
LUCENE-1043
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Affects Versions: 2.2
Reporter: Michael McCandless
Assignee: Michael McCandless
Priority: Minor
Fix For: 2.3
Attachments: LUCENE-1043.patch
Robert Engels suggested the following idea, here:
http://www.gossamer-threads.com/lists/lucene/java-dev/54217
When merging in the stored fields from a segment, if the field
name ->
number mapping is identical then we can simply bulk copy the entire
entry for the document rather than re-interpreting and then re-
writing
the actual stored fields.
I've pulled the code from the above thread and got it working on the
current trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
