Before 2.3, each doc was in fact a separate segment in memory, and
then these segments were merged together to flush a single segment in
the Directory.
As of 2.3, IndexWriter now writes directly into RAM the data
structures that are needed to create the segment, and then flushing
the segment is a matter of copying these data structures into the
Directory. This gave a substantial speedup to indexing throughput,
much better RAM efficiency (documents per MB that IndexWriter can
buffer), etc.
In any event, for all versions of Lucene, when flush happens that
flush adds a single new segment to the index.
Mike
David Lee wrote:
ok, thanks. I knew that the documents were buffered in memory until
they
were flushed, but I thought that in memory, they were still separate
documents/segments until they were merged together at the
appropriate time
(dependent on the mergeFactor).
Do you mean that when the IndexWriter flushes the documents in
memory to the
disk, it will merge all the documents in that flush to one segment?
Thanks!
David
On Sat, Aug 23, 2008 at 2:40 AM, Karsten F.
<[EMAIL PROTECTED]>wrote:
Hi David,
this is not true, please take a look to
IndexWriter#setRAMBufferSizeMB
and
IndexWriter#setMaxBufferedDocs
But you can produce 9 segments (each with only one document), if
you call
IndexWriter#flush
or
IndexWriter#commit
after each addDocument
so from my knowledge about lucene there is no difference between
#flush
and
#optimize(getMergeFactor())
(btw #optimize() is equal to optimize(1) ).
Best regards
Karsten
p.s. and yes, searching goes through every segment.
David Lee-26 wrote:
So from what I understand, is it true that if mergeFactor is 10,
then
when
I
index my first 9 documents, I have 9 separate segments, each
containing 1
document? And when searching, it will search through every segment?
Thanks!
David
--
View this message in context:
http://www.nabble.com/Clarification-about-segments-tp19117115p19120086.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
