What do you mean by "does your test do any merging"? All I do is create IndexWriter w/ the RAM and MBD settings as I've described before. Then I just call addDocument. At the end I call optimize() (it is a one time created index, after that I need it optimized for search). I guess Lucene performs segment merging in between, although I think I noticed that before optimize is called, there were 10 segments (CFS files) of size 200MB each, which matches the default merge factor setting to 10 (if I'm not mistaken). Does it mean that no segments were merged during the indexing process?
Shai On 3/21/08, Michael McCandless <[EMAIL PROTECTED]> wrote: > > > Shai Erera wrote: > > Besides the content field, everything is stored, so that may > > explain the > > large CFS files. > > You could run w/o CFS turned on and then look at the size of fdt/fdx > to see if this explains the size. > > > Regarding the RAM-usage performance, I tried setting to 128, 256 > > and 512, > > all gave the same time measurements (give or take ~5%) as the MBD > > (set to > > 10,000) run. I think it needs further investigation. Was it tested > > before? I > > mean - has someone tried to set RAM to 2GB for example and noticed > > a major > > performance improvement (as I'd expect)? > > Yes, in LUCENE-843 Doron did some tests and found similar tail-off, I > think above 32 or 64 MB RAM buffer. I also saw the same thing in my > tests. > > However one thing to remember is: a larger RAM buffer will result in > less merging. But, with CMS and a machine with good enough > concurrency, net run time may be unaffected by this benefit even > though net resource consumption is. This makes it hard to test > speedups in merging :) Does your test do any merging at all? > > Mike > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- Regards, Shai Erera
