You changed the positionIncrementGap to reduce index size....? Hm, maybe I'm forgetting something at the moment, but I don't think this will make any difference with index size, just with token offsets noted at index time. Are you referring to term index interval (normally 128 by default, I believe, but changeable, at least directly in Lucene)? That is one thing that would have some effect on the index size (but unfortunately also on search performance).
Depending on how big those stored fields are, yes, perhaps it would make sense to keep those elsewhere (e.g. BDB, RDBMS). This would make a difference for index optimization - fewer bytes would have to be read from and then written back to disk. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: F Knudson <[EMAIL PROTECTED]> > To: solr-user@lucene.apache.org > Sent: Friday, February 29, 2008 11:28:32 AM > Subject: RE: Optimization taking days/weeks > > > We will review the java settings. The current settings are a bit low - but > the indexed typically does not reach even 50% of the allocated 1024MB Max > Heap. > > Yes the index is large - only 3 fields are stored - and I have set the > positionIncrementGap to 50 (down from 100) in an attempt to reduce index > size. Would you suggest to build one index used only for searching and a > second index used only for display? Does that fit within your definition of > "partition"? > > Thanks > Frances > > > Alex Benjamen wrote: > > > > This sounds too familiar... > > > >>java settings used - java -Xmx1024M -Xms1024M > > Sounds like your settings are pretty low... if you're using 64bit JVM, you > > should be able to set > > these much higher, maybe give it like 8gb. > > > > Another thing, you may want to look at reducing the index size... is there > > any way you could > > partition the index? Also only index fields which you need and do not > > store the values in the index. > > I've originally had an index which was 50Gb in size, and after removing > > fields I do not need, I'm down > > to 8Gb and not storing any values in the index. > > > > > > > > > > -- > View this message in context: > http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15762156.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
