On Wed, Jun 24, 2009 at 3:38 PM, Nigel<nigelspl...@gmail.com> wrote: > Yes, we're indexing on a separate server, and rsyncing from index snapshots > there to the search servers. Usually rsync has to copy just a few small > .cfs files, but every once in a while merging will product a big one. I'm > going to try to limit this by setting maxMergeMB, but of course that's a > trade-off with having more segments.
OK. > It sounds like surely any swapping out of the JVM memory could cause big and > unpredictable performance drops. As I just mentioned in reply to Uwe, our > poor performance times don't always directly correlate with index updates, > but it may be that the damage is done and the effects are only seen sometime > later. OK... I wonder whether the bytes written by rsync go into the IO cache. I would assume they do. But in your case that might be OK since presumably you'll then cutover to those new files, so, they've been pre-warmed by rsync. > We don't store norms (since we don't care about sort order), and we don't > have any deleted docs (since the index is read-only on the search servers). Ahh excellent. > What exactly is stored in the field cache? FieldCache is used when sorting by field (not relevance) and by function queries, or, if you directly load values, eg FieldCache.DEFAULT.getInts(...). > p.s. I haven't tried SSDs yet, or for that matter faster disks of any sort. > First I'd like to get a better understanding of what I/O is required and > when during the search process, ideally to be able to have an approximate > model that predicts I/O based on the query (the way a DBA might do when > estimating how a SQL query would work with certain tables and indexes). Sounds like a good plan!! And, after that, upgrade ;) There's no going back once you make the switch... Mike --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
