Are the indexes on local disks? Is the same index present on all 6 servers or split or different or what? Do you see the slowdown on all servers/indexes or what? Any IO/memory/CPU problems being reported anywhere? Are you always loading the same fields with approx the same volume of data or do you happen to have some docs with massive fields?
It doesn't really feel like a lucene issue to me. Feels more like an intermittent network problem. -- Ian. On Tue, May 14, 2013 at 9:19 PM, G B <g.b.co...@gmail.com> wrote: > Hi there, > We've been having troubles with performance regarding IndexReader's * > document<http://lucene.apache.org/core/4_1_0/core/org/apache/lucene/index/IndexReader.html#document(int)> > *(int docID) method. > > In summary: > Why would the > *document<http://lucene.apache.org/core/4_1_0/core/org/apache/lucene/index/IndexReader.html#document(int)> > *(int docID) take a few seconds? For some docIds it takes a millisecond and > for some it takes up to a few seconds whereas it used to consistenly take a > millisecond for each doc fetched. > > In depth: > We have a set of 6 app servers and each serve around a million requests per > day. > Specs for these app servers are: Ubuntu x86_64 GNU/Linux 8GB RAM, Java 6 > with an Xmx setting of 4 GB. Using Lucene 4.1. Current index size is 2.1 GB. > > We've started using Lucene more extensively in the last 6 months, our index > size used be ~ 1.5 GB and back then we had no problems. > Recently we acquired more data and the index size jumped to 2.1 GB. When we > pushed this new data to production first we had Out of Memory > org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:248) > exception. The recommendation was to increase the memory so we upped the > Xmx setting 4GB but then we had the > https://issues.apache.org/jira/browse/LUCENE-1566 (Lucene bug caused by the > JVM). Increasing the memory of the servers to 8GB seems to resolve this > problem. Now, the system is stable and no more of those exceptions but the > response times of Lucene searches increased by 50%. Debugging the issue, I > realized that IndexReader's doc is taking an insane amount of time. Used to > be a ms each but now it can take a few seconds to fetch a single doc. Is > readChunkSize setting of FSDirectory have anything to do with it? (now that > our index size is larger than the default chunk size?) --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
