Re: performance implications for an index with large number of documents.

Re: performance implications for an index with large number of documents. hi, Thank you for all the quick and pertinent responses. The index is being optimized every hour due to the number of updates. The JVM has a heap of 2gig and the machine has a total of 4. Currently the JVM is not configu

Re: performance implications for an index with large number of documents.

hi, Thank you for all the quick and pertinent responses. The index is being optimized every hour due to the number of updates. The JVM has a heap of 2gig and the machine has a total of 4. Currently the JVM is not configured with -server parameter and the parallel garbage collection (we are test

Re: performance implications for an index with large number of documents.

Hi Ori, Before taking drastic rehosting measures, and introducing the associated software complexity off splitting your application into pieces running on separate machines, I'd recommend looking at the way your document data is distributed and the way you're searching them. Here are some qu

Re: performance implications for an index with large number of documents.

How much RAM do you have? If you're under linux, can you run something like "iostat -x -d -t 60" and watch your disk usage during searching? If your disk utilization is high, add more RAM (enough to hold your index in RAM) and see if the OS cache solves the problem. I would try this before the c

Re: performance implications for an index with large number of documents.

: : The index is made up of approximately 10 million documents that have : 11 fields. The average document size is less then 1k. The index has : a total of 13 million terms. The total index size is about 2.2 gig. : The index is being updated relatively aggressively. In a 24hr period : there may

Re: performance implications for an index with large number of documents.

Lucene scales with the number of unique terms in the index and not the number of documents nor the size of the documents. Typically, you should have at most 1 million unique terms for a set of 10 million documents. So the fact that you have 13 million unique terms in 10 million documents tells me

performance implications for an index with large number of documents.

Hi, Apologies if this question has being asked before on this list. I am working on an application with a Lucene index whose performance (response time for a query) has started degrading as its size has increase. The index is made up of approximately 10 million documents that have 11 fields. Th