Just want to ask if anyone else has noticed that the index and segments under the searcher dir are causing a hot spot on the hard drive in a heavy transaction use search.
I am on windows, Nutch 7.1, tomcat 5.15, and have tuned the system for some decent performance, Modified both tomcat and Nutch configuration files to increase threads, memory, and limit returns to 1 per site, but on the hardware side I am seeing a hot spot on the drive when making a very large query (returns more than 4000 results). Has anyone else noticed this behavior when dealing with an index that is greater than 750 megs, and 48 gigs worth of search data? If you have noticed this, did you throw hardware at the problem or is there a way in software to fix the issue? Would the nutch dfs work, or should I use the os level dfs? Interested in hearing what others have seen, or if anyone else has even seen this issue. r/d
