On Wed, 2008-08-20 at 00:25 +0800, Cedric Ho wrote: > Search response time. We used the search log from our production > system and test it with SSD. The results shows that 75% of queries > returns within 1 second, 90% returns in 2.5 seconds, the remaining 10% > ranges from 2.5 seconds to less than 100 seconds.
Are the sub-second response times fairly equal or fluctuating? Do you have the figures for conventional harddisks? > Total number of queries is ~40000, so about 10000 queries are kind of > slow, 1000 queries are very slow. But those 10% very slow queries are > not from the first 1000 queries. It's more or less evenly distributed. Is it the same queries that are slow each time? > It's a single OCZ 64G SSD. We just got it yesterday. Is there a big > difference between different SSDs? Depends on who you ask... There's a big difference for bulk reads and writes, but that's not the main issue for searches. While there's also a difference in IO-operations/second, the count is normally so high, compared to discs, that just using a random SSD (discounting slap-4-CF-cards-together hacks) should give you a clear improvement. The OCZ-drives are getting fine reviews and I would definitely count them as serious SSDs. Have you checked to see if your controller can keep up with the drive? Running a standard harddrive-benchmark with IO-operations/second measurements might be an idea. > All queries involves a Date Range Filter and a Publication Filter. > We've used WrappingCachingFilters for the Publication Filter for there > are only a limited number of combinations for this filter. For the > Date Range Filter we just let it run every time which seems to be > doing fine. > > The queries also range from simple term query to phraseQueries to > nested spanQueries. Number of search terms > 10 is not uncommon. Our queries normally has fewer terms but gets expanded to 4-5 fields. We have no filters and very few ranges. Have you tried monitoring the memory-usage of the JVM? At one time we had a lot of short-term allocation of fairly large structures - this triggered a lot of full GC's and killed performance. This still doesn't account for all the > 1 second response times though. > There are 3 returned fields, docId, date and publication, all of which > we retrieve through fieldCaches. We retrieve only a single field for 20 hits, using no caching, but it's ½-1KB in size. Upping the number of retrieved fields doesn't affect performance in any significant way for us. > And we use this method to do the search: > TopFieldDocs Searcher.search(Query query, Filter filter, int n, Sort sort) > where for the test run n=100 I'll see if I can steal time enough to make a test with sorting, but no promises. > We are targeting to get >90% of queries to return under 1 sec. Of > course the more the better =) That's about the same as our original goal, but we've gotten greedier in the meantime. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
