Well, depending on how many docs get served from the cache the time will vary. But this is just ugly, if you can avoid this use-case it would be a Good Thing.
Problem here is that each and every shard must assemble the list of 2,000 documents (just ID and sort criteria, usually score). Then the node serving the original request merges the sub-lists to pick the top 2,000. Then the node sends another request to each shard to get the full document. Then the node merges this into the full list to return to the user. Solr really isn't built for this use-case, is it actually a compelling situation? And having your document cache set at 1M is kinda high if you have very big documents. FWIW, Erick On Fri, Jun 28, 2013 at 8:44 PM, Utkarsh Sengar <utkarsh2...@gmail.com>wrote: > Also, I don't see a consistent response time from solr, I ran ab again and > I get this: > > ubuntu@ip-10-149-6-68:~$ ab -c 10 -n 500 " > > http://x.amazonaws.com:8983/solr/prodinfo/select?q=allText:huggies%20diapers%20size%201&rows=2000&wt=json > " > > > Benchmarking x.amazonaws.com (be patient) > Completed 100 requests > Completed 200 requests > Completed 300 requests > Completed 400 requests > Completed 500 requests > Finished 500 requests > > > Server Software: > Server Hostname: x.amazonaws.com > Server Port: 8983 > > Document Path: > > /solr/prodinfo/select?q=allText:huggies%20diapers%20size%201&rows=2000&wt=json > Document Length: 1538537 bytes > > Concurrency Level: 10 > Time taken for tests: 10.858 seconds > Complete requests: 500 > Failed requests: 8 > (Connect: 0, Receive: 0, Length: 8, Exceptions: 0) > Write errors: 0 > Total transferred: 769297992 bytes > HTML transferred: 769268492 bytes > Requests per second: 46.05 [#/sec] (mean) > Time per request: 217.167 [ms] (mean) > Time per request: 21.717 [ms] (mean, across all concurrent requests) > Transfer rate: 69187.90 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect: 0 0 0.3 0 2 > Processing: 110 215 72.0 190 497 > Waiting: 91 180 70.5 152 473 > Total: 112 216 72.0 191 497 > > Percentage of the requests served within a certain time (ms) > 50% 191 > 66% 225 > 75% 252 > 80% 272 > 90% 319 > 95% 364 > 98% 420 > 99% 453 > 100% 497 (longest request) > > > Sometimes it takes a lot of time, sometimes its pretty quick. > > Thanks, > -Utkarsh > > > On Fri, Jun 28, 2013 at 5:39 PM, Utkarsh Sengar <utkarsh2...@gmail.com > >wrote: > > > Hello, > > > > I have a usecase where I need to retrive top 2000 documents matching a > > query. > > What are the parameters (in query, solrconfig, schema) I shoud look at to > > improve this? > > > > I have 45M documents in 3node solrcloud 4.3.1 with 3 shards, with 30GB > > RAM, 8vCPU and 7GB JVM heap size. > > > > I have documentCache: > > <documentCache class="solr.LRUCache" size="1000000" > > initialSize="1000000" autowarmCount="0"/> > > > > allText is a copyField. > > > > This is the result I get: > > ubuntu@ip-10-149-6-68:~$ ab -c 10 -n 500 " > > > http://x.amazonaws.com:8983/solr/prodinfo/select?q=allText:huggies%20diapers%20size%201&rows=2000&wt=json > > " > > > > Benchmarking x.amazonaws.com (be patient) > > Completed 100 requests > > Completed 200 requests > > Completed 300 requests > > Completed 400 requests > > Completed 500 requests > > Finished 500 requests > > > > > > Server Software: > > Server Hostname: x.amazonaws.com > > Server Port: 8983 > > > > Document Path: > > > /solr/prodinfo/select?q=allText:huggies%20diapers%20size%201&rows=2000&wt=json > > Document Length: 1538537 bytes > > > > Concurrency Level: 10 > > Time taken for tests: 35.999 seconds > > Complete requests: 500 > > Failed requests: 21 > > (Connect: 0, Receive: 0, Length: 21, Exceptions: 0) > > Write errors: 0 > > Non-2xx responses: 2 > > Total transferred: 766221660 bytes > > HTML transferred: 766191806 bytes > > Requests per second: 13.89 [#/sec] (mean) > > Time per request: 719.981 [ms] (mean) > > Time per request: 71.998 [ms] (mean, across all concurrent > requests) > > Transfer rate: 20785.65 [Kbytes/sec] received > > > > Connection Times (ms) > > min mean[+/-sd] median max > > Connect: 0 0 0.6 0 8 > > Processing: 9 717 2339.6 199 12611 > > Waiting: 9 635 2233.6 164 12580 > > Total: 9 718 2339.6 199 12611 > > > > Percentage of the requests served within a certain time (ms) > > 50% 199 > > 66% 236 > > 75% 263 > > 80% 281 > > 90% 548 > > 95% 838 > > 98% 12475 > > 99% 12545 > > 100% 12611 (longest request) > > > > -- > > Thanks, > > -Utkarsh > > > > > > -- > Thanks, > -Utkarsh >
