When you ask for "a large response (~1mb of data)", you are asking for Solr to do tons of disk accesses and sorting before it sends the first response. That is going to be slow.
I strongly recommend requesting smaller results. One of those requests may be using most of the caching resources in Solr, so that the next response is slow, too. How many rows are you requesting and why? wunder Walter Underwood On Nov 21, 2011, at 9:46 PM, Shawn Heisey wrote: > On 11/21/2011 10:19 PM, Stephen Powis wrote: >> Thanks for the reply Shawn. >> >> The solr server currently has 8gb of ram and the total size of the dataDir >> is around 30gb. I start solr and give the java heap up to 4gb of ram, so >> that leaves 4gb for the OS, there are no other running services on the >> box. So from what you are saying, we are way under on the amount of ram we >> would ideally have. >> >> Just trying to get a better understanding of this.....Wouldn't the indexes >> not being in the disk cache make the queries themselves slow as well (high >> qTime), not just fetching the results? > > Having 4GB for disk cache isn't much compared to 30GB, but it's probably > enough to get most of the important bits cached. On the other hand, unless > your query is really complex, 1 second is a pretty slow response time. > (Yonik said the same thing, only better.) > >> We currently store all the fields that we index, my reasoning behind that >> is that debugging results we get from solr w/o being able to see what is >> stored in solr would be near impossible (in my head anyhow..). Generally >> our original source (mysql) and solr are consistent, but we've had cases >> where some updates have been missed for one reason or another. >> >> So my options are: reduce index sizes, increase ram on the server, increase >> disk speed (SSD drives)? > > You're right about debugging being harder if you don't see all the data in > the result. It's something I have to deal with in my index. Each of my > index shards is already 20GB in size, it would be easily triple that if I > stored everything. > > One thing you can do to help with debugging is include faceting on the field > that you are trying to debug. You won't see the original field values, but > you will see what your index analyzer chain has done with it, which sometimes > is even more useful. > > SSD is an awesome option, but it becomes absolutely critical that you have > redundant servers if you go that route. This is because there are few (maybe > none) RAID1, RAID5, or RAID10 solutions that support TRIM, which is > absolutely required for good SSD performance. With RAID0 or no RAID, a > single SSD failure takes the server out. Things can get very expensive very > quickly with SSD. > > With your current index size, if you can bump the server RAM to 32GB, your > performance will be very good. If you can go higher, it would be stellar. > You'll want to be running a 64-bit OS and 64-bit JVM. > > Thanks, > Shawn >
