Joan, Thanks for the advice. In some initial tests across only 3 design docs I'm seeing a noticeable speed increase. Granted with the workload I'm talking about we're dealing with times measured in fractions of milliseconds, but it's enough of an improvement to know it's a proper way to take advantage of the concurrency / parallelism on the server side as well.
Thank you, Cameron On 08/06/2013 04:29 PM, Joan Touzet wrote: > Hi Cameron, > > When you split up your views, did you split them into separate view > groups (different design docs) or keep them in the same view group > (ddoc)? If not, try to do so. This should lead to additional couchjs > processes being spawned for each view group, which (if your server can > support it) will allow increased parallelism. > > That said, given your data set, I would be inclined to implicate your > libevent code. If you can attempt to reproduce the situation using > something "lower tech" such as multiple shell scripts + curl, this might > be more useful information. > > On Tue, Aug 06, 2013 at 11:50:24AM -0400, Cameron Jacobson wrote: >> Hi, >> >> I apologize if this has already reached a list. I tried sending >> previously, but based on the online archives it doesn't appear that it >> was received by the list. >> >> >> Original Message: >> ----------------------------------- >> >> I've got a question regarding how CouchDB handles concurrency in views. >> >> Background: I'm writing a CouchDB client where I'm trying to leverage >> libevent so the client can handle requests concurrently. >> >> When I started to benchmark the "async" functionality I noticed that no >> matter what I did the time-spent in the closure where all the buffers >> were being retrieved was growing linearly. My expectation, given that >> this is concurrent accesses, would be that time would at worst grow >> logarithmically (at least while the server had enough resources to >> handle the requests concurrently). No noticeable issues in other >> critical areas, such as I/O traffic thresholds, CPU or memory bottlenecks. >> >> I made several adjustments in the default.ini to ensure there were >> enough resources allocated to the server. I tried the "stale" >> parameter, and "warmed" the views with pre-test queries. Thinking it >> was possible the view handling was the bottleneck I broke out my queries >> against views1-10, all doing exactly the same thing. Nothing seemed to >> help. This got me thinking that I may be implementing the libevent code >> incorrectly. I realized this wasn't the issue once I ran the code >> against a remote environment (non-couchdb) and was getting no noticeable >> increase in time over as many as 100 concurrent requests (tests on >> couchdb until now have not exceeded 50 concurrent requests). >> >> I don't have the background in CouchDB to know that it's the bottleneck, >> or how I would go about proving this, but based on all the tests so far >> I think I've narrowed in on CouchDB as the cause. I'm running version >> 1.3.0. >> >> I came across the following link, which may explain part of the problem, >> but I don't think it explains why multiple concurrent queries across >> multiple views showed the same performance characteristics as multiple >> concurrent queries against a single view. Also the comments in the >> following thread are talking about large documents / views. My tests >> were against 10 "cloned" views which all only contained 2 documents, >> each of which was less than 1kb in size: >> http://comments.gmane.org/gmane.comp.db.couchdb.user/22200 >> >> Any insights into how I can tweak my configuration or what measures I >> can take to improve this would be greatly appreciated. >> >> Thanks
