On Jul 8, 2010, at 10:43 AM, Norman Barker wrote: > Hi, > > I have been thinking about how to query multiple views at one time. > > I have an erlang handler in couchdb that takes a http post containing > N view queries, each query contains a startkey and an endkey, I then > open up each view in parallel (using pmap) and accumulate the doc ids, > then I use the erlang sets module to get the unique values. All good > and looks pretty (and works), though it doesn't scale since I am > holding all the results on the server (potential memory overload!) > whereas I would like to stream the results to the client one by one. > > I am thinking of doing the following but have some questions; > > My first question is when I do > > couch_view:fold(View, FoldlFun, FoldAccInit, > couch_httpd_view:make_key_options(Args)), > > is there a way to call the _count reduce function in code to find the > number of rows in the slice between startkey and endkey? > > If so, I would like to order all the views in the posted query > document by the result of _count from smallest to largest. > > I would then fold over the smallest result view and pull each document > id (*) in turn. > > With each document id I would then call each of the other views in > turn with their startkey and endkey and in addition include > startkey_docid and endkey_docid with the docid in * above, again > calling _count I can test for inclusion. If the doc id is in all views > then I will immediate stream this to the client. > > Am I doing something stupid, is this optimal? >
It sounds like you are on the right track. this could be a very valuable patch to CouchDB once you have it working. > Any help with the programmatic _count call would be great. > One hint: maybe the call to reduce_to_count will help. Here's an implementation of a reduce query in Erlang. http://github.com/jchris/hovercraft/blob/master/hovercraft.erl#L217 Sorry I can't be more helpful. I've successfully bootstrapped this stuff in my head before, but it always takes a couple of hours of turning my brain into a step debugger. Good luck! Once you get deeper into the code you might have better luck getting responses on the dev@ list or maybe the #couchdb IRC channel on freenode. Chris > thanks, > > Norman
