On Jul 11, 2010, at 7:27 AM, Norman Barker wrote: > Afshin > > I have got the all clear from my work to release this as a patch, I > expect to be putting something up on github by the end of the week > (internal paperwork permitting). I am going to implement it as an > external handler so it can be used and reviewed and from there it will > be under a do what you want with it license so it can go into couchdb > if accepted. >
If you're implementing in Erlang, you probably don't need to go full hog as an external. Because the configuration system is so modular, you are probably best adding it as a new httpd_design_handler, or httpd_db_handler. It should be easy to create a new module and link it in via the configuration file. I don't know if you plan to allow querying across databases (I'd suggest restricting the queries to a single database if you want to stay within CouchDB's security model, and have it more likely to be accepted as a patch.) We should really move this discussion to dev@ -- a lot of the developers only give a cursory glance at the user list, so you will get more valuable feedback there. Thanks for taking the time to write and release the patch! Chris > Chris, thanks for the help with the reduce function and confirming the > concept! > > Norman > > On Sun, Jul 11, 2010 at 8:02 AM, afshin afzali <[email protected]> wrote: >> Hi Norman, Chris >> >> I just wanted to say this is the same problem we are currently facing >> with. We are implementing a Local Business Directory application on >> couchdb. Our searches need to combine several keys together to find >> right entries. To do something like that in server side, we had the >> paging mechanism problem, so we have chosen that do Norman's algorithm >> in client side! I'll appreciate if there will be a successful progress >> in this issue. >> >> BEST, >> -- afshin >> >> On 7/8/10, J Chris Anderson <[email protected]> wrote: >>> >>> On Jul 8, 2010, at 10:43 AM, Norman Barker wrote: >>> >>>> Hi, >>>> >>>> I have been thinking about how to query multiple views at one time. >>>> >>>> I have an erlang handler in couchdb that takes a http post containing >>>> N view queries, each query contains a startkey and an endkey, I then >>>> open up each view in parallel (using pmap) and accumulate the doc ids, >>>> then I use the erlang sets module to get the unique values. All good >>>> and looks pretty (and works), though it doesn't scale since I am >>>> holding all the results on the server (potential memory overload!) >>>> whereas I would like to stream the results to the client one by one. >>>> >>>> I am thinking of doing the following but have some questions; >>>> >>>> My first question is when I do >>>> >>>> couch_view:fold(View, FoldlFun, FoldAccInit, >>>> couch_httpd_view:make_key_options(Args)), >>>> >>>> is there a way to call the _count reduce function in code to find the >>>> number of rows in the slice between startkey and endkey? >>>> >>>> If so, I would like to order all the views in the posted query >>>> document by the result of _count from smallest to largest. >>>> >>>> I would then fold over the smallest result view and pull each document >>>> id (*) in turn. >>>> >>>> With each document id I would then call each of the other views in >>>> turn with their startkey and endkey and in addition include >>>> startkey_docid and endkey_docid with the docid in * above, again >>>> calling _count I can test for inclusion. If the doc id is in all views >>>> then I will immediate stream this to the client. >>>> >>>> Am I doing something stupid, is this optimal? >>>> >>> >>> It sounds like you are on the right track. this could be a very valuable >>> patch to CouchDB once you have it working. >>> >>>> Any help with the programmatic _count call would be great. >>>> >>> >>> One hint: maybe the call to reduce_to_count will help. >>> >>> Here's an implementation of a reduce query in Erlang. >>> >>> http://github.com/jchris/hovercraft/blob/master/hovercraft.erl#L217 >>> >>> Sorry I can't be more helpful. I've successfully bootstrapped this stuff in >>> my head before, but it always takes a couple of hours of turning my brain >>> into a step debugger. >>> >>> Good luck! >>> >>> Once you get deeper into the code you might have better luck getting >>> responses on the dev@ list or maybe the #couchdb IRC channel on freenode. >>> >>> Chris >>> >>> >>>> thanks, >>>> >>>> Norman >>> >>> >>
