Hi Norman, As Chris said, it's a valuable patch.
Cheers, -- afshin On Fri, Jul 23, 2010 at 3:57 AM, Norman Barker <[email protected]>wrote: > ok, I have this working in practise, though there are still many > improvements that could be made. Sorry for the delay, 1.0 > experimenting took some of my time last week (yay for 1.0!!). > > Since couchdb now has an eccn number and ssl components this will take > me a little longer to release this code, but it will go out for > comment shortly. > > I did sort the views from smallest to largest (by seeing the number of > hits in each view) and the I used the process ring example where you > send a message in a ring, each node in the ring represents a view and > if there is an intersection it sends a message to the next and if it > gets back to the start then it is sent as a chunk to the user. If the > match fails then the ring stops (actually it sends an empty message > back to the main node). > > Interesting problem, but I am sure this is just a start, handling > externals which return all their data in one blob (as they don't > stream) is also an issue. > > I will work on getting the code out. > > Norman > > On Sun, Jul 11, 2010 at 11:13 AM, afshin afzali <[email protected]> > wrote: > > Thank You Very Much Guys. > > > > -- afshin > > > > On Sun, Jul 11, 2010 at 7:27 PM, J Chris Anderson <[email protected]> > wrote: > > > >> > >> On Jul 11, 2010, at 7:27 AM, Norman Barker wrote: > >> > >> > Afshin > >> > > >> > I have got the all clear from my work to release this as a patch, I > >> > expect to be putting something up on github by the end of the week > >> > (internal paperwork permitting). I am going to implement it as an > >> > external handler so it can be used and reviewed and from there it will > >> > be under a do what you want with it license so it can go into couchdb > >> > if accepted. > >> > > >> > >> If you're implementing in Erlang, you probably don't need to go full hog > as > >> an external. > >> > >> Because the configuration system is so modular, you are probably best > >> adding it as a new httpd_design_handler, or httpd_db_handler. > >> > >> It should be easy to create a new module and link it in via the > >> configuration file. > >> > >> I don't know if you plan to allow querying across databases (I'd suggest > >> restricting the queries to a single database if you want to stay within > >> CouchDB's security model, and have it more likely to be accepted as a > >> patch.) > >> > >> We should really move this discussion to dev@ -- a lot of the > developers > >> only give a cursory glance at the user list, so you will get more > valuable > >> feedback there. > >> > >> Thanks for taking the time to write and release the patch! > >> > >> Chris > >> > >> > Chris, thanks for the help with the reduce function and confirming the > >> concept! > >> > > >> > Norman > >> > > >> > On Sun, Jul 11, 2010 at 8:02 AM, afshin afzali < > [email protected]> > >> wrote: > >> >> Hi Norman, Chris > >> >> > >> >> I just wanted to say this is the same problem we are currently facing > >> >> with. We are implementing a Local Business Directory application on > >> >> couchdb. Our searches need to combine several keys together to find > >> >> right entries. To do something like that in server side, we had the > >> >> paging mechanism problem, so we have chosen that do Norman's > algorithm > >> >> in client side! I'll appreciate if there will be a successful > progress > >> >> in this issue. > >> >> > >> >> BEST, > >> >> -- afshin > >> >> > >> >> On 7/8/10, J Chris Anderson <[email protected]> wrote: > >> >>> > >> >>> On Jul 8, 2010, at 10:43 AM, Norman Barker wrote: > >> >>> > >> >>>> Hi, > >> >>>> > >> >>>> I have been thinking about how to query multiple views at one time. > >> >>>> > >> >>>> I have an erlang handler in couchdb that takes a http post > containing > >> >>>> N view queries, each query contains a startkey and an endkey, I > then > >> >>>> open up each view in parallel (using pmap) and accumulate the doc > ids, > >> >>>> then I use the erlang sets module to get the unique values. All > good > >> >>>> and looks pretty (and works), though it doesn't scale since I am > >> >>>> holding all the results on the server (potential memory overload!) > >> >>>> whereas I would like to stream the results to the client one by > one. > >> >>>> > >> >>>> I am thinking of doing the following but have some questions; > >> >>>> > >> >>>> My first question is when I do > >> >>>> > >> >>>> couch_view:fold(View, FoldlFun, FoldAccInit, > >> >>>> couch_httpd_view:make_key_options(Args)), > >> >>>> > >> >>>> is there a way to call the _count reduce function in code to find > the > >> >>>> number of rows in the slice between startkey and endkey? > >> >>>> > >> >>>> If so, I would like to order all the views in the posted query > >> >>>> document by the result of _count from smallest to largest. > >> >>>> > >> >>>> I would then fold over the smallest result view and pull each > document > >> >>>> id (*) in turn. > >> >>>> > >> >>>> With each document id I would then call each of the other views in > >> >>>> turn with their startkey and endkey and in addition include > >> >>>> startkey_docid and endkey_docid with the docid in * above, again > >> >>>> calling _count I can test for inclusion. If the doc id is in all > views > >> >>>> then I will immediate stream this to the client. > >> >>>> > >> >>>> Am I doing something stupid, is this optimal? > >> >>>> > >> >>> > >> >>> It sounds like you are on the right track. this could be a very > >> valuable > >> >>> patch to CouchDB once you have it working. > >> >>> > >> >>>> Any help with the programmatic _count call would be great. > >> >>>> > >> >>> > >> >>> One hint: maybe the call to reduce_to_count will help. > >> >>> > >> >>> Here's an implementation of a reduce query in Erlang. > >> >>> > >> >>> http://github.com/jchris/hovercraft/blob/master/hovercraft.erl#L217 > >> >>> > >> >>> Sorry I can't be more helpful. I've successfully bootstrapped this > >> stuff in > >> >>> my head before, but it always takes a couple of hours of turning my > >> brain > >> >>> into a step debugger. > >> >>> > >> >>> Good luck! > >> >>> > >> >>> Once you get deeper into the code you might have better luck getting > >> >>> responses on the dev@ list or maybe the #couchdb IRC channel on > >> freenode. > >> >>> > >> >>> Chris > >> >>> > >> >>> > >> >>>> thanks, > >> >>>> > >> >>>> Norman > >> >>> > >> >>> > >> >> > >> > >> > > >
