+1 on Benoît's comment. Using erlang_js, emonk or something equivalent should be top priority
On Mon, Aug 2, 2010 at 11:55 AM, Benoit Chesneau <[email protected]> wrote: > On Mon, Jul 26, 2010 at 11:35 PM, Mikeal Rogers <[email protected]> > wrote: >> After some conversations I've had in NYC this week and Mathias' great post >> on the 10 biggest issues with CouchDB ( >> http://www.paperplanes.de/2010/7/26/10_annoying_things_about_couchdb.html ) >> I wanted to formally propose some changes to the view server/protocol. >> >> The first issue I want to tackle is the lack of CommonJS modules in >> map/reduce. The reason for this is that we use a deterministic hash on all >> the views in a design document in order to query it. >> >> First off, it would be great if we could separate out each view and cache it >> based on it's own hash. This way updating one view doesn't blow away the >> entire design document. This has some large ramification, for one thing it >> means that each view needs to keep it's own last sequence and while one view >> is getting up to date it can't be included in generation when other views >> are getting updated. >> >> Once each view has it's own deterministic hash I would propose that we move >> the responsibility for generating the has to a new view server call. This >> call would get triggered during every design doc update and look something >> like. >> >> request : ["hash", {"_id":"_design/foo", .......} ] >> response ["views/bar","aoivniuasdf8ashd7zh87vxxz87gf8sd7"] >> >> The view server can inspect each map/reduce function and determine which >> modules it imports and include those strings in the hash for that particular >> view. >> >> The second issue I'd like to tackle is two fold, parallelized view >> generation and unnecessarily chatty IO for large view generations. >> >> Currently, every single document is passed to the view server one at a time >> and the response is read back one at a time. I would suggest that we allow a >> user configuration upper limit to "batch" documents to the view server (100 >> by default). The request/response would remain exactly the same as it is now >> except there would be an extra array around the request and response. >> >> This would also open up the ability for the view server to break up that >> batch and pass it to different view servers and then return the responses >> all together (this obviously means it's limited to the speed of the client >> handling that last chunk). >> >> Thoughts? >> >> Somewhere on github I actually have the changes to the view server for that >> batching but it doesn't includes the changes on the erlang side. >> >> -Mikeal >> > > Before doing any changes to view server protocol, I would prefer to > start working on a better implementation of js insde CouchDB. The > current implementation (ie using one os process for each call/request > ) even limited in a pool of process limit what you can do for obvious > reason. > > I would prefer that we start working on a js implementation using an > erlang driver or nif (erlang_js or emonk) that would allow us to add > any needed command using erlang. So we could do what nodejs do with > its own library more easily (ie adding features in pure js but using > erlang call). One other advantage of such implementation is speed, we > won't need to pass messages in json and such, also it could be > eventually in the future distributed. > > - benoit > -- Filipe David Manana, [email protected] "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men."
