Göran, Are there links to your map/reduce functions along with some example data? If I get some time tonight or tomorrow I'll try and reproduce and see if I can't figure out what's going on.
Like Adam says, it does seem like the flushing is getting overly aggressive when it shouldn't be. Paul 2009/7/4 Göran Krampe <[email protected]>: > Adam Kocoloski wrote: >> >> Not sure if it's described, but it is by design. The reduce function >> executes when the btree is modified. We can't afford to cache KVs from an >> index update in memory regardless of size; we have to set some threshold >> when we flush them to disk. > > And I presume you can't write KVs *without* doing the reduce? > > When I wrote "described" I am referring to the blog post by Ricky Ho btw. It > seems to imply a strict ordering, map -> reduce -> rereduce. IIRC. > >> I think the fundamental question is why the flush operations were >> occurring so frequently the second time around. Is it because you were >> building up a largish hash for the reduce value? Probably. Nevertheless, >> I'd like to have a better handle on that. > > Yeah, well, I am on vacation now - but some other guys are not. We could of > course start by trying to rewrite this the Right Way first as Chris said. > > I am curious if it can be done using grouping because we dismissed grouping > due to its relatively slow performance (it runs lots of reduces at query > time IIRC) :) > > Btw, the solution used now DOES return the map for a full year in about 230 > ms, including parsing on client side. So query time was perfectly fine, but > view generation was not. This shows to me that it *can* work. > > regards, Göran > >
