On Mon, Mar 15, 2010 at 11:10 AM, Henrik Thostrup Jensen <[email protected]> wrote: > Hi > > On 13 March 2010 20:44, Benoit Chesneau <[email protected]> wrote: > >> delayed_commits = true ; set this to false to ensure an fsync before >> 201 Created is returned >> batch_save_size = 1000 ; number of docs at which to save a batch >> batch_save_interval = 1000 ; milliseconds after which to save batches >> >> are the settings you may want to change in ini. > > This appears to work in 0.10, but is (as far as I can tell) ignored in > 0.11, due to the changed view collection architecture, which relies on > work queues, which have statically defined sizes (see line 42-43 in > couch_view_updater.erl). > > I have a synthetic benchmark for view generation over 70K documents. > In stock CouchDB 0.10, the view will be checkpointed about 15-17 > times. Around 9 times with the batch_save_size and batch_save_interval > set to 10000. CouchDB 0.11 on the other hand performs a whopping > 108/109 checkpoints of the view. Due to shadow B-trees this generates > significantly larger view files (2-3x much) and more time is spend > writing to disk. Generating the view takes roughly twice as long in > 0.11 as it does in 0.10. > > I've tracked down the problem to the new view generation architecture; > particularly the small sizes of the work queues defined in > couch_view_updater.erl. The attached patch decreases the number of > checkpoints to around 15, and makes view generation slightly faster > than 0.10. It basically increases the size of the write queue. > Inserting a 500 ms sleep in do_writes increased the performance a bit > more, but that is not a nice or right solution. > > I suspect the patch is not "the completely right solution (tm)", as a > lot checkpoints are performed initially whereafter it backs off and > starts to take longer time/revisions between the checkpoints. I > suspect that the code is just writing repeatedly and as writes start > to take longer time, more revisions are added per checkpoint. Though I > am not really sure of this. > > Still, it is a 2 line patch, and it significantly increases view > generation performance. I'd very much like to see this in 0.11, to > avoid a rather large performance regression between 0.10 and 0.11. If > 0.11 comes out as it is, we would either have to stick with 0.10 or > build our own patched 0.11. > > -- > - Henrik >
I don't know if it's the right solution but could you open a ticket on jira and add your patch here ? It would help to track the problem :) - benoît
